Crawling audits documentation

The following are all the audits related to not being able to obtain a satisfactory response from servers when requesting main articles HTML or related assets (images, author pages…). Some of these may be caused by WAFs blocking Marfeel crawlers, which may be solved by whitelisting the IPs from which they crawl all the content.

non indexable pages: Pages marked as non indexable for search engines will not be analyzed and their audits will not appear on the module.
Note that any url that results in a crawling error won't be analyzed and therefore other audits will not show up. Once a crawling issue is solved, many new errors may appear as a result of this analysis becomeing available.

Connection failed

code: connectionFailed

The system has not been able to validate the information in these URLs.

Read more

Connection timed out

code: connectionTimeout

The system has received a timeout when trying to connect to these URLs.

Read more

Received an empty response

code: emptyResponse

The system has not been able to trace the content of these pages as they are apparently blank. Please double-check to ensure this is the expected behaviour.

Read more

Redirection to same URL found

code: selfRedirection

URL responds with a redirection HTTP code (3XX) and a location header pointing to the same URL, which results in an endless loop.

Read more

301 response code

The system has found 301 redirects in these URLs. This type of redirect tells search engines such as Google that it should ignore the crawled URL and is now part of a different URL.

This does not have to be incorrect, but we recommend that you double-check the behaviour to make sure that this is what you were looking for.

Read more

302 response code

The system has found 302 redirects in these URLs. This type of redirect tells search engines such as Google that it should ignore the crawled URL and is now part of a different URL.

This does not have to be incorrect, but we recommend that you double-check the behaviour to make sure that this is what you were looking for.

Read more

400 response code

Your server has returned a 400 http status code - Bad request for these URLs.

Please note that both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.

Read more

401 response code

Your server has returned a 401 http status error code - Unauthorized for these URLs.

Please note that both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.

Read more

403 response code

Your server has returned a 403 http status error code - Forbidden for these URLs.

Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent. This could be the reason for the issue.

Please note that both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.

Read more

404 response code

Your server has returned a 404 http status error code - Not found status for these URLs.

This behaviour is not necessarily wrong, especially if your intention is that this content is not visible to anyone. Double-check these URLs since both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.

Read more

410 response code

code: 410

Your server has returned a 410 http status error code - Gone status for these URLs.

This behaviour is not necessarily wrong, especially if your intention is that this content should not visible to anyone. Double-check these URLs, since both search engines and social networks will not be able to index or publish this content correctly as this error is considered invalid for the functioning of their systems.

Read more

429 response code

Your server has returned a 429 http status error code - Too many requests for these URLs.

Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent. This could be the reason for the issue.

Please note that both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.

Read more

451 response code

Your server has returned a 451 https status error code - Unavailable For Legal Reasons for these URLs.

Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent. This could be the reason for the issue.

Please note that both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.

Read more

500 response code

Your server has returned a 500 http status error code - Internal server error for these URLs.

Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent. This could be the reason for the issue.

Please note that both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.

Read more

Server 503 response

Your server has returned a 503 https status error code - Service unavailable for these URLs.

Make sure that if you have a WAF system, it is not blocking the Marfeel Compass user-agent, as this could be the reason for the issue detected.

Please note that both search engines and social networks will not be able to publish this content correctly as it is considered an invalid status code.

Read more

image.url did not return an HTTP 200 response

code: image_url_correct_response

The url property is a required property of the image structured data type which, in turn, is part of the Article type. This image is used by some search engines such as Google to identify the main image to be used in Top Stories, Google News and Google Discover carousels and others Google products.

The system has found problems accessing the image. Please make sure that Google can access the URL correctly otherwise it can negatively impact the performance of your Search and Discover traffic.

Read more