The Marfeel Editorial Crawler crawls a url and builds the editorial profile of a page using its metadata. The Marfeel Editorial Crawler crawls a url under different situations:
- As soon as the Marfeel SDK triggers the first hit to the url.
- When a content modification is detected through the
article:modified_timemeta property or the article’s structured data fielddateModified. See the whole chain. Make sure these fields are correctly updated for Marfeel and Googlebot to be always up-to-date. - Whenever a crawl attempt fails, the crawler will retry every 5 minutes for up to 200 times before giving up on a url.
- When a manual re-crawl is triggered from an article details
