Editorial Crawler Frequency

The Marfeel Editorial Crawler crawls a url and builds the editorial profile of a page using its metadata. The Marfeel Editorial Crawler crawls a url under different situations:

  1. As soon as the Marfeel SDK triggers the first hit to the url.
  2. When a content modification is detected through the article:modified_time meta property or the article’s structured data field dateModified. See the whole chain. Make sure these fields are correctly updated for Marfeel and Googlebot to be always up-to-date.
  3. Whenever a crawl attempt fails, the crawler will retry every 5 minutes for up to 200 times before giving up on a url.
  4. When a manual re-crawl is triggered from an article details