Heap obtains a copy of your website content via a web crawler that periodically crawls your site. We apply the following policies to minimize the number of crawling requests:
- A successfully crawled resource will be cached for 30 days.
- Crawled resources are shared across all sessions. For example, a CSS file that is included in all pages will be crawled only once every 30 days.
- If the crawling fails, we will cache the response for 24 hours and try again after 24 hours.
If we discover any missing assets when processing a captured session, we will automatically initiate a fresh re-crawl.