Heap obtains a copy of your website content via a web crawler that periodically crawls your site. We apply the following policies to minimize the number of crawling requests:
- A successfully crawled resource will be cached for 30 days.
- Crawled resources are shared across all sessions. For example, a CSS file that is included in all pages will be crawled only once every 30 days.
- If the crawling fails, we will cache the response for 24 hours and try again after 24 hours.
If we discover any missing assets when processing a captured session, we will automatically initiate a fresh re-crawl.
Can I prompt my site to be re-crawled?
At this time, we do not provide the functionality to trigger a manual re-crawl. However, any changes to your site, for example if a new version of your CSS file is uploaded, will automatically trigger a re-crawl with the updated version.
Note that if your website is updated but the filename is kept the same (for example, a new version of your CSS file) then Heap will continue to use the cached (older) version of the file. As a best practice, we recommend renaming your file per the new version when uploading any code changes to your website.