AppSupplyIndexBot
AppSupplyIndexBot is the crawler behind the App Supply Index. It reads the public files that the IAB Tech Lab standards publish for automated verification, and nothing else.
What it fetches
/app-ads.txton the developer website listed in an app's store page./sellers.jsonof ad systems that app-ads.txt files list.- Public App Store and Google Play pages, to find apps and their developer websites.
It doesn't log in, submit forms, run scripts, or fetch any other page on your site.
How often
Once a week (Mondays, from 04:00 UTC), at most one request per file per run, plus one retry after a failure. Requests are spread out; a single site sees a handful of requests per week.
How to recognize it
- User agent
- AppSupplyIndexBot/1.0 (+https://appsupplyindex.com/bot)
- IP address
- 165.227.6.41
- Hostname
- crawler.appsupplyindex.com
Requests that claim this user agent but come from another address are not ours.
If your file is blocked
If your server or CDN blocks automated requests, the index shows your sellers.json or app-ads.txt as
"Blocks automated access", and lines that depend on it aren't held against anyone. Buyers' verification systems
usually run from data centers too, so they may meet the same block. To let us (and other verifiers) read the
file, allow the address above for /sellers.json and /app-ads.txt.
Contact
Questions, corrections, or a request to stop crawling your domain: [email protected]. See the methodology for how the files are used.