Web crawler
NightroBot
Mozilla/5.0 (compatible; NightroBot/1.0; +https://nightro.ai/bot)What it reads
- Ad landing pages
- The pages a business's Google Ads send people to: every 20 minutes, and every address in its ads once a day
- Pages in an ad change
- Before Nightro sends the change to Google Ads, each with one made-up address beside it
- Website Doctor
- The business's own site, when a person starts a check: up to 40 pages and a sample of links and images, 6 checks a day at most
- Speed test
- Up to 3 page loads in Chrome per Website Doctor check, as a phone and a laptop, with the files each page loads
- Other
- OpenStreetMap place lookups, one a second at most, and Google's connection check page
Only for businesses that run their Google Ads with Nightro.
Rate
- Website Doctor crawl
- At most 4 requests at a time; a 150 ms pause after each page or link checked
- Speed test
- One page load at a time, beside the crawl
- Ad landing pages
- Every 20 minutes; a page that fails, once more
- robots.txt
- Before the first request to a site, then every 60 minutes
robots.txt
- User-agent: NightroBot
- Followed by every request
- User-agent: *
- Followed by the Website Doctor's crawl. Other checks follow only the NightroBot group, as Google's ad robot follows only its own.
Block NightroBot
User-agent: NightroBot Disallow: /
Nightro then can't check those pages, and sends Google Ads no change that points to them.
Verify a request
- Method
- Web Bot Auth: HTTP Message Signatures (RFC 9421), Ed25519
- Signature-Agent
"https://nightro.ai"- Signed
- @authority and Signature-Agent, tag web-bot-auth, valid 60 seconds
- IP addresses
- None fixed: requests come from Vercel's network
Speed test user agents
Mozilla/5.0 (Linux; Android 11; moto g power (2022)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Mobile Safari/537.36 (compatible; NightroBot/1.0; +https://nightro.ai/bot) Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 (compatible; NightroBot/1.0; +https://nightro.ai/bot)