Skip to content

Web crawler

NightroBot

Mozilla/5.0 (compatible; NightroBot/1.0; +https://nightro.ai/bot)

What it reads

Ad landing pages
The pages a business's Google Ads send people to: every 20 minutes, and every address in its ads once a day
Pages in an ad change
Before Nightro sends the change to Google Ads, each with one made-up address beside it
Website Doctor
The business's own site, when a person starts a check: up to 40 pages and a sample of links and images, 6 checks a day at most
Speed test
Up to 3 page loads in Chrome per Website Doctor check, as a phone and a laptop, with the files each page loads
Other
OpenStreetMap place lookups, one a second at most, and Google's connection check page

Only for businesses that run their Google Ads with Nightro.

Rate

Website Doctor crawl
At most 4 requests at a time; a 150 ms pause after each page or link checked
Speed test
One page load at a time, beside the crawl
Ad landing pages
Every 20 minutes; a page that fails, once more
robots.txt
Before the first request to a site, then every 60 minutes

robots.txt

User-agent: NightroBot
Followed by every request
User-agent: *
Followed by the Website Doctor's crawl. Other checks follow only the NightroBot group, as Google's ad robot follows only its own.

Block NightroBot

User-agent: NightroBot
Disallow: /

Nightro then can't check those pages, and sends Google Ads no change that points to them.

Verify a request

Method
Web Bot Auth: HTTP Message Signatures (RFC 9421), Ed25519
Signature-Agent
"https://nightro.ai"
Signed
@authority and Signature-Agent, tag web-bot-auth, valid 60 seconds
IP addresses
None fixed: requests come from Vercel's network

Speed test user agents

Mozilla/5.0 (Linux; Android 11; moto g power (2022)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Mobile Safari/537.36 (compatible; NightroBot/1.0; +https://nightro.ai/bot)
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 (compatible; NightroBot/1.0; +https://nightro.ai/bot)

Contact

admin@nightro.ai