NetAPI Bot is the web crawler used by NetAPI to collect publicly available homepage content for domain research and dataset enrichment.
Requests from our crawler include the following HTTP User-Agent header:
NetAPI/1.1 (+https://netapi.com/bot.html)
The URL in the User-Agent points to this page so site owners can learn what the crawler does and how to control access.
A User-Agent can be faked. Genuine NetAPI Bot requests come only from these IPv4 addresses (the crawler does not use IPv6):
136.243.61.201 (crawl-136-243-61-201.bot.netapi.com)162.55.129.246 (crawl-162-55-129-246.bot.netapi.com)162.55.132.97 (crawl-162-55-132-97.bot.netapi.com)94.130.14.179 (crawl-94-130-14-179.bot.netapi.com)The same list in machine-readable form: https://netapi.com/bot-ips.json
You can also verify an address with DNS, the same way as for Googlebot: a reverse lookup of the IP
returns a host name ending in .bot.netapi.com, and a forward lookup of that name
returns the same IP.
$ host 136.243.61.201 201.61.243.136.in-addr.arpa domain name pointer crawl-136-243-61-201.bot.netapi.com. $ host crawl-136-243-61-201.bot.netapi.com crawl-136-243-61-201.bot.netapi.com has address 136.243.61.201
Requests with our User-Agent from any other address are not ours.
Our crawl policy is intentionally light:
We fetch publicly available HTML for research and enrichment purposes related to domain data offered through the NetAPI service.
NetAPI Bot respects robots.txt.
If crawling is disallowed for our product token (or for all bots), we will not fetch the page.
In robots.txt, matching uses the product token
NetAPI (the name before the version), not the full User-Agent string.
This follows the standard Robots Exclusion Protocol (RFC 9309), the same as Googlebot and other crawlers.
To disallow NetAPI Bot specifically:
User-agent: NetAPI Disallow: /
Or, to disallow all crawlers:
User-agent: * Disallow: /
If you prefer that we do not visit your site:
robots.txt disallow rule (preferred).Questions about NetAPI Bot, crawl behavior, or exclusion requests: [email protected]