Dashboard

NetAPI Bot


NetAPI Bot is the web crawler used by NetAPI to collect publicly available homepage content for domain research and dataset enrichment.

How to identify NetAPI Bot

Requests from our crawler include the following HTTP User-Agent header:

NetAPI/1.1 (+https://netapi.com/bot.html)

The URL in the User-Agent points to this page so site owners can learn what the crawler does and how to control access.

How to verify NetAPI Bot

A User-Agent can be faked. Genuine NetAPI Bot requests come only from these IPv4 addresses (the crawler does not use IPv6):

  • 136.243.61.201 (crawl-136-243-61-201.bot.netapi.com)
  • 162.55.129.246 (crawl-162-55-129-246.bot.netapi.com)
  • 162.55.132.97 (crawl-162-55-132-97.bot.netapi.com)
  • 94.130.14.179 (crawl-94-130-14-179.bot.netapi.com)

The same list in machine-readable form: https://netapi.com/bot-ips.json

You can also verify an address with DNS, the same way as for Googlebot: a reverse lookup of the IP returns a host name ending in .bot.netapi.com, and a forward lookup of that name returns the same IP.

$ host 136.243.61.201
201.61.243.136.in-addr.arpa domain name pointer crawl-136-243-61-201.bot.netapi.com.
$ host crawl-136-243-61-201.bot.netapi.com
crawl-136-243-61-201.bot.netapi.com has address 136.243.61.201

Requests with our User-Agent from any other address are not ours.

How NetAPI Bot accesses your site

Our crawl policy is intentionally light:

  • Frequency: each site is fetched at most twice per month.
  • Scope: we request only the site’s main page (homepage). We do not crawl deep into sitemaps, blogs, product catalogs, or secondary URLs.
  • Load: because of this limited scope and low frequency, NetAPI Bot does not create meaningful server load for typical websites.

We fetch publicly available HTML for research and enrichment purposes related to domain data offered through the NetAPI service.

robots.txt

NetAPI Bot respects robots.txt. If crawling is disallowed for our product token (or for all bots), we will not fetch the page.

In robots.txt, matching uses the product token NetAPI (the name before the version), not the full User-Agent string. This follows the standard Robots Exclusion Protocol (RFC 9309), the same as Googlebot and other crawlers.

To disallow NetAPI Bot specifically:

User-agent: NetAPI
Disallow: /

Or, to disallow all crawlers:

User-agent: *
Disallow: /

Blocking NetAPI Bot

If you prefer that we do not visit your site:

  • Use a robots.txt disallow rule (preferred).
  • Or contact us and we can exclude your domain manually.

Contact

Questions about NetAPI Bot, crawl behavior, or exclusion requests: [email protected]