NetAPI Bot

NetAPI Bot is the web crawler used by NetAPI to collect publicly available homepage content for domain research and dataset enrichment.

How to identify NetAPI Bot

Requests from our crawler include the following HTTP User-Agent header:

NetAPI/1.1 (+https://netapi.com/bot.html)

The URL in the User-Agent points to this page so site owners can learn what the crawler does and how to control access.

How NetAPI Bot accesses your site

Our crawl policy is intentionally light:

• Frequency: each site is fetched at most twice per month.
• Scope: we request only the site’s main page (homepage). We do not crawl deep into sitemaps, blogs, product catalogs, or secondary URLs.
• Load: because of this limited scope and low frequency, NetAPI Bot does not create meaningful server load for typical websites.

We fetch publicly available HTML for research and enrichment purposes related to domain data offered through the NetAPI service.

robots.txt

NetAPI Bot respects robots.txt. If crawling is disallowed for our product token (or for all bots), we will not fetch the page.

In robots.txt, matching uses the product token NetAPI (the name before the version), not the full User-Agent string. This follows the standard Robots Exclusion Protocol (RFC 9309), the same as Googlebot and other crawlers.

To disallow NetAPI Bot specifically:

User-agent: NetAPI
Disallow: /

Or, to disallow all crawlers:

User-agent: *
Disallow: /

Blocking NetAPI Bot

If you prefer that we do not visit your site:

• Use a robots.txt disallow rule (preferred).
• Or contact us and we can exclude your domain manually.

Contact

Questions about NetAPI Bot, crawl behavior, or exclusion requests: [email protected]