Crawler
OmniImpactBot
Last Updated: August 12, 2026
OmniImpactBot is the web crawler operated by Omni Impact LLC. It reads websites so their owners can see how AI assistants and answer engines read them. It is not a search-engine spider and it does not crawl the open web: it visits sites our customers have added to their own account, and the competitor sites those customers choose to track.
If you run a site, a host, or a WAF and you want to identify or allow this traffic, everything you need is below. A plain-HTML version of the same identity details — no JavaScript required, for curl and automated verification tooling — lives at omniimpact.net/bot.
At a glance
Identifying OmniImpactBot
Every request the crawler makes carries this exact User-Agent string:
Mozilla/5.0 (compatible; OmniImpactBot/1.0; +https://omniimpact.net/bot)The same string is sent on all of our fetch paths — the plain HTTP fetch, the headless Chromium fallback we use for JavaScript-rendered pages, and the robots.txt, sitemap, and llms.txt lookups that go with a crawl. Allowing this one user-agent covers all of them.
We deliberately do not disguise the crawler as a consumer browser. A site that blocks AI crawlers should block us too — that block is itself one of the findings our product exists to report.
Why it visits your site
OmniImpactBot fetches a site for one of three reasons:
- A customer added the site to their account. Our platform audits how answer engines see a site: whether pages are reachable without JavaScript, whether structured data is present, how content is organized. That requires reading the pages.
- A customer tracks the site as a competitor. Customers can benchmark themselves against a handful of named competitor domains. Those crawls are smaller and read public pages only.
- A scheduled refresh. Sites already on the platform are re-read periodically so the analysis does not go stale.
We never bypass logins, never submit forms, never solve CAPTCHAs, and never attempt to defeat bot protection. If a site blocks us, we record that it blocked us and tell the customer.
robots.txt
Site crawls run by OmniImpactBot honor robots.txt. Before crawling a site we fetch /robots.txt and evaluate the rules against the token OmniImpactBot (falling back to your User-agent: * group where no OmniImpactBot group exists). A disallowed URL is not fetched, and a site that disallows us entirely is reported to the customer as blocked by robots.txt rather than crawled anyway.
A site with no robots.txt (a 404 response) is treated as permission to crawl, per RFC 9309.
Opting out
To stop OmniImpactBot from reading your site, add this to your robots.txt:
User-agent: OmniImpactBot
Disallow: /Customers can also remove a site from their account, which stops all crawling of it immediately. If a site is being crawled and you believe no one authorized it, email [email protected] and we will stop it.
Crawl rate and volume
Our crawls are small by design. Concretely:
- At most 10 pages per site crawl — a hard cap, applied on top of the customer's plan limit, so many crawls read fewer.
- At most 3 requests in flight at a time against a single host (one at a time for competitor-site crawls).
- A scheduled re-crawl roughly once a week for a customer's own site and once a month for competitor sites, plus a crawl when a customer explicitly asks for one.
- Individually tracked pages are fetched one page per request, on the same weekly-refresh basis.
- At most two retries on a failed fetch, with exponential backoff. A
429 Too Many Requestsgets a longer backoff (5 seconds, growing to a 30-second maximum), and repeated failures trip a circuit breaker that ends the crawl early. - Alongside pages we read
robots.txt,sitemap.xml, andllms.txtwhen they exist.
If our traffic is still causing you a problem, tell us at [email protected] — we would rather slow down than be blocked.
Allowlisting the crawler
Allowlist by user-agent (recommended)
The reliable way to allow OmniImpactBot is to match the user-agent string above, or the substring OmniImpactBot. It is stable, we publish it here, and it covers every path our crawler uses.
Cloudflare
On Cloudflare Pro/Business, add a WAF custom rule with the Skip action (Security → WAF → Custom rules) using this expression — and in the Skip settings, bypass only Super Bot Fight Mode, leaving managed rules, rate limiting, and the rest of the WAF active:
(http.user_agent contains "OmniImpactBot")Leave the rule switched on. We re-crawl your site every week, so a rule that is only on while a crawl runs leaves us blocked the rest of the time. A user-agent string is public and spoofable, so the rule should never bypass more than the bot challenge. Make it the first rule in the list.
On the Cloudflare Free plan, Bot Fight Mode cannot be skipped by a custom rule. The only option there is to turn Bot Fight Mode off (Security → Bots) and leave it off.
To check that it worked, re-run the crawl from your dashboard. If your site still shows as blocked, the rule is off, below another rule, or skipping the wrong product. We have applied to Cloudflare's verified bot program. Once Cloudflare approves us, Bot Fight Mode will let us through without a rule, but a custom rule or "Block AI bots" setting on your site can still block us.
WordPress plugins and managed hosts
- Wordfence: Wordfence's firewall allowlist is IP-based and bypasses all rules for a listed address. Because our crawl traffic egresses from shared cloud-provider ranges, putting those ranges on an all-rules bypass would also exempt unrelated traffic from the same provider — we don't recommend it. Instead, temporarily raise Wordfence's rate-limiting thresholds (Firewall → Rate Limiting) during a crawl window, or ask us to schedule the crawl.
- Sucuri: the self-service whitelist accepts IP addresses only, and we don't publish IPs to allowlist — open a ticket with Sucuri support asking for a firewall exception for the
OmniImpactBotuser-agent. - SiteGround (Imunify360): there is no self-service allowlist. Open a Site Tools → Help Desk ticket asking support to allowlist the
OmniImpactBotuser-agent. - GoDaddy Website Security: Firewall → Access Control; if AI Crawl Control is enabled, allow the crawler there too.
- AWS WAF: add a scope-down statement on the Bot Control managed rule group excluding requests with our user-agent — an ACL-level Allow rule is terminating and would bypass every other protection in the ACL.
Why we don't publish IP ranges to allowlist
Our crawl traffic egresses from our cloud hosting provider's shared address ranges — they are not reserved exclusively for us and can change if we move infrastructure. Allowlisting them would also allowlist unrelated traffic from the same provider, so we deliberately do not publish them as an allowlist target. Identify us by the user-agent above, scoped to bot-challenge bypasses only.
OmniImpactBot signs its requests with Web Bot Auth (HTTP Message Signatures), the emerging IETF standard that lets a crawler cryptographically prove its identity on every request. That, not an IP list, is the durable answer. Our public keys are published at:
https://oiv-prod-1.onrender.com/.well-known/http-message-signatures-directoryAgent Journeys
Separately from the crawler, Omni Impact operates OmniImpactJourneyBot, a browser agent that attempts real tasks on a site — find a product, reach a quote form, look up a policy — to report where an AI agent would get stuck. It only visits the domain a customer sets for a journey, and only when that customer starts or schedules a run. It identifies itself by appending this token to a standard desktop browser user-agent, and sends an X-OmniImpact-Journey header on every request:
OmniImpactJourneyBot/1.0 (+https://omniimpact.net/bot)Journey runs never bypass bot protection, never solve CAPTCHAs, and stop before checkout and before submitting any form other than a site search.
What we do with what we read
Crawled pages are used to produce the analysis shown to the customer who requested it — readability for answer engines, structured-data coverage, content gaps, and the like. We do not sell crawled content, and we do not use it to train general-purpose AI models. Our Privacy Policy describes how the data is stored and handled.
Contact
Questions, allowlisting help, or a complaint about our crawler's behavior — a real person reads this address:
Omni Impact LLC