AI Bot Identification Cheat Sheet — web scraping bots, named before they are blocked

Reviewed October 25, 2026 — Ingenious AI security checks.

Key facts

• Identification precedes blocking: a bot you cannot name is a bot you cannot deliberately allow, throttle or stop.
• 5 signals combine: user-agent declaration, published IP ranges, behaviour, robots compliance and reverse-DNS or registry records.
• The database is dated and vendor-documented: agent identities come from vendors' own published documentation, with the date checked — not from folklore lists.

How do you identify a web scraping bot?

You identify a bot by combining signals, because no single one is proof. Honest crawlers declare a recognisable user-agent and publish the IP ranges and verification methods they use; dishonest scrapers borrow those names. Matching the declared identity against the vendor's own documentation — ranges, behaviour and robots conduct — is what separates identification from guessing.

• User-agent declaration: the name the visitor claims — necessary, never sufficient.
• Published IP ranges: does the traffic arrive from the ranges that vendor actually publishes?
• Reverse DNS and registry records: does the address resolve back to the claimed operator?
• Behaviour: request rate, path order, asset loading and session shape — crawlers, agents and scrapers move differently.
• Robots and policy conduct: does the visitor respect robots.txt and your published agent policy, as its documentation promises?

What do you do with an identification?

• Allow the verified agents you want — search, answer engines and your customers' assistants.
• Throttle the heavy but legitimate — rate limits keep a keen crawler from becoming an outage.
• Block the undeclared, the spoofed and the abusive — by identity and behaviour, not by country-sized net blocks.
• Log and revisit: new agents launch constantly; an identification list ages in weeks, so the database behind it must be maintained and dated.

What is the blocking kit?

Identification becomes protection when it ships as deployable rules. AgentBlock builds the same decisions into three forms — a Cloudflare Worker, a WordPress plugin and a JavaScript snippet — so the allow, throttle and block lists you chose are enforced at the edge, in the CMS, or on the page, without hand-editing server rules.

Run the free check

The fastest way to apply this cheat sheet is the free check on AgentBlock: it identifies the known agents in the database and builds your blocking kit. Start on the AgentBlock homepage, compare tiers on AgentBlock pricing, and keep the full hosted reference open at the AI Bot Identification cheat sheet on AgentBlock. Questions or orders: ask Ingenious AI on WhatsApp or email orders@getingeniousai.com.

Run the free check on AgentBlock → Open the hosted cheat sheet →