Good bots and bad bots: How to tell them apart
What bots do, five kinds of good bot and five kinds of bad bot, and the robots.txt, IP, user-agent and behaviour checks that separate them on your site.
On this page
By one estimate, 56% of all website traffic comes from bots. Most of it is useful: search crawlers, uptime monitors, feed readers. The rest is scraping your prices, stuffing stolen credentials into your login form and testing stolen cards at your checkout.
Some bad bots exist to spread malware and are hard to spot. The rest leave traces you can read. Below: what separates good bots from bad ones, and the checks that tell them apart on your own traffic.
What is a bot?
A bot is a program that runs automated tasks over the internet. Bots do simple jobs, like posting on social media or answering emails, and complex ones, like buying and selling products, analysing data for trends, or handling customer service.
The most common types are web crawlers (which index the web), chatbots (which talk to people by text) and spambots (which send spam). Social media bots spread fake news; political bots spread propaganda.
What is a good bot?
A good bot does work you want done. Search engines run the best-known ones: crawlers that index web pages so people can find them.
Google, for example, ranks pages using signals such as the links pointing at a site and the match between the query and the page's content.
None of that works unless the crawler can read the page. Without access, the engine is guessing what you mean when you type "pizza".
The crawler is what gets every pizza restaurant's details into the results, so a searcher finds the right one without paging through dozens of sites.
Examples of good bots
Five kinds you want on your site:
1. Search engine bots
They crawl your content and index it, so your site appears when someone searches for a topic you cover.
2. Social network bots
They run on platforms such as Facebook and Instagram, serving content or carrying out tasks such as liking posts or sending messages on a user's behalf.
3. Aggregator bots
They collect information from many sites into one place, such as a news feed. They are also a source of referral traffic for your business.
4. Site monitoring bots
Uptime monitors check your site on a schedule, track response times, catch errors and alert you when something breaks.
5. Voice engine bots
Siri, Alexa and Google Assistant answer spoken questions out loud instead of returning a page of results, and they fetch those answers from the web.
What is a bad bot?
A bad bot is a program that interacts with a website without the owner's permission. Its operators use it to reach data they should not have, such as credit card numbers and other personal information.
Bad bots also spam search engines with links to malicious sites and adverts, run denial-of-service attacks, and steal information from the visitors to the sites they compromise.
The most common kind is a malicious script that scans the internet for vulnerable websites and uses them to send spam. These scripts run on hacked machines whose owners have no idea they are funding fake sales and subscriptions.
Examples of bad bots
Five kinds you want off your site:
1. Scraper bots
They copy content and data from your site at scale: prices, product catalogues, inventory levels. Competitors use them to undercut you and spammers use them to clone your pages.
2. Spam bots
They flood forms, comments and inboxes with unsolicited adverts and links, for their operator or for a third party paying for the traffic.
3. Scalper bots
They buy up limited stock the moment it goes on sale: concert tickets, sneakers, consoles.
The bot completes checkout faster than any human can, clears the inventory, and the goods reappear on secondary markets such as eBay or Craigslist at a markup.
4. Account takeover bots
They take stolen username and password pairs and try them against your login form at machine speed. Every pair that works is an account the attacker now controls, along with whatever personal and payment data sits inside it.
5. Carding and card-cracking bots
They run stolen card details through your checkout, usually with small purchases, to find which cards are still live. Card-cracking bots go one step further and guess the missing expiry date or CVV by brute force.
The merchant gets the chargebacks for the test purchases and the cardholders get fraudulent charges on cards that now circulate as verified.
How to identify a good bot and a bad bot
Good bots announce themselves and follow your rules. Bad bots pretend to be browsers and break them.
The tells are request rate, behaviour on the page and where the traffic comes from. Three places to look:
1. Robots.txt
This file tells crawlers which parts of your site they may visit. A good bot fetches it first and obeys it. A bad bot either never reads it or reads it and uses it as a map: a request for a path you disallowed is a bot that is not playing by your rules.
2. IP addresses
Legitimate crawlers come from address ranges their operators publish, and a reverse DNS lookup on the address resolves to the crawler's own domain. Bad bots arrive from residential proxies, cloud hosts and addresses that change between requests. Run the address through an IP lookup tool and see which group it falls into.
3. User-agent
Good bots send a user-agent string that names them, such as Googlebot or Bingbot, and you can verify the claim against the IP. Bad bots send a browser's user-agent, or a forged crawler's, and give themselves away by behaviour: dozens of pages in a few seconds, requests for links no human would click, forms filled faster than a person can type.
How to keep bad bots out
Six controls, from cheapest to most effective:
1. Block known bad IPs
Keep a list of addresses you refuse to serve, or subscribe to a third-party list of known bot IPs.
It stops the lazy bots and saves bandwidth. It does little against bots on rotating residential proxies, which is why it is the floor, not the fix.
2. Use CAPTCHA
A CAPTCHA asks the visitor to prove they are human by typing distorted characters or solving a small puzzle. It stops simple scripts. Solver farms and vision models now beat most CAPTCHAs, and every one you show costs real customers, so keep it for the point of risk rather than every page.
3. Block bad user-agents
The user-agent is the string a client sends to identify itself. Block the ones belonging to known scrapers and attack tools. Like IP blocking, it removes the bots that do not bother to disguise themselves.
4. Put HTTP authentication on non-public pages
HTTP authentication asks for a username and password before the page is served. Bots cannot supply credentials they do not have.
It only fits pages that were never meant to be public: staging sites, admin panels, internal tools.
5. Watch your traffic
Bot activity shows up as patterns: spikes at odd hours, one IP hitting hundreds of pages, sessions with no mouse movement, conversion rates that fall while visits climb.
Google Analytics or your server logs will show the pattern; from there you can block the source or tighten the rule it slipped through.
6. Score the visitor, not just the request
Everything above checks a single request against a list. Bots on fresh IPs with clean user-agents pass those checks. Sensfrx scores each signup, login and payment from device, network and behaviour signals together, so a bot that looks fine on IP and user-agent still fails on how it moves and what it runs on, and your rules decide whether it is allowed, reviewed or blocked.
Where to start
Good bots are the crawlers, monitors and assistants that bring you traffic and keep you informed. Bad bots scrape, spam, scalp, take over accounts and test stolen cards.
Telling them apart on your own traffic comes down to three checks: did it read robots.txt, does its IP match who it claims to be, and does it behave like a person.
Start by pulling a week of server logs and sorting by requests per IP. The top entries that are not a known crawler are your first block list.
Bots are not going away, so the job is to let the useful ones through and stop the rest at the login and the checkout. See how Sensfrx stops account takeover before the login succeeds.