Cloudflare has introduced “Friendly Bots,” a new way to verify an online bot’s identity.
Bots are applications designed to automatically perform specific, repetitive tasks online without the need for human oversight.
Many bots are set to beneficial tasks such as crawling web pages, for analytics, providing payment services, chatting to website users, and giving them advice or pointing them to the right customer service department — but not all.
So-called ‘bad’ bots can be used to scrape user data, send spam, overwhelm a domain with traffic and disrupt services (DoS/DDoS attacks), or perform automatic account access attempts in what is known as credential stuffing.
In an effort to stop malicious bots from causing too much havoc online, some online service providers implement allow and deny lists to stop known bad bots from accessing resources.
However, according to Cloudflare, there are many “well-behaved” bots online — and so it can be a challenge to maintain a balance between the good and the bad.
“At Cloudflare, we manually “verify” good bots, so they don’t get blocked,” the firm says. “Our customers can choose to allowlist any bot that is verified. Unfortunately, new bots are popping up faster than we can verify them.”
Therefore, Cloudflare has developed new functionality for customers called “Friendly Bots.”
Normally, bots are verified through public forms and documentation provided by a developer, including its IP addresses — whether static or dynamic — rDNS, user agents, and machine learning (ML), the use of smart algorithms that detect patterns in bot behavior and aim to profile the innocent ones.
It can take a few weeks for bots to be verified, but smaller developers may have to join a long queue unless the bot is working at a vast scale.
In the meantime, Cloudflare hopes that by considering a bot ‘friendly’ while it is waiting to be verified, this can cut some of the legwork and time required for good bots to be given the seal of approval.
Friendly Bots will allow users to “auto validate” bot traffic through the Cloudflare dashboard. Users can provide information about a bot, and the company will then be better equipped to verify bots based on their traffic.
“In the past, we’ve struggled to verify bots that did not crawl the web at a large scale,” Cloudflare says. “[…] Bots were sometimes difficult to verify if they did not make thousands of requests to Cloudflare. With Friendly Bots, we’ve eliminated that requirement, introducing a new, dynamic cache that optimizes for fun-sized projects.”
In addition, if users in large numbers are submitting the same bot to allow lists, such as through a specific IP address, this bot will be automatically added to the ‘to verify’ list.
“Previously, we required bot operators (e.g. Google) to submit verification data themselves,” the firm added. “If there was a bot you wanted to verify but did not own, you were out of luck. Friendly Bots eliminates this dependency on bot operators. Anyone who can find identifying information can register a bot on their site.”
Cloudflare says that Friendly Bots will be launched “soon” and will “reduce false positives, improve crawl-ability, and generally stabilize sites.” Verified bots to are also being added to the Logs feature under Cloudflare Radar.
See also
Have a tip? Get in touch securely via WhatsApp | Signal at +447713 025 499, or over at Keybase: charlie0
