If you’re worried about AI ChatBots scrapping your website’s contents, you can simply block them from doing so in your robot txt file. On a normal level, AI ChatBots can access your website and scrap all your contents without license or permission.
So, if you’re concerned about contents being scrapped by such tools, here, I’m going to show you how to block the bots using the robots.txt file for your website. I’ll explore the benefits and consequences of using robot txt file to block AI bots.
A chatbot is a computer program that uses artificial intelligence (AI) and natural language processing (NLP) to understand customer questions and automate responses to them, simulating human conversation.
AI Chatbots get their data from a variety of sources, including websites, APIs, databases, social media, machine learning algorithms, and user input. Chatbots combine information from the above highlighted sources to provide personalized recommendations and improve user’s experience.
How does AI Chatbots Access your Website Content?
AI ChatBots are trained using multiple databases (including open-source and publicly databases), to serve personalized recommendation and improve user’s experience.
Using OpenAI’s ChatGPT as an example. GPT3 was trained using five (5) datasets. This was published in a research paper made by OpenAI, an Artificial Intelligence company that built and designed ChatGPT.
Here are the five (5) datasets OpenAI’s GPT3 was trained with, according to research paper published by OpenAI:
- Common Crawl (60% weight in training)
- WebText2 (22% weight in training)
- Books1 (8% weight in training)
- Books2 (8% weight in training)
- Wikipedia (3% weight in training)
Common crawl is the petabyte (i.e thousands of TBs) of data collected from websites since 2008, similarly to how Google’s search algorithm crawls web content.
WebText2 which take 22% of the datasets is a dataset created by OpenAI, containing roughly 45 million web pages linked to from Reddit posts with at least three upvotes.
Now, you should know how valuable Reddit’s post are.
So, in the case of ChatGPT, the OpenAI bot isn’t accessing and crawling your web pages directly. Indirectly, maybe! However, the company has announced a ChatGPT-hosted web browser which has raised concerns that this could be a way of scrapping web contents.
In the meantime, website owners should keep an eye on other AI chatbots, as more of them hit the market. Bard, a recently launched Google’s AI ChatBot, is the other big name in the field, and very little is known about the datasets being used to train the AI bot.
Apparently, we know Google’s search bots are constantly crawling web pages and social media pages, but this doesn’t necessarily mean Bard has access to the same data as Google’s crawlers does.
Why should you concerned as a website owner?
I should say “why are some website owners concerned?” The point is what happens to AI chatbots when websites stop publishing the content they rely upon to function?
Actually, website owners are mostly concerned about the devaluation AI chatbots like ChatGPT, Bing Chat and Google’s Bard bring to their contents.
I can boldly say 99% of AI chatbots are using existing publishers contents to generate responses, which is also meant to reduce the need for users to look out for the original source.
In other words, instead of visiting a website to access piece of information you need, you could easily get GPT3 or Bard to summarize the contents.
Furthermore, if AI ChatBot are devaluating website owners contents, then they’re losing traffics. But in Bard’s case, the Google AI chatbot rarely includes citations in its generative responses, telling users which pages the information was gathered.
In a nutshell, the current models of generative AI tools and Chatbots are using the work of content creators to systematically replacing the need for content creators.
How to block AI Bots from scrapping your website’s contents.
You can simply stop AI bots from using your web contents using robot.txt file. However, one line of txt command cannot block all AI bots. You’ll need to block each individual bot and specify them by name.
For example, Common Crawl’s bot is called CCBot and you can block it by adding the following code to your robots.txt file:
User-agent: CCBot
Disallow: /
This will block Common Crawl’s bot (Ccbot) from crawling your website. However, its never going to remove any data already collected from previous crawls.
To block ChatGPT bot, OpenAI has already gave instructions for blocking its bot. In this case, ChatGPT’s bot is named ChatGPT-User and you can block it by adding the following code to your website’s robots.txt file:
User-agent: ChatGPT-User
Disallow: /
So, to block other generative AI bots fron accessing your web contents, you’ll need to know the name and add a command to your robot.txt file to block the crawling bots whenever they visit your site.
Blocking search engine bots from crawling your content is another problem entirely, though. As Google is highly secretive about the training datasets it uses in moderating Bard, it’s impossible to identify which bots you’ll need to block and whether they’ll even respect commands in your robots.txt file (many web crawlers don’t usually respect robot.txt commands).
In a nutshell, have it in mind that there’s no simple way to block all generative AI bots and chatbots from accessing your website’s contents. Meanwhile, manually blocking each individual bot is a mission impossible.
Even if you keep up with the latest AI bots roaming the web, there’s no guarantee they’ll all adhere to the commands in your website’s robots.txt file.
Please drop a comment before you leave. Thanks for reading this article, cheers!
