Which crawler are you blocking, and what do you think you gained

One of the most common conversations I have had this year starts like this: “I blocked GPTBot, I don’t want them taking my content.” And two sentences later it continues: “But I do want ChatGPT to recommend me when people ask about my area.”

Both are perfectly reasonable. The problem is that the person believes they did one thing when they did another. Let us see which.

There is no “the crawler”. There are many, with different jobs

The basic misunderstanding is that each company runs one crawler that does everything. It does not work that way. Each runs several, under separate names, each serving a different purpose. This is not a detail for technical people. It is exactly the point where the wrong decision gets made.

At OpenAI, the documentation describes three:

OAI-SearchBot, which in its own words is “used to surface websites in search results in ChatGPT’s search features”.

GPTBot, which crawls “content that may be used in training our generative AI foundation models”. And it adds: “Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models”.

And ChatGPT-User, for actions the user themselves starts inside ChatGPT.

You can see where this is going. GPTBot and OAI-SearchBot are different things. Closing the first does not close the second. Your content does not go into training, and you carry on appearing in the results. Which, if you think about it calmly, is probably exactly what my client wanted.

The reverse mistake is worse, and I see that too: someone closes everything with a generous Disallow: / against every name they found on a list, then wonders why nobody mentions them anywhere.

ChatGPT-User and the detail people skip

For the third one, the same documentation adds something worth knowing: for ChatGPT-User “robots.txt rules may not apply”, because these are user initiated actions rather than automatic crawling.

Meaning that when a person explicitly asks ChatGPT to open your page, the page opens. Sensible, when you think it through: that is what a person does with a browser too. robots.txt does not govern people.

At Google, the same pattern under different names

Google’s documentation lists eleven crawler names. One of them concerns us today: Google-Extended.

It is, in its own words, “a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models”.

And then comes the sentence that settles the worry for half the people who ask, so here it is in full: “Google-Extended does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.”

You can say no to training without paying anything in ranking. Google writes it itself, explicitly, on its own page.

With one caveat that has to be said: Google-Extended also controls grounding, meaning the model pulling content from the index as it answers so the answer stands on real sources. So it is not entirely free if what you want is to be cited by those answers.

Let us talk about what is worth closing

Here I stop with the names and ask the question that should have come first: what do you sell?

If you sell the content itself, if you are a publisher or you hold an archive somebody pays to read, then the training question is a real one and I am not the person to decide it for you. It is a commercial decision, not a technical one.

But if you run a hotel, an accountancy, a law practice, a workshop, a shop, if your site exists so that people find you, then closing doors to engines that recommend businesses is like taking down your sign so nobody copies your name. You gained something. It was not what you wanted.

And a reminder for anyone who thinks they shut the door: robots.txt is not a lock. It is a statement of intent. The serious ones respect it, the rest do not. Whatever genuinely must not be read is not protected by a text file, it is protected by access control. Different conversation, different article.

What we put in ours, and why

The robots.txt on mslogic.gr fits in four lines and says: everything open to everyone, and here is the sitemap. No disallow, no excluded name.

And we carry two files more than the average site: llms.txt and llms-full.txt, in plain text, explaining who we are, what we do and what each product does. Not because a standard demands it, but because we would rather a machine read it written by us than infer it from our menus.

The reasoning is simple and not hidden: we want to be read. We do not sell text. We sell work. The text is how the person who needs that work finds us.

Before you touch your file

Open mslogic.gr/robots.txt, or your own, by adding /robots.txt to the end of your address. It is public, everyone sees it, and so can you. If you find names in there you do not remember putting in and cannot explain, do not delete them blindly. Do not leave them alone either.

If you want to go through what your site allows and forbids today, and above all whether that matches what you want from it, tell me. It is one of the shortest conversations I have, and one of the most often useful.