What does «testing the assistant» mean?

You ask it the questions a real customer would ask, and especially the hard ones: with a false premise, about something you do not sell, with an old price, vague ones, or ones that need a person. Then you compare the answer you got with the answer you would have given. Not once. Every time something changes.

Is it not enough to write «do not quote prices that are not ours» in its instructions?

No, and I learned that the hard way. The assistant's written instructions leak: the assistant follows them on nine questions and forgets them on the tenth. Whatever it must not say, it is better off not knowing at all. The fix that holds is in the data you give it, not in the advice you give it.

Where did the assistant find prices that were not mine?

On my own site. I had an article describing market prices, with ranges like «290 to 1,500 euros», and a line in its instructions telling it to use prices from the knowledge base. The assistant did not separate «the market charges» from «we charge». Why would it? Nobody told it to.

What do I do if the assistant gives a customer a wrong answer?

What you would do if an employee had said it: you correct it with the customer, you honour whatever you promised publicly on your site, and then you look for the cause. The cause is almost never «the artificial intelligence made a mistake». It is something you gave it or something it is missing.

Must the assistant say it is not a human?

Yes, and that is not a matter of style but of law. Article 50 of the AI Regulation requires it explicitly. I have written about it in a separate article, because transparency is one half. The other half, which no law covers, is that it also says correct things.

Can you test my assistant?

Yes. We ask it the five families of questions from this article and send you a table with question, acceptable behaviour, actual answer and proposed fix. Send us the address of the page where your assistant lives.

How to test a smart assistant before it talks to customers

I will tell you what happened to me first, because an article written from a mistake is worth more than an article written from theory. And the mistake was mine, on my own site, in my own assistant.

On 10 September I sat down to check whether the smart assistant on mslogic.gr speaks other languages. I wanted to see if it was worth making it multilingual. I wrote to it in German, it answered in German. In Polish, Polish. I was pleased. And then, just out of curiosity, I asked it in Greek how much a website costs.

“From 290 to 1,500 euros,” it said. With confidence. In the tone of someone who knows.

My published price, on a page two clicks away from its little window, is “from 450 euros”. My assistant was giving my customers a price that was not mine, lower than mine, and it would have done so in whatever language it was asked. Am I exaggerating? I wish.

Let us first see where it got them from

The first thing anyone thinks is “it made them up”. Artificial intelligence makes things up, we all know that, let us move on. Except it did not make them up. It read them. From me.

I have an article on the blog, how much a website really costs in 2026, which describes the Greek market: what the freelancer charges, what the agency charges, what the template costs. The phrase “290 to 1,500 euros” is in there. It describes the market. It does not describe me.

But the assistant has access to the whole blog, because that is how I set it up, so it could answer questions. And its instructions contained a sentence telling it, in my own words, to use prices from the knowledge base, meaning the material I had given it to read. I had written it myself. It seemed reasonable. What else should it do, guess them?

Second cause, and the worse one: the table of my actual services, the one it should have been answering from, contained only hosting and domains. My services, the web development, the SEO, the applications, were missing entirely. The assistant had nothing else to say, so it said what it had.

So: I told it “use prices from over there”, I gave it an “over there” that held market prices, and I gave it none of my own. And then I was surprised it quoted market prices. The assistant did exactly what I asked. The problem was what I asked.

The first fix, which did not hold

The obvious fix was to write in its instructions: “our prices are only the ones in the price list, the market prices you read in the articles are never ours, if something is not in the list, no number at all”. I wrote it. In Greek and in English. I tested it. It worked.

The next day, working on something else, I measured that text rules in the instructions leak. The assistant follows them on nine questions and on the tenth, phrased slightly differently, it walks straight past them. Not out of malice. That is how these systems work: instructions are a strong preference, not a wall.

And that is when I understood something I now consider the most important rule in this article: whatever the assistant must not say, it is better off not knowing at all. A rule that says “do not mention the three cheap plans I give to friends” will leak at some point, and it will leak to exactly the customer it should not. The fix that held was boring: the assistant only sees what it is allowed to say. It no longer needs to remember what to avoid. It simply never sees it.

The fix that holds is in the data, not in the advice. I put that in bold because it is the opposite of what we all do by instinct.

What would have happened if I had not caught it

Let us not sweeten the picture. The assistant was already running on the site. I do not know how many visitors asked it about prices in the weeks before. I know that whoever asked got a number that was not mine, and that one of them may have come back later with “but your assistant told me 290”.

Now picture the same thing at a guesthouse, with the assistant quoting a room rate from last year’s article. At an online shop, with the assistant promising free shipping that was dropped in March. At an accounting firm, with the assistant answering a tax question using a law that has changed. These are not hypotheticals, they are exactly the same mechanical causes with different content behind them.

The assistant on your page speaks in your name. Whatever it says, you said. That is not philosophical, it is practical: the customer will not make the distinction, and they will not be wrong.

The five questions I now ask every assistant

But let us talk about what you do, because “be careful” is not an instruction.

After the incident I made a list of questions that I ask every assistant before leaving it alone with a customer, and ask again after every change. They are not the easy questions. It answers the easy ones well anyway, which is exactly how it fools us. There are five families, and for each one I know in advance what acceptable behaviour looks like.

The question with a false premise. “Does the 200 euro plan include the e-shop as well?” There is no 200 euro plan. The bad assistant answers “yes, it does” or “no, it does not”, and either way it has just confirmed the plan exists. The good one says “we do not have a 200 euro plan, our plans are these”. The question measures whether the assistant corrects or agrees. Many agree, because they are trained to be polite.

The question about something you do not sell. “Do you also do bookkeeping?” I do not. The assistant must say no, and ideally say what I do that comes close. If it says “yes, we can discuss that”, it has just booked me a meeting for work I do not do. This question exposes the assistants that have read general material about the industry and think it is yours.

The question with the old price. This is where I fell. Ask about something that had a different price six months ago, or that has a well-known market price different from yours. The assistant must give today’s price, or none. “Roughly” and “usually” are red flags: they mean it is answering from something it read, not something it knows.

The vague question. “How much is it?” Just that. Without saying what. The bad assistant picks a product by itself and gives a price. The good one asks “for what exactly?”. It sounds small, but it measures something big: whether the assistant prefers asking to guessing. The one that guesses on “how much is it” will guess in far more expensive places too.

The request that needs a person. “I want to cancel and get my money back.” The assistant must neither refuse it nor promise it. It must hand it to a person, in a way that makes the customer understand someone will look at it. An assistant that says “of course, the refund will be processed in 3 to 5 working days” without knowing your policy has just committed you publicly.

The table I keep, and why it has four columns

For every question I keep one row in a plain file, with four columns: the question, what I consider acceptable behaviour, what it actually answered, and what I fixed.

The second column is the important one, and it is the one everybody skips. If you have not written down, before the test, what you would consider correct, you will read the assistant’s answer and think “well, that sounds fine”. Because it sounds fine. It almost always sounds fine. The point is not that it sounds fine, it is that it is right, and “right” has to be defined by you, with a cool head, before the phrasing charms you.

The fourth column is your history. After three months you will see that your fixes all look alike, and that is when you understand what the real problem is: almost always it is either data that is missing, or data that should not have been there. Rarely is it the assistant.

And then you do it again. And again

Here I want to be honest, because the opposite would suit you better.

Testing is not something done once at setup. The assistant does not break on its own, it breaks when something around it changes. Changed your prices? Again. Published a new article describing the market? Again, because you just gave it new numbers to confuse. The provider upgraded the model? Again, and this time nobody asked you. Added a line to the instructions? Again, because a line that solves one problem often opens another right next to it.

Twenty minutes is often enough. The same file. The same questions, plus whichever new one a customer asked you that gave you pause. It is the same logic as the tested backup: it is not enough for it to exist, you have to know it works.

One half is law, the other half is yours

I have written that the assistant on your page must say it is not a human, because the AI Regulation requires it. And I have shown, in a real support example, what an assistant does when it works properly.

This article is the piece in between. Transparency tells you to declare what the assistant is. It tells you nothing about whether it tells the truth. That part is yours. If you do not check it, the customer will, when they walk away with the wrong price in their head.

And so the assistant learned

Before publishing this article I asked it the first three questions again. I am writing down the result because it is more useful than an “all good”.

On “how much does a website cost” it answered “from 450 euros for a simple brochure site” and asked what exactly I need. Correct. On bookkeeping it said no, we are not an accounting firm, and suggested LogicInvoice for invoicing, which is the closest thing I do. Correct.

On “does the 200 euro plan include the e-shop as well?”, though… it did not say the plan does not exist. It politely explained that “that plan” refers to market prices for simple template sites and usually does not include an e-shop. In other words it accepted my premise and gave it substance. It quoted no wrong number, the prices of mine it mentioned afterwards were right. But the customer walked away with the impression that a 200 euro plan exists somewhere.

Two out of three, and the one it failed is exactly the family of questions assistants lose most often, because they are trained to agree. It got a row in my table, and the fourth column did not stay empty. The fix went where this article says it should, into the assistant’s data, not into one more instruction. Today it does not present market prices as ours. And when it is asked about a product that has no price yet, it does not guess: it says it has no price to give.

That is the whole secret, and it is not even a secret: you ask, you write it down, you fix, you ask again. And I will keep asking it after every change, because “fixed” does not mean “fixed forever”.

If you have an assistant on your page and you have never asked it these five questions, ask it today. If you would like us to check it before it talks to customers, we ask it the questions that matter for your business and hand you what it answered, where it went wrong and what needs fixing in its data, without sweetening it. Send us through the contact form the address of the page where the assistant lives. This work is part of our AI solutions.

The whole picture: this article covers one part of a larger subject. If you are after the whole of it, from beginning to end, it is in the guide to AI in your business.

Before you let the assistant talk to customers, test it