It answers customers’ most common questions, helps them choose a product, checks their orders, and takes pressure off the support team. AI chatbots can significantly reduce the burden on customer support, and a growing number of software development companies now offer such solutions. What their sales representative may not mention, however, is that a chatbot introduces serious security risks into your company’s infrastructure—risks that should be addressed before deployment.
A working demo can be built relatively quickly. But a functional chatbot is not the same as a secure one. If security is what you want, simply connecting company data to a language model is not enough. In practice, every business that does so is opening a new public-facing interface into its environment. If, for example, the vendor stores credentials insecurely, gives the solution excessive permissions or fails to monitor its behaviour, an attacker may get much further than answers to questions about opening hours.
“When choosing a vendor, I would pay very close attention to its credibility, track record, and demonstrable expertise. Just because a solution appears to work does not mean it is secure,” warns Michal Korchanik from Binary Confidence. He adds that the market is seeing a sharp rise in the number of companies offering and integrating these solutions. Without robust security, deploying them could have devastating consequences for a business. A chatbot is merely the interface. The impact of an attack depends on the systems and permissions connected to it.
An attack through an AI chatbot can open a path into AWS
Imagine an attacker targets your company and, rather than searching for an exotic vulnerability, begins by talking to its publicly available chatbot. Using carefully crafted prompts, the attacker gradually manipulates it into revealing its hidden system instructions. These rules describe what the chatbot should do, which resources it works with, and how it should respond. Buried somewhere among them is an access key.
The key has far broader permissions than the chatbot needs to perform its job. The attacker can therefore use it to gain access to the company’s AWS cloud environment. They create a new administrator account and lock out the original administrator. What began as an ordinary conversation with a chatbot suddenly becomes full access to the infrastructure. The company has lost the keys to its own digital server room.
Although this was a training scenario used in the Guardians 2026 final, it is a textbook example of what can happen in real life. Chains of mistakes like this are entirely common in practice: sensitive information stored in the wrong place, unnecessarily broad permissions, and missing layers of control.
Prompt injection: How attackers manipulate AI chatbots
The technique in which an attacker attempts to change a model’s behaviour through their own instructions is known as prompt injection. They may tell the chatbot to ignore its original rules, assume a different role, reveal internal instructions, or use a connected tool in a way its developer never anticipated. A malicious instruction does not even have to come directly from a person. It can also be hidden in a document, email, or website that the AI has been asked to process.
OWASP ranked prompt injection first in its list of the most serious risks facing applications that use large language models. Other risks include the disclosure of sensitive information, system prompt leakage, excessive agency, and the uncontrolled consumption of paid resources.
One important detail is that a hidden system prompt cannot be treated as a vault. It can guide the model’s behaviour, but it must never replace authorisation or contain passwords, API keys or access tokens. If a chatbot can “see” such information at all, the company must assume that someone will try to extract it.
“Access tokens have real value to attackers. Stolen keys are either resold or used for the attackers’ own activities, leaving the compromised company to pay the resulting bill for ‘AI credits’—often hundreds or thousands of euros,” explains Michal Korchaník.
The risk grows with every system connected to the chatbot. If it only answers questions using a public product catalogue, the potential impact is limited. If it reads from a CRM, handles orders, sends emails, creates users or controls cloud services, every permission granted to it may become a permission available to an attacker.
Prompt injection alone will not magically unlock AWS or a company database. It will, however, expose architectural weaknesses that are already there. If the model has no access to secrets, it cannot reveal them. If its account can only read public information, a manipulated chatbot cannot create an administrator. The scale of the incident therefore depends not only on whether an attacker manages to deceive the AI, but primarily on what the company has allowed it to see and do.
Why a working AI chatbot may still be insecure
AI has significantly lowered the technical barrier to developing these solutions. That is good news for innovation, but not automatically for security.
“Even a less experienced team can now build a working demo in a short time. But the ability to create a convincing demo is not proof that the team knows how to design the entire solution securely,” adds Michal Korchaník.
A chatbot is not merely a window on a website. Behind it sit a language model, application code, a database, cloud infrastructure, user identities, company documents, and third-party interfaces. The security of the final solution depends on the weakest link in this entire chain.
Financial loss is not the only possible consequence. A poorly restricted chatbot might show one customer another person’s order, make an unauthorised change, or confidently provide incorrect information. When it is connected to a critical process, that becomes a security incident.
Nor is this a product that will remain unchanged forever. The vendor will update the model, system instructions, data sources, and connected tools. Even a seemingly minor adjustment may alter the solution’s behaviour or create a new avenue for abuse. Security testing and monitoring must therefore continue throughout its entire operational life.
Five questions to ask before deploying an AI chatbot
1. What is the vendor’s track record?
References matter, as do the names of the people responsible for security and evidence that the solution has been tested for more than functionality alone. An impressive demo is not a security audit.
2. What data and systems will the chatbot access?
The vendor should be able to state precisely what the solution can read, what it can change, where the data is processed, and whether it is used for further model training.
3. What permissions will it receive?
Under the principle of least privilege, a chatbot should receive only the access required for a specific task. If all it needs to do is read an order status, it must not use an account capable of modifying users or administering the entire cloud environment. Sensitive actions should also require approval outside the model itself.
4. Where are the keys stored, and who monitors their use?
Secrets do not belong in a prompt or hard-coded into the application. They should be stored securely, rotated regularly, and monitored whenever they are used. The company needs spending limits, anomaly alerts, and a way to revoke compromised access immediately.
5. Has anyone tried to break the chatbot?
The solution needs a penetration test before deployment and after every significant change. The tester should examine the entire chain, from user input and the model itself to databases, APIs, and cloud permissions.
Secure AI chatbot deployment: Blind trust is the real problem
Companies should not avoid chatbots. They can accelerate support, improve service availability, and relieve employees of repetitive tasks. They should, however, be procured and deployed as responsibly as any other internet-facing application that works with company systems.
“Security is always about layers. It is not enough for a vendor to declare a chatbot secure. You need to restrict its access, independently verify sensitive actions, monitor its behaviour, and then make a genuine attempt to break the entire solution,” stresses Michal Korchaník.
At Guardians, we create scenarios like this so that defenders can experience emerging attacks before encountering them in their own infrastructure. The same principle applies to real-world AI deployment. The goal is to review the architecture, minimise permissions, protect credentials, monitor operations, and test the solution’s boundaries before an attacker starts looking for them.
A chatbot should be a helpful assistant at your company’s front door. Before deploying one, bring in cybersecurity experts to ensure it does not become a doorman with keys to your company’s most closely guarded rooms.
This activity is supported by the European Cybersecurity Competence Centre (ECCC) as part of the project under grant code 101145856, and by the Ministry of Investments, Regional Development and Informatization as part of the state programme of the Recovery and Resilience Plan of the Slovak Republic under project grant code 17I04-04-V02-00001.
