Responsible AI Use in the Age of LLMs

In early November ‘25, Luleå-based consulting firm Agio approached with a request for a seminar on the topic of AI security. After some initial exchange, AI chatbots built on Large Language Models (LLMs) were selected as the focal topic of the talk. A screenshot from the lunch seminar held online with over 20 participants from various roles at Agio.

The seminar’s core points are summarised in short below; for more in-depth information or a presentation of the seminar at your organisation, please do reach out to us!

“Real Attackers Don’t Compute Gradients”

Actual attacks on modern AI systems are not as complicated as one might expect. The titular quote here comes from a research paper of the same name by Apruzzese et al. which documents how economical factors are integral to the decision making process of both attackers and defenders. Attackers tend to use the cheapest attack that is likely to work while defenders need to justify security-related spending over more direct investments in company success.

The most common threat are ‘prompt injections’ which are malicious instructions sent to an LLM (usually a chatbot). These instructions are specifically designed to ‘convince’ the LLM that it should ignore any existing safety rules in order to follow the new orders. Once this happens, the potential consequences are only limited by what the LLM is technically capable of doing. For example, a public-facing chatbot could be made to divulge company-internal information, an email assistant could add phishing links to message summaries, or a programming assistant could suggest insecure code snippets for use in the company code base.

Prompt Injections Are More Feasible Than You Think

Contrary to many classical cyberattacks, prompt injections do not require substantial technical skill or direct access to a company’s network. Since LLMs take regular language as input, no programming is required to construct and send malicious instructions. Model safeguards can often be circumvented by simple techniques like misspelling keywords, splitting questionable instructions up into multiple queries or asking the model to answer questions within a fictional scenario.

The range of starting points for such attacks are also becoming more broad as AI products extend their features. Models that can check additonal sources for more information can contract prompt injections from these sources. If your AI assistant is allowed to search the web, read external files or has access to your email inbox, all these places gain the potential to infect your assistant with malicious instructions. To fully prevent such infections, you would need to be sure that those additional sources do not contain any dubious content, meaning you would need to verify it by hand. This quickly defeats the purpose of AI summarising tools used on emails or internal documents and becomes practically impossible in the face of the endless information on the internet.

Our Recommendations

AI chatbots can not comply with proper software security standards from classical cybersecurity as their risk profile is more similar to that of a careless employee. We recommend making yourself aware of the tradeoff between security and convenience that comes with using these tools for each use case. To get you started, we have compiled a question list for sensible LLM use:

1. Is the problem actually solved best by an LLM?
   LLMs are not the only type of AI that can solve problems for you. Investigate whether established methods from machine learning are suitable for your use case. If they are, they will likely be both computationally cheaper and safer than a chatbot.
2. How much access does the LLM really need?
   Assuming an LLM is the right tool for the job, limit its access permissions to the absolute necessities. For example, an assistant that answers questions about internal documents does not need internet access. Generally, copy-pasting selected information into a chat is less risky than letting the LLM freely access the information source directly.
3. Can the model be hosted locally​?
   Using one of the established commercial models (ChatGPT, Copilot, etc) requires you to send all model input over the internet to their servers and count on their security practices to avoid data leaks. Running a model directly on computers on company premises allows you to limit the amount of data handled online while making you less dependent on 3rd parties.
4. How much are you willing to spend on defenses?
   More sophisticated defenses don’t come for free and will cost you both money and employee time. There is no single right amount to spend on security as your risk depends on your situation.
5. What’s the worst that could happen?
   Consider worst-case scenarios where internal information is leaked, assistants are manipulated to give wrong or harmful responses or the AI system becomes dysfunctional. Create backups of important data and don’t feed highly sensitive informaton to the LLM.