Access Control In LLMs For Safe Multi-User Usage

Problem Scenario

When deploying LLM-based AI systems in an organisation, it’s efficient to have a single large instance of the model with all relevant information available to it. Various users in the organisation can then interact with that single model to access a shared knowledge base. But what if different users should have different levels of access to the knowledge base? This is where role-based access control becomes important.

Before AI chatbots became a popular tool to process information, access control usually was about setting permissions in a file system to decide who could see and read what. Each document would require certain permissions and each user had individual permissions assigned to them. This scheme can’t be simply carried over to LLM assistants as they process information across all documents and generate tailored responses to user queries. Not only can it be difficult to decide which part of the response is based on a specific document but also for the LLM to understand nuanced levels of access permissions.

Current Approaches

The simplest way of introducing access control to an LLM-powered knowledge base would be to include written rules in the model’s system prompt (i.e. the guiding instructions that are prepended to every user query). This is better than nothing but quickly falls apart under pressure. Complex or conflicting sets of permissions often cause LLMs to make the wrong decision about giving access and due to integrating the guardrails at the prompt-level, prompting techniques can be used to circumvent these rules. The user could talk the LLM into providing access even after initial refusal by making up circumstances that would justify an exception.
Generally speaking, LLMs should not be trusted to keep information at their disposal confidential. This leads us to more advanced approaches to solve the problem.

Models can be given special training to channel their responses into more specific patterns. This is much more effective than prompting the model with access rules but such training requires data in the form of example queries and desired or undesired responses. For best results, the data should be based on relevant topics and actual roles from the organisation in question, so external datasets are not as useful.

Rather than retraining the LLM in question, additional models can be set up to serve as watchdogs. They are trained from the ground up to judge responses from the main LLM with regard to the access rules and can force the LLM to generate another, more appropriate answer before showing it to the user. This approach can be quite effective but comes with even more additional setup cost and maintenance as access control becomes dependent on

Other approaches go the other way and split up the data underlying the knowledge base according to its sensitivity. The LLM then receives limited access to that data based on each individual user query, making it impossible to accidentally mix sensitive information into the response. This approach combines a high level of data safety with relatively little computational overhead but requires a well-structured system to keep track of data and corresponding permissions.

Our Work

We are currently experimenting with elements of various existing methods to find a solution that is both feasible and effective. For testing purposes, a demonstration environment is being developed that will let us showcase different approaches and their advantages versus disadvantages.

More information on this is to come