Secure your RAG: Where to start?

To minimize the risks with RAG, organizations must implement access control that ensures AI systems operate within secure and controlled parameters.

Paddy SR   ·   Monday, March 24th, 2025   ·   Reading Time: 6 minutes

Generative AI and large language models (LLMs) are reshaping our digital landscape, emerging as the next-generation search engines for both public information and enterprise internal systems.

Recent reports from Microsoft and IDC say that generative AI usage has jumped from 55% in 2023 to 75% in 2024. Even this article could have been written by a LLM (hmm maybe, or partly?).

Well… that doesn’t matter anymore given the rapid evolution of intelligent multimodal agents, seamless integrations and context aware querying finding its use everywhere.

RAG needs no introduction

While LLMs have demonstrated impressive capabilities, they are prone to hallucinations, fine-tuning constraints, outdated knowledge bases, and challenges in maintaining factual accuracy.

Retrieval Augmented Generation (RAG) addresses these limitations by integrating dynamic retrieval mechanisms with generative models, which enables context-sensitive document sourcing and minimizes the risks of misinformation. In RAG systems, the model queries a curated external knowledge base or index in tandem with its generation process, thus grounding the generated responses in verifiable data.

With great RAG comes great responsibility

Even if SkyNet sounds a bit far-fetched for today, GenAI is already prone to several security concerns such as prompt-injection, adversarial attacks, etc. from its human users and smart agents.

In the context of RAG, integrating retrieval components with generative models introduces new security vectors that demand robust defenses.

In this article, we will focus on a specific item from OWASP’s Top 10 for LLM & GenAI list that surged from 6th place in 2023 to 2nd place in 2024: ‘Sensitive Information Disclosure.’ This category highlights the risks associated with leaking proprietary and confidential information.

For the skeptics, sensitive information disclosure also deserved its place in the MITRE ATLAS matrix and was exposed by the recent DeepSeek breach.

As a rule of thumb, fine-tuning models won’t scale for authorization needs and we don’t want the generative models to handle authorization. They are black boxes with several layers of neural networks and are good at handling public information based on its training data. We can’t just rely on the system prompts to do the access control of internal data or redact sensitive information.

That leaves us with the retrieval source as the holy grail to be secured.

Enter: Policy-driven authorization for the retriever

Dynamic authorization through attribute-based access control (ABAC) is critical for RAG-based LLM applications as it proactively adapts access controls based on context and granular user attributes, ensuring secure and compliant data management.

This approach offers several key benefits including:

• Fine-grained & dynamic: Enforces real-time access controls using detailed user and context attributes, when the sensitivity and relevance might change rapidly depending on the query parameters and current user state.
• Scalable security across diverse sources: Applies congruous policies across vector databases, APIs, and relational database sources.
• Seamless integration: With intelligent agents, multiple sources of data and actuators involved, an external policy based on dynamic attributes would ensure consistent access control across all the components.
• Robust auditability and compliance: Provides traceable access decisions that meet regulatory standards and reduce risks.

Use cases demanding access control

RAG-based applications serve diverse industries where secure and context-aware access to information is essential. Here are some key use cases that highlight the need for dynamic authorization:

• An autonomous stock market advisory application that can provide insights based on categorized information to public, paid-subscribers, internal employees and leadership.
• Corporate legal firm providing precedents or case information based on users, case sensitivity, status or location based attributes.
• Pharmaceutical company researchers accessing information related to ongoing trials and past results based on their department or clearance levels.
• Support technicians accessing troubleshooting information only related to their authorized compartments in an aircraft.
• Doctors, law-enforcement or government organizations providing information based on cases assigned, or hierarchies or document categories.

Architecture patterns

We’ve established that access control should be enforced at the data retrieval stage — before augmentation — rather than directly on the model itself, ensuring sensitive information is protected from the start.

Regardless of the data source — whether a vector database, API, or RDBMS — it must be fully secured following the principle of least privilege (PoLP) and infrastructure standards. Additionally, data should be properly tagged using manual or intelligent catalogs. Below are two examples of how access control can be enforced:

Enforcement at Orchestrator

The Orchestrator is the central component, responsible for retrieving data, chaining agents, and constructing and propagating prompts to the LLM.

This pattern is particularly useful when building in-house GenAI applications using tools like LangChain or LlamaIndex, which expose APIs and SDKs for customization. One key advantage of these orchestrators is that they provide a homogeneous API to access multiple data sources, eliminating the need for custom integrations for policy enforcement.

Additionally, they help when data is migrated between one vector database to another as the same LangChain API can support dozens of vector DBs, ensuring seamless transitions without requiring extensive modifications or vendor lock-in.

Enforcement at data source

This pattern is suitable for commercial off-the-shelf (COTS) solutions or software-as-a-service (SaaS) based frameworks that do not provide many options for customizing the orchestrator. This way, the access control mechanism is transparent to the orchestrator.

In this case, the proxy to intercept the query and filter needs to be chosen according to the data source. For example it could be an API gateway or custom vector DB integration or Trino query engine.

How can Axiomatics help?

Axiomatics is the originator and leading provider of runtime, fine-grained authorization powered by attribute-based access control. It delivers dynamic, externalized access control for applications, databases, big data, API gateways, microservices, and RAG solutions. With a standards-based approach, Axiomatics ensures that the same policy can be consistently enforced across various data sources and applications. Additionally, multiple policies can be combined at different levels to meet enterprise needs.

In the context of RAG, these policies can be applied seamlessly to the Orchestrator, Vector Database, API Gateway, RDBMS, etc. Attributes can be dynamically resolved from nearly any source, including databases, APIs, identity providers, and agents. Furthermore, Axiomatics offers the capability to mask sensitive data based on policies during query time or even at the model training time.

Sample Demonstration

Below is a sample implementation using OpenAI LLM model, LangChain as the orchestrator and a vector DB.

• Alice is the subject with the role as ‘USER’ and belongs to ‘Strategy’ department
• The authorization policy is written in ALFA language, and allows Alice access only data that is public or owned by her or if it is internal to the department she belongs to
• During runtime, the Axiomatics’ Contextual Authorization Query engine resolves the user and contextual attributes dynamically, evaluates the access-control-policy and generates the filter expression that is included in the query towards the vector DB
• The actual prompt generated by the orchestrator only contains the data that she is entitled to access (as highlighted in green in the image below)

The same query, when Bob asks, returns almost the opposite response from the LLM, because based on the policy:

• He had access to the information about Acme’s plans to cut 10% of the workforce.
• He is not supposed to know about the partnership with the leading tech firm.

Securing RAG with ABAC

If you have any questions, feel free to contact our solution experts to learn more about how ABAC can help secure your RAG solutions.

Going to Gartner IAM EMEA from March 24 to 25? Our CTO, David Brossard, will be speaking at the event on how you can harness authorization and generative AI for enhanced customer experiences.