Every organisation I have worked in has a folder like this. Technically, a few hundred people can open it. Practically, nobody ever has, because it sits four levels down in a site nobody remembers, under a filename that meant something to one person in 2019.
That folder has been safe for years — not because the permissions were right, but because finding it required already knowing it was there. Then someone connects an assistant to the document store, and a person who would never have gone looking types a question in ordinary English and gets the answer in three sentences, with a citation.
Search friction was the control. Nobody wrote it down, and nobody decided to remove it.
Obscurity was doing more work than the permissions
Access control in a real organisation is two layers: the formal one — groups, sites, entitlements — and a thick layer of practical friction underneath it. A decade of drift lives in that friction. Folders that inherited the wrong parent. A migration that flattened permissions. “Anyone with the link” switched on for one deadline in 2021 and never switched off. Every internal audit finds some of it, nobody ever finishes fixing it, and that was a defensible call: exploiting it meant knowing what to look for and where.
An assistant exists to remove exactly that friction. Microsoft’s own guidance for Copilot is largely a programme of work to fix oversharing before you switch it on, because Copilot honours your existing permissions faithfully — and your existing permissions are the problem. The tool behaves correctly. The assumption underneath it fails.
The index is a second copy of your permission model
Index content for retrieval and you have created a second place where access decisions get made. Four things go wrong there, and I have seen all four.
The indexing account sees more than any human does. Connectors are usually given a service account with broad read access, because that is the fastest route to complete coverage. Fine, if retrieval then filters results against the asking user’s entitlements. If it doesn’t, your index is a flat pool of everything the crawler could reach, and the model is a very cooperative librarian standing in front of it.
Permissions move and the index lags. Someone comes off a deal team and their access is revoked that afternoon. The index refreshed overnight. There is a window — hours, sometimes days — in which the assistant answers from documents the source system has already locked. Nobody watches that window, because a stale hit and a legitimate hit look identical in an answer.
The extract loses the label. To make a document searchable you chunk it, strip the formatting and store the text. The classification, the header reading “Privileged & Confidential”, the protection label on the file — those are properties of the file, not of the paragraph. Unless you carry them onto each chunk and enforce them at query time, the text reaches the model with nothing attached.
The answer is a new document with no lineage. The output gets pasted into a deck, a ticket, a message. Whatever classification governed the source does not follow it — you are manufacturing unlabelled derivatives of labelled documents, at volume.
Information barriers are about people. An index has no people in it.
This is where it stops being an IT hygiene problem and becomes a supervisory one. Inside information and the wall around a deal team. Conflicts between clients. An HR investigation file. Legally privileged advice.
Each of those controls is enforced in a source system built for it — the matter management tool, the case system, the data room. Copy content from both sides of a barrier into one searchable store and the barrier is no longer enforced by any of them. It is enforced by the filtering logic in a retrieval pipeline, configured by someone who has quite reasonably never heard the phrase “information barrier”. That is not an edge case; it is the default outcome of the most common architecture on the market: index everything, filter later.
There is a harder version I have no tidy answer for. Even with correct trimming, an assistant answers by combining sources — and two documents you may see can support a conclusion neither states and you were not meant to reach. No rule broken, no filter failed. The only control I know is deciding, deliberately, what goes in the corpus.
What actually holds
Filter at query time, against the asking user. Not at index time, and not against a copy of the group membership taken when the connector was configured. Retrieval has to resolve the current user’s entitlements against the current document, on every request. If a vendor cannot describe how that works in one clear paragraph, assume it does not.
Scope the corpus before you tune anything else. The strongest control available is not indexing something. A narrow assistant over a deliberately chosen set of sources beats a firm-wide one nobody can reason about — and most of the value sits in the boring corpus anyway: policies, procedures, product documentation.
Put entitlement cases in the eval set. The discipline I set out in you can’t unit-test an agent, pointed at access rather than accuracy: real questions, asked from specific test accounts, where the correct answer is “I don’t have anything on that.” It is the only mechanism I know that catches a reindex bug or a quietly changed connector.
Log what was retrieved, not just what was said. I have argued that the retrieved material, with versions, is the highest-value log line in a regulated AI system. For an access question it is the only one that counts: which documents were served, to which user, when. Without it you cannot answer “did anyone actually see something they shouldn’t have” — asked under time pressure, by someone who needs a number rather than a reassurance.
The test I would run this week
Take your most sensitive live matter. Write three questions that can only be answered from documents belonging to it. Ask them from a test account that has no business seeing any of it. Do it before go-live, and again after every connector or scope change. It takes an afternoon.
If you get a useful answer, you have found the real state of your permissions — cheaply, privately, and before anyone else did. And the finding is never really about the AI. Your access controls have been approximately right for years because nothing was ever capable of testing them. The assistant is not the risk. It is the audit, and it arrives whether or not you were ready for one.
Sources: OWASP Top 10 for Large Language Model Applications — sensitive information disclosure rose to second place in the 2025 list; Get ready for Microsoft Copilot with SharePoint Advanced Management.
If your team is putting an assistant over its own documents and wants the access side to hold up before it goes live, that is exactly the kind of thing we work through with teams. Talk to us if it’s useful, or see how we run it in-house.