Most accounting firms already possess a large body of institutional knowledge. It lives in prior-year workpapers, tax position memos, engagement letters, internal procedures, templates, training notes, and the accumulated judgment of partners and managers. The problem is not that the knowledge is missing. The problem is that it is fragmented across shared drives, email, document-management systems, and individual memory, and that making it queryable creates new risks the firm must deliberately manage.
Retrieval-augmented generation (RAG) is the technical pattern most often proposed for this situation. In simple terms, the system indexes selected documents, retrieves passages that appear relevant to a question, and uses those passages to ground an answer. The appeal is obvious: a staff member could ask how the firm handled a particular revenue-recognition issue last year, or what the current SOP is for a specific notice response, and receive an answer that points back to the firm’s own material rather than a generic internet summary.
That retrieval step is only part of the problem. For accounting work, the harder requirements are permissions, provenance, version status, authority ranking, and confidentiality. A system that surfaces a relevant document is still unusable if the person asking was not authorized to see it, if the document has been superseded, if its source metadata disappeared during ingestion, or if another equally relevant document contradicts it.
The actual information problem inside a firm
Consider a typical mid-size practice. Methodology and quality-control guidance sit in one repository. Prior tax memos live in engagement folders or partner email. Templates and checklists are scattered across shared drives. Standards and public guidance are bookmarked or stored locally. Client-specific history—prior positions, unusual facts, negotiated settlements—is often known only to the people who worked the engagement.
When a junior asks a question, the usual path is interruption: find the senior who remembers, wait for a reply, or dig through folders hoping the right file surfaces. The cost is not only time. It is inconsistency. Different staff members receive different answers depending on who they asked and which version of a memo they found.
A retrieval system can reduce some of that friction by making the corpus searchable in natural language and returning the underlying passages. It does not, by itself, decide which version is current, who is allowed to see which client’s material, or what to do when two memos conflict.
Retrieval quality versus trustworthiness
Retrieval quality asks whether the system found passages that are topically related to the question. Trustworthiness asks whether those passages are the right ones for that user, at that moment, under the firm’s professional obligations.
Four failure modes appear repeatedly once firms move past demos.
Permissioning. Vector search ranks by similarity. It does not rank by authorization. If client A’s workpapers and client B’s workpapers sit in the same index without retrieval-time filters, a question about a technical issue can surface material from the wrong engagement simply because the language is similar. In a multi-client firm this is not theoretical. Engagement isolation and role-based visibility must be enforced at the moment of retrieval, not merely at login. Metadata that records client, engagement, and access rights has to survive chunking and embedding. A poorly designed ingestion pipeline can fail to preserve it.
Provenance. When a document is split into chunks and stored as vectors, the link back to the original file, page, and paragraph can be lost. An answer that cites “the firm’s revenue recognition memo” without a usable path to the source is difficult to review and difficult to defend. Professional work requires the ability to open the underlying document and verify the passage.
Version status and authority. Preliminary figures and audited restatements, draft guidance and final guidance, last year’s policy and this year’s policy often share similar language. A standard retrieval system will return both. Without explicit status fields (draft, final, superseded) and effective dates, the model has no principled way to prefer the authoritative version. The same problem appears when a later amendment modifies an earlier clause: the system may retrieve the original language because it is longer or more densely worded.
Conflicting sources. Two internal memos can reach different conclusions on similar facts. Public standards can be updated while older firm guidance remains in the corpus. When the system retrieves both, a confident synthesis that papers over the disagreement is worse than a clear flag that the sources conflict. The safer behavior is to surface the conflict and require human resolution.
These are document-control and quality-control problems. They are not solved by choosing a better embedding model.
Confidentiality constraints
Client tax return information and other confidential client data introduce additional hard limits. For U.S. tax preparers, IRC §7216 and its implementing regulations can restrict disclosures and uses of tax return information, including disclosures to third-party service providers. Whether a particular AI deployment is permissible depends on the circumstances, applicable exceptions, and, where required, taxpayer consent. AICPA confidentiality rules and Circular 230 duties continue to apply regardless of the technology used. A firm should not place client-identifiable material into consumer AI services unless that specific use has been reviewed and approved under the firm’s confidentiality and data-handling controls.
Even when a private or enterprise deployment is used, embeddings themselves must be treated with care. Research has shown that text embeddings can be inverted to recover substantial portions of the original content under certain conditions. The vector store therefore inherits many of the same access, retention, and deletion obligations as the source documents.
These constraints do not prohibit internal knowledge systems. They do require that the architecture respect existing professional boundaries rather than treating every document as interchangeable training or retrieval material.
A realistic starting point for a small firm
The practical recommendation is narrow.
Begin with a corpus that is internal, non-client-specific, and already intended for firm-wide use: procedures and SOPs, methodology and quality-control guidance, templates and checklists, training materials, and authoritative public standards or regulations the firm relies on. Index only final, current versions. Attach clear status and effective-date metadata. Preserve source links so every retrieved passage can be opened in its original context.
Keep client-specific workpapers, prior engagement memos that contain client facts, and any material subject to engagement-level confidentiality out of the first version. Those materials raise permissioning, isolation, and consent questions that are substantially harder. They can be considered later, once the firm has working controls for the simpler corpus and has decided how engagement isolation will actually be enforced.
The first system should also enforce retrieval-time authorization even on the internal corpus (so that, for example, certain quality-control documents remain visible only to reviewers), log queries and retrieved sources, and be designed to flag rather than smooth over conflicts or low-evidence results. Human review remains the control that turns a retrieved passage into professional work product.
This scope is deliberately limited. It addresses the highest-frequency internal questions—how the firm does something, what the current template is, what the relevant standard says—without immediately confronting the harder problem of making client history safely queryable.
What version one should contain, and what it should exclude
Include
- Current firm procedures, methodology, and SOPs marked as final.
- Standard templates and checklists with version identifiers.
- Training and onboarding materials intended for firm-wide use.
- Selected public or regulatory guidance the firm treats as authoritative, with clear source attribution.
- Metadata for status, effective date, and source location on every document.
- Retrieval-time access controls and an audit log of queries and citations.
Exclude from version one
- Live client tax return information and other client-identifiable data.
- Draft workpapers still under review.
- Superseded memos or policies left unmarked.
- Engagement-specific history that would require cross-client isolation.
- Any consumer or public AI endpoint that would constitute an uncontrolled disclosure.
The goal of the first implementation is not a comprehensive “firm brain.” It is a controlled, permission-aware index of material the firm already considers internal reference, with enough provenance and version discipline that a retrieved answer can be checked. Client-specific knowledge remains a later, more heavily governed project.
Retrieval can make existing firm knowledge easier to find. It does not remove the need for permissions, document control, or professional judgment. Those requirements determine whether the system is usable in an accounting practice.
