RAG Is the Easy Part: Designing AI Systems Around Messy Enterprise Data
RAG Is the Easy Part: From Messy Enterprise Data to Production-Grade AI Systems
Why connectors, ingestion, retrieval, permissions, provenance, evaluation, observability and backend architecture matter more than the chatbot demo.
There is a particular kind of AI demo that has become almost trivial to build.
Take a few documents. Split them into chunks. Generate embeddings. Put those embeddings into a vector database. Retrieve the nearest chunks when a user asks a question. Send the retrieved text to a language model.
A surprisingly small amount of code later, you have something that feels intelligent.
Ask:
What is our refund policy?
The system retrieves the relevant paragraph and produces a coherent answer.
That pattern is useful.
It is also dangerously easy to mistake the pattern for the system.
The moment the same idea enters a real organisation, the architecture changes dramatically.
The information is no longer sitting conveniently in a /documents directory.
It may live across:
relational databases,
internal APIs,
SharePoint,
Confluence,
object storage,
support systems,
CRM platforms,
operational tools,
spreadsheets,
PDFs,
source repositories,
internal dashboards,
ticketing platforms,
legacy applications,
and systems nobody has touched in years.
Some of that information changes once a quarter.
Some changes every minute.
Some is globally visible.
Some is restricted by team, geography, customer account, role or individual employee.
Some is structured perfectly.
Some arrives as a 180-page PDF containing tables, diagrams, headers, appendices and scanned pages.
Some questions tolerate approximation.
Others influence financial decisions, customer support, compliance workflows, production operations or clinical processes.
At that point, the interesting question is no longer:
How do we connect an LLM to a vector database?
It becomes:
How do we turn fragmented organisational information into a system that can produce useful, current and verifiable answers without violating the rules that govern the underlying data?
That is a much more substantial engineering problem.
And it leads to a useful observation:
In production AI, the language model is often only the final component of a much larger information system.
The architecture looks simple until the data becomes real
The simplified RAG architecture usually looks something like this:
Documents
↓
Chunking
↓
Embeddings
↓
Vector Database
↓
Retrieval
↓
LLM
↓
Answer
This is not wrong.
It is simply incomplete.
A production system often looks closer to:
ENTERPRISE DATA SOURCES
APIs SQL PDFs SaaS Wikis Logs
│ │ │ │ │ │
└──────────┴─────────┴──────────┴───────────┴──────────┘
│
Connector Layer
│
Discovery + Change Detection
│
Ingestion Pipelines
│
Parsing + Normalisation
│
Structural Extraction
│
Chunking + Metadata Enrichment
│
┌───────────────┴────────────────┐
│ │
Search Index Vector Index
│ │
└───────────────┬────────────────┘
│
Hybrid Retrieval
│
Filtering + Reranking
│
Access Enforcement
│
Context Assembly
│
LLM / Agent
│
Answer Generation
│
Citations + Source Attribution
│
User
│
Feedback Loop
│
Evaluation + Observability
Once the architecture reaches this stage, calling it an "LLM application" starts to hide more than it explains.
The system is simultaneously:
a data integration platform,
a search engine,
a distributed backend,
an access-control boundary,
a document-processing system,
an observability problem,
and a probabilistic application built around language models.
This distinction matters because many failures happen outside the model.
The model can behave exactly as designed while the overall system still gives the wrong answer.
Perhaps the retrieved information is stale.
Perhaps the ingestion pipeline indexed the wrong document version.
Perhaps a table was flattened incorrectly during parsing.
Perhaps the semantic search results were relevant in topic but wrong in time.
Perhaps the user should never have been able to retrieve the source in the first place.
Perhaps the citation looks authoritative but does not actually support the generated claim.
These are not primarily model problems.
They are systems-engineering problems.
1. The first hard problem is usually ingestion
Before a system can answer questions about organisational knowledge, it has to know where that knowledge lives.
This becomes difficult remarkably quickly.
Consider a company where useful information is spread across:
PostgreSQL
Internal REST APIs
SharePoint
Confluence
S3
Google Drive
Jira
ServiceNow
Git repositories
CRM platforms
Support tickets
PDF reports
CSV exports
Internal applications
Each source behaves differently.
A relational database gives you:
rows
columns
schemas
relationships
transactions
An API gives you:
authentication
pagination
rate limits
versioning
failure modes
A document platform gives you:
folders
versions
permissions
owners
comments
metadata
A PDF gives you something that appears structured to a human but may be surprisingly unstructured to software.
An event stream gives you potentially enormous data volume where time may matter more than semantic similarity.
This is why one of the first useful abstractions is often not an embedding function.
It is a connector layer.
Conceptually:
class DataConnector:
def discover(self):
"""Discover available resources."""
...
def fetch(self, resource):
"""Retrieve the current representation."""
...
def changes_since(self, cursor):
"""Return incremental changes."""
...
def permissions(self, resource):
"""Return the resource's access rules."""
...
def metadata(self, resource):
"""Return useful descriptive metadata."""
...
The exact interface will differ by system.
The architectural principle is what matters.
The rest of the platform should consume a reasonably consistent representation of information even when the underlying sources behave very differently.
Without that abstraction, source-specific assumptions leak everywhere.
Soon there is SharePoint-specific logic in the chunking layer.
CRM assumptions inside retrieval.
Database rules inside indexing.
Permission handling duplicated across half a dozen services.
That architecture may survive a proof of concept.
It becomes expensive once the number of sources, users and workflows grows.
2. Production ingestion is a state-management problem
A demo can rebuild everything.
Production often cannot.
Suppose a knowledge system contains several million resources and only a few thousand change during the day.
Rebuilding every embedding and every search index entry is wasteful.
A production pipeline therefore needs to understand state.
Conceptually:
Initial Discovery
↓
Content Fingerprint
↓
Store Version
↓
Later Discovery
↓
Compare State
↓
┌────┼────────┐
│ │ │
new changed deleted
│ │ │
↓ ↓ ↓
add update remove
Now ordinary distributed-systems questions appear.
What happens if a worker crashes halfway through a document?
What happens if an upstream system sends the same event twice?
How do we retry safely?
How do we represent deletions?
How do we know which source version produced a particular chunk?
What happens if the document store succeeds but the vector-index update fails?
How do we reconcile inconsistent downstream state?
This introduces familiar concerns:
idempotency
retries
checkpoints
deduplication
queue backpressure
partial failure
dead-letter handling
versioning
reconciliation
None of them are unique to AI.
But AI systems still need them.
3. Parsing is really information recovery
A PDF is not necessarily a document in the semantic sense.
It is often a description of how a document should appear on a page.
Humans look at a page and perceive:
Title
Section
Paragraph
Table
Subsection
List
A basic extractor may produce:
Title Section paragraph column A column B footer page 8
confidential subsection item...
The visual structure has disappeared.
Tables may lose relationships between cells.
Headers may be repeated inside the body.
Multi-column layouts may interleave.
Footnotes may appear between sentences.
Scanned pages may contain no machine-readable text at all.
This matters because retrieval quality cannot exceed representation quality indefinitely.
If the ingestion pipeline destroys important structure, the embedding model is being asked to infer information that the pipeline already discarded.
A better objective is therefore not:
Convert the file to text.
It is:
Recover as much meaningful structure as possible.
That may include:
document title
heading hierarchy
paragraphs
tables
lists
captions
page numbers
authors
timestamps
source identifiers
document versions
This representation becomes the foundation for everything downstream.
4. Chunking is an information architecture problem
A common RAG implementation starts with something like:
for i in range(0, len(text), 1000):
chunks.append(text[i:i + 1000])
This creates chunks.
It does not guarantee that those chunks remain meaningful.
Imagine a policy document:
Customer Refund Policy
Eligibility
Customers may request a refund within...
Exceptions
Subscriptions purchased through...
Suppose arbitrary chunking separates:
Chunk 21:
Exceptions
from:
Chunk 22:
Subscriptions purchased through third-party marketplaces...
The second chunk contains the actual rule but has lost information about what kind of rule it represents.
Or consider a healthcare document:
Medication Guidance
Contraindications
...
A paragraph detached from its heading can mean something substantially different.
A better representation might look like:
{
"text": "Subscriptions purchased through third-party marketplaces...",
"document_id": "refund-policy-v14",
"section": "Exceptions",
"document_type": "customer_policy",
"effective_date": "2026-08-01",
"source": "internal_knowledge_base"
}
Now the chunk has context.
This leads to an important principle:
Metadata frequently matters as much as embeddings.
Sometimes it matters more.
5. Structure-aware chunking usually beats arbitrary chunking
Different content types have different natural boundaries.
For source code:
module
class
function
method
For contracts:
agreement
section
clause
sub-clause
For support systems:
ticket
conversation
message
resolution
For knowledge bases:
page
section
subsection
paragraph
For research documents:
abstract
method
result
table
discussion
A useful hierarchy might therefore look like:
Document
│
├── Section
│ ├── Paragraph
│ ├── Paragraph
│ └── Table
│
├── Section
│ ├── Subsection
│ │ ├── Paragraph
│ │ └── List
│ └── Paragraph
│
└── Appendix
A chunk can then retain its ancestry:
document_title
section_title
subsection_title
page
source
version
There is no universal best chunk size because the real objective is not:
Generate N-token chunks.
The objective is:
Preserve enough semantic coherence for the information to remain useful when retrieved independently.
That is a very different optimisation problem.
6. Retrieval is a search problem, not a vector-database problem
Embeddings are powerful because they allow semantically related language to match even when the words differ.
A user may ask:
Why did checkout failures increase?
while the internal incident report says:
Payment authorisation errors rose immediately following the gateway deployment.
Semantic retrieval can connect those ideas.
But consider another query:
Show me the notes for incident INC-84721.
The exact identifier may matter far more than semantic similarity.
Or:
Which policy applies to customers in Germany after 1 September?
Now geography and date matter.
Production retrieval therefore often combines several strategies.
User Query
│
↓
Query Understanding
│
┌──────────────┼──────────────┐
│ │ │
↓ ↓ ↓
Metadata Keyword Vector
Filtering Retrieval Retrieval
│ │ │
└──────────────┼──────────────┘
↓
Fusion
↓
Reranking
↓
Context Selection
Each component serves a different purpose.
Metadata filters
Useful for deterministic constraints:
country = Germany
customer_id = 4712
created_at >= 2026-09-01
document_type = policy
Keyword search
Strong for:
IDs
product names
error codes
acronyms
legal phrases
technical identifiers
Semantic search
Strong for:
paraphrases
concepts
descriptions
natural-language questions
Reranking
Useful when an inexpensive first stage retrieves broad candidates and a more capable model determines which ones actually answer the query.
The exact architecture varies.
The important principle is:
Retrieval should be designed as a search system, not reduced to nearest-neighbour lookup.
7. Sometimes the query itself needs processing
Consider:
Why did order fulfilment slow down after yesterday's deployment?
The sentence contains multiple concepts:
{
"intent": "root_cause_analysis",
"domain": "order_fulfilment",
"time_range": "since_yesterday_deployment",
"event_type": "deployment"
}
The system may need to:
identify yesterday's deployment,
retrieve its timestamp,
find relevant operational events after that point,
retrieve incident notes,
compare previous behaviour,
gather evidence,
generate an explanation.
This is no longer simple retrieval.
It is query decomposition and tool-assisted investigation.
The same pattern could appear in other domains.
For finance:
Why did reconciliation failures increase this week?
For support:
Why does this customer keep reopening the same issue?
For healthcare operations:
Which current procedure applies to this referral type?
For SaaS:
What changed before API latency increased?
The model may help interpret intent.
But deterministic software should still own the parts of the workflow where correctness can be explicitly encoded.
8. Indexed knowledge and live data are different things
One distinction becomes increasingly important as systems mature.
Some questions ask:
What does our documentation say?
Others ask:
What is happening right now?
These should not necessarily use the same data path.
For example:
"What is the severity-one escalation policy?"
↓
Knowledge Retrieval
versus:
"What is the current status of ticket 1847?"
↓
Live API
Or:
"What are the eligibility rules for refunds?"
↓
Policy Index
versus:
"Has customer 8421 already received a refund?"
↓
Transaction System
An AI interface may make both interactions feel identical.
The underlying operations are completely different.
This leads to an important architectural principle:
Do not use semantic retrieval to imitate systems that already know the exact answer.
If the answer lives in a transactional database, query the database.
If it lives in an API, call the API.
If it lives in unstructured knowledge, retrieval makes sense.
9. RAG and tool use solve different problems
RAG generally answers:
What does the available information say?
Tool use answers:
What can the system inspect or do?
A mature application may combine both.
Imagine:
Why has customer 318 contacted support three times about the same issue?
The system might:
retrieve previous support conversations,
query the account system,
inspect current subscription state,
retrieve relevant product documentation,
check known incidents,
assemble evidence,
generate a summary.
That system is no longer simply a chatbot over documents.
It is an application orchestration layer with language-model capabilities.
The model may decide which evidence is relevant.
But some operations should remain deterministic.
For example:
customer ID validation
permission checks
database constraints
financial calculations
tenant isolation
should not depend on the model deciding whether they are necessary.
10. Retrieval should produce evidence, not just context
There is a major difference between:
information useful to the model
and:
evidence useful to a human
Suppose a system says:
This request is not eligible because it falls outside the current policy window.
A user may reasonably ask:
According to what?
A useful system should ideally expose:
Refund Policy
Section: Eligibility
Version: 14
Effective: 1 August 2026
and the relevant passage.
Now the user has something they can verify.
This changes RAG from:
Give the model text.
into:
Build an evidence chain from the answer back to authoritative sources.
That evidence chain improves:
trust
debuggability
auditability
expert review
error detection
It also changes the human relationship with the system.
The question becomes less:
Do I trust the AI?
and more:
Does the evidence support this answer?
That is a healthier interface for probabilistic software.
11. Citation presence is not citation correctness
Adding a source link is easy.
Ensuring the source supports the claim is harder.
Suppose an answer says:
The contract renews automatically.
Cancellation requires 30 days' notice.
The customer must submit the request in writing.
One cited paragraph may support only the first statement.
Yet the interface may make the entire answer appear sourced.
A more careful system preserves provenance with each retrieved unit:
{
"chunk_id": "chunk-9381",
"source_id": "contract-117",
"section": "Renewal",
"page": 12,
"version": 4,
"text": "...",
"retrieval_score": 0.91
}
The application can then associate generated statements with evidence.
This creates an important distinction:
Citation presence
≠
Citation correctness
A system with many clickable references can still be poorly grounded.
12. Permissions need to travel with the data
Enterprise AI can create a new path to existing information.
That is useful.
It is also dangerous.
Imagine indexing:
HR documents
financial forecasts
customer records
engineering documentation
executive material
security procedures
internal contracts
If everything enters one retrieval system without preserving access rules, the AI layer may accidentally expose information that users could never access through the original systems.
A simple principle helps:
If a user cannot access information in the source system, the AI layer should not make that information accessible.
Conceptually:
User Identity
↓
Roles / Groups / Tenant
↓
Authorised Resource Set
↓
Retrieval
↓
Permitted Context
↓
Model
Possible implementations include:
metadata-based ACL filters
separate indexes
tenant-specific namespaces
source-level permission checks
direct retrieval from authoritative systems
The correct approach depends on the threat model.
The important thing is that access control exists inside the retrieval architecture, not as an afterthought around the chatbot.
13. Multi-tenancy raises the stakes
A SaaS platform may serve:
Customer A
Customer B
Customer C
One of the worst possible failures is:
Customer A request
↓
retrieves Customer B information
Traditional software already knows how serious this is.
AI systems do not receive an exemption.
Tenant identity should flow through the system:
Request
↓
Authentication
↓
Tenant Context
↓
Retrieval Filters
↓
Tool Calls
↓
Context
↓
Generation
A vector database does not replace application security.
Neither does an agent framework.
14. Freshness is a product requirement disguised as infrastructure
Different information has different useful lifetimes.
Consider:
Employee handbook → days
Legal policy → hours or days
Support tickets → minutes
Inventory → seconds or minutes
Operational telemetry → near real time
A simple product question—
How current does the answer need to be?
—has major architectural consequences.
If daily freshness is acceptable, scheduled ingestion may be sufficient.
If seconds matter, the system may need:
event streams
message queues
CDC
webhooks
live APIs
event-driven workers
This is another reason not to push everything into a vector database.
Sometimes the best architecture is hybrid:
Historical knowledge → indexed retrieval
Current state → live source
The conversational interface can hide that complexity from the user.
The backend should not.
15. AI engineering starts to look a lot like backend engineering
The deeper production AI systems become, the more familiar many of the problems look.
| Traditional backend engineering | AI application engineering |
|---|---|
| API design | Model and tool interfaces |
| ETL | Knowledge ingestion |
| Database indexes | Search/vector indexes |
| Access control | Retrieval permissions |
| Background jobs | Embedding/indexing workers |
| Caching | Retrieval/context caching |
| Monitoring | Model/retrieval observability |
| Unit tests | Evaluation suites |
| Exception handling | Model/tool fallback behaviour |
| Database migrations | Index/schema evolution |
| Distributed systems | Multi-stage AI workflows |
The technology has changed.
The need for engineering discipline has not.
You still care about:
interfaces
retries
idempotency
latency
cost
permissions
failure modes
maintainability
observability
In fact, probabilistic components make deterministic engineering more important.
The conventional software around the model defines the boundaries within which uncertainty is allowed to operate.
16. Evaluation is where AI projects become engineering projects
Traditional software has a useful property.
A deterministic function can often be tested with:
input → expected output
Language-model systems make that less straightforward.
For a retrieval-based application, several independent failures can occur.
Retrieval failure
The relevant evidence exists but was never retrieved.
Filtering failure
Relevant evidence existed but was accidentally removed.
Context failure
The correct information was retrieved but assembled poorly.
Generation failure
The model received good evidence but interpreted it incorrectly.
Grounding failure
The answer contains claims unsupported by the evidence.
Citation failure
The cited source does not actually support the associated claim.
Calling all of these:
bad answer
makes debugging difficult.
A better evaluation pipeline separates them.
Evaluation Question
↓
Known Relevant Evidence
↓
Retrieval
↓
Retrieval Evaluation
↓
Context Assembly
↓
Generation
↓
Correctness Evaluation
↓
Groundedness Evaluation
↓
Citation Evaluation
Now teams can answer much better questions.
Is semantic retrieval poor?
Is metadata filtering too strict?
Is the reranker failing?
Is the model ignoring the context?
Are older document versions outranking newer ones?
Are citations being assigned incorrectly?
That is actionable engineering information.
17. Build the evaluation set before you desperately need it
A useful pattern is maintaining representative questions.
For example:
Question:
"What is the escalation process for severity-one incidents?"
Expected sources:
- Operations Handbook § 7.2
- Incident Management Policy § 3
Important facts:
- Notify responsible lead
- Create incident channel
- Begin escalation within defined SLA
Or:
Question:
"Why was order 8214 not fulfilled?"
Expected systems:
- Order service
- Inventory service
- Fulfilment incident record
Expected relationship:
The item became unavailable after order placement.
These examples form a regression suite.
When changing:
chunking strategy
embedding model
retrieval algorithm
metadata
reranker
prompt
LLM provider
the same evaluation dataset can be rerun.
Without this, AI improvement becomes dangerously anecdotal.
A prompt changes.
Five examples look better.
Everyone celebrates.
Twenty other categories quietly regress.
Evaluation makes iteration measurable.
18. Observability needs to include the reasoning pipeline
Traditional monitoring asks:
Did the request succeed?
How long did it take?
Did an exception occur?
AI systems require additional visibility.
For a request, you may need to understand:
What did the user ask?
How was the query interpreted?
Which filters were applied?
Which documents were retrieved?
What were their scores?
Which sources entered the final context?
Which tools were called?
How many tokens were used?
Which model produced the answer?
How long did each stage take?
Did the user accept or reject the result?
A trace might look like:
request
│
├── intent detection 64 ms
│
├── metadata extraction 31 ms
│
├── keyword search 22 ms
│
├── vector search 58 ms
│
├── reranking 141 ms
│
├── permission filtering 11 ms
│
├── context assembly 8 ms
│
└── generation 1,760 ms
Now when quality falls, engineers have somewhere to look.
When latency increases, the slow stage is visible.
When a strange answer appears, the exact retrieved evidence can be inspected.
Without this, AI debugging becomes guesswork.
19. Latency becomes an architectural budget
AI systems accumulate latency easily.
Imagine:
authentication
↓
query classification
↓
embedding
↓
keyword retrieval
↓
vector retrieval
↓
reranking
↓
tool call
↓
LLM
Every stage adds something.
Individually, each one may look reasonable.
Together, they may feel slow.
It helps to treat latency as a budget.
query processing 100 ms
retrieval 180 ms
reranking 200 ms
external tool 450 ms
generation 1,500 ms
--------------------------------
total 2,430 ms
This makes architectural trade-offs explicit.
Does every request need reranking?
Can retrieval operations run concurrently?
Can a smaller model classify the query?
Can frequently accessed evidence be cached?
Can independent tool calls run in parallel?
Can the response stream before every downstream operation finishes?
Performance engineering does not disappear because a model is involved.
20. Cost is also architecture
A production AI request may involve:
embedding
search infrastructure
reranking
LLM input tokens
LLM output tokens
external APIs
compute
storage
At low usage, inefficient decisions can remain invisible.
At scale, they become expensive.
Suppose every request retrieves 40 chunks.
Perhaps eight are enough.
Suppose every query goes to the most expensive model.
Perhaps 80% could use a smaller one.
Suppose unchanged documents are repeatedly embedded.
Perhaps incremental indexing removes most of the work.
A routing architecture might look like:
Query
│
↓
Complexity Router
/ \
/ \
simple complex
│ │
smaller model larger model
│ │
└───────┬───────┘
↓
Answer
The goal is not to minimise spending at all costs.
It is to spend computation where it creates actual value.
21. Context windows are not databases
Large context windows are useful.
They can also encourage poor architecture.
Why not simply send everything?
Because:
more context
does not automatically mean:
more useful information
Large contexts can introduce:
greater latency,
higher cost,
conflicting versions,
irrelevant material,
weaker signal-to-noise ratio,
harder debugging.
A good retrieval layer attempts to maximise something closer to:
useful evidence
───────────────
total context
The objective is not to fill the context window.
It is to construct the smallest evidence set sufficient for the task.
22. Structured data is still extremely valuable
AI can make unstructured information much easier to work with.
That does not make structured data obsolete.
Suppose an application stores transactions as:
transaction_id
customer_id
amount
currency
status
created_at
failure_code
Embedding every transaction as prose may be less useful than querying the structured system directly.
For example:
SELECT *
FROM transactions
WHERE customer_id = 318
AND status = 'failed'
AND created_at >= NOW() - INTERVAL '7 days';
The model can then explain the result.
This produces a useful combination:
structured queries
+
semantic retrieval
+
natural-language explanation
rather than forcing semantic search to pretend it is a database.
A principle worth keeping:
LLMs do not make databases obsolete. They make good data architecture more useful.
23. The source of truth should remain the source of truth
An AI index should usually be treated as a derived representation.
If policy documents live in a document platform, that platform remains authoritative.
If customer status lives in PostgreSQL, PostgreSQL remains authoritative.
If inventory comes from an ERP system, the ERP system remains authoritative.
The AI layer sits downstream:
Authoritative Source
↓
Derived Representation
↓
Search / Retrieval
↓
AI Application
This makes provenance easier to reason about.
Every derived object should ideally answer:
Where did this come from?
When was it retrieved?
Which version produced it?
Does the source still exist?
When was it last validated?
These questions become more important as people begin relying on the generated answers operationally.
24. Contradictory sources should remain contradictory when necessary
Organisations are messy.
Their documentation often reflects that.
Imagine:
Policy A:
Requests must be approved within 24 hours.
Policy B:
Requests must be approved within 48 hours.
The tempting approach is to let the language model resolve the contradiction.
That may be exactly the wrong behaviour.
Perhaps one document is newer.
Perhaps the rules apply to different regions.
Perhaps one is obsolete.
Perhaps the organisation genuinely has conflicting policies.
A good system should retain enough metadata to investigate:
effective date
department
jurisdiction
owner
status
version
authority
And sometimes the best answer is:
Two active sources contain different instructions.
That is better than an elegant fabrication.
25. Failure should be designed explicitly
What happens when:
retrieval finds nothing,
an API is down,
an ingestion job is behind,
the model provider times out,
two sources disagree,
evidence is insufficient,
the user asks something outside the system's scope?
A weak application improvises.
A stronger one has explicit behaviour.
For example:
Insufficient evidence
↓
Do not invent
↓
State what could not be established
↓
Show the closest relevant sources
Or:
Live source unavailable
↓
Use latest cached snapshot
↓
Display snapshot timestamp
This is ordinary reliability engineering.
A production AI system should not merely succeed elegantly.
It should also fail predictably.
26. What should the language model actually own?
A useful architecture exercise is deciding where probabilistic reasoning adds value.
Things that generally belong in deterministic software:
authentication
authorization
tenant isolation
database constraints
money calculations
resource ownership
critical validation
Areas where models can provide significant leverage:
intent interpretation
semantic retrieval
query rewriting
classification
information extraction
summarisation
tool selection
language generation
reasoning over evidence
This suggests a useful rule:
Use deterministic software where correctness can be encoded explicitly. Use probabilistic models where language, ambiguity and interpretation are the problem.
The model becomes a powerful component.
It does not need to become the operating system for the entire application.
27. The interface does not need to be a chatbot
Enterprise AI is frequently presented through chat.
That does not mean every useful AI system should look like ChatGPT.
The same underlying capabilities can appear as:
search enhancement
incident summarisation
document intelligence
workflow automation
customer-support assistance
analyst tooling
developer tooling
knowledge discovery
compliance review
case summarisation
decision support
Imagine a support interface that automatically shows:
Customer: ABC Ltd
Recent issues:
- Ticket 1847
- Ticket 1921
Relevant knowledge:
- Troubleshooting Guide § 4.2
- Known Issue KI-118
Current account state:
Active subscription
Suggested next investigation:
...
There may be embeddings, retrieval, language models and tool calls underneath.
The user does not necessarily need to see a chat box.
The goal is a better workflow, not a fashionable interface.
28. Start with the workflow, not the model
A technology-first project begins with:
We need RAG.
A workflow-first project begins with:
Engineers spend 25 minutes locating information before they can investigate an incident.
Or:
Support staff routinely search four systems before answering a customer.
Or:
Analysts manually combine database results and reports before producing a weekly summary.
Or:
Employees repeatedly ask questions that are already answered somewhere in internal documentation.
These are measurable problems.
The architecture can then follow the need.
Perhaps the answer is RAG.
Perhaps it is structured search.
Perhaps it is an agent.
Perhaps it is a conventional data pipeline with a natural-language interface.
Perhaps the correct system contains less AI than expected.
Good architecture begins with the workflow.
29. Production AI eventually becomes an organisational problem too
Enterprise information has owners.
Operations owns one dataset.
Finance owns another.
Security governs access.
Legal defines retention.
Individual teams maintain documentation.
Platform teams own infrastructure.
AI systems cut across these boundaries.
Eventually somebody must answer:
Who owns the indexed representation?
Which source is authoritative?
Who approves new data sources?
Who handles stale documents?
Who can request deletion?
Who reviews answer quality?
Who defines acceptable failure?
Who decides whether a workflow is safe enough to automate?
Those sound like governance questions.
They rapidly become engineering requirements.
30. The best production architecture is often boring in the right places
AI engineering currently moves very quickly.
New models, agent frameworks, orchestration libraries and vector databases appear constantly.
Some are genuinely useful.
But mature systems benefit from boring infrastructure.
A strong AI application may contain:
PostgreSQL
Redis
object storage
message queues
workers
REST APIs
search indexes
ordinary authentication
ordinary logging
ordinary monitoring
plus a language model.
That is not a lack of ambition.
It is often good architecture.
Novel technology should be concentrated where it provides novel capability.
Solved engineering problems should remain solved whenever possible.
31. The durable value often sits around the model
Models improve quickly.
Providers change.
Costs decline.
Context windows expand.
Open models get stronger.
A product whose entire technical advantage is:
we call model X
may not have much of a moat.
The harder-to-replicate system usually includes:
data access
connectors
clean metadata
domain modelling
retrieval quality
evaluation datasets
workflow integration
permissions
feedback loops
operational history
The model matters enormously.
But the organisation-specific value is often in the infrastructure surrounding it.
That is good news for engineers.
Good systems architecture becomes more important as models become commoditised, not less.
32. A practical production-readiness checklist
Before calling an enterprise AI system production-ready, I would want reasonable answers to questions like these.
Data
Where does the information originate?
Which sources are authoritative?
How are updates detected?
How are deletions propagated?
Can every derived object be traced back to a source?
How are versions represented?
Retrieval
Why was each result retrieved?
Are exact identifiers handled properly?
Can deterministic metadata filters be applied?
Do we need hybrid search?
Do we need reranking?
How are outdated versions prevented from outranking current ones?
Security
Are source permissions preserved?
Is tenant isolation enforced?
Can restricted content enter prompts accidentally?
Are live tool calls independently authorised?
Generation
Is sufficient evidence available?
What happens when evidence is missing?
Are claims tied to sources?
Can the model distinguish retrieved content from instruction?
Evaluation
Do representative test questions exist?
Can retrieval and generation be measured separately?
Can regressions be detected?
Is user feedback captured?
Operations
Can ingestion jobs retry safely?
Are workers idempotent?
Can individual requests be traced?
Can model providers be changed?
Can bottlenecks be scaled independently?
Economics
What does an average request cost?
Which stages dominate latency?
Where does caching help?
Does every task require the most capable model?
The checklist is less exciting than a chatbot demo.
It is also much closer to what determines whether the system becomes useful infrastructure.
33. A better mental model for enterprise AI
The simplest mental model is:
LLM
+
company data
A more useful one is:
Enterprise AI
=
Data Engineering
+
Backend Engineering
+
Search
+
Security
+
Distributed Systems
+
Evaluation
+
Language Models
The language model remains a critical component.
It gives software a remarkably flexible interface for interpreting and producing language.
But its usefulness depends heavily on everything underneath it.
Poor retrieval produces poor evidence.
Stale ingestion produces outdated evidence.
Weak permission enforcement produces security problems.
Missing provenance produces unverifiable answers.
Missing evaluation produces subjective improvement.
Missing observability produces difficult debugging.
Poor architecture turns successful adoption into a cost or scaling problem.
The LLM is the last few centimetres of a much longer pipeline
The most visible part of an AI application is usually the answer.
A user types something.
A few seconds later, fluent text appears.
But by the time that answer is generated, a considerable amount of engineering may already have happened.
A source had to be discovered.
Its current version had to be identified.
Its access rules had to be retained.
Its structure had to be extracted.
Its information had to be represented.
Its metadata had to survive.
Its searchable representations had to be created.
The user's request had to be interpreted.
Relevant evidence had to be found.
Irrelevant evidence had to be rejected.
Permissions had to be enforced.
Live data may have needed to be queried.
Context had to be assembled.
Only then did the language model receive something useful.
That is why the model can be thought of as the last few centimetres of a much longer information pipeline.
The capability of modern language models is extraordinary.
But production systems still depend on familiar engineering concerns:
data quality
interfaces
search
security
distributed processing
testing
observability
reliability
cost
The real question is therefore no longer:
Can we connect a language model to our data?
We know we can.
The more important question is:
Can we build the surrounding information system so that the answer remains useful, current, secure, observable, affordable and verifiable when people actually depend on it?
That is where the real engineering begins.
About the author
Ahmad W Khan is a software engineer working across backend systems, data-intensive applications and applied AI.
His work has included high-concurrency platforms, analytics systems, healthcare software and distributed backend applications. His current technical interests sit at the intersection of Python, data platforms, retrieval systems, LLM applications and production software architecture, particularly the problem of turning fragmented organisational information into reliable software systems.
More engineering writing and projects are available at ahmadwkhan.com.
