RAG vs. Connected Tools: How Company AI Really Works

Company AI works one of two ways: it copies your data into a search index it controls (classic RAG), or it reads directly from your live tools when you ask (connected-tool access). RAG builds a static snapshot for fast retrieval; connected access queries the source in real time. That single choice, where and when the data is read, shapes freshness, permissions, security, and setup, and it decides which approach fits your team.

The two architectures at a glance

Strip away the jargon and both approaches are solving the same problem: your knowledge lives in dozens of apps, and no single tool can answer a question that spans all of them. Where they diverge is when and where the data is read.

Neither is "wrong." But they make very different tradeoffs, and those tradeoffs land on the people who have to trust the answers.

How classic RAG works: copy everything into a vector index

RAG, retrieval-augmented generation, is the architecture most people picture when they think of company AI. It works in two phases.

Ingest (happens ahead of time):

• Connect to each source and pull the documents out.
• Split them into chunks and convert each chunk into a vector embedding.
• Store those vectors in a dedicated database the AI system owns.

Query (happens when you ask):

• Turn your question into a vector.
• Find the nearest chunks in the index.
• Feed those chunks to the model to write an answer.

The strength here is speed and control over messy, unstructured content. Once everything is embedded, semantic search across a large corpus is fast, and you can tune the ranking. The catch is that everything the AI "knows" is a copy frozen at ingest time. The index is a photograph of your company, and photographs age.

How connected-tool access works: read from the source on demand

Connected-tool access flips the order. Instead of copying data in advance, the AI holds authorized connections to your tools and reads from them the moment a question arrives. This is the model behind the Model Context Protocol and modern connectors: a standard way for an assistant to call a live tool, run a scoped query, and get back exactly what it needs, nothing pre-hoarded.

When you ask "what's the status of the Acme renewal," a connected assistant queries your CRM, your shared drive, and your inbox at that instant, reads the current records, and answers with a note about which tool each fact came from. There's no ingest job to schedule and no separate copy to keep in sync, because the source is the copy.

So the RAG-vs-connected-tools distinction is less a religious debate than a question of where the read happens: RAG reads at ingest, connected tools read at query time. That single choice cascades into everything below.

Freshness: stale snapshots vs. always-current answers

This is the difference your team will feel first.

A RAG index is only as fresh as its last sync. If the pipeline runs nightly, an answer at 4 p.m. may reflect the world as it was at 2 a.m., before this morning's deal closed, before the pricing page changed, before the ticket was reopened. Teams paper over this by syncing more often, but frequent re-indexing is expensive and still leaves a gap.

Connected access has no gap. Because it reads at question time, the answer reflects the current state of the record. For anything that changes during the day, pipeline stages, support tickets, payment status, on-call ownership, real-time reads are the difference between a useful answer and a confidently wrong one. It's also why "real-time RAG" has become a selling point: it's an attempt to close a freshness gap that connected tools never open.

Permissions: re-indexing access vs. respecting live scopes

Permissions are where indexing quietly gets dangerous.

When you copy documents into a vector store, you also have to copy, and continuously reconcile, who is allowed to see what. If someone loses access to a Notion space or a Drive folder, the index has to learn that and re-index, or the AI can surface content that person should no longer see. Access control becomes a second system you have to keep perfectly in step with the first: every source's permission model re-implemented inside the index, every change propagated.

Connected access sidesteps this. Because the read happens live through the user's own authorized connection, each tool enforces its own scopes at query time. If you can't open a document in Salesforce, an assistant reading Salesforce on your behalf can't either. There's no separate permission mirror to drift out of date, the source of truth for "who can see this" stays where it already lived.

Security surface: a second data store vs. data that stays put

Every copy of your data is a new thing to secure. A RAG index is, by definition, a consolidated second store holding embeddings of your most sensitive material, contracts, financials, customer records, sitting outside the tools built to protect them. That store is a high-value target precisely because it aggregates everything in one place.

Connected access keeps data in the tools you already vetted. Nothing is bulk-copied into a new database, so there's no central honeypot to defend. A well-designed connected assistant adds two guardrails on top:

• Read-only by default, so the assistant answers questions without changing anything.
• Encrypted access tokens rather than stored copies of the underlying data.

The result is a materially smaller surface. You're securing connections, not a warehouse.

Setup burden: build-an-index vs. connect-and-go

There's also a plain operational difference in what it takes to go live.

Standing up RAG is an engineering project: build ingest pipelines for each source, decide on chunking, choose an embedding model, run and monitor sync jobs, and re-index as data and permissions change. It's ongoing infrastructure someone has to own.

Connected access is closer to plumbing than construction. You authorize each tool, click through OAuth for Slack, Gmail, your CRM, your drives, and start asking. Adding a source later is another authorization, not another pipeline. For a small team without a data-engineering function, that gap in effort is often the whole decision. You can see the full set of supported tools on the integrations page.

Where each approach wins

To be fair to both, each architecture has a home.

RAG tends to win when:

• The corpus is huge, mostly static, and unstructured, years of archived docs or a large knowledge base.
• Content rarely changes, so a snapshot ages slowly.
• You need heavily tuned semantic ranking over one big pile of text.
• You have the engineering capacity to run and secure the pipeline.

Connected access tends to win when:

• Answers must reflect the current state of live systems.
• Data spans many tools with different, changing permission models.
• You want to avoid creating and defending a second data store.
• You don't have a team to babysit ingest jobs.

Plenty of deployments blend both, index the slow-moving archive, connect the fast-moving systems. The mistake is assuming indexing is the only "real" way company AI can work.

Why connected + read-only is the safer default for SMBs

For a small or mid-sized company, the calculus is lopsided. You don't have a data-engineering team to run sync jobs, you can't afford a breach of a consolidated index, and your most valuable questions, where does this deal stand, what did the customer say, what changed today, are exactly the ones a stale snapshot answers badly.

Connected, read-only access matches that reality. Data stays in the tools you already trust, permissions are enforced by those tools in real time, nothing new is created to leak, and any action that would change something is surfaced as a draft for you to approve rather than executed silently. You get answers that are current, attributable, and scoped, without standing up infrastructure to get them. Indexing is a powerful technique, but for most SMBs it solves a problem they'd be better off never creating: a second copy of everything, always one sync behind.