Skip to content

alt text

Databricks to Azure AI Search

Overview

Our main goal for search is to make information more discoverable and accessible. Users will likely want the ability to search through the following data sources:

  • Data tables hosted in Databricks (DBX)
    • Ex. Clinical notes
  • Data tables hosted externally
    • Ex. Archer policies
  • Documents hosted in SharePoint
    • Ex. Nursing practice guidelines
  • Documents hosted externally
    • Ex. Real estate leasing docs

Each use case will require a specific solution to index data and make it searchable. Below, we compare two architectures that cover these different use cases.

Architecture 1 - Databricks-first

flowchart TD
 A[DBX Tables] -- DBX Job --> B[Chunked DBX Tables]
 C[External Tables] -- DBX Job --> D[DBX Document Volume]
 E[SharePoint Documents] -- DBX Job --> D
 F[External Documents] -- DBX Job --> D
 D -- DBX Job --> B
 B -- DBX Vector Search --> G[DBX Vector Search Index]
 G -- DBX Vector Search Endpoint --> H[Search Azure Container App/Bot]

This is essentially how we've been doing things up to this point with Spotlight.

  1. Data lands in one of two places:
    • A Databricks table containing rows of raw data
    • A Databricks volume with folders for each type of document domain
    • In this case, a document ingestion model must be called to first remove styling, extract text, and understand images and tables.
  2. A chunking strategy is developed to split up text blocks into smaller sections, which happens on a scheduled basis.
  3. Chunks are embedded using a custom model or a built-in Databricks model.
  4. Embeddings are stored into a Databricks-hosted vector search index, which is refreshed on a regular basis.
  5. The vector search index is served from a vector search endpoint.

The majority of the difficulty for this process comes from writing a custom app to perform hybrid search across documents. It is simple to perform basic full-text search when keyword spelling is correct and the terms are known. It is also simple to perform vector search by embedding a user query and finding the closest matching chunks. However, marrying both search results together using a reranker requires some custom code. This code could be hosted as an Azure container app or bot, but writing it would be a separate project.

Creating a chatbot using vector search is simple for foundation models hosted in Databricks. The user interface allows addition of multiple vector search indexes as tools that the model can call to generate an answer. This is not possible yet for custom models hosted in Azure that are added to Databricks. You would need to create another custom Azure container app or bot capable of agentic search.

Overall, this strategy is the most viable for keeping development effort low on the data side, but requires significant effort on the user interface side.

Architecture 2 - Azure-first

flowchart TD
 A[DBX Tables] -- DBX Job --> D
 C[External Tables] -- Container App Job --> D
 subgraph Azure Storage Account
 D[Azure Table Storage]
 G[Azure Blob Storage]
 end
 H[External Documents] -- Container App Job --> G
 D -- Azure Table Indexer --> E[Azure AI Search Index]
 D -- Azure Table Indexer --> E
 G -- Azure Blob Indexer --> E
 F[SharePoint Documents] -- SharePoint AI Search Indexer --> E
 E --> I[Azure AI Search Agent]

This architecture requires a significant engineering effort, but results in two products instead of one.

  1. Table data and document data are treated as separate, each landing in a particular location in an Azure Storage Account.
    • Databricks table data is synced to Azure Table Storage by taking advantage of the Delta table change feed. A Databricks Job inspects changes on a regular basis (upserts and deletes) and performs the same changes on the Azure Table Storage.
    • External table data is synced to Azure Table Storage through a Container App Job. This requires a known schema with valid primary keys that will be used to track changes on a scheduled basis.
    • External document data is synced to Azure Blob Storage through a Container App Job. File updates are detected from the MD5 hash. Updated or new documents are upserted while removed documents are deleted.
  2. SharePoint documents are treated separately as Azure AI Search has built-in connectors for those.
  3. No matter where data lands, a custom indexer must be written for each source of data.
    • Indexers extract text/images from documents as well as take care of chunking and embedding text.
    • The easiest way to create an indexer is through the Azure Portal, but a Python SDK is available as well as a JSON definition.
    • Indexers can be tracked with Git by exporting the JSON definition.
  4. Indexers create search indexes that can be synchronized on a scheduled basis if documents are updated regularly.
  5. Search indexes are available to be queried via REST API immediately, but are also capable of being turned into knowledge sources. Knowledge sources can be turned into knowledge bases capable of performic agentic search through the Azure portal.

Once data reaches an Azure resource, it is capable of being turned into a data source for Azure AI search. From there, you can follow this flow to create an AI Search knowledge base capable of agentic search:

flowchart LR
A["Resource (Table/Blob)"] --> B[Data Source]
B -- Indexer --> C[**Search Index**]
C --> D[Knowledge Source]
D --> E[**Knowledge Base**]

This results in the creation of a REST API-accessible search index as well as an agentic search endpoint in the form of the knowledge base. All search is also performed using hybrid search which takes into account both the vector embedding and full text, reranking based on a metric of how well results match the user query.

For users who wish to use the service from Databricks, the simplest method is to call the REST API through a notebook. A network exception would need to be carved out to allow data transfer. We know this is possible as our Open AI endpoints have been added already.

Cost Comparison

The following calculations are for the most expensive use case in mind: clinical notes ingestion. Based on the last discussion had with the data platform team on 2/3/2026, we can approximate that there are about 30 million data rows in their table with a size close to 100 GB. All costs shown are on a monthly basis. Another assumption I make is that there are about 200 notes added daily, which comes out to about 6000 notes each month.

Note

The embedding and LLM costs are not included in the calculation as they are identical for both solutions.

Azure Breakdown

Azure Upfront

Resource Cost
Azure Table Storage (Data storage) $0.045 * 100 GB = $4.50
Azure Table Storage (Write Transactions) $0.075/10K Batch Write 1 * 30M rows = $225
Azure Table Storage (Read Transactions) $0.005/10K Read * 30M rows = $15

Upfront Total: $244.50

Azure Monthly

We would need an Azure AI Search Standard S1 service: Source

Resource Cost
Azure AI Search Standard S1 SKU $245.28
Agentic Retrieval $0 (first 50M tokens free)
Semantic Ranker $3 (First 1000 requests free, then $1/1000 reqs)
Azure Table Storage (Write Transactions) $0.075/10K Batch Write * 6000 rows = $0.05
Azure Table Storage (Read Transactions) $0.005/10K Read * 6000 rows = $0

Monthly Total: $248.33

Databricks Breakdown

This assumes we are using the Storage Optimized vector search, which is the only one that can fit this number of vectors: Source

Databricks Upfront

There are no upfront costs to creating Databricks-hosted vector search indexes.

Total: $0

Databricks Monthly

Resource Cost
Databricks Vector Search Compute $1.28/hr * 730 hr/month = $934.40
Databricks Vector Search Storage $0.046/GB * 100 GB = $4.60

Total: $939

  1. Note: If we were to use regular writes instead of batch writes (which would be slower), we would knock approximately $150 off the cost.  ↩