From Static Dashboards to Conversational Insights: Using Databricks Genie, Unity Catalog & NYC Open Data
Context
In my previous post, I explored using Tableau to build a BI dashboard for NYC restaurant health inspection data. While it was an effective exercise in visual storytelling, the user experience is shifting in the era of Generative AI. Sometimes, you don’t want to navigate a complex dashboard—you want to have a conversation with your data.
While my previous attempt to build a custom AI agent faced reliability hurdles, leveraging the Databricks Data Intelligence Platform fundamentally changed the timeline. I moved from a raw API endpoint to a governed, conversational AI assistant in under 15 minutes.
Notably, the recent release of the Databricks Free Edition has made this experience accessible and extremely easy to set up, lowering the barrier for developers to move from idea to prototype.
User Story
“As a health-conscious NYC diner, I want a reliable conversational assistant to answer specific questions about restaurant inspection grades and violations so that I can make informed dining decisions on-the-go without navigating complex dashboards.”
My goal was to bridge the gap between Data and Intelligence, proving that with the right foundation, AI can provide immediate utility for everyday decision-making. Here is the architectural breakdown of how I used Databricks Genie to make that happen.
Development method of choice
To move fast while ensuring enterprise-grade reliability, I chose the Databricks Data Intelligence Platform. Specifically, I leveraged Databricks Genie, which allows you to create a natural language interface over your data without writing custom bot logic from scratch.
Basic features
- Conversational Interface: A chat-based space to ask ad-hoc questions about NYC restaurants.
- Real-time API Ingestion: Direct connection to NYC DOHMH Restaurant Inspection Results via the OData API.
- Governed Metadata Layer: Use of Unity Catalog to define “ground truth” business definitions.
- Automatic Visualization: The ability for the agent to autogenerate charts and trends based on query results.
Time it took
The entire first version was built and deployed in under 15 minutes. Because the platform handles the heavy lifting of infrastructure, the focus remained entirely on the data.
The Strategy: Governance as a Prerequisite for AI
The industry often obsesses over the “model,” but the real bottleneck for enterprise AI is Data Readiness. If your metadata is poor, your AI is useless. My goal was to build a system that wasn’t just fast, but reliable enough for real-world decision-making.
Step 1: Data Ingestion (5 minutes)
The foundation is the NYC DOHMH Restaurant Inspection Results. Rather than relying on manual exports, I used a Databricks notebook to call the data via the Open Source OData API.
A huge shoutout to the NYC Open Data team for providing such a robust, well-defined dataset. Having access to high-quality public data was the ultimate game-changer for this project.
The Super-User Move: When dealing with large-scale public datasets, standard API calls often timeout or truncate. By optimizing the $limit parameter to 500,000, I ensured the entire historical record—including violation codes, scores, and grade dates—was ingested into Delta Lake in a single, clean pass.
Step 2: Governance & Metadata (7 minutes)
To prevent AI hallucinations, you must provide a semantic map. I registered the dataset in Unity Catalog, Databricks’ unified governance layer.
Because the NYC Open Data was already so well-structured, the “readiness” of the data allowed for a seamless transition into the AI layer. I used Unity Catalog’s AI-assisted metadata features to rapidly validate business definitions, mapping technical identifiers (like CAMIS or DBA) to intuitive terms (“Restaurant ID” and “Business Name”). In the context of AI, Governance is Guidance. By defining these attributes, I ensured the LLM knew exactly how to join tables and interpret descriptions without manual prompt engineering.
Step 3: Genie Setup (3 minutes)
Once the metadata was in place, the chat interface was functional in seconds. I initialized a Genie Space and connected it to my governed table. Because the metadata was already curated, Genie didn’t need to “guess” the schema; it immediately began translating natural language questions into precise SQL queries.
The Outcome: From Dashboards to Dialogue
The result is a conversational layer that allows for complex discovery that traditional BI often misses:
- Root Cause Analysis: Instead of filtering a chart, I can ask: “Why did [Restaurant Name] lose its ‘A’ rating last quarter?” Genie instantly surfaces the specific violation description.
- Zero-Lag Aggregation: “Identify the top 3 most common critical violations for bakeries in Brooklyn.” Genie handles the filtering on CRITICAL_FLAG and CUISINE_DESCRIPTION automatically.
- Visual Synthesis: As I ask questions, Genie autogenerates the necessary visualizations, transforming a chat into a dynamic report.
Here’s a look into the health ratings of famous NYC restaurant Carbone, with reasons for the B ratings received in 2024 and 2025
Observed limitations
- Metadata Sensitivity: The agent is only as good as its definitions. If I missed a definition for a specific violation ID, the agent struggled to explain it.
- Ambiguous Names: NYC has many restaurants with similar names (e.g., “Joe’s Pizza”). The agent occasionally required me to specify a borough for accuracy.
- Real-time Drift: While the OData API is live, “freshness” depends on the DOHMH update schedule, not the AI itself.
Overall thoughts
In just 15 minutes, I moved from raw data to a production-ready assistant. This project reinforces a core tenet of modern data strategy: Data Readiness is the real AI multiplier.
When your data is well-defined and governed, AI implementation becomes a secondary, almost trivial task. For leaders looking to drive ROI, the lesson is clear: Stop searching for the “perfect” model and start building a robust data foundation. If you have 15 minutes and a tool like Unity Catalog—now easily accessible via the Databricks Free Edition—you have everything you need to turn static data into an interactive asset.
TLDR
I built a conversational AI assistant for NYC restaurant data in under 15 minutes using Databricks Genie and Unity Catalog. The high-quality structure of the NYC Open Data set was a total game-changer, allowing me to focus on governance rather than data cleaning. By establishing a solid data foundation, I moved past static dashboards to a reliable, chat-based interface. With the new Databricks Free Edition, this workflow is now faster and more accessible than ever.
Leave a Reply