<![CDATA[Evandro Paula]]>https://evandropaula.hashnode.devRSS for NodeSat, 07 Nov 2026 09:35:19 GMT60<![CDATA[Databricks at Data + AI Summit 2026: The Battle for the Data Tier and Workload Diversification Towards $1T Valuation]]>https://evandropaula.hashnode.dev/databricks-at-data-ai-summit-2026-the-battle-for-the-data-tier-and-workload-diversification-towards-1t-valuationhttps://evandropaula.hashnode.dev/databricks-at-data-ai-summit-2026-the-battle-for-the-data-tier-and-workload-diversification-towards-1t-valuationWed, 08 Jul 2026 13:26:36 GMT

Disclaimer: The views here are entirely my own and based only on publicly available information — the Summit keynotes, vendor announcements, and published articles. Nothing in this post is confidential, proprietary, or internal to any employer.

Executive Summary

I spent this year's Data + AI Summit trying to answer one question: what is Databricks actually trying to become? After sitting with the keynotes and product sessions, the answer that kept surfacing is that the company is fighting on two fronts at once. One front is defensive — making sure enterprise data, compute, and governance never have a reason to leave the platform. The other is offensive — pushing aggressively into territory it never used to own: transactional databases, autonomous agents, and cybersecurity. Read together, the announcements look less like a product roadmap and more like a campaign plan for the trillion-dollar valuation Ali Ghodsi keeps talking about. Here's how I'd break down what was shown, and what I think it means.

The defensive front: give the data no reason to leave

The through-line on the defensive side is gravity. Every announcement in this bucket is designed to keep workloads inside the Databricks platform instead of forcing customers to copy or integrate data out to a specialized system.

Lakehouse//RT was the headline. It's a real-time warehouse running on a brand-new engine (internally "Reyden") that promises latencies down to around 10 milliseconds, sub-100ms responses at roughly 12,000 queries per second, and — critically — it runs directly on Delta and Iceberg tables under Unity Catalog with no data movement. The pitch is that you no longer need to replicate hot data into a separate serving database to get operational-grade speed. Databricks claims up to 16x lower end-to-end latency than architectures that require that replication step. The engine itself is undocumented publicly, but the characteristics they describe — NUMA-aware scaling, vectorized execution, high concurrency — echo the morsel-driven, many-core query research that Peter Boncz (MonetDB, VectorWise, DuckDB) is known for, and he's advised Databricks for years. Make of that what you will.

Lakebase is the other big structural move, and arguably the more strategic one. Databricks re-engineered large parts of PostgreSQL to decouple compute from storage, pulling the local disk out into a distributed, multi-tenant storage service so that Postgres becomes effectively stateless. That unlocks sub-second provisioning, branching, autoscaling, scale-to-zero, and analytics directly on operational data. The motivation they gave is telling: in an agent-driven world, it isn't humans spinning up databases anymore — it's agents creating, branching, and dropping databases by the millions. That's a fundamentally different set of demands than a human-operated OLTP system, and it reframes Postgres as infrastructure for machines. Microsoft made a similar bet with the Azure HorizonDB preview at Build. The industry consensus right now seems to be that PostgreSQL is the database of the agentic era.

Apache Iceberg interoperability matured into something real rather than aspirational. Apache Iceberg is now a first-class citizen: UniForm keeps Delta and Iceberg metadata over the same data, sharing works directly with Iceberg REST Catalog clients, inbound federation of foreign Iceberg catalogs (Snowflake, AWS Glue, Hive) is GA, and outbound sharing reaches the broader Iceberg and open-Delta ecosystem. One data point stuck with me: by Databricks' own telemetry, the overwhelming majority of outbound sharing is to Snowflake — a quiet acknowledgment of who the customer overlap actually is.

Rounding out the defensive side were the quieter operational wins: Predictive Optimization, which automates table maintenance (OPTIMIZE, VACUUM, ANALYZE) for Unity Catalog managed tables so nobody has to hand-tune it, and Zerobus, a serverless ingestion service that streams data straight into governed Delta tables and lets you skip the intermediate message bus entirely. Neither is glamorous, but both remove a reason to reach for something outside the platform. That's the whole point of the defensive front.

The offensive front: expand into everything adjacent

If the defensive announcements were about retention, the offensive ones were about expansion — moving the platform into workloads that used to belong to somebody else.

The clearest expression of this is the Genie agentic suite, built around a sharp observation: general-purpose coding agents are bad at data work. They can write flawless Python that produces wrong data, because the hard part lives in the catalog and the semantics, not the source code — and a pipeline can stay "green" while silently emitting garbage after an upstream schema change. Genie tries to close that gap with three pieces. Genie One is a conversational, agentic coworker for pipeline authoring and deployment, generally available across web and mobile. Genie Ontology is the self-improving context layer underneath it, pulling business meaning from the systems people actually work in — Drive, Jira, Slack, Confluence, SharePoint, Gmail — and ranking relevance with a mechanism they call OntoRank. They reported 84.5% first-attempt accuracy on an internal benchmark versus 52.4% for the strongest competing coding agent, the whole point being that grounded, governed context beats a strong model working blind. Genie ZeroOps is the background operator: it detects issues, runs root-cause analysis off Unity Catalog lineage, remediates safely in a sandbox using shallow clones, and validates the fix before it lands.

Sitting above the agents is Unity AI Gateway, a runtime governance layer that stretches Unity Catalog past data and models to cover agents, MCP services, tools, skills, and prompts. It enforces policy at runtime, imposes hard spend caps with smart routing across providers, and gives you unified tracing — with security integrations wired into CrowdStrike, Palo Alto, Zscaler, and Okta. This is the connective tissue that makes the agentic ambitions safe enough for a regulated enterprise to actually adopt.

Then there's Omnigent, an open-source (Apache 2.0) meta-harness for agents. Rather than compete head-on with LangGraph, CrewAI, the OpenAI Agents SDK, or Claude Code, Omnigent positions itself as a "harness of harnesses" — a common execution and governance layer across all of them, with evaluation and benchmarking built in. It's a smart posture: don't fight the frameworks, sit underneath them.

The most surprising offensive move was LakeWatch, a native SIEM built directly on the lakehouse and pitched as an "open, agentic SIEM" aimed squarely at Splunk, Microsoft Sentinel, CrowdStrike, and the rest. The argument is threefold — legacy SIEMs charge punishing per-byte ingestion, security data is siloed by business unit, and human-run SOCs can't keep pace with AI-driven attacks. Unifying all security telemetry in open formats on your own storage, then turning agents loose on it, is a genuinely different economic and operational model. It's in gated preview now, with beta targeted for mid-summer.

The numbers behind the ambition

The financials give the strategy its credibility. Databricks was valued at \(134B in its December 2025 round, reported a revenue run-rate that grew from \)4.8B (55%+ YoY) to \(5.4B (65%+ YoY) by February 2026, crossed a \)1.4B run-rate in AI products alone, and passed $1B in data warehousing. It's been a Gartner CDBMS Magic Quadrant leader five years running. When a company growing that fast tells you it wants to be worth a trillion dollars, the workload diversification above is how it plans to get there: capture the transactional database market with Lakebase, scale agents with Omnigent and Agent Bricks, and enable a new wave of applications built directly on the lakehouse — with governance and security as the trust layer that makes the whole expansion possible.

Where the competition actually stands

The real-time push puts Databricks up against a specific set of players, and it's worth being honest about their strengths rather than waving them away.

  • ClickHouse Cloud remains the cost/performance benchmark for real-time OLAP and observability, with a mature vectorized engine and a serious customer roster. What it lacks against Lakehouse//RT is a unified governance layer and native agent integration — and it still requires moving data out of the lakehouse.

  • SingleStore delivers excellent mixed OLTP/OLAP performance and low-latency serving, but on a proprietary format with a separate governance model and data duplicated out of Delta and Iceberg.

  • Snowflake brings the strongest governance story and the deepest enterprise footprint, but has no native real-time engine and typically needs a separate serving stack for operational analytics.

  • StarTree (Apache Pinot) offers millisecond latency and enormous concurrency for user-facing analytics, at the cost of a separate serving architecture, extra operational complexity, and no unified governance.

The pattern is consistent: each competitor is strong on its own axis, and Databricks' entire counter-argument is "and you don't have to move your data or manage a second governance model". Whether that convenience outweighs best-of-breed specialization is the question every architect will now have to answer for their own workloads.

Final Thoughts

What impresses me isn't any single product — it's the coherence. Defense and offense reinforce each other. The governance and security investments (Unity AI Gateway, LakeWatch) aren't separate bets; they're the enabling infrastructure that makes the offensive expansion into databases and agents palatable to a risk-averse enterprise. Without trust, the land-grab stalls. The open question I'm left with is whether "good enough and already governed here" beats "best-in-class but separate" for the workloads that matter most. For a lot of enterprises carrying real operational and platform-consolidation pressure, I suspect it increasingly will. That's exactly the bet Databricks is making — and this year's Summit was the clearest articulation of it yet.

References

Data + AI Summit

Important: please note not all sessions are available through on-demand.

Articles

]]>