Content by wiliam_rosa (13)

Wiliam_Rosa breaks down Azure Databricks multi-table, multi-statement transactions (GA as of Oct 16, 2026), explaining the two transaction modes, how conflict detection works, and why external clients need explicit retry logic to be reliable under real concurrency.

Wiliam_Rosa explains how Azure Databricks cross-engine ABAC enforces Unity Catalog row filters, column masks, and ABAC policies for external Apache Spark reads by routing queries through a Databricks-managed serverless enforcement layer, including required metastore settings, grants, Spark configuration, known limitations, and cost tracking.

Wiliam_Rosa explains how to get earlier, more actionable cost visibility in Azure Databricks using tags, Unity Catalog system tables, account-level budgets, and scheduled SQL alerts—plus Azure-specific features like serverless usage policies that propagate tags into Azure Cost Analysis.

Wiliam_Rosa breaks down Databricks’ LTAP Direct Writes (beta) for Azure Databricks Lakebase Postgres, explaining how it shifts bulk-load work off the primary to avoid impacting live OLTP traffic. The post covers the architecture, key mechanics (sandboxed Postgres, COPY FREEZE, WAL handoff), and shows CLI/SDK examples for creating and monitoring Synced Tables.

Wiliam_Rosa explains Azure Databricks Lakeflow Jobs “Multiple Triggers” (beta), showing how to run a single job from several conditions (schedule, file arrival, table/model updates) and how to configure, monitor, and reason about concurrency and idempotency when triggers overlap.

Wiliam_Rosa breaks down Lakeflow in Azure Databricks, explaining how it unifies ingestion, transformation, and orchestration, and why Unity Catalog-based governance and lineage can reduce the “glue work” of multi-tool data engineering stacks.

Wiliam_Rosa breaks down a practical rollout pattern for testing newly released “frontier” AI models without letting costs spike: give immediate experimental access, but isolate spend risk using layered budgets in Azure Databricks Unity AI Gateway, then decide promotion or rollback using benchmarks, user feedback, and OpenTelemetry cost traces.

Wiliam_Rosa explains how Azure Databricks’ Unity Gateway CLI (ug) centralizes authentication, routing, governance, and usage tracking for multiple coding agents (Claude Code, Codex, Gemini, and GitHub Copilot CLI) by reusing Unity Catalog permissions, rate limits, and budgets instead of scattering provider keys across developer machines.

Wiliam_Rosa explains how Azure Databricks (Databricks Runtime 18) lets you change the partition count of a stateful Spark Structured Streaming query after it’s already in production, without abandoning the checkpoint. The post covers how the repartition works, how to apply it, and what to measure to confirm the impact.

Wiliam_Rosa explains why Azure Databricks Genie Agents need the same ongoing evaluation discipline as production ML models, and walks through Genie Agent Benchmarks, including SQL-based “Chat mode” scoring, LLM-judged “Agent mode”, and a practical workflow for monitoring and improving agent accuracy over time.

Wiliam_Rosa explains a public preview update to Azure Databricks Genie in Slack: when Genie answers include charts, Slack now shows the visualization as an image (not just text). The post also covers the required Slack app reinstall to grant files:write and a SQL query to audit Slack-originated Genie requests.

Wiliam_Rosa breaks down how Azure Databricks’ internal “AI SRE” agent speeds up incident investigation by gathering context automatically before engineers start debugging. The post focuses on an operational pattern: deterministic health checks and version-controlled runbooks first, then LLM-based synthesis—plus the guardrails needed to keep observability access safe.

Wiliam_Rosa breaks down how Spark Structured Streaming’s Real-time mode (as implemented in Azure Databricks) can cut anomaly-detection latency from seconds to sub-millisecond, using an Ethereum transaction pipeline that classifies events as ALLOW or QUARANTINE based on protocol invariants and sensitive-payload checks.