Browse Machine Learning Community (42)

isaiahhoyer describes using Microsoft Discovery agents to adapt Microsoft Premonition biological monitoring systems for citrus pests. The case study covers grounding agents in engineering documents, designing field experiments, preparing permit requests, and generating synthetic wingbeat data for preliminary edge classifiers, with scientific review and human oversight throughout.

madhurinrao explains ABFS Vectored Read for Azure Data Lake Storage and how it reduces request overhead for selective Parquet reads by coordinating noncontiguous byte ranges through the ABFS driver, enabling compatible Spark workloads to adopt the optimization via configuration.

Wiliam_Rosa breaks down Azure Databricks multi-table, multi-statement transactions (GA as of Oct 16, 2026), explaining the two transaction modes, how conflict detection works, and why external clients need explicit retry logic to be reliable under real concurrency.

Ben Sickle outlines Microsoft’s announcements at NVIDIA GTC Berlin 2026 focused on running AI across Azure and customer-controlled environments, including Azure Local sovereign deployments, Foundry Local for on-prem model inference, confidential VMs, open-model endpoints for NVIDIA Nemotron via Fireworks AI on Azure, and GPU-accelerated analytics in Microsoft Fabric.

Johnson Shi explains how AKS GPU memory profiling helps you move from “GPU memory pressure” alerts to the exact functions responsible for allocations, using eBPF-based tracing and Grafana flame graphs so you can right-size GPU memory and validate fixes with repeatable tests.

Wiliam_Rosa explains how Azure Databricks cross-engine ABAC enforces Unity Catalog row filters, column masks, and ABAC policies for external Apache Spark reads by routing queries through a Databricks-managed serverless enforcement layer, including required metastore settings, grants, Spark configuration, known limitations, and cost tracking.

RicardoDuncan rounds up the latest SQL developer tooling updates for VS Code, including new MSSQL extension features, GitHub Copilot schema-aware agent workflows, and Microsoft SQL “skills” for assessing and migrating SQL Server to Azure SQL, plus driver releases aligned with SQL Server 2025.

kinfey explains how to use small language models (SLMs) as typed “decision components” inside an application, then walks through a concrete pipeline using Microsoft Olive to LoRA fine-tune Qwen3.5-0.8B, export to CUDA FP16 ONNX, and validate decisions via option-letter probabilities.

vinilv shows how to build a small, two-node kubeadm Kubernetes cluster on Azure using an N-series GPU VM, then validate GPU scheduling with the NVIDIA device plugin and a PyTorch CUDA test Job. The guide focuses on a transparent dev sandbox for debugging GPU workloads rather than a production-ready design.

Wuyi Weng shows how to run Qwen3.8-Flash-Next across two H100 GPUs on an Azure Spot VM, then instrument and troubleshoot it with OpenTelemetry, Application Insights/Azure Monitor, and Grafana. The post focuses on KV-cache behavior under agentic load, dashboard pitfalls, and a vLLM KV-cache FP8 setting that improves concurrency and time-to-first-token.

Wiliam_Rosa explains how to get earlier, more actionable cost visibility in Azure Databricks using tags, Unity Catalog system tables, account-level budgets, and scheduled SQL alerts—plus Azure-specific features like serverless usage policies that propagate tags into Azure Cost Analysis.

Wiliam_Rosa breaks down Databricks’ LTAP Direct Writes (beta) for Azure Databricks Lakebase Postgres, explaining how it shifts bulk-load work off the primary to avoid impacting live OLTP traffic. The post covers the architecture, key mechanics (sandboxed Postgres, COPY FREEZE, WAL handoff), and shows CLI/SDK examples for creating and monitoring Synced Tables.

Rafia Aqil lays out a practical, end-to-end troubleshooting approach for slow Oracle-to-Power BI refresh and query performance, covering provider choice, FetchSize, query folding, Import vs DirectQuery trade-offs, gateway considerations, and when to shift to Microsoft Fabric Mirroring and OneLake for a different analytics architecture.

Wiliam_Rosa explains Azure Databricks Lakeflow Jobs “Multiple Triggers” (beta), showing how to run a single job from several conditions (schedule, file arrival, table/model updates) and how to configure, monitor, and reason about concurrency and idempotency when triggers overlap.

bobmital announces the general availability of Anyscale on Azure, a managed Ray runtime that runs on Azure Kubernetes Service in the customer’s subscription. The post explains the control-plane/data-plane split, Azure-native identity and governance integration, private connectivity options, and the kinds of AI/ML workloads teams can run end-to-end.

NehaNellepalli outlines an architecture for building a reusable analytics layer around Odoo without modifying the ERP, using Azure Data Factory ingestion into ADLS Gen2, Databricks/Spark transformations, and Synapse/SQL modeling for governed Power BI reporting with row-level security.

rickliev outlines Microsoft’s message for TM Forum Innovate Americas 2026: scaling agentic AI in telecom depends on trusted, governed data plus shared operational context. The post connects “agent-ready” data concepts (meaning, provenance, permissions) with Microsoft Fabric capabilities like OneLake, Fabric IQ, and Real-Time Intelligence.

Wiliam_Rosa breaks down Lakeflow in Azure Databricks, explaining how it unifies ingestion, transformation, and orchestration, and why Unity Catalog-based governance and lineage can reduce the “glue work” of multi-tool data engineering stacks.

Christine Caggiano explains how a multidisciplinary team used the Microsoft Discovery app to design an inspectable, reproducible bioinformatics workflow for phage therapy—connecting curated literature, genomic analysis, and external tools via MCP-based adapters, with validation steps and safety checks built into each stage.

varghesejoji introduces a local “Assessment & Optimization Workbench” that pulls together Azure billing and Databricks workspace evidence so teams can review Databricks cost drivers, utilization, job health, and selected posture checks. It walks through the UI workflow (configure, validate, run, visualize, export) and how to build and run it locally.

Wiliam Rosa explains how to use Azure Databricks ai_query to run a “decision model” (closed-set classification with probabilities) directly from SQL against Unity Catalog-governed tables, avoiding chat-style outputs and extra orchestration. The post covers required warehouse/runtime constraints, structured return types, hosting options, and governance caveats in Unity Gateway.

RajyaLaxmiYellajosyula explains how Oracle GoldenGate is now managed as a native Azure experience within Oracle AI Database@Azure, including Azure portal/API lifecycle operations, Azure control-plane governance (RBAC, tags, Activity Log), and generally available observability via Azure Monitor metrics and Azure Log Analytics events.

Simon Frost explains how he used the Microsoft Discovery app to speed up exploratory analysis of a large wastewater metagenomics dataset, while keeping human scientific judgment in the loop. The post walks through agent-assisted data wrangling, API enrichment, phylogenetics, and visualization to produce auditable, reproducible findings.

Wiliam_Rosa explains why Azure Databricks Genie Agents need the same ongoing evaluation discipline as production ML models, and walks through Genie Agent Benchmarks, including SQL-based “Chat mode” scoring, LLM-judged “Agent mode”, and a practical workflow for monitoring and improving agent accuracy over time.

Sayanika walks through building a conversational analytics agent that queries data in OneLake using Microsoft Fabric semantic models and Fabric IQ Ontology, with a Copilot Studio setup that normalizes messy location inputs (e.g., abbreviations and misspellings) before running ontology-backed queries.

Sayanika outlines an architecture pattern for a retail “growth intelligence” agent that uses Azure AI Foundry (Foundry IQ) to orchestrate reasoning across Microsoft Fabric (Fabric IQ + ontology), Dynamics 365 Finance & Operations via an ERP MCP server, SharePoint policy documents, and web intelligence, backed by Azure AI Search.

Wiliam Rosa explains Azure Databricks’ new (beta) configurable retention for supported system tables, including the 30–3,650 day range, the default behavior when enabled, and how to set the st_retention value via the Databricks Account Settings API for governance and long-term auditing needs.

Mark Gitau and Azin Heidarshenas summarize Azure’s MLPerf Inference v6.1 submissions for DeepSeek-R1 on NVIDIA GB300/GB200 NVL72, highlighting throughput and latency-focused results across Interactive, Server, and Offline scenarios at 1-rack and 4-rack scale.

Wiliam_Rosa explains a public preview update to Azure Databricks Genie in Slack: when Genie answers include charts, Slack now shows the visualization as an image (not just text). The post also covers the required Slack app reinstall to grant files:write and a SQL query to audit Slack-originated Genie requests.

Wiliam_Rosa breaks down how Spark Structured Streaming’s Real-time mode (as implemented in Azure Databricks) can cut anomaly-detection latency from seconds to sub-millisecond, using an Ethereum transaction pipeline that classifies events as ALLOW or QUARANTINE based on protocol invariants and sensitive-payload checks.

KimVaddi explains why many oil & gas AI initiatives stall on inconsistent business meaning rather than data volume, and argues that the “meaning layer” is the real readiness bottleneck. The post connects OSDU, Azure Data Manager for Energy, and Microsoft Fabric IQ’s preview ontology capability as a practical path to governed, cross-domain reasoning.

bharadwajr explains how the Azure team built GEM (Great Experiences Matter), an AI-enabled “feedback-to-fix” loop that aggregates customer feedback at scale, turns it into prioritized insights, and creates actionable Azure DevOps work items with an auditable evidence trail and human review.

newmancheng explains how Microsoft Discovery combined agentic AI, computational predictors, and wet-lab feedback to design and validate a novel organic negolyte for redox flow batteries, with a focus on preserving negative results as durable knowledge that improves subsequent design rounds.

pranabpaul walks through the first part of building a retail assistant demo using Microsoft Fabric and Azure AI Foundry, focusing on setting up the Azure environment, provisioning Fabric capacity, loading sample data into a Lakehouse, creating an ontology, and wiring a Fabric Data Agent to query that data.

wolfgangdesalvador breaks down Azure’s MLPerf Storage v3.0 results for Azure Managed Lustre, focusing on how sustained throughput and fast checkpoint I/O help keep large AI training clusters from stalling.

Rafia Aqil explains how to diagnose and resolve Azure Databricks “capacity” incidents by separating true regional VM SKU shortages from subscription quota limits and Databricks platform ceilings, then applying practical mitigations like SKU swaps, off-peak retries, instance pools, and capacity reservation groups.

Rafia Aqil (co-authored with Sanjeev Nair) lays out a three-phase, practical approach to reducing Azure Databricks spend: first getting visibility into where DBUs and cloud costs go, then tightening cluster and code practices, and finally setting up ongoing cost observability with Unity Catalog system tables, tagging, dashboards, and budgets.

Vishnu Charan TJ explains how new integrations between Azure Blob Storage and the NVIDIA Dynamo stack can speed up LLM inference on AKS, focusing on faster model weight loading for cold starts and KV cache offloading to reduce time-to-first-token latency.

Paul_VicenteH explains why retrieval and reranking are separate stages in a RAG pipeline, then demonstrates hybrid retrieval (including RRF) and learned reranking across Azure AI Search, Azure SQL, PostgreSQL Flexible Server, and Azure Cosmos DB using the SQuAD dataset.

Pamela Fox shares the materials from the “Microsoft IQ Deep Dive with Python” livestream series, showing how to ground AI apps and agents using Web IQ, Work IQ, Fabric IQ, and Foundry IQ via MCP endpoints, plus code samples, slides, and write-ups for building and deploying Python agents.