Weekly ML Roundup: Foundry evaluation, Fabric pipelines, agent memory

by TechHub

This week's ML roundup focuses on tightening the path from data to deployed models, with Microsoft Foundry expanding model options and leaning into trace-based evaluation that works across clouds. On the data side, Microsoft Fabric added features that reduce day-to-day pipeline overhead, including incremental Delta maintenance, CDC in Copy job, richer IoT streaming metadata, and new preview tooling for Excel ingestion and scheduled Spark pools. We also look at practical building blocks around ML work, from governed data exploration in Data Formulator to persistent agent memory with SQL, plus an infrastructure take on single-GPU training at the 100B+ scale and a simpler approach to Python data pipelines with dlt.

This Week's Overview

Microsoft Foundry expands model choice and production evaluation

Microsoft Foundry's May 2026 roundup adds more options to the model catalog (including Grok 4.3, DeepSeek V4, and Fireworks-hosted models), which makes it easier to swap models based on cost, latency, or capability without changing your surrounding app architecture. On the training side, GPT-5 Reinforcement Fine-Tuning moved to gated general availability (GA), signaling a more stable path for teams that want RL-style tuning workflows but still need controlled access.

A key platform theme this month is tightening the feedback loop between prompts, tool calls, and outcomes. Foundry now emphasizes trace-based evaluation that works across clouds, so you can evaluate agent runs using the traces your app already generates instead of relying only on offline datasets or ad hoc manual review.

Networking and local development also got attention. Managed VNET (virtual network) reached GA, which matters for enterprises that need private networking boundaries and controlled egress/ingress for model calls and connected data sources - and it lines up with last week's OneLake/Foundry thread where governed data access and environment boundaries were becoming part of the default RAG workflow. Foundry Local advanced to 1.1/1.2, and the azure-ai-projects SDK added new skills/toolboxes support across Python, JavaScript/TypeScript, and .NET (including Model Context Protocol (MCP) scenarios), which helps teams keep agent tooling consistent across languages while moving between local and cloud environments.

Microsoft Fabric puts more of the data-to-ML pipeline on rails

This week's Fabric updates cluster around a common goal: reduce the operational overhead between ingestion, transformation, performance tuning, and real-time streaming so teams can spend more time on modeling and analytics and less on platform mechanics.

Incremental Liquid Clustering in Fabric Runtime 2.0

Fabric Runtime 2.0 adds Incremental Liquid Clustering so Delta Lake OPTIMIZE no longer has to repeatedly process the entire table. Instead, OPTIMIZE targets only files that actually need work (clustering, compaction, or deletion-vector cleanup), which is especially useful when tables see continuous streaming writes, frequent MERGE operations, or append-overlap patterns.

The post backs this up with benchmark results across several workload shapes and includes Spark SQL examples plus a reproducible benchmark repo. For teams that rely on Delta tables as feature stores or high-churn analytical tables, this shifts clustering toward an always-on maintenance approach without the same recurring compute tax, which pairs with last week's Fabric operations push (capacity visibility and Spark automation improvements) by making the “keep tables fast over time” work less disruptive.

Data Factory Copy job: CDC GA, plus finer-grained control

Fabric Data Factory Copy job now has Change Data Capture (CDC) with SQL estate generally available, expanding the “keep sources and targets in sync” story beyond simple full loads. The announcement highlights broader connector coverage across SQL platforms and points to additional sources/destinations still in preview, which is a practical signal about where you can standardize today versus where you'll still need alternatives.

The same announcement also calls out expanded Slowly Changing Dimension (SCD) Type 2 support for Fabric Warehouse and Synapse SQL Pool. That matters if you're building analytics-ready tables that preserve history and need consistent handling of deletes (via soft deletes) without custom pipelines.

Separately, Copy job gained operational flexibility: expanded auto partitioning support (including Oracle, SAP HANA, and Fabric Lakehouse tables), JSON payload editing for advanced configurations, and the ability to switch an existing job between full and incremental modes, building on last week's theme of making ingestion and transformation more maintainable (for example via reusable Power Query assets) by reducing rework when pipelines evolve.

Eventstream IoT Hub connector preserves richer event metadata (Preview)

An enhanced Azure IoT Hub source connector for Fabric Eventstream is now in preview, focusing on a common pain point in streaming pipelines: losing IoT Hub system properties on the way into downstream analytics. The connector copies IoT Hub system properties into event metadata so routing, filtering, and diagnostics can use the same context that existed at ingestion time.

This is especially relevant if you're pushing events into Eventhouse and analyzing them with KQL (Kusto Query Language). When device identifiers, message properties, or system headers survive the hop, you can build more reliable real-time monitoring, alerting, and enrichment without inventing your own wrapper schema - a practical continuation of last week's real-time emphasis where fresher signals only help if the streaming context is preserved end to end.

Shortcut Transformations: Excel workbooks to Delta tables (Preview)

Fabric Shortcut Transformations adds a preview path for ingesting multi-sheet Excel workbooks directly into Delta tables with a no-code configuration flow. The feature includes sheet selection patterns, schema validation, schema drift handling, continuous sync, and monitoring, which targets the messy reality that “source of truth” data often starts in spreadsheets.

For ML and analytics teams, this can shorten the time from “new file dropped in a SharePoint or file share” to “queryable Delta table ready for feature engineering.” The drift and validation hooks are the key detail here, since they reduce silent breakage when business-owned workbooks change column names or add new fields, and they fit neatly with last week's schema-evolution theme (for example ALTER COLUMN) where the goal is to absorb change without constant rebuilds.

Custom Live Pools for Fabric Data Engineering (Preview)

Custom Live Pools (preview) introduce scheduled, ready-to-run Spark capacity tied to Environment artifacts, with explicit lifecycle controls and monitoring via the Monitoring Hub. The practical promise is fewer cold starts and more predictable job execution when you run recurring notebooks, transformations, or feature engineering workloads on a schedule.

Because the pool is an environment-aware artifact, teams can align Spark runtime and dependency management with the compute lifecycle, rather than treating clusters as an external detail. For production pipelines, the monitoring and lifecycle controls are as important as the raw scheduling since they determine how debuggable and cost-governable the setup is, and it extends last week's Fabric reliability work (high-concurrency Livy sessions and centralized failure notifications) by pushing more scheduled Spark execution into a controlled, observable shape.

Agent memory, data exploration, and the “small things” that unblock ML work

Several items this week focused on the building blocks around modeling: persistent agent state, iterative analytics workflows, community performance guidance, and pragmatic pipeline tooling. They share a theme of making ML-adjacent work (data prep, orchestration, and operationalization) more repeatable.

Data Formulator 0.7 (open source) adds governed Data Connectors and context-aware agents designed to help teams iteratively prepare data, analyze it, and refine visualizations in a single workspace. The emphasis on governed connectors signals an enterprise angle where provenance and access control matter as much as the charts, echoing last week's OneLake governance and catalog work where “can we use this data safely” was treated as a first-order product surface.

On the app side, a Microsoft Agent Framework walkthrough shows how to implement persistent agent memory by wiring a history/context provider to SQL Server, with guidance for moving the same approach to Azure SQL Database. For developers building assistants that need continuity across sessions, the key takeaway is that “memory” can be treated as a pluggable storage concern rather than something embedded into prompts alone.

For practitioners thinking about infrastructure limits, an analysis of the MegaTrain paper explains how 100B+ parameter training on a single GPU becomes plausible by streaming layers from host memory/NVMe and treating GPU memory as a cache. The Azure-specific angle is practical: your NC-series VM choice, storage throughput, and PCIe bandwidth become first-order constraints, so “one big GPU” only works if the rest of the system can feed it.

Finally, GitHub Open Source Friday featured dlt (from dltHub), a Python SDK aimed at reducing the complexity of production-grade ETL/ELT pipelines. And the Fabric Influencers Spotlight collected community deep-dives on topics like Power BI/VertiPaq performance, lakehouse schema choices, capacity cost optimization, and embeddings (Word2Vec/GloVe), which is a useful follow-on to last week's Spotlight in showing the same practitioner concerns (governance, performance, real-time, and cost) continuing to drive how teams tune the platform under ML.