Weekly Machine Learning Roundup: Faster Pipelines, Smarter Agents
Microsoft is rolling out updates to improve analytics pipelines, automate agent training, and streamline data prep. This work covers the entire journey from pipeline optimization and data engineering to advanced real-world deployment of AI agents.
Vectorized Execution and Data Preparation in Microsoft Fabric
Microsoft Fabric introduces a vectorized C++ execution layer under Apache Spark, bypassing the JVM for faster performance. Technologies like Velox and Gluten route supported Spark jobs directly to the new backend, delivering up to 6× faster batch execution and reduced compute costs. Features are enabled through familiar Spark APIs, with adaptive execution and column pruning. Unsupported tasks still use JVM execution, and Spark Advisor assists with performance monitoring and diagnostics. Dataflow Gen2 in Fabric offers Recent Data recall, storing access history for files and tables so developers can easily revisit important sources. Automated source discovery and easy folder browsing minimize manual navigation, so teams focus on transformation work. Both features aim to support more productive and responsive data engineering workflows.
- Under the Hood: An Introduction to the Native Execution Engine for Microsoft Fabric
- Recent Data Feature: Streamlining Data Preparation in Microsoft Fabric
Scalable Multimodal Agents and Recommender Systems
Engineering teams at Microsoft share methods for improving robustness and scaling of multimodal AI agents. Production RL can struggle with stability and reward design, so five approaches are recommended: staged curricula, adaptive reward segmentation, gradient normalization, constraint shaping, and mixed-horizon training. These enable better live agent performance, more reliable coding tasks, and orchestration at scale. GenRec Direct Learning (DirL) updates move traditional ranking out of feature engineering pipelines by forming unified token embeddings for users, items, and context. New models apply multi-task heads and attention mechanisms for direct generative ranking, simplifying real-time recommendations and providing code examples for batch scaling. Research also addresses RL for multimodal agents and automated verification tools, supporting complex audio, visual, and document workflows, and improving workflow automation.