Weekly Machine Learning Roundup: Spark Control and Real-Time Fabric
This week’s machine learning news includes more advanced Spark pipeline management, improved real-time analytics, and visual upgrade options in Microsoft Fabric and Azure Synapse.
Smarter Pipeline Orchestration in Shared Spark Environments
Admins of Spark workloads can now use a priority-based orchestration approach using job tags like Light/Critical, Medium/High, or Heavy/Best Effort with metadata for optimized job scheduling. This strategy, compatible with Microsoft Fabric and Synapse, supports both fixed and adaptive classification. Copilot-style agents monitor and adjust workload class, reducing human input and increasing stability. Ready-to-use sample notebooks and template tools are available to get started. These changes deepen pipeline control and protection, continuing last week’s surge management efforts.
Real-Time Analytics and ML in Spark Notebooks with Eventstreams Integration
Direct Eventstreams and Spark Notebook integration (preview) in Fabric means instant access to over 30 streaming data sources—like CDC databases or brokers—right from the Real-Time Hub. PySpark code is auto-generated, Entra ID secures access, and one-click imports enable fast prototyping and migration to production. Early community feedback is encouraged. This builds on earlier additions like “Get Data with Cloud Connection,” supporting a smooth transition from batching to real-time analytics.
Improved Visualization for Real-Time Dashboards: Custom Series Colors
Microsoft Fabric’s dashboard series now features customizable data series colors for any chart, so teams can visually separate operational data for easier monitoring and clarity. Documentation covers usage, supporting ongoing dashboard improvements.