Content by santhosh_ravin1 (5)
Job concurrency and queue monitoring for Spark, now with a capacity-level view (Generally Available)
Santhosh_Ravin1 announces general availability of job concurrency and queue monitoring for Apache Spark in Microsoft Fabric, adding a capacity-level view to help diagnose why Spark jobs are queued or throttled and to understand how individual workspaces consume shared capacity over time.
Santhosh_Ravin1 announces that Microsoft Fabric now extends customer-managed key (CMK) encryption to the Spark processing layer, covering data that Spark jobs read, write, and temporarily stage on compute clusters—aimed at meeting stricter security and compliance expectations for key control during active processing.
Santhosh_Ravin1 explains how Microsoft Fabric Spark’s Native Execution Engine now accelerates CSV reads with a native, SIMD-optimized reader, improving common data engineering workloads without requiring code changes.
Santhosh_Ravin1 introduces Efficient Scaledown (Preview) for Microsoft Fabric Spark, explaining how remote shuffle storage and shuffle migration reduce recomputation during scale-down, improve resiliency, and cut compute costs, with concrete benchmark results and the Spark configuration needed to enable the feature.
Santhosh_Ravin1 explains how Microsoft Fabric’s Native Execution Engine (NEE) speeds up Spark workloads that use Python/Scala UDFs and nested data types. It covers why UDFs and complex types have historically forced costly serialization and row-based fallbacks, what NEE changes in the execution path, and the benchmarked performance gains.
End of content