ChiefOSby ConversintStart free
Menu
← Back to the library
Data science and analytics

Fabric spark performance

Fabric spark performance. Troubleshoot and optimize Delta Lake and Apache Spark performance in Microsoft Fabric. Use when diagnosing slow Spark jobs, small file problems, data skew, shuffle bottlenecks, out-of-memory errors, V-Order tuning, OPTIMIZE/VACUUM operations, partition strategy, resource profile selection (writeHeavy, readHeavyForSpark, readHeavyForPBI), autotune configuration, Native Execution Engine, broadcast joins, AQE (Adaptive Query Execution), or when Spark notebooks or Spark Job Definitions run slower than expected in Fabric Lakehouse workloads. Diagnose and resolve performance issues in Microsoft Fabric notebooks running Apache Spark. Use when notebooks are slow, Spark jobs are timing out, sessions hit HTTP 430 throttling errors, data skew causes straggling tasks, shuffle operations are inefficient, Delta Lake tables.

Works with the context you provideVersion 1.0.0

Fabric spark performance

Use this skill for the outcome in its description. Start with the supplied request and evidence, identify the specific task, and distinguish planning from execution. Do not trigger solely because the subject happens to be fashion, retail, or a technology named in an example. Read `references/execution-contract.md` before applying a retained source method.

Select the narrowest matching method below, then read its full entrypoint and the resources it requires. If multiple methods cover the same outcome, choose the one matching the user's environment and inputs; do not run them all. Retain each method's input requirements, decision steps, validation, output structure, and failure handling. If prerequisites are absent, use its honest proposal-only fallback or state that its documented path is blocked.

  • **PatrickGallucci/fabric-delta-spark-perf**: Troubleshoot and optimize Delta Lake and Apache Spark performance in Microsoft Fabric. Use when diagnosing slow Spark jobs, small file problems, data skew, shuffle bottlenecks, out-of-memory errors, V-Order tuning, OPTIMIZE/VACUUM operations, partition strategy, resource profile selection (writeHeavy, readHeavyForSpark, readHeavyForPBI), autotune configuration, Native Execution Engine, broadcast joins, AQE (Adaptive Query Execution), or when Spark notebooks or Spark Job Definitions run slower than expected in Fabric Lakehouse workloads. Read `references/PatrickGallucci/skills/fabric-delta-spark-perf/SKILL.md`.
  • **PatrickGallucci/fabric-notebook-perf-remediate**: Diagnose and resolve performance issues in Microsoft Fabric notebooks running Apache Spark. Use when notebooks are slow, Spark jobs are timing out, sessions hit HTTP 430 throttling errors, data skew causes straggling tasks, shuffle operations are inefficient, Delta Lake tables need optimization, V-Order configuration is needed, capacity utilization is high, OOM errors occur, or Spark Advisor warnings appear. Covers Spark session tuning, Native Execution Engine, autotune, custom Spark pools, partition optimization, and Monitoring Hub diagnostics. Read `references/PatrickGallucci/skills/fabric-notebook-perf-remediate/SKILL.md`.
  • **PatrickGallucci/fabric-pyspark-perf-remediate**: Diagnose and resolve Apache Spark performance issues in Microsoft Fabric notebooks and Spark Job Definitions. Use when PySpark jobs are slow, notebooks take too long, Spark stages are skewed, shuffles are excessive, out-of-memory errors occur, Delta Lake writes are slow, or Fabric capacity is throttled. Covers data skew, shuffle optimization, broadcast joins, partition tuning, VOrder, Optimized Write, resource profiles, autotune, native execution engine, small file compaction, and Spark UI interpretation. Keywords include slow notebook, OOM, spill, shuffle, skew, broadcast, repartition, coalesce, OPTIMIZE, VACUUM, Z-ORDER, checkpoint, cache, persist, executor memory, driver memory, spark.sql Read `references/PatrickGallucci/skills/fabric-pyspark-perf-remediate/SKILL.md`.
  • **PatrickGallucci/fabric-spark-compute-perf-remediate**: Diagnose and resolve performance issues in Microsoft Fabric Delta Lake and Spark workloads. Use when remediate slow Spark notebooks, long-running Spark jobs, Delta table query degradation, small file problems, VOrder configuration, resource profile tuning, autotune configuration, capacity throttling (HTTP 430), session startup delays, executor memory errors, skewed partitions, or streaming ingestion bottlenecks. Covers Lakehouse, SQL analytics endpoint, Spark compute, environment configuration, Delta OPTIMIZE, VACUUM, Z-Order, bin-compaction, and Fabric capacity SKU sizing. Read `references/PatrickGallucci/skills/fabric-spark-compute-perf-remediate/SKILL.md`.
  • **PatrickGallucci/fabric-spark-compute-remediate**: Diagnose and resolve Microsoft Fabric Spark compute issues including HTTP 430 throttling errors, job queueing failures, slow session startup, autoscale problems, capacity SKU limits, environment publishing failures, library conflicts, node sizing, burst factor configuration, and VNet provisioning delays. Use when remediate Spark notebooks, Spark job definitions, lakehouse operations, starter pools, custom pools, or concurrency limits in Fabric workspaces. Read `references/PatrickGallucci/skills/fabric-spark-compute-remediate/SKILL.md`.
  • **PatrickGallucci/fabric-spark-perf-remediate**: Diagnose and resolve Apache Spark performance issues in Microsoft Fabric. Use when asked to troubleshoot slow Spark notebooks, optimize Spark SQL queries, fix data skew or shuffle bottlenecks, tune spark.sql.shuffle.partitions or autoBroadcastJoinThreshold, configure resource profiles (writeHeavy, readHeavyForSpark, readHeavyForPBI), enable autotune, resolve HTTP 430 throttling errors, analyze Spark UI stages and executors, optimize Delta Lake writes with VOrder or Optimized Write, run table maintenance (bin-compaction, vacuum, Z-Order), fix small files problems, tune streaming throughput, or right-size Fabric Spark pools and capacity SKUs. Read `references/PatrickGallucci/skills/fabric-spark-perf-remediate/SKILL.md`.

Return the requested deliverable with evidence, assumptions, validation performed, unresolved dependencies, and actual execution status. Source-specific results must be verified independently; importing these methods does not establish a working service connection.