Ms Fabric Spark Pools take a long time to spin up
11:24 27 Mar 2026

My workspace in MS Fabric is private endpoint enabled so Starter Pools aren't available so I've created a Custom Pool.

My notebooks take 3-5 minutes to start up from cold and don't honour the 20 min timeout set. What I mean is that if I restart the sae pipeline, for example, that contains a notebook, the notebook activity will again take 3-5 minutes to start up.

I've attached screenshots of the custom pool settings.

enter image description here

enter image description here

enter image description here

enter image description here

This is the json of the Pipeline that I'm testing with. As you can see there's only activity.

{
    "name": "pl_Test",
    "objectId": "30445b45-xxx",
    "properties": {
        "activities": [
            {
                "name": "nb_SparkPoolWarmUp",
                "type": "TridentNotebook",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.00:10:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "typeProperties": {
                    "notebookId": "b6dde9a9-xxx",
                    "workspaceId": "29f8113f-xxxx",
                    "sessionTag": {
                        "value": "@replace(pipeline().DataFactory,'-', '_')",
                        "type": "Expression"
                    }
                }
            }
        ],
        "lastModifiedByObjectId": "9a11117d-xxx",
        "lastPublishTime": "2026-03-27T15:04:05Z"
    }
}

And notebook python code

# Welcome to your new notebook

print("This is just to prebake/warmup spark pools to be utlised by notebooks downstream")

This is a screenshot from the Spark Monitoring

enter image description here

How can I better configure so that my notebooks start quicker, execute quicker and honour the timeout setting?

pyspark azure-service-fabric spark-notebook