Compiling a model

Put the compiler's own cache on the volume, so only the first worker pays the compile cost.

Introduction

torch.compile on an image classifier, with the compiler's own cache on the volume, so only the first worker pays the compile cost.

The first thing this example teaches is why it is a container at all: the code path has no compiler.

Building the image

Dockerfile
# A -devel image, not a -runtime one. torch.compile hands its kernels to Triton,
# and Triton shells out to a C compiler at runtime, so the toolchain has to be in
# the image. This is the reason this example is a container at all.
FROM pytorch/pytorch:2.14.1-cuda13.0-cudnn9-devel

# --break-system-packages because this image's python is system managed and pip
# refuses otherwise (PEP 668). Inside a container there is no other environment
# to break.
RUN pip install --no-cache-dir --break-system-packages timm==1.0.30 pillow==13.0.0

# Set here rather than in code: torch reads all three as it loads, and the first
# import anywhere in the process has already fixed them.
ENV HF_HOME=/weights/huggingface \
    TORCHINDUCTOR_CACHE_DIR=/weights/inductor \
    TRITON_CACHE_DIR=/weights/triton \
    PORT=8080

# Fail the build, not the first request. There is no GPU here, so this checks the
# imports rather than the model: enough to catch a wheel that does not load.
RUN python -c "import torch, timm; from PIL import Image; print(torch.__version__, timm.__version__)"

WORKDIR /app
COPY server.py /app/

EXPOSE 8080
CMD ["python", "/app/server.py"]

The cache paths are set in ENV, not in code, because torch reads all of them as it loads, and the first import anywhere in the process has already fixed them. Setting them after that silently does nothing.

--break-system-packages is needed to pip install at all, because this image's python is system managed and pip refuses without it. A container has one environment, so there is nothing else to install into.

The last RUN imports torch, timm and pillow at build time. There is no GPU during a build, so it checks the imports and not the model. That catches a wheel that does not load. Without it, a broken wheel shows up only after a deploy and a request.

Declaring the contract

container.yaml
configVersion: 2
port: 8080
probes:
  readiness: /ready
  liveness: /health
timeouts:
  requestSeconds: 120
  startupSeconds: 2400
endpoints:
  - path: classify
    default: true
    input:
      type: object
      additionalProperties: false
      required: [image]
      properties:
        image:
          type: string
          description: The image to classify, base64 encoded, with no data URI prefix.
        top:
          type: integer
          minimum: 1
          maximum: 10
          description: How many labels to return.
    output:
      type: object
      properties:
        labels:
          type: array
          description: The most likely labels, best first.
          items:
            type: object
            properties:
              label:
                type: string
              index:
                type: integer
              probability:
                type: number
        compiled:
          type: boolean
          description: Whether the graph this request ran was a compiled one.
        seconds:
          type: number

startupSeconds asks for 2400, the maximum, because a first compile is longer than the 300 second default and the platform adds 30 minutes on top of that. A worker that has not answered by then is restarted, so a long compile looks like a crash loop.

The toolchain the compile needs

torch.compile passes its kernels to Triton, and Triton shells out to a C compiler at runtime. A code worker has none.

Measured from inside one with shutil.which:

ToolFound
cc, gcc, g++, clangnone
as, ldnone
make, ninjanone
nvcc, ptxasnone
python3-config/usr/local/bin/python3-config

A code app that compiles anyway crash-loops: the container exits with code 3 on every attempt, and the task stays pending with no error while apps errors stays empty. The reason appears only in apps logs, inside a Triton traceback that ends Failed to find C compiler.

So the image is a -devel one, which ships the toolchain. A dependency you cannot install any other way is the clearest reason to choose the container path. A server you already wrote is a weaker reason.

Where the compile runs

server.py
"""The server the platform talks to, and the readiness gate that hides the compile.

Runware hands this server the request's ``payload`` member alone, not the
envelope around it, so a local curl and a real invocation send the same body.
"""

import base64
import io
import json
import os
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import torch
from PIL import Image

import timm

PORT = int(os.environ.get("PORT", "8080"))
MODEL = "resnet50.a1_in1k"
SIZE = 224
DEFAULT_TOP = 5

state = {"ready": False, "compiled": False, "error": None}


def warm():
    """Load, compile and exercise the model. Runs in a thread so the server can
    answer its readiness probe with a 503 while this is still going."""
    try:
        model = timm.create_model(MODEL, pretrained=True).eval().cuda()
        config = timm.data.resolve_data_config({}, model=model)
        transform = timm.data.create_transform(**config)

        try:
            labels = timm.data.ImageNetInfo()
        except Exception:
            labels = None

        # No cudagraphs. Capturing a graph records the addresses its tensors live
        # at, and a worker that is checkpointed and restored does not get the same
        # addresses back, so a captured graph replays against memory that moved.
        compiled = torch.compile(model, mode="max-autotune-no-cudagraphs", dynamic=False)

        # This pass is the compile. It writes the cache the volume keeps, and it is
        # why readiness takes minutes the first time and seconds afterwards.
        with torch.inference_mode():
            compiled(torch.zeros(1, 3, SIZE, SIZE, device="cuda"))

        state.update(model=compiled, transform=transform, labels=labels, compiled=True, ready=True)
    except Exception as problem:  # noqa: BLE001
        state["error"] = f"{type(problem).__name__}: {problem}"


def classify(raw, top):
    image = Image.open(io.BytesIO(base64.b64decode(raw, validate=True))).convert("RGB")
    batch = state["transform"](image).unsqueeze(0).cuda()

    started = time.time()
    with torch.inference_mode():
        probabilities = state["model"](batch).softmax(dim=-1)[0]
    seconds = time.time() - started

    scores, indices = probabilities.topk(top)
    labels = state["labels"]

    return {
        "labels": [
            {
                "label": labels.index_to_description(int(i)) if labels else str(int(i)),
                "index": int(i),
                "probability": round(float(s), 5),
            }
            for s, i in zip(scores.tolist(), indices.tolist())
        ],
        "compiled": state["compiled"],
        "seconds": round(seconds, 4),
    }


class Handler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def _send(self, status, body):
        raw = json.dumps(body).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(raw)))
        self.end_headers()
        self.wfile.write(raw)

    def do_GET(self):
        if self.path == "/health":
            return self._send(200, {"status": "ok"})

        if self.path == "/ready":
            if state["error"]:
                return self._send(500, {"status": "failed", "error": state["error"]})
            if not state["ready"]:
                # The whole cold start mechanism: no traffic until the compile is
                # done, so no caller ever waits on it inside a request.
                return self._send(503, {"status": "compiling"})

            return self._send(200, {"status": "ok"})

        return self._send(404, {"error": "not found"})

    def do_POST(self):
        if self.path.rstrip("/") not in ("/classify", ""):
            return self._send(404, {"error": f"no endpoint at {self.path}"})
        if not state["ready"]:
            return self._send(503, {"error": "still compiling"})

        try:
            length = int(self.headers.get("Content-Length", "0"))
            payload = json.loads(self.rfile.read(length) or b"{}")
        except (ValueError, json.JSONDecodeError) as problem:
            return self._send(400, {"error": f"bad body: {problem}"})

        top = payload.get("top")
        try:
            return self._send(200, classify(payload["image"], int(top) if top else DEFAULT_TOP))
        except KeyError:
            return self._send(400, {"error": "image is required"})
        except Exception as problem:  # noqa: BLE001
            return self._send(500, {"error": str(problem)})

    def log_message(self, fmt, *args):
        print(f"{self.address_string()} {fmt % args}", flush=True)


threading.Thread(target=warm, daemon=True).start()
ThreadingHTTPServer(("0.0.0.0", PORT), Handler).serve_forever()

The server starts, returns 503 on /ready, and compiles in a background thread. The platform sends no traffic until that turns 200, so no caller ever waits on a compile inside a request.

A server that reports ready before its weights are loaded receives a request it cannot serve.

mode="max-autotune-no-cudagraphs" is deliberate. Capturing a CUDA graph records the addresses of its tensors, and a worker that is checkpointed and restored does not get the same addresses back, so a captured graph replays against memory that moved.

Deploying the app

compiled-modelrtx-pro-6000source
From a URL
runware serverless deploy --from https://runware.ai/docs/serverless/examples/compiled-model.zip
From a clone
runware serverless deploy --container . \
  --id compiled-model \
  --gpu-type rtx-pro-6000 \
  --volume /weights

--container points at the directory, not at a file, and that directory has the Dockerfile and the container.yaml beside whatever the Dockerfile copies.

Caches on a volume

A volume stores anything your app generates once and should not generate again, not only weights. The volumes page lists a compiled kernel cache as one of them.

The weights here are 100MB and the compile is the expensive part. Loading a model is the inverse, and both examples exist for that reason.

Calling the app

import { createClient } from '@runware/sdk'

const client = await createClient({ apiKey: process.env.RUNWARE_API_KEY })

const task = await client.invoke({
  appId: 'compiled-model',
  endpointPath: 'classify',
  payload: {
    image: '<base64 JPEG or PNG>',
    top: 5
  }
}, {
  deliveryMethod: 'sync'
})
import asyncio
import os

from runware import InvokeOptions, Runware


async def main():
    client = Runware(api_key=os.environ["RUNWARE_API_KEY"])
    try:
        task = await client.invoke(
            "compiled-model",
            "classify",
            {
                "image": "<base64 JPEG or PNG>",
                "top": 5
            },
            options=InvokeOptions(delivery_method="sync"),
        )
    finally:
        await client.close()


asyncio.run(main())
curl https://api.serverless.runware.ai/v1/apps/compiled-model/invoke-sync/classify \
  -H "Authorization: Bearer $RUNWARE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
    "payload": {
      "image": "<base64 JPEG or PNG>",
      "top": 5
    }
  }'
cat <<'EOF' | runware serverless apps invoke compiled-model classify --sync -f -
{
  "image": "<base64 JPEG or PNG>",
  "top": 5
}
EOF
{
  "taskId": "7c9e6679-7425-40de-944b-e07fc1f90ae7",
  "payload": {
    "image": "<base64 JPEG or PNG>",
    "top": 5
  }
}

A container app receives the payload your server expects, with no parameter name wrapping it, because there is no handler signature to bind to.

The labels come back at output.labels, best first. A container app has no predictions wrapper: output is your server's response body as it was. A code app puts its handler result at output.predictions[0].

seconds is the forward pass alone, so you can see the compile's effect with no queue time in the number.

What the run cost

CallTime
Build10m41s
First call, empty volume: weights plus the compile14m24s
A restart, compile cache already on the volume19s
A warm worker1.1s
The forward pass itself, warm3.2ms

The first compile runs 45 times as long as a restart that reads the cache. Every worker after the first one is cheap for that reason.

The first forward pass after a restart took 2.59 seconds against 3.2ms warm, so a worker that has read the cache still needs one pass to reach that speed.

max-autotune makes the first start this long. mode="default" compiles in a fraction of the time and keeps most of the benefit. Use it when you do not want to pay a 14 minute cold start even once.