Writing a model
Declare endpoints with the decorator, load your weights once, and let the build derive the request schema from your handler signatures.
Introduction
A code application is one Python class in one file. You write the class, mark the methods you want callable, and mark the class itself. The platform builds the image, starts the worker, and routes each request to the matching method.
from runware_serverless import endpoint, serve
@serve
class ImageTools:
def load(self) -> None:
self.pipe = load_my_pipeline()
@endpoint
def generate(self, prompt: str, steps: int = 4) -> dict:
steps = 4 if steps is None else steps
return {"image": self.pipe(prompt, steps).encode()}
@endpoint
def upscale(self, image: str, scale: int = 2) -> dict:
scale = 2 if scale is None else scale
return {"image": self.pipe.upscale(image, scale).encode()}That file serves two endpoints, generate and upscale, on one worker.
load
load runs once per worker, before any request reaches it. Put your weights, your pipeline and anything else expensive here, so the cost is paid at startup and callers get a warm worker.
Everything you assign to self in load is available to every handler for the life of that worker.
Keep your heavy imports out of the top of the file. The build imports your file to find your endpoints, and it does that without those dependencies installed, so a module-level import torch fails the build. Import inside load or inside a handler, which is what every example with a real model does, or put the imports in a with imports(): block when a name has to be visible at module scope.
load is required, even when there is nothing to load. The build identifies your class by looking for one that defines load alongside its handlers, so a class without it is not a candidate, and the build fails with no deployment class found when none is left. Write def load(self) -> None: pass when you have nothing to put in it.
There is no matching shutdown hook. load is retried when the GPU is out of memory at startup, waiting for a larger allocation each time. The retry re-instantiates your class, so both __init__ and load have to be safe to run twice.
Serving-time imports
with imports(): is the other place a serving-time import can go, and the one to use when a name has to be visible outside a method.
from runware_serverless import endpoint, imports, serve
with imports():
from my_model.manager import InferenceManager
@serve
class MyModel:
manager: "InferenceManager | None" = None
def load(self) -> None:
self.manager = InferenceManager()
@endpoint
def predict(self, prompt: str) -> dict:
return {"text": self.manager.run(prompt)}At serving time the block runs and binds its names at module scope, so load and your handlers use them without importing again. At build time the platform suppresses whatever the block raises, so the build imports your file with none of your dependencies installed.
The quotes on the annotation are needed. At build time the import was suppressed, so the name is unbound, and a class-level annotation is evaluated when the class body runs. A module with from __future__ import annotations can leave them off, because every annotation in it is already a string.
The block holds imports and nothing else, at the top level of your file. A statement in it that fails for its own reason is suppressed at build time too, and silently. A block inside a function binds its names as locals, where no handler can reach them.
A file with a block needs those dependencies to import at all. A test suite that imports your model file needs them in its dev group, which importing inside load does not ask of you.
The endpoint decorator
@endpoint marks a method as callable. Bare, the path is the method name with each underscore replaced by a hyphen, so remove_background serves remove-background.
It accepts two optional keyword arguments:
pathsets a route the method name cannot produce, such as@endpoint(path="v1/chat/completions").default=Truemarks the endpoint a root invocation calls, one sent without an endpoint path. An app marks one at most.
A positional argument such as @endpoint("upscale"), a keyword other than those two, a second default=True, or a path that is not legal fails the build, before a deployment exists.
A class with no @endpoint methods falls back to predict, so a single-endpoint model can skip the decorator. An app with exactly one endpoint is the default, so you can invoke it without naming the path. The moment you declare one endpoint, only declared paths are served, and predict returns 404 like any other path you did not declare.
One app declares at most 20 endpoints. Going over fails the build.
Endpoints you inherit
Decorated methods are found through the whole inheritance chain, so a shared base class serves its endpoints to every model built on it.
class ImageBase:
@endpoint
def upscale(self, image: str, scale: int = 2) -> dict: ...
@serve
class ImageTools(ImageBase):
def load(self) -> None: ...
@endpoint
def generate(self, prompt: str) -> dict: ...ImageTools serves both generate and upscale. An override serves its own implementation whether or not you decorate it again, so removing the decorator from an override keeps the path served.
Request schemas
The build reads each handler's signature and records one manifest entry per endpoint.
A parameter without a default becomes required. A parameter with a default becomes optional. The schema is closed, so a body with a field outside the signature is rejected.
For the class above, a request to generate must include prompt and may include steps. Sending image to generate fails with a 422 before your worker sees it, because image belongs to upscale.
An omitted optional parameter arrives as an explicit null, not as a missing key, so a Python default alone does not apply. Apply the default inside the handler, as steps = 4 if steps is None else steps does above. A parameter's default is not published, so the platform has no value to send. A field on a model is different, and the Pydantic section below says how.
Every parameter has to be fillable by name, because each request is dispatched as handler(model, **fields). A positional-only parameter, *args or **kwargs fails the build.
Typed requests with Pydantic
A parameter annotated with a Pydantic model is validated into that model before your handler runs, and the model's JSON Schema becomes that field's schema in the contract.
from pydantic import BaseModel
from runware_serverless import endpoint, serve
class Transcribe(BaseModel):
audio_url: str
language: str | None = None
@serve
class Whisper:
def load(self) -> None:
self.pipe = load_my_pipeline()
@endpoint
def transcribe(self, req: Transcribe) -> dict:
return {"text": self.pipe(req.audio_url, language=req.language)}req arrives as a Transcribe instance, not a dictionary, so the annotation is a contract the platform enforces. It behaves the way the same signature does on a FastAPI route.
A field publishes its default and description, and a parameter cannot. The platform sends the published default, so a field with a default arrives as that default, not as null, and your handler does not have to apply it again. Declare a model even for a single optional flag.
Every field you declare on a model is a field the platform can reject before your worker is woken. With a bare dict the caller has to guess the body, and nothing about its contents is checked for them.
An annotation the build cannot resolve becomes "anything", and the build still succeeds. A type that only exists at serving time, imported under if TYPE_CHECKING, is the usual cause: the build reads your file without your heavy dependencies installed, so the name is missing at build time. The endpoint still works, and its schema stops describing that field.
Returning a model declares your response shape in the same way. Only the request side is coerced: the value your handler returns goes out as it is, and the declared type is not enforced. A response type that is not JSON needs encode_response below.
Coercion applies to a model, or a model that may be null. It does not reach a model inside a container: a parameter annotated list[Item] arrives as a list of plain dictionaries, not a list of Item. Declare a wrapper model with the list as a field when you want a batch validated.
Images that arrive as base64
An image arrives at an endpoint as a base64 string, and runware_serverless.schema converts that string to a PIL.Image before your handler runs. Base64RGBImage and Base64RGBAImage are the two annotations, and they differ only in the alpha channel: the first converts to RGB, the second keeps the channels that arrived.
from pydantic import BaseModel
from runware_serverless import endpoint, serve
from runware_serverless.schema import Base64RGBImage
class UpscaleRequest(BaseModel):
image: Base64RGBImage
scale: int = 2
@serve
class Upscaler:
def load(self) -> None:
self.pipe = load_my_pipeline()
@endpoint
def upscale(self, req: UpscaleRequest) -> dict:
return {"image": self.pipe(req.image, req.scale)}The field publishes as a string of format base64, and a payload that is not valid base64, or that decodes to something that is not an image, is refused with a validation error on that field.
These work on a field of a model, not on a parameter. Binding validates a parameter against a Pydantic model, and these types are PIL images, so a bare def upscale(self, image: Base64RGBImage) is never coerced and your handler receives the string exactly as it arrived.
They are the one part of the package that has dependencies, Pillow and Pydantic, so they come as an extra. Add it in the app directory, where the package is installed.
uv add --dev 'runware-serverless[schema]'Without uv, install it with the Python inside the app directory's .venv, at .venv/bin/python or .venv\Scripts\python on Windows. That route does not touch pyproject.toml, so record the extra yourself: find the runware-serverless entry in the dev list under [dependency-groups] and write it as runware-serverless[schema].
serve
@serve registers the class and returns it unchanged, so the file stays importable for the class it defines. Your own tests can import ImageTools and call its methods directly without any of the serving machinery running.
Use it as a decorator above the class, or call serve(ImageTools) after the class body for the same result.
Endpoints are resolved when the module is imported, not on the first request, so a class with nothing to serve fails in the build, with the reason in its log.
A model file where serve registered no class fails the build too, before the manifest is written, so a missing decorator shows up in the build log.
Encoding the response
The schema is derived from your handler's return type. An image model that returns a PIL image cannot declare its wire format, and encoding inside the handler would make the published schema wrong.
encode_response does that conversion. It maps the handler's results to what goes on the wire, and the signature keeps describing what the handler returns.
@serve(encode_response=to_base64_png)
class FluxDev:
...Use it when your response needs a transport encoding. Leave it unset otherwise. Every use of it is a deliberate difference between the published schema and the wire format.
One queue and one worker pool
Every endpoint on an app shares one queue and one worker pool. Hardware and scaling are set once, on the app, and apply to all of them.
The platform enforces that at three points: the decorator accepts only those two keywords, the create and update routes reject an endpoints field, and a container configuration with a hardware key returns 422.
Two endpoints that need different hardware are two apps.
Dependencies
List what your model imports beyond the standard library. There are three places to put that list, and the build reads the first one it finds: a requirements attribute on your model class, then whatever you passed as --requirement on the deploy, then a pyproject.toml and uv.lock in your source directory.
The attribute keeps the list beside the code that needs it, so a file that moves between projects keeps its dependencies.
@serve
class MyModel:
requirements = ["torch==2.5.1", "diffusers>=0.31"]
def load(self) -> None:
...It has to be a list or tuple of strings. The name is reserved, so a handler called requirements is refused and has to be renamed. Declaring the list on the class and on the deploy at the same time fails the build instead of picking one, because picking one would silently drop the other.
Leave runware-serverless out of it. The platform's client libraries are installed into the image at build time and again when the container starts, so a version pinned in your requirements would drift from the services it talks to.
System libraries come from the base image, not from this list. The build starts from python:3.12-slim, which has an interpreter and little else, so ffmpeg and libGL are missing at runtime. --base-image on the deploy pins one that has them, and it has to include Python 3.12 or newer, which the build checks. Bringing a container is the other way.
OpenCV is the common case: opencv-python needs libGL, and opencv-python-headless does not.
What the build includes
The build includes importable code only: a .py, .so or .pyd file, or a directory with an __init__.py. A folder of loose files beside your model file is not a package, so the build skips it.
Each included entry is capped at 100 MiB. Weights belong on a volume or in the mounted cache, fetched in load.
A model file that reads a data file next to itself, say a prompts.txt at the root, will not find it in the worker. Move it inside a package that ships, or read it at runtime from a mounted path.
The build log lists every skipped entry with the reason.
The examples are working apps you can deploy as they are.