---
title: Monitoring | Runware Docs
url: https://runware.ai/docs/serverless/monitoring
description: Read the workers serving an app, the traffic on each endpoint, the events behind a scaling decision, and the requests that failed.
relatedDocuments:
  - https://runware.ai/docs/serverless/troubleshooting
  - https://runware.ai/docs/serverless/compute-and-scaling
  - https://runware.ai/docs/serverless/errors
---
## Introduction

Five surfaces tell you what an app is doing, and most questions need just one of them. **Health is the verdict**, in one field. **Workers are the present tense.** **Endpoint traffic is the recent past**, per path. **Events are the control plane's record** of what the platform did. **Errors are the failed requests themselves.**

```bash
runware serverless apps workers my-app
runware serverless apps events my-app
```

## Whether the app can serve at all

Reading an app returns a `health` object, and **it is a different question from `status`**. Status is what you asked for. Health is what the cluster managed. An app can read `active` and serve nothing.

<table><tbody><tr><td>State</td><td>What it means</td></tr><tr><td><code>healthy</code></td><td>Can serve</td></tr><tr><td><code>degraded</code></td><td>Can serve, with less capacity than it asked for</td></tr><tr><td><code>unavailable</code></td><td>Has nothing able to serve</td></tr></tbody></table>

The `reason` says why, and **three of the eight are the platform being short of GPUs**. `capacity_below_floor` means the app keeps a warm floor and has fewer workers than that floor. `demand_unserved` means the app scales to zero and has work queued with nothing running it. `placement_refused` is that same wait once the platform could not place a worker of this GPU type, because it has no capacity on record for it or no complete block of that size is free, and the refusal has outlasted the scale-up grace. Those three are the ones that refuse an invocation with `capacity-unavailable`. A refusal for credit, for a placement policy the worker misses, or for contention is a different reason, and the explanations below say which.

`start_failed` points at your own start instead of the shortfall. A worker kept failing to start for about three attempts without becoming ready, so **the platform stops the app**, records the cause on its events and notifies you. Fix the cause, then resume or redeploy. A worker that keeps failing while other workers serve is a different case, and the app keeps running.

**The other four are not capacity.** `workload_present`, `autoscaler_unhealthy`, `workload_terminating` and `workload_missing` describe a workload that is fine, unmanaged, going away or gone, and a request refused for one of those gets the plain `service-unavailable`.

> [!WARNING]
> **`health` is absent until the platform has observed the app**, and absence is not a claim that it is healthy. Read it as a report that lags the cluster by one observation rather than as a prediction of whether your next call will be served.

An `initializing` app that already has a version to route to **accepts invocations while reporting `unavailable`**. That is the ordinary case during a first deploy. Only an `active` app is refused on health.

## What is holding the app

Health says the app cannot serve. `health.explanations` says what the platform is waiting on, **at most one placement wait and one worker limit at a time**, each including a `message`, a `nextStep` and the `since` it was first observed. A limit also includes the `workers` count it reports.

<table><tbody><tr><td>Type</td><td>What the platform observed</td></tr><tr><td><code>waiting_for_placement</code></td><td>A worker exists and the cluster has not found a node for it. The platform reports the wait and does not guess the cause</td></tr><tr><td><code>capacity_refused</code></td><td>The capacity on record cannot place a worker of that GPU type and count, and the platform is retrying</td></tr><tr><td><code>credit_refused</code></td><td>Credit or billing does not allow another worker</td></tr><tr><td><code>create_refused</code></td><td>A worker could not be created, for a reason the platform does not report</td></tr><tr><td><code>at_max_workers</code></td><td>The worker target sits at the applied maximum while calls wait in the queue</td></tr><tr><td><code>credit_max_workers</code></td><td>A credit hold caps the app, and <code>workers</code> is the maximum it applies</td></tr><tr><td><code>at_min_workers</code></td><td>The target sits at the keep-warm minimum, which is the floor the app keeps while idle rather than a count of running workers</td></tr></tbody></table>

`capacity_refused` reports that the recorded capacity cannot take that shape. Whether the GPUs are busy, not yet recorded, or offered in no group that size is outside what it says.

**`health.applicationLine` is the last line your application printed before its container exited**, so a start that fails for your own reason says so on the app itself. The exit has to belong to the version the app runs now. A capacity shortfall shows a line only when that exit was recorded at or after `since`, and a start failure looks back two minutes further, because the exit commits before the observation that opens the spell. The same sentence is appended to that exit on the events list, and a `503` from an invocation includes it after its own detail.

**Explanations go quiet when there is nothing to report**: while no wait and no limit applies, and also once the platform has not observed the app for ten minutes, so a stale explanation is never shown as current.

## Where a worker is in its life

A worker moves through a fixed sequence, and **the state it is stuck in tells you what is slow**.

<table><tbody><tr><td>State</td><td>What is happening</td></tr><tr><td><code>pending</code></td><td>Waiting for capacity. Nothing of yours is running yet. A <code>statusReason</code> of <code>Unschedulable</code> means no node could give it the GPUs the app asks for</td></tr><tr><td><code>pulling</code></td><td>Fetching the image. Large images pay here</td></tr><tr><td><code>loading</code></td><td>The container is up and ready has not passed. That is your <code>load</code> on the Python path, your own readiness route in a container, and the first checkpoint write on both. <strong>Weights that are not on a volume pay here</strong></td></tr><tr><td><code>ready</code></td><td>Idle and able to accept work</td></tr><tr><td><code>busy</code></td><td>Reserved in the API contract and <strong>not reported today</strong>. A worker serving a task reads <code>ready</code></td></tr><tr><td><code>unhealthy</code></td><td>The pod exists but never became ready, or stopped being ready</td></tr><tr><td><code>draining</code></td><td>Finishing in-flight work before it goes</td></tr><tr><td><code>stopping</code>, <code>stopped</code></td><td>Shutting down, or gone</td></tr></tbody></table>

> [!NOTE]
> That sequence **is** the cold start, broken into its parts. A worker sitting in `pulling` says your image is too big. A worker sitting in `loading` is up and has not passed ready yet. On the Python path that is usually weights being fetched, which a [volume](https://runware.ai/docs/serverless/volumes) fixes. In a container it is your own readiness route, and on both the first checkpoint is written before ready passes. The state tells you which.

`unhealthy` is worth reading precisely. It usually means **the pod came up and did not become ready**, which for a container app points at the readiness route. It covers a second case: a worker whose node stopped being ready reads `unhealthy` with the reason `node-not-ready`, and it keeps that reading even after a stop was requested, until the node comes back and reports how the worker ended. So a stop that leaves a worker here is waiting on the node rather than on your app.

## Traffic, per endpoint

Listing an app's endpoints returns more than paths. Each row may include a `runtime` object with **the latest five-minute window** of that endpoint's traffic, so a single call says which path is busy and which one is slow.

```bash
GET /v1/apps/{appId}/endpoints
```

<table><tbody><tr><td>Field</td><td>What it reports</td></tr><tr><td><code>status</code></td><td>A badge derived from the error ratio: <code>healthy</code> at zero, <code>degraded</code> below 5%, <code>unhealthy</code> at 5% and above</td></tr><tr><td><code>requestsPerMinute</code></td><td>The five-minute request rate</td></tr><tr><td><code>p95RequestDuration</code></td><td>95th percentile request duration, in seconds, across all requests</td></tr><tr><td><code>p99RequestDuration</code></td><td>99th percentile, same basis</td></tr></tbody></table>

**The object is omitted rather than zeroed** when the endpoint had no requests in the window, or when the metrics store cannot be read. A zero that is present is a reading: it means no traffic was measured, not that nothing is known.

> [!NOTE]
> `status` is **computed, not stored**. There is no status column on an endpoint and nothing sets one, so it reflects the last five minutes and nothing longer. An endpoint that failed badly an hour ago and has been quiet since reads as absent, not as `unhealthy`.

Durations cover **all requests regardless of outcome**, because the duration histogram has no status class. A p99 that climbs while the error rate climbs with it may be measuring how long failures take.

`apps endpoints` lists the paths. **The `runtime` numbers come from the API read above.**

## Events

Events are the record of **what the platform did and why**, and they are filterable by kind.

<table><tbody><tr><td>Type</td><td>Covers</td></tr><tr><td><code>deploy</code></td><td>Rollouts, activations, rollbacks</td></tr><tr><td><code>scaling</code></td><td>Workers added and removed, and what triggered it</td></tr><tr><td><code>audit</code></td><td>Configuration and attachment changes</td></tr><tr><td><code>error</code></td><td>Failures the platform recorded against the app</td></tr></tbody></table>

```bash
runware serverless apps events my-app --type scaling
```

An `audit` event belongs to one app. For the organization's own record of who changed what, with the credential behind each write, read [the audit log](https://runware.ai/docs/serverless/api-audit-logs).

**Scaling events are the ones worth reading first** when an app costs more or responds slower than you expected. They say when workers appeared and disappeared, which is the difference between a scaling policy that is wrong and traffic that is spikier than you thought.

A `deploy` event also **lists the endpoint paths a deploy adds and removes**, because renaming a handler method moves a public URL. The deploy goes ahead and the event records the change, so it is where you find out that yesterday's callers are now getting a `404`.

The sentences a `deploy` event records mark different moments, and the one a client polls for is the last.

<table><tbody><tr><td>Sentence</td><td>What it means</td></tr><tr><td><code>deploy accepted: rolling to version N</code></td><td>The rollout was accepted, and <code>activeVersionId</code> already points at N</td></tr><tr><td><code>version N applied; waiting for M keep-warm worker(s) to be able to serve</code></td><td>The workload is applied and activation waits for that floor. Written only when the floor is above 0</td></tr><tr><td><code>version N deployed; app is active</code></td><td>The rollout activated. With a floor of 0 it is written as soon as the apply is accepted</td></tr><tr><td><code>deploy of version N failed (...); the previous version keeps serving</code></td><td>The roll did not come up. The other form of it ends <code>app marked failed</code></td></tr></tbody></table>

A secret or OIDC connector change records its own accept sentence and **re-applies the version already pinned**, so `activeVersionId` stays where it was.

**The default workers page follows `activeVersionId`**, which hides an older version that is still responding while a rollout drains it. `GET /v1/apps/{appId}/workers?versionId=all` shows that process, and the new version is serving once one of its workers is `ready` and the previous version's are `draining`, `stopping` or `stopped`.

> [!WARNING]
> **A worker's `createdAt` does not prove a new secret is serving.** It is when the worker's row was recorded, and the accept event comes before the template is applied, so a worker created just after the change can still be running the previous configuration.

## What the GPU is doing

**Four charts read the GPU itself**: power draw, power utilization, temperature and memory used.

They accept a `worker` selector, so you can follow one pod, and **without it they return the mean across the app's workers**. Metric charts are read through the query catalog rather than by writing your own: see the [Monitoring API](https://runware.ai/docs/serverless/api-monitoring).

**Memory used against the card's capacity is the one to check** when a model loads on one GPU type and fails on a smaller one.

## Failed requests

Events record what the platform did. **Errors record what your callers got.** One page of failed inference requests, newest first, is available per app.

```bash
GET /v1/apps/{appId}/errors?window=24h&statusClass=5xx
```

`window` accepts `1h`, `6h`, `24h`, `7d` or `30d` and defaults to the last day. Omitting `statusClass` returns both `4xx` and `5xx` together, usually what you want first: **the split between the two is the diagnosis.** A wall of `4xx` is callers sending the wrong shape, and a wall of `5xx` is your handler.

Each entry has a timestamp, a level, the message body and a set of string fields.

> [!NOTE]
> The cursor is opaque and **only valid with the `window` and `statusClass` it was issued under**. Changing either mid-pagination means starting again.

## What to look at, in order

When an app is not responding, the questions have an order and each one rules out a layer.

**Is it deployed?** An app still building has no workers by definition. Check its status before anything else.

**Are there workers, and what state?** No workers on an app scaled to zero is correct and not a fault. Workers stuck in `pending` mean capacity, and `unhealthy` means your image came up and did not become ready.

**Is one endpoint the problem, or all of them?** The `runtime` object on each endpoint shows that in one call, and it is the fastest way to tell a broken path from a broken app.

**Did a deploy change something?** `deploy` and `audit` events cover both the code and the configuration, and a configuration-only change still records a version you can compare.

**Is the request even reaching an endpoint?** A `404` with `endpointPath` never reached a worker. See [Errors](https://runware.ai/docs/serverless/errors).

## Logs and handler errors

Read what your workers print with `apps logs`, and `--follow` to watch it live. See the [CLI reference](https://runware.ai/docs/tools/cli/serverless).

A page has **up to `1000` entries**, `20` by default. That is where `--limit` and the API's `limit` stop, and `--errors` narrows both a page and a `--follow` stream to the error lines. A Python traceback arrives as **one entry** rather than one per frame.

**Both streams are collected**, stdout and stderr, and an entry becomes readable **about a minute** later. Lines are kept for **about seven days**, so a longer window goes no further back than the oldest line still held. An empty page just after a worker printed usually means the read was too early, not that the worker printed nothing.

> [!WARNING]
> An app that fails inside your handler surfaces its reason on the task's `error`, and that string includes the message your own exception raised. **Make it say something useful.** The failed-request listing above tells you that a call failed and when, not what your code was doing at the time.