OIDC connectors

Let your workers call AWS, Google Cloud or your own services with short-lived tokens that Runware signs, instead of cloud keys you store and rotate.

Introduction

An OIDC connector lets your workers call AWS, Google Cloud or any service that trusts OpenID Connect without a stored key. Runware signs a short-lived token for each worker, your cloud account trusts that token through federation, and your code uses the AWS or Google SDK unchanged.

import boto3

s3 = boto3.client("s3")   # no access key anywhere

Compare that with a secret that stores an access key. The key is valid until someone rotates it, it works from any machine that has a copy, and it is one more thing to keep out of logs. A connector has no credential at all. The token proves which app in which environment is asking, it expires within the hour, and your cloud decides what that app may do.

Every Runware step on this page is shown for the CLI and the dashboard. For the HTTP API, see the OIDC connectors reference.

The CLI commands assume you are signed in with runware auth login. The apps commands and connectors trust address production unless you pass --environment, set RUNWARE_ENV, or set a default in the CLI config, in that order of precedence. The connectors commands take none, because a connector belongs to your organization, and the environments commands name the environment as their argument. See the CLI reference for both.

How it works

  1. You create a connector in your organization. It says where the token goes: an IAM role, a Google workload identity provider, or an audience of your own.
  2. You attach it to an app. Its workers start getting a token file and the environment variables the SDKs read.
  3. You tell your cloud to trust the Runware issuer, and only for tokens whose subject identifies your app.
  4. When the SDK needs credentials, it sends the token to the cloud's security token service and gets temporary credentials back.

The platform renews the token file in place while the worker runs, so your code only ever reads it.

The values your cloud needs

Every trust rule you write is built from three claims of the token.

ClaimValue
isshttps://oidc.serverless.runware.ai, also returned as issuer on every connector
audWhat your cloud checks the token against, one string that depends on the kind. Returned as audience on the connector
subWhich app in which environment is asking, returned as subject on an attachment

An aws connector has runware:<organization uuid> as its audience unless you set your own, and a generic one has the value you set. A gcp connector always has the provider's full resource name, the value Google expects.

//iam.googleapis.com/projects/<project number>/locations/global/workloadIdentityPools/<pool id>/providers/<provider id>

The subject lists the organization, the environment and the app, in that order.

worker:org:<organization uuid>:env:<environment uuid>:app:<app id>

The order of the sub segments and the : between them are stable, so a pattern such as worker:org:<organization uuid>:env:<environment uuid>:* keeps matching every app of one environment. The worker: prefix means a rule written for workers matches no other kind of Runware token.

The environment segment is the environment's id, not its name, so a staging copy of an app never matches a production trust rule, and a new environment that reuses a deleted one's name gets none of its trust. The app segment is the app id, so an app you delete and create again with the same id in the same environment matches the same rule.

The audience does not keep other Runware customers out. You choose a gcp provider name and a generic audience, and anyone could choose the same one. Every trust rule you write must check sub, or org_id where your verifier reads claims. A rule that checks only aud can accept tokens from apps that are not yours.

Find your values

Every command on this page reads its values from the shell, so set them once and the rest is copy and paste. Start with the two you already have.

export RUNWARE_API_KEY=...   # the key you call the Serverless API with
export APP_ID=my-app         # the app the connector is attached to

The environment has its own prefix, so a trust rule can name it before any app or connector exists. GET /v1/environments/{name} returns oidcSubjectPrefix, which every worker token in that environment starts with.

runware serverless environments show production --format json
{
  "name": "production",
  "id": "0190a6a8-0000-7000-8000-0000000000e1",
  "oidcSubjectPrefix": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:"
}

environments show prints the prefix in its table as well. The dashboard does not show it, so it is read from the CLI or the API.

The value stops at the colon and has no *. An AWS StringLike condition appends one, and a GCP attribute condition uses it as it stands, with assertion.sub.startsWith(...).

The prefix follows the environment's id, not its name. An environment deleted and created again under the same name gets a new id, so it gets a new prefix, and a rule that named the old one stops matching. An environment's name cannot change, so that is the only way a prefix moves.

For one app, the organization and environment ids come with the connector too, and it reports them rather than you assembling them. Attach it to an app and list the app's connectors: each one includes the exact subject its tokens use, and a subjectPattern for every app of that environment.

runware serverless apps connectors list $APP_ID

Open the app, go to Settings and pick OIDC connectors. Each attached connector shows its Subject with a button that copies it, and the environment variables the runtime sets for it.

{
  "data": [
    {
      "name": "s3-reader",
      "kind": "aws",
      "audience": "runware:0190a6a8-0000-7000-8000-00000000000a",
      "issuer": "https://oidc.serverless.runware.ai",
      "aws": { "roleArn": "arn:aws:iam::123456789012:role/runware-s3-reader" },
      "subject": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:app:my-app",
      "subjectPattern": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:*",
      "id": "...",
      "createdAt": "...",
      "updatedAt": "..."
    }
  ]
}

Both uuids sit inside the prefix, as worker:org:<organization uuid>:env:<environment uuid>:, so cut them out of it.

export PREFIX=worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:

export ORG_UUID=$(echo "$PREFIX" | cut -d: -f3)
export ENV_UUID=$(echo "$PREFIX" | cut -d: -f5)

An app's subject has them in the same places, followed by app:<app id>.

An app's own subject exists once the connector is attached to it, so for an app rule the app comes first. For an environment rule the prefix above is enough. The cloud side can wait: the connector only records the role or provider, and the worker asks for credentials only when your code does.

Last, your cloud account's own identifier, which is the one value Runware does not hold. Set the one your cloud uses.

# on AWS
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

# on Google Cloud
export PROJECT_ID=my-project
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')

Google wants the project number, not the project id. It is the all-digit one, and a provider name built with the id is rejected.

What your cloud should trust

A trust rule covers one app or every app of an environment, and that choice decides the value you write in your cloud.

ScopeWhat it admitsThe value to write
One appThe app the connector is attached to, in one environmentsubject, the exact sub its tokens have
One environmentEvery app of your organization in that environment, including apps created latersubjectPattern, or the environment's oidcSubjectPrefix

On AWS the first is a StringEquals condition on sub, and the second a StringLike with * appended. On Google Cloud the first grants to subject/<the subject>, and the second to attribute.environment_id/<the environment id>.

--scope environment trusts every app of your organization in that environment, so pick it only when all of them may reach the resource.

Each rule covers one environment. A rule written for staging does not admit a token from production.

Connecting to AWS

Create and attach the connector

Name the IAM role you will create. The role does not have to exist yet, because the connector only records its ARN.

export ROLE_NAME=runware-s3-reader

runware serverless connectors create s3-reader \
  --kind aws \
  --role-arn arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME

runware serverless apps connectors attach $APP_ID s3-reader

In Serverless settings, under Developer, open OIDC connectors and press Create connector. Pick AWS as the provider, give it a name, and paste the role ARN into Role ARN. The name cannot be changed later. Leave Audience (optional) empty to get the default.

Then open the app, go to Settings, pick OIDC connectors in the side list and press Attach connector. The dialog has no fields: choose the connector from the list and confirm.

Leave the audience out, as here, and it is runware:$ORG_UUID. The connector belongs to your organization, and attaching binds it to one app in one environment. For an app outside production, add --environment <name>.

Attaching rolls the app's live deployment. The attach returns the connector with the exact subject its tokens use and the subjectPattern that covers every app of the environment. An app rule needs the first, an environment rule the second.

Make your AWS account trust it

connectors trust reads the attached connector and prints the aws iam commands with every value in place. The command only prints: the output is shell, and you run it yourself.

runware serverless connectors trust s3-reader --app $APP_ID

To run the output instead of reading it first, pass it to a shell.

sh -c "$(runware serverless connectors trust s3-reader --app $APP_ID)"

Read the output before you trust the exit status. When connectors trust fails it prints the error and no commands, so the shell has nothing to run and exits without an error of its own.

It prints one chain of three steps: an OIDC identity provider for the Runware issuer with the connector's audience as its client ID, the trust policy written to a file named after the connector and the scope, and the role the connector names with that policy attached. Each command runs only when the one before it worked, so a role is never created against a provider that does not accept the audience.

One provider serves every app of your organization, because the default audience is the same for all of them. The create therefore fails with EntityAlreadyExists when your account already has the provider, and the command after || adds the audience to the existing one instead.

The policy goes to a file named runware-<connector>-<scope>-trust-policy.json, so a run cannot overwrite a trust-policy.json of your own and cannot read the file a run for the other scope left behind.

A custom audience has to be printable as an AWS CLI argument. connectors trust refuses one that holds a comma, starts with -, or starts with file://, fileb://, http:// or https://, because the AWS CLI would read those as a file or a URL to load the value from. The default audience is never affected.

Add --scope environment to write the rule for every app of the environment. Grant the role its permissions afterwards, with aws iam attach-role-policy.

For one app the dashboard has the same policy ready to copy: open the app's Settings, pick OIDC connectors, and expand Role trust policy on the attached connector.

create-role fails when the role already exists. The output suggests aws iam update-assume-role-policy for that case, which replaces the role's trust policy and does not merge it. When other principals rely on the role, add the printed statement to its existing policy by hand.

Doing it by hand

Register the issuer in IAM. Create an IAM OIDC identity provider for the issuer, with the connector's audience as its client ID.

aws iam create-open-id-connect-provider \
  --url https://oidc.serverless.runware.ai \
  --client-id-list runware:$ORG_UUID

If you already have the provider and set a custom audience on a connector, add it with aws iam add-client-id-to-open-id-connect-provider.

Create the role and its trust policy. The trust policy sets the provider as a federated principal and pins both claims: StringEquals on aud, and StringEquals or StringLike on sub.

cat > trust-policy.json <<JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/oidc.serverless.runware.ai"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.serverless.runware.ai:aud": "runware:$ORG_UUID",
          "oidc.serverless.runware.ai:sub": "worker:org:$ORG_UUID:env:$ENV_UUID:app:$APP_ID"
        }
      }
    }
  ]
}
JSON

The exact subject above admits one app in one environment. To admit every app of the environment, move sub into a StringLike block with the subjectPattern value, which ends in :*. Then create the role with the name the connector records, and give it only the permissions the app needs.

aws iam create-role \
  --role-name $ROLE_NAME \
  --assume-role-policy-document file://trust-policy.json

aws iam attach-role-policy \
  --role-name $ROLE_NAME \
  --policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess

Use it from your app

The worker has AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE set, which every AWS SDK reads on its own. Create the client once, in load, and keep it: it exchanges the token for credentials and does it again when they expire.

import boto3

@serve
class Thumbnails:
    def load(self) -> None:
        self.s3 = boto3.client("s3", region_name="eu-west-1")

    @endpoint
    def list_inputs(self, payload: dict) -> dict:
        page = self.s3.list_objects_v2(Bucket="my-inputs", MaxKeys=10)
        return {"keys": [item["Key"] for item in page.get("Contents", [])]}

Make sure that it works

A connector test checks the trust setup with nothing deployed. To check it from inside your own app, sts:GetCallerIdentity needs no permissions of its own and is free to call, so it proves the exchange without touching a bucket. Deploy this endpoint and call it once.

import boto3

@serve
class Check:
    def load(self) -> None:
        self.sts = boto3.client("sts")

    @endpoint
    def identity(self, payload: dict) -> dict:
        return {"arn": self.sts.get_caller_identity()["Arn"]}

The ARN it returns is the assumed role, so a successful response means the token reached AWS, the trust policy matched and the role was assumed. Anything else is a trust problem, and the AWS entries below name each one.

Connecting to Google Cloud

Create and attach the connector

The connector identifies the workload identity provider by its full resource name. The name is fixed by the ids you choose, so you can create the connector before the pool exists.

export POOL_ID=runware
export PROVIDER_ID=runware-workers
export SERVICE_ACCOUNT=gcs-reader@my-project.iam.gserviceaccount.com

export POOL=projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL_ID
export MEMBER=principal://iam.googleapis.com/$POOL/subject/worker:org:$ORG_UUID:env:$ENV_UUID:app:$APP_ID
runware serverless connectors create gcs-reader \
  --kind gcp \
  --workload-identity-provider $POOL/providers/$PROVIDER_ID \
  --service-account-email $SERVICE_ACCOUNT

runware serverless apps connectors attach $APP_ID gcs-reader

In Serverless settings, under Developer, open OIDC connectors and press Create connector. Pick Google Cloud, name it, and paste the full resource name into Workload identity provider. Service account (optional) is impersonated after the exchange, and an empty one means the workers reach your resources directly.

Google Cloud has no audience field, because the audience of a gcp connector is always the provider's name.

Then open the app, go to Settings, pick OIDC connectors and press Attach connector.

serviceAccountEmail is optional. With it, the worker impersonates that service account after the exchange. Without it, you grant roles to the federated identity directly. To stop impersonating, runware serverless connectors update gcs-reader --clear service-account-email removes it. Grant the federated identity access to your resources before you clear, because the clear rolls every live app that attaches the connector and the new workers lose whatever only the service account could read. A clear on a connector that has no service account is refused when it is the only change, and goes through alongside another one. The audience of a gcp connector is always the provider's name, the value Google expects, so leave it out.

Make Google Cloud trust it

The same command prints the gcloud commands for a gcp connector, with the pool, the provider and the member string filled in. The command only prints: the output is shell, and you run it yourself.

runware serverless connectors trust gcs-reader --app $APP_ID

It prints the pool, the provider with its attribute mapping and its condition on the subject, and then the IAM binding. With a serviceAccountEmail on the connector the binding lets the workers impersonate that account. Without one, the binding grants a role on a bucket, and you set BUCKET first, because that is the one value the connector does not hold.

Both creates fail with ALREADY_EXISTS when the pool or the provider is already there, from another environment for example. The binding step then reads the existing provider and fails closed: it grants only when the provider's condition admits this environment and its attribute mapping has what the grant relies on, and otherwise it stops before granting, with a non-zero exit status, and prints what to add.

It admits a condition made of assertion.sub.startsWith(...) terms joined by ||, because any one of those admitting the subject is enough. A condition that puts such a term next to an && or a negation, or that compares with ==, needs a human to read it, so the command prints the grant for you to run once you have checked. It also stops when the provider is disabled, uses an issuer other than the connector's, lists allowedAudiences, or cannot be read without the iam.workloadIdentityPoolProviders.get permission. An --attribute-mapping replaces the whole mapping, so an existing one has to be repeated entry by entry with the new one appended.

Add --scope environment to grant every app of the environment instead of the one.

For one app the dashboard has the binding ready to copy, under Grant access on the attached connector.

Doing it by hand

Create the pool and the provider. Point the provider at the issuer URL, map the claims you want to write conditions on, and restrict it with an attribute condition on the subject.

gcloud iam workload-identity-pools create $POOL_ID \
  --location=global \
  --display-name="Runware Serverless"

gcloud iam workload-identity-pools providers create-oidc $PROVIDER_ID \
  --location=global \
  --workload-identity-pool=$POOL_ID \
  --issuer-uri=https://oidc.serverless.runware.ai \
  --attribute-mapping="google.subject=assertion.sub,attribute.org_id=assertion.org_id,attribute.environment_id=assertion.environment_id,attribute.app_id=assertion.app_id" \
  --attribute-condition="assertion.sub.startsWith('worker:org:$ORG_UUID:env:$ENV_UUID:')"

Leave --allowed-audiences out. Google then accepts the provider's own name, and that is the value in the token.

Do not upload a copy of the Runware keys to the provider. Google lets a provider store an uploaded JWKS instead of reading it from the issuer, but Runware rotates its signing keys, and an uploaded set stops working at the next rotation. Use --issuer-uri as above, so Google fetches the current keys itself.

Grant access. With a service account on the connector, let the app's identity impersonate it:

gcloud iam service-accounts add-iam-policy-binding $SERVICE_ACCOUNT \
  --role=roles/iam.workloadIdentityUser \
  --member="$MEMBER"

Without one, grant the role on the resource to the same member:

gcloud storage buckets add-iam-policy-binding gs://my-inputs \
  --role=roles/storage.objectViewer \
  --member="$MEMBER"

To grant every app of the environment at once, swap the subject for a mapped attribute, and keep the attribute condition that pins your organization.

gcloud storage buckets add-iam-policy-binding gs://my-inputs \
  --role=roles/storage.objectViewer \
  --member="principalSet://iam.googleapis.com/$POOL/attribute.environment_id/$ENV_UUID"

Use it from your app

The worker has GOOGLE_APPLICATION_CREDENTIALS pointing at an external account credential file, which google-auth and every Google client library read as Application Default Credentials. Create the client once and keep it.

from google.cloud import storage

@serve
class Thumbnails:
    def load(self) -> None:
        self.gcs = storage.Client(project="my-project")

    @endpoint
    def list_inputs(self, payload: dict) -> dict:
        blobs = self.gcs.list_blobs("my-inputs", max_results=10)
        return {"names": [blob.name for blob in blobs]}

Pass project explicitly: a federated credential does not always have one. For lower level use, google.auth.default() returns the same credentials.

Make sure that it works

A connector test checks the trust setup with nothing deployed. To check it from inside your own app, reading the project's own metadata proves the chain on a project with no billing, so you can confirm the setup before you put a bucket behind it. Grant roles/browser, which reads that and nothing else.

# with a service account on the connector
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --role=roles/browser \
  --member="serviceAccount:$SERVICE_ACCOUNT"

# without one, the app's federated identity
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --role=roles/browser \
  --member="$MEMBER"

Then call Resource Manager from an endpoint, with the credentials the worker already has.

import google.auth
from google.auth.transport.requests import AuthorizedSession

@serve
class Check:
    def load(self) -> None:
        credentials, _ = google.auth.default(
            scopes=["https://www.googleapis.com/auth/cloud-platform"]
        )
        self.session = AuthorizedSession(credentials)

    @endpoint
    def project(self, payload: dict) -> dict:
        response = self.session.get(
            "https://cloudresourcemanager.googleapis.com/v3/projects/my-project"
        )
        return {"status": response.status_code}

A 200 means the token was accepted, exchanged and authorized. A 403 means the exchange worked and the grant did not reach this member, and the exchange itself fails earlier with one of the Google entries below.

Connecting to your own service

The generic kind is for anything else that can verify an OpenID Connect token: your own API, a partner's service, or a third-party product with an OIDC trust setting. You choose the audience, and the worker gets the raw token file.

runware serverless connectors create billing-api \
  --kind generic \
  --audience https://billing.example.com

runware serverless apps connectors attach $APP_ID billing-api

In Serverless settings, under Developer, open OIDC connectors, press Create connector and pick Generic OIDC. The only field it needs is Audience, the aud claim your verifier checks. Attach it from the app's Settings, under OIDC connectors.

An audience that starts with runware: is reserved, except your own organization's runware:<organization uuid>.

A connector test covers aws and gcp connectors only, so verify a generic one from your own service.

After the connector exists, runware serverless connectors trust billing-api --app $APP_ID prints the iss, aud and sub your verifier checks, as OIDC_ISS, OIDC_AUD and OIDC_SUB. With --scope environment it prints OIDC_SUB_PREFIX and OIDC_SUB_PATTERN in place of OIDC_SUB. The dashboard shows the same three under Claims to verify on the attached connector.

Sending the token

RUNWARE_OIDC_TOKEN_FILE points at the file. Read it on every request instead of once at startup, because the platform replaces it before it expires.

import os
import requests

def call_billing(body: dict) -> dict:
    with open(os.environ["RUNWARE_OIDC_TOKEN_FILE"]) as f:
        token = f.read().strip()

    response = requests.post(
        "https://billing.example.com/usage",
        json=body,
        headers={"Authorization": f"Bearer {token}"},
        timeout=10,
    )
    response.raise_for_status()
    return response.json()

Verifying the token

Your service reads the issuer's discovery document at https://oidc.serverless.runware.ai/.well-known/openid-configuration, which gives the JWKS URL. Fetch the keys from there and cache them by kid, and fetch again when a token arrives with a kid you have not seen, because a rotation adds new keys. Then check:

  • the signature, with RS256 only,
  • iss equals https://oidc.serverless.runware.ai,
  • aud equals the connector's audience,
  • exp is in the future and nbf is not, with a small allowance for clock skew,
  • sub equals the subject you trust or starts with your environment's prefix, or org_id equals your organization's id.
import jwt
import requests

ISSUER = "https://oidc.serverless.runware.ai"
AUDIENCE = "https://billing.example.com"
TRUSTED_PREFIX = "worker:org:<organization uuid>:env:<environment uuid>:"

discovery = requests.get(f"{ISSUER}/.well-known/openid-configuration", timeout=5).json()
keys = jwt.PyJWKClient(discovery["jwks_uri"])

def verify(token: str) -> dict:
    signing_key = keys.get_signing_key_from_jwt(token)
    claims = jwt.decode(
        token,
        signing_key.key,
        algorithms=["RS256"],
        audience=AUDIENCE,
        issuer=ISSUER,
        leeway=60,
        options={"require": ["iss", "sub", "aud", "exp", "iat"]},
    )
    if not claims["sub"].startswith(TRUSTED_PREFIX):
        raise PermissionError("token is not from a trusted app")
    return claims

The example uses PyJWT with its crypto extra, pip install "pyjwt[crypto]" requests. Any OIDC library that reads a JWKS URL does the same checks.

The jti claim is unique per token, so a service that must refuse a replay can record the ones it has seen until they expire.

Test the trust setup

A test exchanges a real token with your cloud without deploying any code. The platform runs a short-lived probe as a worker of the app. The probe gets the app's token the way a worker does, exchanges it with AWS AssumeRoleWithWebIdentity or Google STS, impersonates the service account when the connector has one, and then discards the credentials. No token or credential is returned, stored or logged. On AWS the role session is runware-connector-test-<test id>, so a test shows up in CloudTrail.

A test uses the app's own subject, so an AWS trust policy matches a test exactly as it matches a worker. On Google Cloud the token's deployment_id claim is connector-test-<test id> for a test, so an attribute condition can exclude tests.

runware serverless apps connectors test $APP_ID s3-reader

The command starts the test, waits for the result and prints it. A result normally arrives within a minute and always within two, and the wait defaults to 2m30s. It exits non-zero on any outcome other than ok, so a deploy script can gate on it. -F json prints the test as the API returns it. A --timeout under two minutes stops the wait early, and the test keeps running.

OutcomeWhat it means
okThe cloud accepted the token. identity is the role it assumed, or the service account it impersonated. A gcp connector without a service account has none
cloud_errorYour cloud refused the token. code and message are its own, stage is the step it refused at (issuer, sts, impersonation or assume_role), and troubleshooting says what that code usually means. A message that held a token or a credential is replaced with cloud error message withheld. Troubleshooting has the same codes
platform_errorThe platform could not run the test, so your cloud never saw a token. reason says what stopped it

The reasons behind a platform_error:

reasonWhat happened
tenancy_not_readyYour organization's workers cannot run yet, so no probe started. The POST returns this reason itself and there is no test id to poll
unconfiguredThe environment cannot run tests
unschedulableThe probe found no node to run on
image_pullThe probe's image could not be pulled
issuer_refusedThe platform's issuer would not sign the app's token, so your cloud was not asked. code is the issuer's own, such as invalid_grant
probe_failedThe probe could not finish for a reason outside the trust setup, such as throttling or an error on the provider's side
deadlineThe test was still running at deadlineAt, two minutes after it started
deployer_unavailableThe platform could not start the test
  • The connector has to be attached to the app in that environment, and be of kind aws or gcp. A generic connector has no cloud to exchange with, so a test of one returns 422.
  • One test of a connector runs at a time for each app and environment. While one is running, a second returns 409.
  • An organization starts 20 tests an hour, counting those running and those that ended ok or cloud_error, plus 60 an hour that end platform_error. Past either, the API returns 429 and Retry-After says how long to wait.
  • A test is kept for 24 hours. After that, reading it by id returns 404.

What the worker gets

The platform mounts a read-only directory at /var/run/secrets/runware/oidc and sets the variables for each attached kind. The file variables have a path inside that directory, at the name below. The first two come with an aws connector, the third with a gcp one and the fourth with a generic one.

VariableValue
AWS_ROLE_ARNThe role on the aws connector
AWS_WEB_IDENTITY_TOKEN_FILEaws-token
GOOGLE_APPLICATION_CREDENTIALSgcp-credential-configuration.json
RUNWARE_OIDC_TOKEN_FILEgeneric-token

AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and RUNWARE_OIDC_TOKEN_FILE belong to the connector only while one of that kind is attached. Attaching to an app that already sets one of them as an environment variable or a secret returns 422, and so does setting one later. Whatever else sets one of them, such as an older version the app is pinned to, the worker gets the connector's value. An app with no connector of that kind can set them as before.

GOOGLE_APPLICATION_CREDENTIALS and RUNWARE_OIDC_CONFIG_DIGEST are reserved on every app, with or without a connector, and setting either returns 422. The platform sets RUNWARE_OIDC_CONFIG_DIGEST on workers that have a connector, and your code has no use for it. The token directory is also reserved for volumes: a mount path on it, under it or over it, such as /var, returns 422.

The token

A token is a JWT signed with RS256, and its header includes the kid of the signing key.

ClaimValue
iss, aud, subAs described above
org_idYour organization's id
environment_idThe id of the environment the app runs in
app_idThe app id
deployment_idThe id of the deployment the worker belongs to
workerThe worker's name
connectorThe connector's name
jtiUnique per token
iat, nbf, expWhen it was issued, when it becomes valid, and when it expires

AWS trust policies can match only aud and sub. A Google attribute condition or your own verifier can match any claim, for example deployment_id to trust one deployment only.

A token lasts at most one hour. Do not depend on an exact lifetime, because it may get shorter. The platform replaces the token file at half its lifetime, including on a worker that was resumed after scaling to zero, and the AWS and Google SDKs read the new file when their credentials run out.

Runware rotates the signing keys. New keys are published at the issuer before they sign anything, and old keys stay until every token they signed has expired. If a signing key is ever compromised, Runware removes it and tells every affected organization when the exposure started and ended, which subjects were used, and what to revoke.

Limits

  • One connector of each kind per app. Attaching a second aws connector, or the same one twice, returns 409.
  • A connector belongs to the organization, not to an environment. Any app in any environment can attach it, and the environment shows up only in sub.
  • Changes reach workers when they roll. Attaching, updating and detaching roll the app's live deployment, and an app that is not live picks the change up on its next rollout.
  • A change made during a rollout rolls again on its own, about 15 seconds after that rollout finishes. The app's events show OIDC connector change accepted: rolls when the rollout in progress finishes and then change made during the previous rollout: rolling to pick it up. Neither moves activeVersionId.
  • A roll the platform cannot start is retried for about 7.5 minutes, five attempts waiting 30 seconds, then 1, 2 and 4 minutes. After the last one an error event says credential change not applied: the active version could not be deployed after 5 attempts, and the workers keep the configuration they have until the app is redeployed. An error event that says credential change not applied yet means a roll failed and could not record its retry. The change is rolled again once the rollout in progress ends.
  • Detaching does not revoke access at once. A worker that still runs keeps getting tokens until it rolls, a token stays valid until it expires, and the cloud credentials it was exchanged for stay valid for their own lifetime. The worst case is one hour of token plus the credential lifetime: one hour on Google, and on AWS the role's maximum session duration, one hour by default and up to twelve. To cut access at once, remove the trust or the grant on your cloud's side, and on AWS revoke the role's active sessions.
  • Every connector write is recorded in the audit log, with the credential that made it and the fields it changed. Creating, updating, deleting, attaching and detaching a connector each record an entry, and so does running a test. See Audit logs.
  • A connector cannot be deleted while any app attaches it, a stopped or failed one included. Detach it first, or delete the app.
  • The audience is one string, never a list, of at most 256 characters with no whitespace.
  • A name is 2 to 63 characters: lowercase letters, digits and hyphens, starting with a letter and ending with a letter or a digit. The name and the kind are fixed once created, and site-store is reserved for a platform connector, so it is not available for one of yours.
  • An API update replaces the whole configuration, so a PUT has to send every field you want to keep. An aws update that leaves audience out puts it back to the default, and a custom audience is then gone while your IAM provider still lists the old client ID. runware serverless connectors update reads the connector first and keeps the fields you do not name.
  • An update can be guarded with If-Match. GET /v1/oidc-connectors/{connectorName} returns an ETag, and sending it as If-Match on the PUT applies the update only while the connector is unchanged since that read. An update to a connector someone else changed first returns 412. runware serverless connectors update sends it for you, so two people editing one connector cannot overwrite each other.

Troubleshooting

On AWS

The IAM OIDC provider is missing, or its URL is not exactly https://oidc.serverless.runware.ai, with no trailing slash.

The provider's client ID list does not contain the connector's audience.

The trust policy does not match the token. Compare its sub with the subject the API returns, check the environment, and check that aud matches. Also check that the connector's roleArn points at this role.

AWS could not reach the issuer for a moment.

The IAM OIDC provider rejected a claim of the token. Its client IDs have to list the connector's audience.

AWS refused the request's parameters. The connector's roleArn has to be a full IAM role ARN.

boto3 raises it when the worker has no connector. Check that the connector is attached in the app's environment, and that the deployment has rolled since.

On Google Cloud

The pool or the provider does not exist, is disabled, or the connector's workloadIdentityProvider points at another one. Use the project number, not the project id.

When the message mentions the audience, the provider has --allowed-audiences set to something else. Clear it, so Google accepts its own name.

When it mentions the signature or the key on a setup that worked before, the provider has an uploaded JWKS. Switch it to the issuer URL.

Google STS refused the exchange itself. The connector's workloadIdentityProvider has to be the full provider resource name.

The condition does not match the token. Compare it with the subject the API returns.

The app's identity has no roles/iam.workloadIdentityUser on the service account, the member's subject does not match, or the IAM Service Account Credentials API is off in the service account's project.

A binding you have just added can take a few minutes to apply, so run the call again before changing anything.

At the impersonation step, the connector's serviceAccountEmail points at a service account that does not exist.

The service account cannot be impersonated, for example because it is disabled. Check its state in its own project.

The exchange worked and Google refused the call itself. The service account, or the federated identity without one, lacks a role on the resource.

From the Serverless API

StatusCause
409 on createYour organization already has a connector of that name
409 on attachThe app already has a connector of that kind, or this one
409 on deleteAn app still attaches the connector
409 on deploying an older versionThat version sets one of the connector's variables. Remove it, or detach the connector first
409 on testA test of that connector is already running for the app in that environment
422 on attach, or on setting a variable or a secretA name the connector sets is already set on the app, or you are setting one while the connector is attached. GOOGLE_APPLICATION_CREDENTIALS and RUNWARE_OIDC_CONFIG_DIGEST return it on every app
422 on create or updateA field breaks its rule, for example a custom audience on a gcp connector, or another organization's runware: audience
422 on testThe connector is generic, so there is no cloud to exchange the token with
429 on testThe organization used up its hourly test allowance. Retry-After says how long to wait
403Your organization is not active, so its connectors cannot change