OIDC connectors
Let your workers call AWS, Google Cloud or your own services with short-lived tokens that Runware signs, instead of cloud keys you store and rotate.
Introduction
An OIDC connector lets your workers call AWS, Google Cloud or any service that trusts OpenID Connect without a stored key. Runware signs a short-lived token for each worker, your cloud account trusts that token through federation, and your code uses the AWS or Google SDK unchanged.
import boto3
s3 = boto3.client("s3") # no access key anywhereCompare that with a secret that stores an access key. The key is valid until someone rotates it, it works from any machine that has a copy, and it is one more thing to keep out of logs. A connector has no credential at all. The token proves which app in which environment is asking, it expires within the hour, and your cloud decides what that app may do.
Every Runware step on this page is shown for the CLI and the dashboard. For the HTTP API, see the OIDC connectors reference.
The CLI commands assume you are signed in with runware auth login. The apps commands and connectors trust address production unless you pass --environment, set RUNWARE_ENV, or set a default in the CLI config, in that order of precedence. The connectors commands take none, because a connector belongs to your organization, and the environments commands name the environment as their argument. See the CLI reference for both.
How it works
- You create a connector in your organization. It says where the token goes: an IAM role, a Google workload identity provider, or an audience of your own.
- You attach it to an app. Its workers start getting a token file and the environment variables the SDKs read.
- You tell your cloud to trust the Runware issuer, and only for tokens whose subject identifies your app.
- When the SDK needs credentials, it sends the token to the cloud's security token service and gets temporary credentials back.
The platform renews the token file in place while the worker runs, so your code only ever reads it.
The values your cloud needs
Every trust rule you write is built from three claims of the token.
| Claim | Value |
iss | https://oidc.serverless.runware.ai, also returned as issuer on every connector |
aud | What your cloud checks the token against, one string that depends on the kind. Returned as audience on the connector |
sub | Which app in which environment is asking, returned as subject on an attachment |
An aws connector has runware:<organization uuid> as its audience unless you set your own, and a generic one has the value you set. A gcp connector always has the provider's full resource name, the value Google expects.
//iam.googleapis.com/projects/<project number>/locations/global/workloadIdentityPools/<pool id>/providers/<provider id>The subject lists the organization, the environment and the app, in that order.
worker:org:<organization uuid>:env:<environment uuid>:app:<app id>The order of the sub segments and the : between them are stable, so a pattern such as worker:org:<organization uuid>:env:<environment uuid>:* keeps matching every app of one environment. The worker: prefix means a rule written for workers matches no other kind of Runware token.
The environment segment is the environment's id, not its name, so a staging copy of an app never matches a production trust rule, and a new environment that reuses a deleted one's name gets none of its trust. The app segment is the app id, so an app you delete and create again with the same id in the same environment matches the same rule.
The audience does not keep other Runware customers out. You choose a gcp provider name and a generic audience, and anyone could choose the same one. Every trust rule you write must check sub, or org_id where your verifier reads claims. A rule that checks only aud can accept tokens from apps that are not yours.
Find your values
Every command on this page reads its values from the shell, so set them once and the rest is copy and paste. Start with the two you already have.
export RUNWARE_API_KEY=... # the key you call the Serverless API with
export APP_ID=my-app # the app the connector is attached toThe environment has its own prefix, so a trust rule can name it before any app or connector exists. GET /v1/environments/{name} returns oidcSubjectPrefix, which every worker token in that environment starts with.
runware serverless environments show production --format json{
"name": "production",
"id": "0190a6a8-0000-7000-8000-0000000000e1",
"oidcSubjectPrefix": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:"
}environments show prints the prefix in its table as well. The dashboard does not show it, so it is read from the CLI or the API.
The value stops at the colon and has no *. An AWS StringLike condition appends one, and a GCP attribute condition uses it as it stands, with assertion.sub.startsWith(...).
The prefix follows the environment's id, not its name. An environment deleted and created again under the same name gets a new id, so it gets a new prefix, and a rule that named the old one stops matching. An environment's name cannot change, so that is the only way a prefix moves.
For one app, the organization and environment ids come with the connector too, and it reports them rather than you assembling them. Attach it to an app and list the app's connectors: each one includes the exact subject its tokens use, and a subjectPattern for every app of that environment.
runware serverless apps connectors list $APP_IDOpen the app, go to Settings and pick OIDC connectors. Each attached connector shows its Subject with a button that copies it, and the environment variables the runtime sets for it.
{
"data": [
{
"name": "s3-reader",
"kind": "aws",
"audience": "runware:0190a6a8-0000-7000-8000-00000000000a",
"issuer": "https://oidc.serverless.runware.ai",
"aws": { "roleArn": "arn:aws:iam::123456789012:role/runware-s3-reader" },
"subject": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:app:my-app",
"subjectPattern": "worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:*",
"id": "...",
"createdAt": "...",
"updatedAt": "..."
}
]
}Both uuids sit inside the prefix, as worker:org:<organization uuid>:env:<environment uuid>:, so cut them out of it.
export PREFIX=worker:org:0190a6a8-0000-7000-8000-00000000000a:env:0190a6a8-0000-7000-8000-0000000000e1:
export ORG_UUID=$(echo "$PREFIX" | cut -d: -f3)
export ENV_UUID=$(echo "$PREFIX" | cut -d: -f5)An app's subject has them in the same places, followed by app:<app id>.
An app's own subject exists once the connector is attached to it, so for an app rule the app comes first. For an environment rule the prefix above is enough. The cloud side can wait: the connector only records the role or provider, and the worker asks for credentials only when your code does.
Last, your cloud account's own identifier, which is the one value Runware does not hold. Set the one your cloud uses.
# on AWS
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
# on Google Cloud
export PROJECT_ID=my-project
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')Google wants the project number, not the project id. It is the all-digit one, and a provider name built with the id is rejected.
What your cloud should trust
A trust rule covers one app or every app of an environment, and that choice decides the value you write in your cloud.
| Scope | What it admits | The value to write |
| One app | The app the connector is attached to, in one environment | subject, the exact sub its tokens have |
| One environment | Every app of your organization in that environment, including apps created later | subjectPattern, or the environment's oidcSubjectPrefix |
On AWS the first is a StringEquals condition on sub, and the second a StringLike with * appended. On Google Cloud the first grants to subject/<the subject>, and the second to attribute.environment_id/<the environment id>.
--scope environment trusts every app of your organization in that environment, so pick it only when all of them may reach the resource.
Each rule covers one environment. A rule written for staging does not admit a token from production.
Connecting to AWS
Create and attach the connector
Name the IAM role you will create. The role does not have to exist yet, because the connector only records its ARN.
export ROLE_NAME=runware-s3-reader
runware serverless connectors create s3-reader \
--kind aws \
--role-arn arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME
runware serverless apps connectors attach $APP_ID s3-readerIn Serverless settings, under Developer, open OIDC connectors and press Create connector. Pick AWS as the provider, give it a name, and paste the role ARN into Role ARN. The name cannot be changed later. Leave Audience (optional) empty to get the default.
Then open the app, go to Settings, pick OIDC connectors in the side list and press Attach connector. The dialog has no fields: choose the connector from the list and confirm.
Leave the audience out, as here, and it is runware:$ORG_UUID. The connector belongs to your organization, and attaching binds it to one app in one environment. For an app outside production, add --environment <name>.
Attaching rolls the app's live deployment. The attach returns the connector with the exact subject its tokens use and the subjectPattern that covers every app of the environment. An app rule needs the first, an environment rule the second.
Make your AWS account trust it
connectors trust reads the attached connector and prints the aws iam commands with every value in place. The command only prints: the output is shell, and you run it yourself.
runware serverless connectors trust s3-reader --app $APP_IDTo run the output instead of reading it first, pass it to a shell.
sh -c "$(runware serverless connectors trust s3-reader --app $APP_ID)"Read the output before you trust the exit status. When connectors trust fails it prints the error and no commands, so the shell has nothing to run and exits without an error of its own.
It prints one chain of three steps: an OIDC identity provider for the Runware issuer with the connector's audience as its client ID, the trust policy written to a file named after the connector and the scope, and the role the connector names with that policy attached. Each command runs only when the one before it worked, so a role is never created against a provider that does not accept the audience.
One provider serves every app of your organization, because the default audience is the same for all of them. The create therefore fails with EntityAlreadyExists when your account already has the provider, and the command after || adds the audience to the existing one instead.
The policy goes to a file named runware-<connector>-<scope>-trust-policy.json, so a run cannot overwrite a trust-policy.json of your own and cannot read the file a run for the other scope left behind.
A custom audience has to be printable as an AWS CLI argument. connectors trust refuses one that holds a comma, starts with -, or starts with file://, fileb://, http:// or https://, because the AWS CLI would read those as a file or a URL to load the value from. The default audience is never affected.
Add --scope environment to write the rule for every app of the environment. Grant the role its permissions afterwards, with aws iam attach-role-policy.
For one app the dashboard has the same policy ready to copy: open the app's Settings, pick OIDC connectors, and expand Role trust policy on the attached connector.
create-role fails when the role already exists. The output suggests aws iam update-assume-role-policy for that case, which replaces the role's trust policy and does not merge it. When other principals rely on the role, add the printed statement to its existing policy by hand.
Doing it by hand
Register the issuer in IAM. Create an IAM OIDC identity provider for the issuer, with the connector's audience as its client ID.
aws iam create-open-id-connect-provider \
--url https://oidc.serverless.runware.ai \
--client-id-list runware:$ORG_UUIDIf you already have the provider and set a custom audience on a connector, add it with aws iam add-client-id-to-open-id-connect-provider.
Create the role and its trust policy. The trust policy sets the provider as a federated principal and pins both claims: StringEquals on aud, and StringEquals or StringLike on sub.
cat > trust-policy.json <<JSON
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::$ACCOUNT_ID:oidc-provider/oidc.serverless.runware.ai"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.serverless.runware.ai:aud": "runware:$ORG_UUID",
"oidc.serverless.runware.ai:sub": "worker:org:$ORG_UUID:env:$ENV_UUID:app:$APP_ID"
}
}
}
]
}
JSONThe exact subject above admits one app in one environment. To admit every app of the environment, move sub into a StringLike block with the subjectPattern value, which ends in :*. Then create the role with the name the connector records, and give it only the permissions the app needs.
aws iam create-role \
--role-name $ROLE_NAME \
--assume-role-policy-document file://trust-policy.json
aws iam attach-role-policy \
--role-name $ROLE_NAME \
--policy-arn arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccessUse it from your app
The worker has AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE set, which every AWS SDK reads on its own. Create the client once, in load, and keep it: it exchanges the token for credentials and does it again when they expire.
import boto3
@serve
class Thumbnails:
def load(self) -> None:
self.s3 = boto3.client("s3", region_name="eu-west-1")
@endpoint
def list_inputs(self, payload: dict) -> dict:
page = self.s3.list_objects_v2(Bucket="my-inputs", MaxKeys=10)
return {"keys": [item["Key"] for item in page.get("Contents", [])]}Make sure that it works
A connector test checks the trust setup with nothing deployed. To check it from inside your own app, sts:GetCallerIdentity needs no permissions of its own and is free to call, so it proves the exchange without touching a bucket. Deploy this endpoint and call it once.
import boto3
@serve
class Check:
def load(self) -> None:
self.sts = boto3.client("sts")
@endpoint
def identity(self, payload: dict) -> dict:
return {"arn": self.sts.get_caller_identity()["Arn"]}The ARN it returns is the assumed role, so a successful response means the token reached AWS, the trust policy matched and the role was assumed. Anything else is a trust problem, and the AWS entries below name each one.
Connecting to Google Cloud
Create and attach the connector
The connector identifies the workload identity provider by its full resource name. The name is fixed by the ids you choose, so you can create the connector before the pool exists.
export POOL_ID=runware
export PROVIDER_ID=runware-workers
export SERVICE_ACCOUNT=gcs-reader@my-project.iam.gserviceaccount.com
export POOL=projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/$POOL_ID
export MEMBER=principal://iam.googleapis.com/$POOL/subject/worker:org:$ORG_UUID:env:$ENV_UUID:app:$APP_IDrunware serverless connectors create gcs-reader \
--kind gcp \
--workload-identity-provider $POOL/providers/$PROVIDER_ID \
--service-account-email $SERVICE_ACCOUNT
runware serverless apps connectors attach $APP_ID gcs-readerIn Serverless settings, under Developer, open OIDC connectors and press Create connector. Pick Google Cloud, name it, and paste the full resource name into Workload identity provider. Service account (optional) is impersonated after the exchange, and an empty one means the workers reach your resources directly.
Google Cloud has no audience field, because the audience of a gcp connector is always the provider's name.
Then open the app, go to Settings, pick OIDC connectors and press Attach connector.
serviceAccountEmail is optional. With it, the worker impersonates that service account after the exchange. Without it, you grant roles to the federated identity directly. To stop impersonating, runware serverless connectors update gcs-reader --clear service-account-email removes it. Grant the federated identity access to your resources before you clear, because the clear rolls every live app that attaches the connector and the new workers lose whatever only the service account could read. A clear on a connector that has no service account is refused when it is the only change, and goes through alongside another one. The audience of a gcp connector is always the provider's name, the value Google expects, so leave it out.
Make Google Cloud trust it
The same command prints the gcloud commands for a gcp connector, with the pool, the provider and the member string filled in. The command only prints: the output is shell, and you run it yourself.
runware serverless connectors trust gcs-reader --app $APP_IDIt prints the pool, the provider with its attribute mapping and its condition on the subject, and then the IAM binding. With a serviceAccountEmail on the connector the binding lets the workers impersonate that account. Without one, the binding grants a role on a bucket, and you set BUCKET first, because that is the one value the connector does not hold.
Both creates fail with ALREADY_EXISTS when the pool or the provider is already there, from another environment for example. The binding step then reads the existing provider and fails closed: it grants only when the provider's condition admits this environment and its attribute mapping has what the grant relies on, and otherwise it stops before granting, with a non-zero exit status, and prints what to add.
It admits a condition made of assertion.sub.startsWith(...) terms joined by ||, because any one of those admitting the subject is enough. A condition that puts such a term next to an && or a negation, or that compares with ==, needs a human to read it, so the command prints the grant for you to run once you have checked. It also stops when the provider is disabled, uses an issuer other than the connector's, lists allowedAudiences, or cannot be read without the iam.workloadIdentityPoolProviders.get permission. An --attribute-mapping replaces the whole mapping, so an existing one has to be repeated entry by entry with the new one appended.
Add --scope environment to grant every app of the environment instead of the one.
For one app the dashboard has the binding ready to copy, under Grant access on the attached connector.
Doing it by hand
Create the pool and the provider. Point the provider at the issuer URL, map the claims you want to write conditions on, and restrict it with an attribute condition on the subject.
gcloud iam workload-identity-pools create $POOL_ID \
--location=global \
--display-name="Runware Serverless"
gcloud iam workload-identity-pools providers create-oidc $PROVIDER_ID \
--location=global \
--workload-identity-pool=$POOL_ID \
--issuer-uri=https://oidc.serverless.runware.ai \
--attribute-mapping="google.subject=assertion.sub,attribute.org_id=assertion.org_id,attribute.environment_id=assertion.environment_id,attribute.app_id=assertion.app_id" \
--attribute-condition="assertion.sub.startsWith('worker:org:$ORG_UUID:env:$ENV_UUID:')"Leave --allowed-audiences out. Google then accepts the provider's own name, and that is the value in the token.
Do not upload a copy of the Runware keys to the provider. Google lets a provider store an uploaded JWKS instead of reading it from the issuer, but Runware rotates its signing keys, and an uploaded set stops working at the next rotation. Use --issuer-uri as above, so Google fetches the current keys itself.
Grant access. With a service account on the connector, let the app's identity impersonate it:
gcloud iam service-accounts add-iam-policy-binding $SERVICE_ACCOUNT \
--role=roles/iam.workloadIdentityUser \
--member="$MEMBER"Without one, grant the role on the resource to the same member:
gcloud storage buckets add-iam-policy-binding gs://my-inputs \
--role=roles/storage.objectViewer \
--member="$MEMBER"To grant every app of the environment at once, swap the subject for a mapped attribute, and keep the attribute condition that pins your organization.
gcloud storage buckets add-iam-policy-binding gs://my-inputs \
--role=roles/storage.objectViewer \
--member="principalSet://iam.googleapis.com/$POOL/attribute.environment_id/$ENV_UUID"Use it from your app
The worker has GOOGLE_APPLICATION_CREDENTIALS pointing at an external account credential file, which google-auth and every Google client library read as Application Default Credentials. Create the client once and keep it.
from google.cloud import storage
@serve
class Thumbnails:
def load(self) -> None:
self.gcs = storage.Client(project="my-project")
@endpoint
def list_inputs(self, payload: dict) -> dict:
blobs = self.gcs.list_blobs("my-inputs", max_results=10)
return {"names": [blob.name for blob in blobs]}Pass project explicitly: a federated credential does not always have one. For lower level use, google.auth.default() returns the same credentials.
Make sure that it works
A connector test checks the trust setup with nothing deployed. To check it from inside your own app, reading the project's own metadata proves the chain on a project with no billing, so you can confirm the setup before you put a bucket behind it. Grant roles/browser, which reads that and nothing else.
# with a service account on the connector
gcloud projects add-iam-policy-binding $PROJECT_ID \
--role=roles/browser \
--member="serviceAccount:$SERVICE_ACCOUNT"
# without one, the app's federated identity
gcloud projects add-iam-policy-binding $PROJECT_ID \
--role=roles/browser \
--member="$MEMBER"Then call Resource Manager from an endpoint, with the credentials the worker already has.
import google.auth
from google.auth.transport.requests import AuthorizedSession
@serve
class Check:
def load(self) -> None:
credentials, _ = google.auth.default(
scopes=["https://www.googleapis.com/auth/cloud-platform"]
)
self.session = AuthorizedSession(credentials)
@endpoint
def project(self, payload: dict) -> dict:
response = self.session.get(
"https://cloudresourcemanager.googleapis.com/v3/projects/my-project"
)
return {"status": response.status_code}A 200 means the token was accepted, exchanged and authorized. A 403 means the exchange worked and the grant did not reach this member, and the exchange itself fails earlier with one of the Google entries below.
Connecting to your own service
The generic kind is for anything else that can verify an OpenID Connect token: your own API, a partner's service, or a third-party product with an OIDC trust setting. You choose the audience, and the worker gets the raw token file.
runware serverless connectors create billing-api \
--kind generic \
--audience https://billing.example.com
runware serverless apps connectors attach $APP_ID billing-apiIn Serverless settings, under Developer, open OIDC connectors, press Create connector and pick Generic OIDC. The only field it needs is Audience, the aud claim your verifier checks. Attach it from the app's Settings, under OIDC connectors.
An audience that starts with runware: is reserved, except your own organization's runware:<organization uuid>.
A connector test covers aws and gcp connectors only, so verify a generic one from your own service.
After the connector exists, runware serverless connectors trust billing-api --app $APP_ID prints the iss, aud and sub your verifier checks, as OIDC_ISS, OIDC_AUD and OIDC_SUB. With --scope environment it prints OIDC_SUB_PREFIX and OIDC_SUB_PATTERN in place of OIDC_SUB. The dashboard shows the same three under Claims to verify on the attached connector.
Sending the token
RUNWARE_OIDC_TOKEN_FILE points at the file. Read it on every request instead of once at startup, because the platform replaces it before it expires.
import os
import requests
def call_billing(body: dict) -> dict:
with open(os.environ["RUNWARE_OIDC_TOKEN_FILE"]) as f:
token = f.read().strip()
response = requests.post(
"https://billing.example.com/usage",
json=body,
headers={"Authorization": f"Bearer {token}"},
timeout=10,
)
response.raise_for_status()
return response.json()Verifying the token
Your service reads the issuer's discovery document at https://oidc.serverless.runware.ai/.well-known/openid-configuration, which gives the JWKS URL. Fetch the keys from there and cache them by kid, and fetch again when a token arrives with a kid you have not seen, because a rotation adds new keys. Then check:
- the signature, with
RS256only, issequalshttps://oidc.serverless.runware.ai,audequals the connector's audience,expis in the future andnbfis not, with a small allowance for clock skew,subequals the subject you trust or starts with your environment's prefix, ororg_idequals your organization's id.
import jwt
import requests
ISSUER = "https://oidc.serverless.runware.ai"
AUDIENCE = "https://billing.example.com"
TRUSTED_PREFIX = "worker:org:<organization uuid>:env:<environment uuid>:"
discovery = requests.get(f"{ISSUER}/.well-known/openid-configuration", timeout=5).json()
keys = jwt.PyJWKClient(discovery["jwks_uri"])
def verify(token: str) -> dict:
signing_key = keys.get_signing_key_from_jwt(token)
claims = jwt.decode(
token,
signing_key.key,
algorithms=["RS256"],
audience=AUDIENCE,
issuer=ISSUER,
leeway=60,
options={"require": ["iss", "sub", "aud", "exp", "iat"]},
)
if not claims["sub"].startswith(TRUSTED_PREFIX):
raise PermissionError("token is not from a trusted app")
return claimsThe example uses PyJWT with its crypto extra, pip install "pyjwt[crypto]" requests. Any OIDC library that reads a JWKS URL does the same checks.
The jti claim is unique per token, so a service that must refuse a replay can record the ones it has seen until they expire.
Test the trust setup
A test exchanges a real token with your cloud without deploying any code. The platform runs a short-lived probe as a worker of the app. The probe gets the app's token the way a worker does, exchanges it with AWS AssumeRoleWithWebIdentity or Google STS, impersonates the service account when the connector has one, and then discards the credentials. No token or credential is returned, stored or logged. On AWS the role session is runware-connector-test-<test id>, so a test shows up in CloudTrail.
A test uses the app's own subject, so an AWS trust policy matches a test exactly as it matches a worker. On Google Cloud the token's deployment_id claim is connector-test-<test id> for a test, so an attribute condition can exclude tests.
runware serverless apps connectors test $APP_ID s3-readerThe command starts the test, waits for the result and prints it. A result normally arrives within a minute and always within two, and the wait defaults to 2m30s. It exits non-zero on any outcome other than ok, so a deploy script can gate on it. -F json prints the test as the API returns it. A --timeout under two minutes stops the wait early, and the test keeps running.
| Outcome | What it means |
ok | The cloud accepted the token. identity is the role it assumed, or the service account it impersonated. A gcp connector without a service account has none |
cloud_error | Your cloud refused the token. code and message are its own, stage is the step it refused at (issuer, sts, impersonation or assume_role), and troubleshooting says what that code usually means. A message that held a token or a credential is replaced with cloud error message withheld. Troubleshooting has the same codes |
platform_error | The platform could not run the test, so your cloud never saw a token. reason says what stopped it |
The reasons behind a platform_error:
reason | What happened |
tenancy_not_ready | Your organization's workers cannot run yet, so no probe started. The POST returns this reason itself and there is no test id to poll |
unconfigured | The environment cannot run tests |
unschedulable | The probe found no node to run on |
image_pull | The probe's image could not be pulled |
issuer_refused | The platform's issuer would not sign the app's token, so your cloud was not asked. code is the issuer's own, such as invalid_grant |
probe_failed | The probe could not finish for a reason outside the trust setup, such as throttling or an error on the provider's side |
deadline | The test was still running at deadlineAt, two minutes after it started |
deployer_unavailable | The platform could not start the test |
- The connector has to be attached to the app in that environment, and be of kind
awsorgcp. Agenericconnector has no cloud to exchange with, so a test of one returns422. - One test of a connector runs at a time for each app and environment. While one is running, a second returns
409. - An organization starts 20 tests an hour, counting those running and those that ended
okorcloud_error, plus 60 an hour that endplatform_error. Past either, the API returns429andRetry-Aftersays how long to wait. - A test is kept for 24 hours. After that, reading it by id returns
404.
What the worker gets
The platform mounts a read-only directory at /var/run/secrets/runware/oidc and sets the variables for each attached kind. The file variables have a path inside that directory, at the name below. The first two come with an aws connector, the third with a gcp one and the fourth with a generic one.
| Variable | Value |
AWS_ROLE_ARN | The role on the aws connector |
AWS_WEB_IDENTITY_TOKEN_FILE | aws-token |
GOOGLE_APPLICATION_CREDENTIALS | gcp-credential-configuration.json |
RUNWARE_OIDC_TOKEN_FILE | generic-token |
AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and RUNWARE_OIDC_TOKEN_FILE belong to the connector only while one of that kind is attached. Attaching to an app that already sets one of them as an environment variable or a secret returns 422, and so does setting one later. Whatever else sets one of them, such as an older version the app is pinned to, the worker gets the connector's value. An app with no connector of that kind can set them as before.
GOOGLE_APPLICATION_CREDENTIALS and RUNWARE_OIDC_CONFIG_DIGEST are reserved on every app, with or without a connector, and setting either returns 422. The platform sets RUNWARE_OIDC_CONFIG_DIGEST on workers that have a connector, and your code has no use for it. The token directory is also reserved for volumes: a mount path on it, under it or over it, such as /var, returns 422.
The token
A token is a JWT signed with RS256, and its header includes the kid of the signing key.
| Claim | Value |
iss, aud, sub | As described above |
org_id | Your organization's id |
environment_id | The id of the environment the app runs in |
app_id | The app id |
deployment_id | The id of the deployment the worker belongs to |
worker | The worker's name |
connector | The connector's name |
jti | Unique per token |
iat, nbf, exp | When it was issued, when it becomes valid, and when it expires |
AWS trust policies can match only aud and sub. A Google attribute condition or your own verifier can match any claim, for example deployment_id to trust one deployment only.
A token lasts at most one hour. Do not depend on an exact lifetime, because it may get shorter. The platform replaces the token file at half its lifetime, including on a worker that was resumed after scaling to zero, and the AWS and Google SDKs read the new file when their credentials run out.
Runware rotates the signing keys. New keys are published at the issuer before they sign anything, and old keys stay until every token they signed has expired. If a signing key is ever compromised, Runware removes it and tells every affected organization when the exposure started and ended, which subjects were used, and what to revoke.
Limits
- One connector of each kind per app. Attaching a second
awsconnector, or the same one twice, returns409. - A connector belongs to the organization, not to an environment. Any app in any environment can attach it, and the environment shows up only in
sub. - Changes reach workers when they roll. Attaching, updating and detaching roll the app's live deployment, and an app that is not live picks the change up on its next rollout.
- A change made during a rollout rolls again on its own, about 15 seconds after that rollout finishes. The app's events show
OIDC connector change accepted: rolls when the rollout in progress finishesand thenchange made during the previous rollout: rolling to pick it up. Neither movesactiveVersionId. - A roll the platform cannot start is retried for about 7.5 minutes, five attempts waiting 30 seconds, then 1, 2 and 4 minutes. After the last one an
errorevent sayscredential change not applied: the active version could not be deployed after 5 attempts, and the workers keep the configuration they have until the app is redeployed. Anerrorevent that sayscredential change not applied yetmeans a roll failed and could not record its retry. The change is rolled again once the rollout in progress ends. - Detaching does not revoke access at once. A worker that still runs keeps getting tokens until it rolls, a token stays valid until it expires, and the cloud credentials it was exchanged for stay valid for their own lifetime. The worst case is one hour of token plus the credential lifetime: one hour on Google, and on AWS the role's maximum session duration, one hour by default and up to twelve. To cut access at once, remove the trust or the grant on your cloud's side, and on AWS revoke the role's active sessions.
- Every connector write is recorded in the audit log, with the credential that made it and the fields it changed. Creating, updating, deleting, attaching and detaching a connector each record an entry, and so does running a test. See Audit logs.
- A connector cannot be deleted while any app attaches it, a stopped or failed one included. Detach it first, or delete the app.
- The audience is one string, never a list, of at most 256 characters with no whitespace.
- A name is 2 to 63 characters: lowercase letters, digits and hyphens, starting with a letter and ending with a letter or a digit. The name and the kind are fixed once created, and
site-storeis reserved for a platform connector, so it is not available for one of yours. - An API update replaces the whole configuration, so a
PUThas to send every field you want to keep. Anawsupdate that leavesaudienceout puts it back to the default, and a custom audience is then gone while your IAM provider still lists the old client ID.runware serverless connectors updatereads the connector first and keeps the fields you do not name. - An update can be guarded with
If-Match.GET /v1/oidc-connectors/{connectorName}returns anETag, and sending it asIf-Matchon thePUTapplies the update only while the connector is unchanged since that read. An update to a connector someone else changed first returns412.runware serverless connectors updatesends it for you, so two people editing one connector cannot overwrite each other.
Troubleshooting
On AWS
The IAM OIDC provider is missing, or its URL is not exactly
https://oidc.serverless.runware.ai, with no trailing slash.
The provider's client ID list does not contain the connector's
audience.
The trust policy does not match the token. Compare its
subwith thesubjectthe API returns, check the environment, and check thataudmatches. Also check that the connector'sroleArnpoints at this role.
AWS could not reach the issuer for a moment.
The IAM OIDC provider rejected a claim of the token. Its client IDs have to list the connector's
audience.
AWS refused the request's parameters. The connector's
roleArnhas to be a full IAM role ARN.
boto3 raises it when the worker has no connector. Check that the connector is attached in the app's environment, and that the deployment has rolled since.
On Google Cloud
The pool or the provider does not exist, is disabled, or the connector's
workloadIdentityProviderpoints at another one. Use the project number, not the project id.
When the message mentions the audience, the provider has
--allowed-audiencesset to something else. Clear it, so Google accepts its own name.When it mentions the signature or the key on a setup that worked before, the provider has an uploaded JWKS. Switch it to the issuer URL.
Google STS refused the exchange itself. The connector's
workloadIdentityProviderhas to be the full provider resource name.
The condition does not match the token. Compare it with the
subjectthe API returns.
The app's identity has no
roles/iam.workloadIdentityUseron the service account, the member's subject does not match, or the IAM Service Account Credentials API is off in the service account's project.A binding you have just added can take a few minutes to apply, so run the call again before changing anything.
At the impersonation step, the connector's
serviceAccountEmailpoints at a service account that does not exist.
The service account cannot be impersonated, for example because it is disabled. Check its state in its own project.
The exchange worked and Google refused the call itself. The service account, or the federated identity without one, lacks a role on the resource.
From the Serverless API
| Status | Cause |
409 on create | Your organization already has a connector of that name |
409 on attach | The app already has a connector of that kind, or this one |
409 on delete | An app still attaches the connector |
409 on deploying an older version | That version sets one of the connector's variables. Remove it, or detach the connector first |
409 on test | A test of that connector is already running for the app in that environment |
422 on attach, or on setting a variable or a secret | A name the connector sets is already set on the app, or you are setting one while the connector is attached. GOOGLE_APPLICATION_CREDENTIALS and RUNWARE_OIDC_CONFIG_DIGEST return it on every app |
422 on create or update | A field breaks its rule, for example a custom audience on a gcp connector, or another organization's runware: audience |
422 on test | The connector is generic, so there is no cloud to exchange the token with |
429 on test | The organization used up its hourly test allowance. Retry-After says how long to wait |
403 | Your organization is not active, so its connectors cannot change |