Configuration
All Hub services are configured via environment variables, managed by envix through each service's ConfigModule.
Common Variables
These variables are shared across all services:
| Variable | Required | Description |
|---|---|---|
PORT | No | HTTP server port (default varies by service) |
AUTHUP_URL | Yes | Authup identity provider URL |
REDIS_CONNECTION_STRING | No | Redis connection URL (for pub/sub and caching) |
AMQP_URL | No | RabbitMQ connection URL |
COOKIE_PREFIX | No | Namespace prefixed onto the access_token cookie the authup middleware falls back to when a request carries no Authorization header (e.g. a bucket/file stream download). See Frontend Variables — must match client-ui's COOKIE_PREFIX. |
TRUST_PROXY | No | Which X-Forwarded-For hops core, storage, telemetry and messenger trust when they derive the client address (audit trail, rate-limit key): true (every hop, the default), false (the socket address), a hop count (1 for one reverse proxy) or a comma list of IPs, CIDRs or loopback / linklocal / uniquelocal. See Client addresses and rate limiting. |
SHUTDOWN_DELAY | No | Seconds core, storage, telemetry and messenger keep serving after SIGTERM/SIGINT while GET /readyz already answers 503, so a load balancer stops routing to them first (0 to 3600, default 0). The worker ignores it. See Graceful shutdown. |
Graceful shutdown
The core, storage, telemetry and messenger CLI starts, and the worker entrypoint, handle SIGINT and SIGTERM by awaiting application teardown once. On the first signal GET /readyz answers 503. When startup had finished, the listeners then keep serving for SHUTDOWN_DELAY seconds (default 0) so a load balancer drains them; a second signal ends that wait early. A signal during startup skips the delay and waits for setup before teardown; a failed setup also triggers cleanup. After the delay, shutdown has a 30-second deadline, including handles left alive after teardown; expiry or a rejected setup/teardown exits with code 1. Successful cleanup lets buffered output drain before the process exits. Allow more than SHUTDOWN_DELAY + 30 seconds of termination grace in your process supervisor (terminationGracePeriodSeconds, stop_grace_period) so Hub's deadline can run before a forced kill. Scheduled components finish an in-flight pass before their dependencies close. Core closes its Socket.IO clients before the HTTP listener. The installed Orkos lifecycle catches individual module teardown errors, so exit code 0 is not proof that every module cleaned up successfully; detecting those failures requires an upstream lifecycle change.
Read cache (Redis)
Services that cache database reads keep the entries in Redis when REDIS_CONNECTION_STRING is set, each service under its own key prefix, and drop them by tag when a record changes.
- Requires Redis 5 or later.
- Set
maxmemory-policytonoevictionor avolatile-*policy; the cache's clock key never expires and must not be evicted on its own. - Entries live 60 seconds, which also bounds how long a lost invalidation (e.g. a write committed during a Redis outage) can serve a stale read.
- A Redis outage does not fail requests or the service start: reads fall through to the database, and the failure is logged at most once every 10 seconds.
- Without Redis each process keeps an in-memory cache. Run more than one replica of a service only with Redis: otherwise a change written through one replica is answered stale by the others for up to 60 seconds.
Client addresses and rate limiting
The audit trail (requestIpAddress of an event) and the rate limiter take the client address from X-Forwarded-For, trusting the hops TRUST_PROXY names. The default trusts every hop, so a client can name any address it likes: behind a reverse proxy set TRUST_PROXY to its hop count (1 for one proxy), or to its addresses.
Every service rate-limits in two stages, per replica:
- Before authentication, per client address: at most 300 requests a minute that carry a credential and answer 401. An expired or revoked token and a request without a credential are not counted, so users behind one NAT address (a clinic network) are not locked out by ordinary token renewal. This stage blocks the whole address, so it runs only once
TRUST_PROXYis set to something other thantrueor a list with a/0range: with every hop trusted, a client could lock out any address by naming it inX-Forwarded-For. - After authentication, per identity (100 requests a second for a user, 1000 for a client), and per client address for an anonymous request (20 a second). A storage capability token counts against its subject, the user or client core minted it for, and shares that subject's budget. Hub's own service clients are not limited.
Health endpoints
Every service answers two anonymous paths, GET and HEAD alike, ahead of authentication, rate limiting and metrics:
| Path | Question | Use it for |
|---|---|---|
/livez | Does the process answer? | A liveness probe: failing it restarts the process |
/readyz | Can the process serve? | A readiness probe, a load balancer health check, depends_on: service_healthy |
Both answer 200 or 503 with Cache-Control: no-store and a report naming each check, never error text:
{ "healthy": false, "checks": [{ "name": "startup", "ok": true }, { "name": "database", "ok": false }, { "name": "shutdown", "ok": true }] }| Service | /livez fails when | /readyz fails when |
|---|---|---|
| core, storage, telemetry, messenger | startup has not finished | startup has not finished, the database does not answer SELECT 1 within one second, or the stop signal arrived |
| worker | startup (bus consumers included) has not finished | startup has not finished, or the stop signal arrived |
- Both fail until every module finished setup (migrations included), so give a Kubernetes
startupProbeon/liveza window longer than your slowest migration. /livezchecks no dependency: a restart does not fix a database outage.- Redis, RabbitMQ and Authup are no readiness checks. They are shared by every replica, so failing on them would take all replicas out of rotation at once; the AMQP driver reconnects on its own. VictoriaLogs is none either for telemetry.
- A database result is reused for one second, so frequent probes add no load.
GET /keeps its status document (version and timestamp, or a timestamp on messenger and worker) and checks nothing; do not probe it.
startupProbe:
httpGet:
path: /livez
port: 3000
periodSeconds: 5
failureThreshold: 60
livenessProbe:
httpGet:
path: /livez
port: 3000
periodSeconds: 30
readinessProbe:
httpGet:
path: /readyz
port: 3000
periodSeconds: 10The image HEALTHCHECK requests /livez (the ui keeps /), so under Docker Swarm or autoheal a database outage does not turn into a restart loop. To make Docker health mean readiness (e.g. for Compose depends_on: service_healthy), set HEALTHCHECK_PATH=/readyz on the container. Kubernetes ignores the image HEALTHCHECK; use the probes above.
Message delivery (RabbitMQ)
When AMQP_URL is set, commands to the worker, storage and telemetry services (build, distribute, bucket create, …) travel over durable RabbitMQ queues.
- A failed or interrupted command is retried, up to 5 attempts. A handler error, or a redelivery after a consumer died mid-command (crash, lost connection, consumer timeout), re-publishes the command with exponential backoff (1s, 2s, 4s, 8s). An exhausted command is dropped with an error log line naming its type and id. Before this, a redelivered command was dropped silently after its first attempt.
- The attempt counter is an AMQP header (
x-attempt). Older versions kept it in the message body (metadata.attempt). While old and new replicas run side by side, a failing command may be retried more often than 5 times, because each version only reads its own counter, but it is never retried forever and never dropped early. No particular upgrade order is needed for this. - The worker retries only failures that can go away on their own. A connection error, a 5xx/429 from the core or storage API, GitHub or the registry, or an unreachable or failing docker daemon is retried as above, and the analysis keeps its
startedstatus meanwhile. Anything else — a broken Dockerfile step, a 4xx, a missing registry — fails the build, distribution or check right away, as before. A retryable failure that persists through the 5th attempt is reported as failed, not dropped. Only a worker that dies on every attempt leaves a build or distributionstarted; server-core fails it once it is 15 minutes stale (see below). An error docker reports inside a build or push stream (e.g. a registry 503 during push) counts as permanent. - Harbor registry commands go through the
registryCommandsqueue (project link/relink/unlink, registry setup/cleanup/delete, webhook processing). They used to run inside server-core's request first and reach the queue only if that call threw. They are now only queued and run by the registry consumer, which retries a command that throws. Ordinary link/unlink handlers still log Harbor errors and give up; the migration handler throws failures for retry. A node's robot account and registry credentials appear shortly after creation rather than with the response. Renaming a registry project'sexternalNamequeues a non-destructive copy, verification and credential cutover; see Registry project renames. Without a message bus server-core runs commands in-process; a command sent before the registry consumer has started waits in the queue. - Set RabbitMQ's
consumer_timeoutabove your longest analysis or master-image build. A command is acknowledged only once its handler finishes. If a build outlasts the timeout (RabbitMQ default: 30 minutes), the broker closes the channel and redelivers the command. A replica still running it ignores the copy, but a redelivery that lands on another worker replica starts a second build. - Worker and storage outcome events (started, progress, finished, failed) go through one durable queue per kind (
analysisBuilderEvents,analysisDistributorEvents,masterImageBuilderEvents,masterImageSynchronizerEvents,storageBucketEvents,storageBucketFileEvents). Each queue is a RabbitMQ single active consumer queue: one server-core replica applies its events in publish order, and another takes over if that replica goes away. Events published while server-core is down wait in the queue instead of being lost. A latestarted/progressevent no longer moves a finished or failed process back tostarted. - Upgrade the worker and storage services before server-core. An upgraded worker publishes its events to these queues, where they wait for the upgraded server-core. An upgraded server-core consumes only the queues, so it would miss events from a worker that still publishes the old way.
- Upgrading server-core across the
TaskManagerremoval: stop the old replicas before new ones serve traffic. An old server-core links a storage bucket only to a create it issued, and it can keep thestorageBucketEventsconsumer slot until it disconnects. A bucket created by a new replica while an old one holds that slot stays unlinked, and the analysis has no bucket. If old and new replicas did overlap, run the storage check (POST /analyses/:id/commandwith{ "command": "storageCheck" }) for every analysis created during the overlap. Old replies are fine in the other direction: a new server-core links every bucket created by an old one. - Before upgrading again after a rollback, delete these six queues. They are durable, so a rollback leaves them on the broker, and the older version neither reads nor removes them. The next upgrade would apply whatever they still hold: a stale
started/progressevent is ignored, but a stalefinished/failedevent would overwrite the current status. - Stuck builds and distributions are recovered by server-core. It re-checks every build/distribution that has been
starting/startedwithout an update for 15 minutes, every 5 minutes, and marks itfailedif the image is not there. Running workers send a heartbeat every 5 minutes, so a long build is not affected (a step that sends no progress for 2 hours is treated as hung and stops the heartbeat). A hung step only makes the process stale: with a single worker replica the re-check queues behind the hung run, so it is recovered only by restarting the worker, running a second worker replica, or RabbitMQ'sconsumer_timeoutending the hung delivery. Upgrade the workers before server-core. An old worker sends no heartbeat and still decides staleness on its own: with a new server-core, the automatic re-check can reach an idle old worker replica, which then reports a healthy build or push with one step longer than 15 minutes asfailed. A new worker with an old server-core is safe: it reports a stuck process as still running, so it stays stuck until server-core is upgraded too. - Analysis buckets are deduplicated on upgrade. server-core now allows one analysis bucket per analysis and type, and server-storage one bucket per
(refType, refId, name). Their migrations clean up duplicates a repeated or concurrent bucket create may have left: server-core keeps the oldest link per analysis and type and deletes the newer ones (with their file rows), and server-storage clearsrefType/refIdon every newer duplicate bucket. No storage bucket or file is deleted; a bucket whose link was dropped stays in server-storage as a ref-less orphan, which an operator can delete withDELETE /buckets/:id. A repeated bucket create command now answers with the existing bucket instead of creating another.POST /bucketsno longer accepts arefId(400): only the service owning the referenced resource sets it, over the bus. - A lost broker connection is retried every 5 seconds (it was 60).
- Registry commands go through the queue. Linking, unlinking or relinking a project in Harbor, registry setup, cleanup and deletion, and Harbor webhook processing are published to the
registryCommandsqueue and run on whichever server-core replica takes them, not inside the HTTP request that triggered them. Before, server-core ran the command itself first and fell back to the queue, so a failing command could run twice. WithoutAMQP_URLthe command still runs inside the same server-core process. Renaming a registry project sends one relink command. Two commands for the same project sent by separate requests (an unlink, then a link) can run on two replicas at once, so wait for the first to finish before sending the second. - Failure events carry the error's
name,messageandcodeinstead of{}.
Registry project renames
Changing a node's externalName, or renaming a registry project directly, while its registry stays the same, starts a same-registry migration. With AMQP_URL set it is queued and runs asynchronously; without it, server-core runs it in-process. Changing a node's registryId is not a migration: Hub unlinks the old project and provisions a new one on the new registry. Hub keeps the active registry-project name, Harbor ID and credentials at the source until it has copied and verified the destination and provisioned destination access. Poll the registry-project record to observe cutover; the update response still contains the active source name. Master-image project renames are refused because workers use their fixed namespace.
Before renaming:
- Drain/stop old server-core registry consumers before deploying this change. Old consumers interpret relink as destructive unlink/link. New consumers refuse legacy relink messages without an explicit destination. Do not allow old and new consumers to handle the new migration commands together.
- Pause all registry writers and drain in-flight distributions, including queued/retried worker commands and external clients. Harbor has no per-project read-only switch. Inventory verification detects changes during copying, but it is not a write fence: a push after verification can remain source-only. Do not resume writers until the active binding has switched and clients have refreshed their namespace and credentials. The migration holds the registry-level lock while it copies artifacts, and that lock times out after 60 seconds: a node registration, a registry-project create or update, or a node update that synchronizes its registry project on the same registry can wait for that long and then fail with
DatabaseLockError. - Do not issue link/unlink/delete, registry reassignment, or overlapping rename requests during migration. A competing rename is refused once its expected source no longer matches; it is not a last-request-wins operation. Retry from the currently active project after investigating the failure.
The migration copies manifests by digest, preserves tags and verifies the result before local cutover. It does not clone arbitrary project configuration such as memberships, retention/immutability rules or scan history. A conflicting unowned destination is refused rather than adopted. Copy, provisioning or verification failure leaves the active source binding and its credentials intact; marked partial destinations are retained for a verified retry. A crash between project creation and marker creation leaves an unowned target; recovery is manual rather than silently adopting it. Check registry error logs if the active name does not change after the message's bounded retries. Retry a failed node-triggered rename through the registry-project update endpoint with the desired target: resubmitting an unchanged node name is intentionally a no-op. If writers added an accessory after its subject was copied, Harbor may refuse to backfill it into an existing destination; verification stops cutover. Such a partial target needs operator recovery after writers are drained, not a forced cutover.
The source project and its robot account are retained, so cached URLs and credentials continue to work. Migration robot descriptions are durable retention records binding source and destination Harbor project IDs. Registry cleanup honours them and takes the same registry-level lock as migration; it refuses to continue if a migration record is malformed or unreadable. Do not remove or edit these robots as routine cleanup. Retention is indefinite: reclaim an old project only after confirming all readers/writers have switched and reconciling any late source writes. There is no automatic retirement or live-writer convergence. Explicit project deletion and cross-registry reassignment are separate destructive operations, not this migration protocol.
Authup upgrade to beta.68
Upgrade the Authup server and all Hub services together. Hub now requires Authup 1.0.0-beta.68 and reads GET /authorization using the service's own CLIENT_ID / CLIENT_SECRET credentials. Grant that client permission_read with realmScope: "ownOrNull" for a single realm, or "any" when serving multiple realms. The default "own" grant cannot read global policy definitions. The built-in system client already has the required access.
Each authenticated HTTP request and socket connection fetches one catalog and combines it with the caller's introspected grants. All permission checks within that request/connection reuse the local evaluator. Catalog errors fail the request; there is no fallback to permission names. Catalogs are not cached across requests. Existing token-introspection caching and socket-session lifetimes still apply, so changes to grants are visible when those are refreshed.
Scope restricts what a caller can do, not who it is. Grants are only evaluated for tokens carrying the global scope; a token issued without it (e.g. a custom login client registered without global) still authenticates and keeps its identity, but is evaluated with no grants, so every permission check denies with 403. Policies that read the identity still see the real caller, and paths that authorize by identity rather than by permission — a node or analysis client reading its own credentials, the messenger mailbox, bucket ownership, realm socket rooms — still honour such a token. Authup applies the same rule since authup#3723. client-ui requests global openid, and Authup's client-credentials and password grants always issue global.
This preserves policy trees and grant realm scopes at the evaluator boundary. Existing endpoints that only perform a permission pre-check still perform a pre-check: this upgrade does not add row-policy checks or collection filtering to those endpoints. Restricting every entity read/write by resource attributes is a separate service-level change.
The frontend kit uses a single POST /authorization/check batch for UI gates and refreshes it at the policy expiry returned by Authup. Hub navigation follows those refreshes. Browser users do not need permission_read merely to render menus.
Authup's beta.67 upgrade also applies its folders/event-aggregate migration and pins database timestamps to UTC. Follow the Authup upgrade notes for existing databases that used local time. Beta.68 quotes newly serialized strings; Hub's text transformer reads both these and legacy unquoted messenger payloads, so no Hub data migration is needed.
server-messenger authenticates against Authup
Breaking. server-messenger now verifies every REST request and socket connection against Authup, like the other services. Before, its REST surface never registered an Authup client and attached one fixed master-realm user to every request: no token was needed, and every caller read and acknowledged the same mailbox. Treat any message stored by an affected version as exposed.
What an operator must do:
- Grant the messenger's service client (
CLIENT_ID, defaultsystem)permission_read, with the samerealmScopeguidance as in Authup upgrade to beta.68 — every authenticated request reads the authorization catalog. The built-insystemclient already has it. - Make sure
AUTHUP_URL,CLIENT_IDandCLIENT_SECRETare set for the messenger. Every Hub service now fails closed without Authup: no identity is attached, so every authenticated route answers 401 and every socket connection is refused. The previous no-Authup fallback (a fixed master-realm user, plus adryRunmode that granted every permission) is removed. - Callers (nodes) must send a bearer token on every request, including the
GET /messages/streamSSE stream.
Malformed bearer tokens no longer authenticate
Security fix, no operator action required in Hub. Authup up to 1.0.0-beta.69 falls back to the token in the introspection request's own Authorization header when the submitted token fails its validator (shorter than 16 or longer than 2048 characters). Hub's token verifier sends that header with the service's own credentials, so a malformed bearer — Bearer x, an oversized string, or a garbage access_token cookie — was introspected as the Hub service client (active, global scope, its grants) and every Hub service acted on it as that client.
Every Hub service now refuses a credential that is not structurally a compact JWS within those bounds before introspecting it, answering 401 (sockets: connection refused). The fallback itself is fixed upstream in authup#3724, released in Authup 1.0.0-beta.70. Authup's own beta.70 token verifier still sends such a token to introspection, so against an older Authup server Hub's guard (IntrospectableTokenVerifier, which refuses the token before introspecting it) is what stops it. Entries cached for such strings sit under Redis token:* keys, which Hub no longer reads since the beta.70 upgrade.
Attaching nodes to a project requires project_update
Breaking for some roles. Adding a node to a project (POST /project-nodes) and removing one (DELETE /project-nodes/:id) both require project_update. Previously, adding required project_create and project_update. Holding only project_create now creates a project but cannot attach nodes to it; grant project_update to every role that creates projects with nodes.
Writing analysis buckets and files requires analysis_update
Breaking for some roles. Analysis buckets and their files are authorized against the analysis's realm, not the caller's:
- Creating, updating or deleting an analysis bucket, and adding, updating or deleting a code file, require
analysis_updatereaching the analysis's realm (see Writes take their realm reach fromrealm_scope). Grant it to every role that uploads analysis code. - Code files are frozen once the analysis configuration is locked: a write answers 400, and so does deleting the code bucket.
- An analysis has at most one bucket per type, and a storage bucket backs at most one analysis bucket (a second one answers 409).
- Uploads made directly to server-storage are checked when server-core registers them: a code file must come from the analysis's realm while unlocked, a result or temp file from the realm of an approved analysis node or from the analysis's own client. A rejected upload is not registered. A rejected code upload is also deleted from storage (an unregistered code object would break every later build), so server-storage must consume the
storageBucketFileTasksqueue (it does from this release on); a rejected result or temp upload stays in storage, so a misjudged uploader never costs result data. This check is realm-level only, and uploads through storage by anyone else are rejected too; upload through a node or the owner realm. - Storage still lets an uploader delete its own object, even a code file of a locked analysis. server-core logs such a deletion as an error; it cannot prevent it.
- A result or temp file may be added by a caller whose
analysis_updatereaches every realm (realm_scope: any, no policy —admin,system), by the realm of a node whose analysis participation is approved, or by the analysis's own client. Only a code file can be the root file. - Result files are updated and deleted by the analysis realm with
analysis_update; temp files only by a caller whoseanalysis_updatereaches every realm.
Uploading into a bucket requires ownership or bucket_update
Breaking for API clients. POST /buckets/:id/upload into a bucket created over HTTP (one without a refId) answers 403 unless the caller created the bucket or holds bucket_update with access to the bucket's realm. Previously any authenticated caller could upload into any bucket. Buckets created before the creator was recorded have no owner, so only bucket_update holders can upload into them. Analysis buckets (which carry a refId) are unchanged: their uploads are still checked by server-core when it registers them.
Analysis buckets created before refType/refId existed carry no ref. Migration 1788900000000-BackfillAnalysisBucketRefs recovers it from the bucket name (analysis-<type>-files.<analysisId>), so node and analysis uploads into older analyses keep working. A duplicate that the 1788600000000 migration detached stays ref-less; server-core does not link it.
Reading nodes, master images, registries, registry projects and events requires *_read
Breaking for every deployment. Listing or reading these entities now needs a read grant. Five permission names are new — node_read, master_image_read, master_image_group_read, registry_read, registry_project_read — and the existing event_read is now realm-scoped.
Which permissions allow a read. Each read accepts its read name or any of the entity's write/manage names, since holding a write implies read:
| Entity | Accepted permissions |
|---|---|
| nodes | node_read, node_create, node_update, node_delete |
| master images | master_image_read, master_image_manage |
| master image groups | master_image_group_read, master_image_group_manage |
| registries | registry_read, registry_manage |
| registry projects | registry_project_read, registry_project_manage, registry_manage |
| events (server-telemetry) | event_read, event_delete |
A caller holding none of them gets 403 — deliberately, with no fallback. The same table also gates:
- Realtime — subscribing to node, master-image and master-image-group updates.
- Includes — a relation requested via
includeneeds the related entity's permissions:/nodes?include=registryand/registry-projects?include=registryneed a registry one,/nodes?include=registryProjecta registry-project one.
Which rows a read returns.
- Nodes, master images, master image groups and registries are global catalogs: the grant is a capability gate only, and every row stays readable across realms.
- Registry projects and events are narrowed by the grant's
realm_scope:ownreturns the caller's realm,ownOrNulladds realm-less rows,anyreturns all. A grant carrying a policy Hub cannot turn into a query is checked row by row instead: the read returns exactly the rows the grant allows, in whichever realm they are. Such a page can hold fewer rows than itslimitor none, whilemeta.totalcounts every candidate row, so paging by total still reaches all allowed rows. - A node client reads its own registry project without any grant, on both
GET /registry-projects(narrowed to that one project) andGET /registry-projects/:id. A read grant it is given applies as for any other caller. - Record reads (
GET /<entity>/:id) apply the same scope, so a row out of reach answers 404 exactly like a missing one. A malformed id (GET /events/not-a-uuid,filter[id]=not-a-uuid) answers 400.
An audit event belongs to the realm of the row it records (a project event to the project's realm). Rows without a single realmId column — master images and groups, registries, and the project-node and analysis-node junctions (which carry one realm per side) — produce realm-less events, visible to ownOrNull and any grants only. Events stored before this release carry no realm, so an own grant does not see them; ownOrNull does (and with it every realm's old events).
Before upgrading:
- Add the new names to your Authup provisioning first. The built-in
adminrole (*) only picks up permissions that exist when provisioning runs. - Grant
node_read,master_image_readandmaster_image_group_readto every role that creates or edits projects or views analyses — project creation lists nodes and master images. - Grant
registry_readto roles that manage a node's registry connection (node_update) — the node's Registry tab lists/registriesto pick one. - Make sure the worker's service client (
system) hasnode_read,registry_readandmaster_image_read;*covers them once they are provisioned.
After upgrading: nothing to do for node clients. server-core re-grants each existing node's client its permissions (analysis_read, node_read, analysis_node_log_create, all at realm_scope: none) in the background — once at start, then hourly — so node-side services (pod orchestration, the storage service, the message broker) keep reading /nodes. Each pass removes any other permission a node client was granted by hand, as node saves always have. A node whose client cannot be updated is logged as an error and retried on the next pass. If node_read is not provisioned in Authup, every node is logged as failed until it is, and the next pass after provisioning grants it without a restart; creating a node fails meanwhile, instead of yielding a client that gets 403 later.
Known gaps:
- Registry projects joined through
include=registryProjectare not narrowed to the caller's realm reach. - Nested includes are not gated:
/project-nodes?include=node.registryProjector/analysis-nodes?include=node.registryreturn the nested rows without the read permissions above, and/nodes?include=registryProject.registrychecks only the first segment. Secret fields stay excluded. - Includes within the project/analysis family (
include=node,include=masterImage) are not gated yet. - The registry-project realtime room still broadcasts every realm's projects to a subscriber holding
registry_manageorregistry_project_manage.
Reading analyses requires analysis_read and reach
Breaking for every deployment. GET /analyses, GET /analysis-nodes, GET /analysis-node-events, GET /analysis-buckets and GET /analysis-bucket-files (list and record) now need a grant and return only rows of the analyses the caller reaches.
Which permissions allow a read. analysis_read, or any analysis write or lifecycle permission: analysis_create, analysis_update, analysis_delete, analysis_approve, analysis_execution_start, analysis_execution_stop. Holding none of them answers 403. analysis_result_read alone no longer lists analyses; pair it with analysis_read.
Which analyses a read returns.
- Users see what their grant's
realm_scopecovers, plus every analysis in which a node of their realm takes part (pending or rejected approval included):any: every analysis. The built-inadminrole and thesystemclient hold their grants atany.own/ownOrNull: analyses of their own realm. Analyses always carry a realm, so the two behave the same here.
- The master realm is no longer special. A master-realm user whose grants are
ownsees only master's own analyses. Give cross-realm roles their analysis grants atrealm_scope: any. - A node's client sees exactly the analyses its node takes part in, and an analysis' client only its own analysis. Neither gains anything from its realm or from its grants' realm scope. New analysis clients are given
analysis_read; clients of analyses created before the upgrade are not. - A grant carrying a policy narrows every one of these reads by that policy, evaluated against the analysis the row belongs to. A policy Hub cannot turn into a query is checked analysis by analysis: the read returns the rows of exactly the analyses the grant allows (plus those reached through a node), in whichever realm they are. Such a page can hold fewer rows than its
limitor none, whilemeta.totalcounts every candidate row. GET /analysis-nodesandGET /analysis-node-eventsreturn every row of a reachable analysis, so nodes see each other's approval and execution status.- Buckets and their files depend on the bucket type:
CODE: everyone who reaches the analysis, including a node realm whose approval is still pending (it needs the code to review it).TEMP: the reach of the grant, plus realms and node clients whose node is approved.RESULT: asTEMPfor the bucket itself. Its files additionally needanalysis_result_read, whose ownrealm_scopecan only narrow that reach further, never widen it. Without it, result files are left out of listings (not refused) and a result file answers 404.
- A record out of reach answers 404, like a missing one.
- Logs (
/analysis-logs,/analysis-node-logs) used to need only a login, because server-core reads and deletes them with its own credentials:- Analysis logs are read with the read permissions above, by the reach of the grant plus realms and node clients whose node is approved.
- A node's logs are read with
analysis_updateoranalysis_approve, by the reach of the grant (every node), or by that node itself (its realm's users or its client) once approved. A node does not see other nodes' logs. - Deleting either needs
analysis_updatethrough the reach of the grant; taking part as a node never allows it. - A malformed
analysisId/nodeIdanswers 400. POST /analysis-node-logsis unchanged.
Before upgrading:
- Add
analysis_readto your Authup provisioning. The built-inadminrole (*) only picks up permissions that exist when provisioning runs. Until it exists, every node save fails (creating or editing a node, and assigning its registry), and every existing node client is logged as failed on each hourly re-sync. - Grant
analysis_readto roles that view analyses without changing them, andanalysis_result_readto roles that download results. - Grant the analysis permissions of every role that must see other realms' analyses at
realm_scope: any. - Make sure the worker's service client (
CLIENT_ID, defaultsystem) holdsanalysis_readatrealm_scope: any; otherwise builds and distributions of other realms' analyses fail with 404.systemis covered by*once the name is provisioned.
After upgrading: node clients need nothing. server-core grants each node's client analysis_read in the background, once at start and then hourly.
Writes take their realm reach from realm_scope
Breaking for every deployment. Hub no longer treats the master realm as special anywhere a record is written. Every create, update, delete and command — analyses, analysis nodes, buckets and files, nodes, projects, project nodes, registry projects, client and registry credentials, storage buckets and files, telemetry events — needs its permission through a grant whose realm_scope reaches the record's realm:
anyreaches every realm. The built-inadminrole and thesystemclient hold their grants atany.ownreaches the caller's own realm,ownOrNullalso records without a realm (realm-less registry projects and telemetry events),nonereaches nothing.- A master-realm user whose grants are
ownnow writes only in the master realm. Grant cross-realm roles their permissions atrealm_scope: any. - A record out of reach answers 403. On an update, the grant's policies must allow both the stored and the updated record.
Behaviour changes on top:
- Analysis commands check a permission. Starting a build or a distribution needs
analysis_execution_start; every other command (buildCheck,distributionCheck,configurationLock/Unlock,storageCheck) needsanalysis_update. Before, only realm membership was checked. - Node clients hold
analysis_read,node_readand the newanalysis_node_log_create, all atrealm_scope: none, and no longeranalysis_update. Their own node is authorized by binding: a node's client reports execution status and progress on its own analysis-node rows, writes its run's logs and reads them, but cannot change other analyses of its realm. The background sync rewrites existing grants. - Analysis clients' permissions are created at
realm_scope: none. - A node's client is assigned by Hub only.
clientIdcan no longer be set throughPOST/PUT /nodes, and a client belongs to at most one node. The upgrade migration keeps a client shared by several nodes on the oldest one; the others get a fresh client on their next save, so their node software needs the new credentials. - Writing a node's logs (
POST /analysis-node-logs) needsanalysis_node_log_create: the node's own client, or a grant reaching the node's realm. The(analysisId, nodeId)participation row must exist; pending participants may write logs, so approval is not required. Telemetry'slog_createis not given to nodes. - Execution progress, like execution status, requires authority over the node (its bound client or
analysis_updatereaching the node's realm). Authority over only the analysis owner's realm does not permit either field. Both remain on the existingPOST /analysis-nodes/:idupdate endpoint. Creating an assignment accepts onlyanalysisIdandnodeId; supplied execution, approval and comment fields are ignored. Set those fields through the authorized update route, not by seeding them when attaching a node. - Registry credentials of a node need
node_updateorregistry_managereaching the node's realm. - Deleting a realm-less telemetry event needs
event_deleteatownOrNullorany. - Creating an analysis needs
analysis_createreaching the project's realm.
Before upgrading:
- Add
analysis_readandanalysis_node_log_createto your Authup provisioning. Until both exist, every node save fails and every node client is logged as failed on each hourly re-sync. The sync checks for both before it changes anything, so a node client keeps its previous grants until then. - Give every role that must write in other realms its permissions at
realm_scope: any, and give roles that start builds or distributionsanalysis_execution_start.
Realtime and rate limiting follow realm_scope
Breaking for cross-realm users. The last master-realm-by-name checks are gone.
- Realtime namespaces. A socket connects to
/resources/<realmId>(that realm's events) or/resources(every realm's events). The own realm's namespace is open. Another realm's namespace and the global one need a hub grant atrealm_scope: anywith no policy. - Realtime subscriptions check the room's permissions against the namespace: in a realm namespace a grant must reach that realm, in the global namespace it must reach every realm. An
anygrant for one entity no longer opens another entity's rooms. - The UI connects to the namespace of the realm a list asks for, else of the current management realm (Admin → Realms, now offered on every realm). Master images, master-image groups, registries and registry projects are published to the global namespace only, so their lists update live only for callers with an unbounded grant — as before, when only the master realm got them. Before, every other user's connection was refused, so their lists updated only on reload.
- The rate limiter no longer throttles a client that holds every hub permission at
realm_scope: anywith no policy (thesystemclient). Being in the master realm no longer exempts a client; users are always limited.
Authup upgrade to beta.69
Hub now requires Authup 1.0.0-beta.69, a security release. Read the Authup upgrade guide before upgrading. Hub needs no code or data changes; these points concern the Authup deployment Hub runs against:
- Tokens are bound to their realm. On a Redis-backed Authup, drop the
oauth2_token_claims*keys once after upgrading, or tokens verified before the upgrade keep bypassing the new check until they expire. - Client reads are realm-gated. Hub's service client creates and reads the per-node and per-analysis Authup clients in the realm of that node or analysis. A Hub client whose
client_*grants areownorownOrNullno longer sees clients in other realms; grantany(the built-insystemclient already has it). - HTTP Basic names resolve in the route realm or the master realm. Hub's Harbor webhook authenticates with the registry client's id, which resolves globally, so it is unaffected.
- Rate limiting is two-stage. Behind one NAT address (a clinic or office network) raise
rateLimitAnonymousMax, and pin Authup'strustProxyto the hop count of your proxy. - Some users sign in once more. Token cookies are now written
SameSite=Lax, andSecureon https. Behind a TLS-terminating proxy, forwardX-Forwarded-Proto(the reverse proxy example already does) so server-side cookie writes in client-ui getSecuretoo.
Authup upgrade to beta.70
Hub now requires Authup 1.0.0-beta.70. Read the Authup upgrade guide for the release. Hub needs no data changes; these points concern Hub's services and the Authup deployment they run against:
- Token verification moved to vouchly. Hub verifies every token by introspection, as before. Results are now cached in Redis under
vouchly:*keys; thetoken:*keys written by earlier versions are not read again and expire with their tokens. A cached result is still served until the token'sexp, so a revoked token keeps working until then. - Cached results are kept per Authup client. Authup only reports a token active to a client allowed to introspect it, so a result is cached for the client that asked. Services using the same Redis and
AUTHUP_URLreuse each other's results only when they authenticate as the same client (the default,systemfor all of them); services with their own clients each introspect a token once. - An Authup outage answers 502. When Hub cannot get an answer about a token (Authup unreachable, answering 5xx or 429, or refusing Hub's own service credential), the request answers 502
upstream_error. A token Authup judged (inactive, expired) still answers 401. Before this release an unreachable Authup answered 401, which madeclient-uisign the user out. - The malformed-bearer fix is complete. Authup now answers a malformed token
active: falseitself (see Malformed bearer tokens). Hub keeps its own check as defence in depth: it still protects a deployment that upgrades Hub before its Authup server. - Realm and self-service grants are checked for realm reach. In Authup,
realm_updateandrealm_deleteat the defaultownreach now manage only the actor's own realm. Grant them atanyto admins who manage other realms from Hub's realm pages. The built-inadminrole already hasany. Auser_self_manageorclient_self_managegrant stored atrealmScope: noneno longer permits self-edits in the account console; grant it atown. - Attribute-policy grants no longer join realm realtime rooms. A grant carrying an
attributes/attributeNamespolicy no longer admits its holder to rooms in a realm namespace (/resources/<realmId>): a room has no row to evaluate the policy against, so it fails closed. HTTP reads and writes evaluate such policies against the record and are unaffected. registry_managereveals registry-project secrets only in its reach. A grant atown(orownOrNull) returnsaccountSecretonly for projects in the realms it reaches; before, anyregistry_managegrant revealed the secret of every registry project the caller could read.- RSA signing keys must be at least 2048 bits. Hub verifies by introspection, so it does not check key size itself. Authup now refuses to import a shorter key, and its own adapters reject tokens signed with one. Replace any shorter imported key as the Authup guide describes.
Authup upgrade to beta.72
Hub now requires Authup 1.0.0-beta.72. Read the Authup upgrade guide for the release. Hub needs no data changes:
- Upgrade the Authup server and Hub together.
POST /authorization/check(the batch behind everyclient-uipermission gate) andGET /authorization(the catalog Hub's services read) changed shape. An olderclient-uiagainst a newer Authup, or the reverse, denies every gate. An older Hub service against a newer Authup rejects the catalog, and every authenticated request fails. - Hub's permissions stay global. A check by bare name matches the global permission definition only, both in Hub's services and in
client-ui. Provision Hub's permission names without arealmId, as before. A copy owned by a realm or client is a different permission, and Hub never checks it. - Roles, permissions, scopes and policies created without a realm are now global. Before, they landed in the creator's realm. This affects the admin pages in
client-ui. A realm admin (not master) gets their own realm filled in by the form. A master user creating a permission now creates a global one, which is what Hub checks. - A token without the
globalscope introspects with no grants. Hub already ignored such grants (see Authup upgrade to beta.68), so nothing changes.
Storage capability tokens
Browsers reach server-storage with a capability token that server-core mints per analysis (POST /analyses/:id/storage/token) by exchanging the caller's access token at Authup (RFC 8693).
Required from this release
The UI's analysis file pages and downloads require the token exchange. There is no fallback to the login token: without the setup below the file lists, uploads and downloads fail. Complete the steps below before rolling out the UI.
Authup 1.0.0-beta.75 or newer. Older servers have no token exchange or capability tokens, and 1.0.0-beta.74 refuses the camelCase capability type
bucketFile.Hub's core client may exchange tokens. The client server-core authenticates as (
CLIENT_ID, defaultsystem) listsurn:ietf:params:oauth:grant-type:token-exchangeingrantTypes(the grant is opt-in) and holds a token exchange capability for the audiencestorage. Declared in an Authup provisioning file:yamlrealms: - attributes: name: master relations: clients: - attributes: name: system grantTypes: client_credentials urn:ietf:params:oauth:grant-type:token-exchange relations: tokenExchangeCapabilities: - strategy: type: merge attributes: audience: storage value: bucket bucketFile maxAge: 3600 realmScope: anyvaluelists the capability types the client may delegate (storage's domain typesbucketandbucketFile),maxAgecaps a token's lifetime in seconds, andrealmScope: anylets it act for users and node clients of every realm. Keep the client's othergrantTypes(a hub client normally also needsclient_credentials).Every realm whose users or node clients mint tokens needs an asymmetric signing key (RSA or EC). Authup's
/jwkspublishes no symmetric (OCT) keys and refuses to sign a capability token with one, so a realm that only has an OCT key cannot issue them, and server-storage could not verify them.server-storage verifies tokens locally against
AUTHUP_URL's/jwks, and accepts only tokens the core client exchanged.CAPABILITY_CLIENT_IDnames that client and defaults to storage's ownCLIENT_ID: when server-core and server-storage authenticate as different clients, set it to the core client's id (or name inREALM; a name is looked up once and needsclient_read). Without it, every capability request answers 401.
Rollout order
- Authup: upgrade to 1.0.0-beta.75 or newer, provision the core client (grant and
storagecapability) and make sure every realm has an RSA or EC signing key. - server-storage: until it runs the new version, a minted token is refused (401).
- server-core: it serves the mint route.
- client-ui: it asks for the tokens.
Upgrading from the bucket-file capability type
The single-file capability type is bucketFile, the storage domain type of a file. Releases before it spelled the type bucket-file; bucket entries are unchanged. Neither spelling is accepted in place of the other: server-core mints only bucketFile, server-storage honours only bucketFile entries, and a server-storage of an earlier release refuses a token carrying one (401).
- Authup: upgrade to 1.0.0-beta.75 and provision both spellings for the rollout, so the server-core of either release can mint:
value: bucket bucket-file bucketFile. Themergestrategy overwrites the storedvalueon Authup's next start. - server-storage and server-core: roll them out together. Between the two, single-file downloads and deletes fail (a
bucket-fileentry covers nothing on the new server-storage, and abucketFiletoken is refused by the old one); bucket uploads, listings and downloads keep working. Abucket-filetoken minted before the switch lives at most an hour (the UI's download tokens 60 seconds) and is not honoured afterwards; a new mint replaces it. - client-ui: the earlier server-core refuses its
bucketFilemints (400). - Authup: drop
bucket-filefrom the value (value: bucket bucketFile).
Symptoms of a missing step
| Symptom | Missing step |
|---|---|
| Mint answers 501 | AUTHUP_URL is not set on server-core |
| Mint answers 502 | Authup is too old or unreachable, the core client lacks the grant or the storage capability, a realmScope does not cover the caller's realm, or the realm has no RSA/EC signing key |
| Mint answers 404 | Unknown analysis |
| Mint succeeds, storage answers 401 | server-storage is not upgraded, or cannot reach Authup's /jwks (a /jwks outage answers 502) |
| File pages and downloads fail in the UI | Any of the above; check the mint response first |
Database Variables
Used by server-core, server-storage, server-telemetry, and server-messenger:
| Variable | Required | Description |
|---|---|---|
DB_TYPE | Yes | mysql, postgres, or better-sqlite3 |
DB_HOST | Yes* | Database hostname |
DB_PORT | Yes* | Database port |
DB_USERNAME | Yes* | Database username |
DB_PASSWORD | Yes* | Database password |
DB_DATABASE | Yes | Database name (or :memory: for SQLite) |
*Not required for SQLite.
SQLite has no upgrade path
Migrations only run for mysql and postgres. On SQLite the schema is built from the entity classes by synchronize(), and only when no schema exists yet — so an existing SQLite database is never migrated or altered on upgrade.
Schema changes therefore break a persistent SQLite deployment. The rename of the analysis table to analyses is one such change: a SQLite database created before it keeps the old table, and queries against the new name fail.
Use SQLite only for tests and throwaway/in-memory (:memory:) instances. For any database you intend to keep, use MySQL or PostgreSQL. To carry an existing SQLite database across a schema change, export its data and re-import it into a freshly created database.
Timestamps are stamped and read in UTC
On MySQL and PostgreSQL every Hub service pins its database sessions to UTC (time_zone on MySQL, TimeZone on PostgreSQL), reads the zone-less created_at / updated_at columns back as UTC, and binds Date parameters in UTC — whatever timezone the database server or the host running Hub is set to. The same pin applies to the migration CLI. There is nothing to configure, and DB_PIN_TIMEZONE is not needed. A database and host already running in UTC (the container image sets no TZ) see no change.
Before, the database stamped these columns in its session timezone, the driver read them in the process timezone, and the messenger's expiry sweep compared its expires_at column against a Date bound in the process timezone, so the three only agreed when every clock ran in UTC.
If the database itself ran in local time, rows it stamped before this upgrade hold local wall-clock values and now read as UTC, shifted by that offset. Rows written afterwards are correct. Check the database's zone before upgrading:
-- PostgreSQL
SHOW timezone;
-- MySQL
SELECT @@global.time_zone, @@system_time_zone;If it is not UTC and the historical values matter, convert them while no Hub service is writing: stop every replica of the old version, run the conversion, then start the new version. Do not convert after the new version has started, or during a rolling upgrade where old and new replicas write side by side: the statements cannot tell a legacy local-time row from a correct UTC one, so they would shift the correct rows too. Convert each table and column once, with the zone the database used:
-- PostgreSQL
UPDATE events SET created_at = (created_at AT TIME ZONE 'Europe/Berlin') AT TIME ZONE 'UTC';
-- MySQL (needs the server's time zone tables loaded)
UPDATE events SET created_at = CONVERT_TZ(created_at, 'Europe/Berlin', '+00:00');A database setting that contradicts the pin stops the boot with an error naming it: a MySQL timezone other than UTC, a MySQL dateStrings or typeCast, or a PostgreSQL TimeZone other than UTC in the driver's startup options (e.g. via DB_DRIVER_EXTRA). Remove it — half a pin would shift values instead of fixing them. A MySQL replication setup is refused the same way for now, since its per-connection session cannot be pinned yet.
Replicas and boot-time migrations
Each service applies pending migrations when it starts. On MySQL and PostgreSQL it now does so while holding a database lock named after the service (server-core:migration, server-storage:migration, …), so replicas that start at the same time apply the chain once: one runs it, the others wait and then find nothing pending. The migration run / migration revert CLI commands take the same lock, so a migration job and a booting replica cannot overlap either. server-core's Harbor registry setup (HARBOR_URL) is serialized the same way.
A replica that cannot get the lock within 60 seconds fails its boot instead of migrating unlocked; the orchestrator restarts it, and by then the chain has been applied. Each lock holds one pooled connection while the migration runs on another, so the connection pool needs at least two connections.
Storage Variables (server-storage)
| Variable | Required | Description |
|---|---|---|
MINIO_ENDPOINT | Yes | MinIO/S3 endpoint URL |
MINIO_ACCESS_KEY | Yes | S3 access key |
MINIO_SECRET_KEY | Yes | S3 secret key |
MINIO_USE_SSL | No | Enable SSL for MinIO connection |
MINIO_PORT | No | MinIO port |
Telemetry Variables (server-telemetry)
| Variable | Required | Description |
|---|---|---|
VICTORIA_LOGS_URL | Yes | VictoriaLogs endpoint URL |
EVENT_RETENTION_DAYS | No | Days an audit/event row is kept before the daily sweep removes it (default 7; 0 = keep forever) |
Frontend Variables (client-ui)
The full list lives in the frontend reference. Two of them are deployment decisions rather than service addresses, and go together:
| Variable | Required | Description |
|---|---|---|
NUXT_PUBLIC_COOKIE_DOMAIN | No | Domain attribute for the UI's session cookies. Leave empty. |
NUXT_PUBLIC_AUTHUP_COOKIE_PREFIX | No | Namespace prefixed onto every session cookie name. Set this whenever NUXT_PUBLIC_COOKIE_DOMAIN is widened, so Authup's own hosted pages can't collide with the UI's cookies on that domain. |
Setting a prefix is only safe once the backend agrees on it: server-core, server-storage, server-telemetry and server-messenger each fall back to reading the access_token cookie (for requests that can't carry an Authorization header — a stream download is a top-level navigation) when the identity middleware finds nothing else. That fallback needs the same prefix, via each service's own COOKIE_PREFIX env var (see Common Variables) — set it to the exact same value as NUXT_PUBLIC_AUTHUP_COOKIE_PREFIX / COOKIE_PREFIX, verbatim (no separator is inserted automatically on either side).
Where Authup is served matters
The UI and Authup's own hosted pages persist their sessions under the same cookie names. If both can see each other's cookies, they hydrate, rotate and revoke each other's tokens, and the user is logged out on the next page reload.
Empty (host-only) cookies are correct in every layout. Setting a Domain delivers the cookies to every subdomain of that value — including Authup's host, if it sits below it — in which case also set NUXT_PUBLIC_AUTHUP_COOKIE_PREFIX to keep the two cookie sets apart. Serving Authup on a path of the UI's own origin no longer needs either variable as of Authup 1.0.0-beta.64 (authup#3495): the consoles scope their own cookies to that sub-path automatically.
The layout matrix and the upgrade caveat are documented under Session cookies.