a73x

9f95183c

feat(ship): the pipeline converges the fleet before it asks the smoke to prove it

a73x   2026-08-11 18:41

Commit message
feat(ship): the pipeline converges the fleet before it asks the smoke to prove it

Agents never upgrade themselves — an offer is a deliberate, per-host act, so a
bad release stops at the first machine instead of running through the fleet.
The consequence landed on v0.0.5 release day, on both planes: the pipeline
rolled the plane and went straight to the smoke, whose VM cycle needs the
release's own agent to prove the release's own legs. The hosts were a release
behind, the smoke failed, and the release waited while an operator upgraded
three machines by hand and resumed the run.

The click now lives between the roll and the proof. Stage 8 lists the fleet,
offers $TAG to every connected host that is not already reporting it, and waits
until they all do — on the same credential the plane already names for the
smoke, chosen the same way the smoke chooses it, so there is no second token to
issue or rotate. A disconnected host is warned about and skipped: it cannot
take an offer while it is dark and the smoke cannot place a VM on it either. A
fleet already at the tag says so and costs one request. A plane with no hosts
is fine and says which tenant it looked in.

Failures are named rather than waited out. A host reporting no version at all,
an offer for some version other than the one being shipped, and a refusal the
plane means (not behind the release, no artifact for this os/arch) each stop
the ship immediately with the reason; a cold server that has not fetched its
manifest and a host that goes dark mid-offer are waited through. Past 300s —
the console's own stuck threshold plus a further offer cycle — the ship fails
naming each host and how long its offer has stood.

The smoke and the report become stages 9 and 10, and --from reaches 10.
deploy/server/README.md
Old New
@@ -133,9 +133,16 @@ failure.
133 manifest at boot and pins the answer for 24 hours, so a server rolled ahead 133 manifest at boot and pins the answer for 24 hours, so a server rolled ahead
134 of its site serves the previous release to the whole fleet for a day. 134 of its site serves the previous release to the whole fleet for a day.
135 7. **Roll the control plane.** Recreate strategy — a brief gap; agents redial. 135 7. **Roll the control plane.** Recreate strategy — a brief gap; agents redial.
136 8. **Hosted smoke**: the same `eitri-smoke` the branch gate runs, against this 136 8. **Converge the fleet's agents** on the tag just rolled. Agents never upgrade
137 themselves, and the smoke below drives a full VM cycle on whichever host the
138 plane picks, so every connected host behind the release is offered the
139 upgrade here and the stage waits (up to 300s) for all of them to report it.
140 A disconnected host is warned about and skipped — it cannot be offered
141 anything and the smoke cannot place a VM on it either. It runs under
142 `--skip-smoke` too: a fleet running the release is part of shipping it.
143 9. **Hosted smoke**: the same `eitri-smoke` the branch gate runs, against this
137 plane's public names. 144 plane's public names.
138 9. **Report** what is actually running. 145 10. **Report** what is actually running.
139 146
140 ### A tag publishes once 147 ### A tag publishes once
141 148
@@ -485,7 +492,7 @@ guests exist at the time.
485 3. `scripts/ship.sh --target stg --tag v0.0.4-pre.1`. Expect stage 4 to fail the 492 3. `scripts/ship.sh --target stg --tag v0.0.4-pre.1`. Expect stage 4 to fail the
486 first time — that is the mechanism working. The fix is a Secret patch plus a 493 first time — that is the mechanism working. The fix is a Secret patch plus a
487 `config.required` classification, both recorded. 494 `config.required` classification, both recorded.
488 4. Migrate mewtwo (above) and re-run stage 8. 495 4. Migrate mewtwo (above) and re-run stage 9.
489 5. **The proof this run exists to produce**: drive `vm_create` over 496 5. **The proof this run exists to produce**: drive `vm_create` over
490 `https://stg.eitri.sh/mcp` — the proxied origin — and observe whether progress 497 `https://stg.eitri.sh/mcp` — the proxied origin — and observe whether progress
491 tokens keep Cloudflare from 524-ing a ten-minute call. Record the answer in 498 tokens keep Cloudflare from 524-ing a ten-minute call. Record the answer in
docs/upgrade.md
Old New
@@ -23,6 +23,11 @@ If a download fails, nothing is swapped; click again to retry. To roll back by
23 hand, stop the agent, move `eitri-agent.prev` back over the binary, and start 23 hand, stop the agent, move `eitri-agent.prev` back over the binary, and start
24 it again. 24 it again.
25 25
26 The release pipeline makes this same offer to every connected host after it
27 rolls a plane, and waits for them to report the new version, so a hosted fleet
28 arrives at a release without anyone clicking through it host by host
29 (`deploy/server/README.md`, stage 8).
30
26 **Requirements.** The button lights up only when the running agent's version 31 **Requirements.** The button lights up only when the running agent's version
27 orders before the published release. Release tags (`vX.Y.Z`), pre-releases 32 orders before the published release. Release tags (`vX.Y.Z`), pre-releases
28 (`vX.Y.Z-pre.N`, which order below the release they lead to), and git-describe 33 (`vX.Y.Z-pre.N`, which order below the release they lead to), and git-describe
scripts/ship.env.example
Old New
@@ -19,6 +19,9 @@ SITE_PLATFORM="linux/arm64"
19 # ── Credentials ─────────────────────────────────────────────────────────────── 19 # ── Credentials ───────────────────────────────────────────────────────────────
20 # Name ONE of the two arrangements. ship.sh forwards whatever it finds here and 20 # Name ONE of the two arrangements. ship.sh forwards whatever it finds here and
21 # clears the rest, so the plane decides and the pipeline stays the same script. 21 # clears the rest, so the plane decides and the pipeline stays the same script.
22 # The same credential converges the fleet's agents one stage earlier, so the
23 # identity named here must be the one whose tenant owns the fleet's hosts — a
24 # credential from another tenant sees no hosts, and stage 8 says so.
22 # 25 #
23 # (a) A plane running the bundled eitri-oidc — stg. The smoke signs in through 26 # (a) A plane running the bundled eitri-oidc — stg. The smoke signs in through
24 # the real code flow, which proves the whole credential chain, and mints its 27 # the real code flow, which proves the whole credential chain, and mints its
scripts/ship.sh
Old New
@@ -8,7 +8,8 @@
8 # scripts/ship.sh --target <stg|prod> --tag <vX.Y.Z[-pre.N]> [options] 8 # scripts/ship.sh --target <stg|prod> --tag <vX.Y.Z[-pre.N]> [options]
9 # 9 #
10 # --from <n> resume at stage n after a partial failure 10 # --from <n> resume at stage n after a partial failure
11 # --skip-smoke stop after the roll (stage 9 still reports) 11 # --skip-smoke deploy and converge the fleet, but do not prove it
12 # (stage 10 still reports)
12 # --render-only print the rendered manifests and exit; touches nothing 13 # --render-only print the rendered manifests and exit; touches nothing
13 # 14 #
14 # Every stage is idempotent: re-running from the top is always safe and is the 15 # Every stage is idempotent: re-running from the top is always safe and is the
@@ -39,7 +40,7 @@ SKIP_SMOKE=0
39 RENDER_ONLY=0 40 RENDER_ONLY=0
40 41
41 usage() { 42 usage() {
42 sed -n '2,32p' "$0" | sed 's/^#\{1,2\} \{0,1\}//' 43 sed -n '2,33p' "$0" | sed 's/^#\{1,2\} \{0,1\}//'
43 exit "${1:-1}" 44 exit "${1:-1}"
44 } 45 }
45 46
@@ -60,7 +61,7 @@ stg | prod) ;;
60 *) echo "ship: --target must be stg or prod (got '${TARGET}')" >&2; usage ;; 61 *) echo "ship: --target must be stg or prod (got '${TARGET}')" >&2; usage ;;
61 esac 62 esac
62 [[ -n "$TAG" ]] || { echo "ship: --tag is required" >&2; usage; } 63 [[ -n "$TAG" ]] || { echo "ship: --tag is required" >&2; usage; }
63 [[ "$FROM" =~ ^[1-9]$ ]] || { echo "ship: --from must be a stage number 1-9" >&2; exit 1; } 64 [[ "$FROM" =~ ^([1-9]|10)$ ]] || { echo "ship: --from must be a stage number 1-10" >&2; exit 1; }
64 65
65 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" 66 REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
66 cd "$REPO_ROOT" 67 cd "$REPO_ROOT"
@@ -525,7 +526,278 @@ if [[ "$FROM" -le 7 ]]; then
525 fi 526 fi
526 fi 527 fi
527 528
528 # ── 8. Hosted smoke ─────────────────────────────────────────────────────────── 529 # ── 8. Converge the fleet's agents ────────────────────────────────────────────
530 #
531 # Nothing in eitri upgrades itself on a schedule. An agent takes a new binary
532 # only when an operator offers it one, so a bad release stops at the first host
533 # instead of running through the fleet. That is deliberate, and it has a
534 # consequence for this script: the smoke below drives a whole VM cycle — create,
535 # boot, gate SSH, a published port — on whichever host the plane places it on,
536 # and a leg this release adds does not exist on an agent from the release
537 # before. On v0.0.5 that is what happened, on both planes. The plane served the
538 # new tag, the hosts still ran the old agent, the smoke failed on the leg their
539 # agents did not have, and the release waited while an operator upgraded three
540 # machines by hand and resumed the run.
541 #
542 # So the click lives here now, once per release, between the roll and the proof.
543 # Nothing about an offer changed: it is still per host, still audited, still
544 # refused for a host that is offline or not behind. What changed is that the
545 # pipeline no longer hands the smoke a fleet it already knows is too old.
546 #
547 # The stage runs even under --skip-smoke. A fleet running the release is part of
548 # shipping it, not part of proving it — a plane at $TAG with agents a release
549 # behind is exactly the half-shipped state this stage exists to end.
550 if [[ "$FROM" -le 8 ]]; then
551 bold "8. Converge the fleet's agents to $TAG"
552
553 # How long a host may take to land the new agent before the ship gives up on
554 # it. The work behind an offer is a download, a sha256, a binary swap and a
555 # re-exec, and the offer itself rides the host's next snapshot — one agent
556 # tick, ~10s — so on a healthy link the whole thing is done in tens of
557 # seconds. The console stops calling an outstanding offer "in flight" and
558 # starts calling it stuck at 180s (UPGRADE_STUCK_S, web/src/lib/fleet.svelte.ts),
559 # and a release must not fail before the console would even raise an eyebrow.
560 # 300s is that threshold plus a whole further offer cycle of slack for a slow
561 # link or a large artifact, and still inside the attention of the person who
562 # started the ship.
563 CONVERGE_TIMEOUT_S=300
564 CONVERGE_POLL_S=5
565
566 API="https://$CONSOLE_HOST"
567
568 # The credential is the one the plane already names for the smoke, chosen the
569 # same way the smoke chooses it for the VM lifecycle: the operator PAT from
570 # CI_PAT_FILE where there is one, and otherwise a console session signed in
571 # as CI_USER. The order matters — a plane may name both, and the PAT is then
572 # the identity whose tenant owns the fleet's hosts, which is exactly the
573 # identity that must offer them an upgrade and the one the smoke will place a
574 # VM as. A stage with a credential of its own would be a third token to
575 # issue, rotate and lose.
576 if [[ -z "${CI_USER:-}" && -z "${CI_PAT_FILE:-}" ]]; then
577 fail "$SHIP_ENV names no credential, so the fleet cannot be converged.
578 Set CI_USER + CI_PASSWORD_FILE for a plane with a password issuer, or
579 CI_PAT_FILE for one fronted by an external identity provider."
580 fi
581
582 # Everything the credential touches is a file in one 0700 directory that goes
583 # away with the run: the bearer header and the cookie jar are files rather
584 # than arguments, because an argument is visible in ps to every user on this
585 # machine.
586 cdir="$(mktemp -d)"
587 chmod 700 "$cdir"
588 trap 'rm -rf "$cdir"' EXIT
589
590 curl_auth=()
591 if [[ -n "${CI_PAT_FILE:-}" ]]; then
592 [[ -r "$CI_PAT_FILE" ]] || fail "CI_PAT_FILE names $CI_PAT_FILE, which cannot be read"
593 (umask 077; printf 'Authorization: Bearer %s\n' "$(cat "$CI_PAT_FILE")" >"$cdir/auth")
594 curl_auth=(-H "@$cdir/auth")
595 echo "acting as the operator whose PAT CI_PAT_FILE names"
596 else
597 : "${CI_PASSWORD_FILE:?set it alongside CI_USER in $SHIP_ENV}"
598 [[ -r "$CI_PASSWORD_FILE" ]] || fail "CI_PASSWORD_FILE names $CI_PASSWORD_FILE, which cannot be read"
599 # The same headless sign-in the smoke's credential chain walks, for the
600 # same reason it works there: GET /auth/login lands on the issuer's login
601 # form after the redirects, and the form takes the credentials posted back
602 # to the URL we landed on. Success is the session cookie in the jar and
603 # nothing else — a wrong password re-renders the form as a plain 200.
604 (umask 077; printf '%s' "$(cat "$CI_PASSWORD_FILE")" >"$cdir/pw")
605 form_url="$(curl -sS -L --max-time 30 -c "$cdir/jar" -b "$cdir/jar" \
606 -o /dev/null -w '%{url_effective}' "$API/auth/login")" ||
607 fail "GET $API/auth/login did not answer, so nothing can sign in to the plane just rolled"
608 curl -sS -L --max-time 30 -c "$cdir/jar" -b "$cdir/jar" -o /dev/null \
609 --data-urlencode "email=$CI_USER" --data-urlencode "password@$cdir/pw" "$form_url" ||
610 fail "posting $CI_USER's credentials to the issuer's login form at $form_url failed"
611 grep -q eitri_session "$cdir/jar" 2>/dev/null ||
612 fail "signing in as $CI_USER left no session — check the identity exists in this
613 plane's issuer and that CI_PASSWORD_FILE holds its current password."
614 # The session in the jar IS the credential from here: the API takes the
615 # console cookie exactly where it takes a bearer token, so this stage needs
616 # no PAT of its own and leaves no token behind it.
617 curl_auth=(-b "$cdir/jar")
618 echo "signed in as $CI_USER"
619 fi
620
621 # One round trip: the body lands in api_body and the status in api_status,
622 # 000 when there was no answer at all (curl's own convention, and stage 3's).
623 api_body=""
624 api_status=0
625 api_call() {
626 api_status="$(curl -sS --max-time 30 -X "$1" "${curl_auth[@]}" \
627 -o "$cdir/resp" -w '%{http_code}' "$API$2")" || api_status=000
628 api_body="$(cat "$cdir/resp" 2>/dev/null || true)"
629 }
630
631 # A host has converged when it reports THIS tag, compared as a string: the
632 # agent's version is stamped from the tag it was built at (scripts/release.sh)
633 # and stage 1 has already refused to ship a HEAD that is not exactly $TAG, so
634 # the two are the same characters or the host is not running this release.
635 # Every ordering question — is this host behind, may it be offered anything —
636 # stays with the plane, which answers it with internal/server/release and
637 # tells us in a status code.
638 started=$SECONDS
639 last_note=$SECONDS
640 offered=" " # hosts this stage has offered the upgrade to
641 announced=" " # hosts already reported as converged
642 skipped=" " # disconnected hosts, warned about once
643 refusal="" # the last thing the plane refused, for the timeout message
644 converged=0
645 while :; do
646 api_call GET /api/v1/hosts
647 [[ "$api_status" == "200" ]] ||
648 fail "GET /api/v1/hosts answered HTTP $api_status: ${api_body:-<empty>}
649 The plane is serving $TAG and the fleet has not been converged."
650 fleet="$(printf '%s' "$api_body" | jq -r '.[] | [
651 .id, .name, (.online|tostring), (.agent_version // "-"),
652 (.pending_upgrade.version // "-"), (.pending_upgrade.age_s // 0 | tostring)
653 ] | @tsv')" ||
654 fail "GET /api/v1/hosts did not answer with a list of hosts: ${api_body:-<empty>}"
655
656 # A plane with no hosts is a plane nobody has joined yet — a fresh one, or
657 # one whose fleet is still being built. There is nothing to converge and
658 # nothing wrong. The tenant is worth saying out loud: hosts are listed for
659 # the credential that asked, so a plane with a fleet and a credential from
660 # some other tenant looks exactly like this.
661 if [[ -z "$fleet" ]]; then
662 echo "no hosts are enrolled here for this credential's tenant — nothing to converge"
663 converged=0
664 break
665 fi
666
667 converged=0
668 waiting=""
669 while IFS=$'\t' read -r id name online version pending_v pending_age; do
670 [[ -n "$id" ]] || continue
671
672 if [[ "$version" == "$TAG" ]]; then
673 converged=$((converged + 1))
674 case "$announced" in
675 *" $id "*) ;;
676 *)
677 case "$offered" in
678 *" $id "*) echo " $name: converged, now reporting $TAG" ;;
679 *) echo " $name: already at $TAG" ;;
680 esac
681 announced="$announced$id " ;;
682 esac
683 continue
684 fi
685
686 # A dark host blocks nothing. It cannot be handed an offer — the plane
687 # refuses one outright — and the smoke cannot place a VM on it either,
688 # so waiting for it would hold a release for a machine that has no part
689 # in proving it. It comes back to a fleet a release ahead of it and the
690 # console offers it the upgrade then.
691 if [[ "$online" != "true" ]]; then
692 case "$skipped" in
693 *" $id "*) ;;
694 *)
695 reported="$version"
696 if [[ "$reported" == "-" ]]; then reported="no version"; fi
697 warn "host $name is disconnected (last reported $reported) — skipped.
698 It cannot take an offer while it is dark, and the smoke cannot place a VM
699 on it either. Offer it the upgrade from the console when it returns."
700 skipped="$skipped$id " ;;
701 esac
702 continue
703 fi
704
705 # Connected and saying nothing about its version: an agent too old to
706 # name itself. The plane will not offer it anything (it has no version
707 # to order), and it would refuse to create a VM there as well, because
708 # that guest would get no certified host key. Neither waiting nor the
709 # smoke can improve on that, so say it now.
710 if [[ "$version" == "-" ]]; then
711 fail "host $name ($id) is connected but reports no agent version, so the plane
712 cannot offer it $TAG — and it would refuse to create a VM there. Upgrade
713 that agent by hand (docs/upgrade.md, 'Agents, by hand'), then resume:
714 scripts/ship.sh --target $TARGET --tag $TAG --from 8"
715 fi
716
717 # An offer for some other version is the plane naming a release that is
718 # not the one being shipped, and no amount of waiting turns one into the
719 # other. The usual cause is the manifest: the server reads it at boot
720 # and pins it for 24 hours, which is why stage 6 rolls the site first.
721 if [[ "$pending_v" != "-" && "$pending_v" != "$TAG" ]]; then
722 fail "host $name ($id) has been offered $pending_v, not $TAG: this plane's release
723 manifest does not name the tag being shipped. Check that the site rolled
724 (stage 6) and that the server restarted after it (stage 7)."
725 fi
726
727 if [[ "$pending_v" == "$TAG" ]]; then
728 waiting="$waiting $name (offered ${pending_age}s ago)"
729 continue
730 fi
731
732 # No offer standing: make one. This is also the re-offer path — a server
733 # restart forgets every outstanding offer, so a host we offered earlier
734 # can show up here again, and offering twice is exactly what a human
735 # would do.
736 api_call POST "/api/v1/hosts/$id/upgrade-agent"
737 case "$api_status" in
738 202)
739 echo " $name: $version → $TAG (offered)"
740 offered="$offered$id "
741 waiting="$waiting $name (offered just now)"
742 ;;
743 503)
744 # The plane has not fetched its release manifest yet, which is
745 # ordinary in the first seconds after stage 7 rolled the server.
746 # Wait for it rather than failing a release on a cold start.
747 refusal="$name: $api_body"
748 waiting="$waiting $name (plane has no release manifest yet)"
749 ;;
750 409)
751 # internal/server/api/upgrade.go refuses three ways. "host is
752 # offline" is a race against the listing above and resolves itself
753 # — the next round sees the host dark and skips it. The other two
754 # (not behind the release, no artifact for this os/arch) need a
755 # person, and waiting out the timeout would only delay telling them.
756 if [[ "$api_body" == *"host is offline"* ]]; then
757 refusal="$name: $api_body"
758 waiting="$waiting $name (went offline mid-offer)"
759 else
760 fail "the plane refuses to offer $TAG to host $name ($id): $api_body"
761 fi
762 ;;
763 *)
764 fail "POST /api/v1/hosts/$id/upgrade-agent answered HTTP $api_status: ${api_body:-<empty>}"
765 ;;
766 esac
767 done <<<"$fleet"
768
769 [[ -n "$waiting" ]] || break
770
771 if [[ $((SECONDS - started)) -ge $CONVERGE_TIMEOUT_S ]]; then
772 fail "the fleet did not converge on $TAG within ${CONVERGE_TIMEOUT_S}s —
773 still waiting on:$waiting${refusal:+
774 last refusal — $refusal}
775 These hosts are connected and not running $TAG. Where the offer was taken
776 and did not land, the reason is on the host, in the agent's log
777 (journalctl -u eitri-agent, or ~/Library/Logs/eitri-agent.log on a Mac);
778 where the plane refused it, the refusal above says why. The plane is
779 serving $TAG and is not proven. Resume with --from 8."
780 fi
781
782 if [[ $((SECONDS - last_note)) -ge 30 ]]; then
783 echo " waiting on:$waiting"
784 last_note=$SECONDS
785 fi
786 sleep "$CONVERGE_POLL_S"
787 done
788
789 skipped_n="$(printf '%s' "$skipped" | wc -w | tr -d ' ')"
790 summary="fleet: $converged host(s) reporting $TAG"
791 if [[ "$skipped_n" != "0" ]]; then
792 summary="$summary, $skipped_n disconnected and skipped"
793 fi
794 echo "$summary"
795 if [[ "$converged" == "0" && "$skipped_n" != "0" ]]; then
796 warn "no connected host is running $TAG, so the smoke has nowhere to place a VM."
797 fi
798 fi
799
800 # ── 9. Hosted smoke ───────────────────────────────────────────────────────────
529 # The same binary the branch gate runs, pointed at this plane's public names. 801 # The same binary the branch gate runs, pointed at this plane's public names.
530 # It needs nothing but those names and a credential: every leg goes through the 802 # It needs nothing but those names and a credential: every leg goes through the
531 # plane's own front door, so a hosted run logs into no host in the fleet. What 803 # plane's own front door, so a hosted run logs into no host in the fleet. What
@@ -538,8 +810,8 @@ fi
538 # PAT; a plane fronted by a real identity provider sets an operator PAT instead, 810 # PAT; a plane fronted by a real identity provider sets an operator PAT instead,
539 # because there is no password to post at Google. No COVER_OUT either way — the 811 # because there is no password to post at Google. No COVER_OUT either way — the
540 # hosted binaries are not coverage-instrumented. 812 # hosted binaries are not coverage-instrumented.
541 if [[ "$FROM" -le 8 && "$SKIP_SMOKE" != "1" ]]; then 813 if [[ "$FROM" -le 9 && "$SKIP_SMOKE" != "1" ]]; then
542 bold "8. Hosted smoke against $TARGET" 814 bold "9. Hosted smoke against $TARGET"
543 if [[ -z "${CI_USER:-}" && -z "${CI_PAT_FILE:-}" ]]; then 815 if [[ -z "${CI_USER:-}" && -z "${CI_PAT_FILE:-}" ]]; then
544 fail "$SHIP_ENV names no credential for the smoke. 816 fail "$SHIP_ENV names no credential for the smoke.
545 Set CI_USER + CI_PASSWORD_FILE for a plane with a password issuer, or 817 Set CI_USER + CI_PASSWORD_FILE for a plane with a password issuer, or
@@ -574,11 +846,11 @@ if [[ "$FROM" -le 8 && "$SKIP_SMOKE" != "1" ]]; then
574 "$REPO_ROOT/bin/eitri-smoke" || 846 "$REPO_ROOT/bin/eitri-smoke" ||
575 fail "hosted smoke FAILED against $TARGET — the plane is serving $TAG and is not proven." 847 fail "hosted smoke FAILED against $TARGET — the plane is serving $TAG and is not proven."
576 elif [[ "$SKIP_SMOKE" == "1" ]]; then 848 elif [[ "$SKIP_SMOKE" == "1" ]]; then
577 bold "8. Hosted smoke SKIPPED (--skip-smoke)" 849 bold "9. Hosted smoke SKIPPED (--skip-smoke)"
578 fi 850 fi
579 851
580 # ── 9. Report ───────────────────────────────────────────────────────────────── 852 # ── 10. Report ────────────────────────────────────────────────────────────────
581 bold "9. $TARGET is serving $TAG" 853 bold "10. $TARGET is serving $TAG"
582 report_deployments=(eitri-server web) 854 report_deployments=(eitri-server web)
583 if [[ "$LOCAL_OIDC" == "1" ]]; then 855 if [[ "$LOCAL_OIDC" == "1" ]]; then
584 report_deployments+=(eitri-oidc) 856 report_deployments+=(eitri-oidc)