# Dashboard — modular dev/company status board A small Deno server that renders a glanceable board of status tiles and updates their markup in place over Server-Sent Events. Every tile is one file with a fixed interface; a single file registers them. ## Run ```bash cd deno task dashboard # = deno run … packages/dashboard/server.ts # open http://localhost:8731 ``` Tiles that read GitHub need `GH_TOKEN` (or `GITHUB_TOKEN`) set in the environment, or a GitHub App configured with `GH_APP_CLIENT_ID` and `GH_APP_PRIVATE_KEY`. The GitHub spend tile can use a separate `GH_BILLING_TOKEN` when billing needs broader account access. The other token-gated tiles gray out cleanly until their env vars are set (see below). The root `deno task dashboard` command starts the server once. Watch mode and dashboard-specific tests are package tasks: ```bash cd packages/dashboard deno task watch deno task test ``` The dashboard uses dark mode on its first visit. The switch at the bottom right of each page cycles through dark, light, and the operating-system preference. It saves that choice for the dashboard and all of its drill-down pages. Inline charts and generated CI Gantt images use the active theme as well. ## Architecture ``` dashboard/ types.ts the interface: Tile, TileView, Status, Ctx, Route config.ts port, repo, tunable status thresholds palette.ts THE ONE PLACE status colors and washes are chosen status-dot.ts THE ONE PLACE the status dot's shapes are chosen green-star.ts THE ONE PLACE the green branch's star is drawn green-branch.ts the commits each repository's green branch is at and was at github-webhooks.ts the GitHub webhook relay's deliveries, and whether what was read from GitHub is still current detail-list.ts the row list a tile's body is built from theme.ts light/dark surface colors, theme switch markup and browser behavior lib.ts shared helpers (github, memo, escapeHtml, sparkline, strip, …) ctx.ts shared data sources handed to every tile (ctx.runs) favicon.ts runtime status priority and access to generated PNG favicon copies favicon-png.generated.ts generated runtime PNG favicon copies favicon-artwork.ts build/test-only SVG source for those favicon copies version.ts the browser/server compatibility version a page reloads on dashboard-message.ts shared message storage and fade timing render.ts renderTile(label, view) + the page shell/CSS detail-page.ts the frame, navigation and type a drill-down page starts from live-page.ts the frame of a live drill-down page, and the event streams that keep it current live-page-client.ts the browser half: puts the changed parts of a fresh rendering on the page ci-jobs-page.ts the page behind the ci tile, and its table sorting repo-page.ts the page for each repository, and the index of them all server.ts generic runtime: scheduler, SSE, route mounting, page assembly registry.ts THE ONE REGISTRATION POINT — the array of tiles tiles/*.ts one tile per file ``` `server.ts` knows nothing about individual tiles. It runs a single ticker, collects each tile that is due (respecting its `intervalMs`), renders the results uniformly, mounts any drill-down routes a tile declares, and pushes new tile markup as each independent collection completes. Every registered tile has a gray placeholder under its label in its registered position until its first collection completes, so slow collectors do not leave holes in the board. A later ticker pass skips a tile or shared workflow fetch that is still updating. It starts every other due collection, so pending work does not pause the rest of the dashboard. A collection that remains active for one minute turns its tile gray with "refresh still pending" while retaining its last value and visuals. Its completed view replaces that pending state. A tile with `showOnlyCompletedViews` keeps its last completed color and values, ignores intermediate views, and still shows the pending warning. An intermediate view that carries no chart keeps the chart already on the tile, so a tile drawing its chart from the slower half of its collection has one throughout. A completed view carries every part of a tile, so a chart its collection no longer draws leaves the tile. A tile whose `collect()` throws is desaturated to a gray "unknown" — it keeps its last-known value and shows a short reason (e.g. "source unreachable"), with the full error in the server log — so one unreachable source never blanks or breaks the board. The text field centered between the dashboard identity and freshness indicator is a shared message. Editing the field and leaving it saves the text for every connected dashboard and persists it in the dashboard cache. The message remains fully visible for two hours. It then fades linearly for four hours, after which the server replaces it with the empty string. A new edit starts the timing again. The trust, duration and recent-run tiles declare the workflow snapshots they read in `runSources`. A snapshot holds either a workflow's runs on main or its runs for pull requests. The scheduler fetches each snapshot independently. When a snapshot fetch completes, the scheduler collects every due tile that reads it from the same stored snapshot and publishes those tile updates together. Each snapshot can trigger a tile once per collection interval. A tile with several snapshots can update once for each snapshot as they arrive. This keeps a repository's trust and recent-run views, which both read its runs on main, in agreement when their intervals coincide. The ci tile reads the labs and loom main-branch snapshots for its two main-build rows, on the same interval as the trust tiles, so the scheduler publishes its verdict on a main build with the trust grid drawn from the same snapshot. It reads every other workflow in a sweep of its own, described below, which runs in the background so that no snapshot's tiles wait for it. A problem with either snapshot turns only that main build's row unreadable, rather than graying the whole tile. A tile that reads snapshots can ask, through `ctx.collectAgain()`, to be collected again, for when it has data of its own that arrived after its last collection. The scheduler collects it from the snapshots it was last published from beside its neighbours, without fetching them again, so the tile still describes the same runs as they do. A tile that asks while it is being collected is collected again once that collection's views are published, so it never gets ahead of the tiles sharing its snapshot. A tile still being collected when one of its sources falls due is left out of that source's collection, and counts as collected with it. Once its own collection is published, it is collected again from the snapshot it missed. It then shares the source's next fetch with the other tiles, rather than falling due on its own and fetching the source for itself, which would leave it describing different runs from its neighbours from then on. A workflow snapshot is read through the run-list reader described under [Reading GitHub's run lists](#reading-githubs-run-lists). Each refresh reads the top hundred runs of the workflow's list, of every branch and event, and reads further down only as far as the snapshot's window reaches past the runs already held. A workflow's two snapshots, of its main runs and of its pull request runs, share one read of the top of the list when they are fetched within twenty seconds of each other. Each snapshot names the list filtered to its branch or event as the one through which runs started again further down are found. A fetch fails when a read of the unfiltered list, or of a run by its id, fails; a filtered list that cannot be read is logged and passed over. The scheduler then keeps the snapshot it has, and each tile reading it turns gray and names the problem, apart from the ci tile, which marks the affected build unreadable. The next fetch that succeeds clears the gray. The recent commits tile reads the labs, loom, and weaver main snapshots beside the commits it reads from every repository in the organization. It rebuilds and sorts its list whenever any of those snapshots arrives. If one snapshot has not arrived yet, it shows the list it can build in gray and names the pending source. If a later refresh fails, it keeps the last good snapshot for that source and shows the list in gray with the error. A repository can have a green branch: a branch its own CI moves to the newest commit of main whose tests passed. `GREEN_BRANCHES` in `config.ts` names it by repository; loom's is `main-green`. Each fetch of a repository's main snapshot also reads the updates of its green branch from GitHub's activity record for the repository, which lists every update of a branch, the ones CI makes through the API included. The first fetch reads back as far as the snapshot's age cutoff, each later one reads only the updates made since, and updates older than the cutoff are forgotten. A read that fails is logged and changes nothing, so the next one reads what it missed, and the snapshot is unaffected. Each run in the snapshot carries what the branch says of its commit. Every view that draws a run on main large enough to carry it marks such a commit with a green star: solid while the branch is at the commit, and hollow once the branch has moved past it. A commit the branch moved past without stopping at gets no star, although the branch contains it. The views are the rows of the recent commits tile, the bars of a repository page's main-branch chart and the runs listed under it, and the heading of the commit CI Gantt page. The star's tooltip names the branch and says whether it is or was at the commit. Each event connection receives the current tile snapshot before it waits for new collections. The browser matches each tile in that snapshot with the tile of the same label on the page, leaving unchanged elements, focus, and scroll positions in place. Inside a tile that changed, an element carrying a `data-focus-key` attribute is matched with the element carrying the same key in the new markup, which takes over its keyboard focus and scroll position; a tile puts that attribute on each element whose focus or scroll position should carry over from one update to the next. Routine data updates never navigate the page. The page shell — the styles and the client script — arrives only with a full page load, so the server hands the browser a compatibility version and the page reloads itself as soon as the server reports a different one. A deployed image uses the commit its publishing workflow checked out, which is fixed for the life of the image, so every display on that image agrees. A server started from a checkout reports the moment it started instead, because the code under it changes between one start and the next: a watched restart therefore pulls every open page onto the code that restart is serving, and the version in the page source says which start served it. An unattended display reloads when it reconnects to a server reporting a different version. An unattended display also survives the server going away. Every serving tick sends a heartbeat down each open event connection, so a browser can tell a server with nothing new to report from a stream that has stopped delivering. The page watches its own stream from the same one-second tick that paints the freshness indicator, and reopens the stream once the server has been silent for the length of time it takes that indicator to turn red. That covers each way a stream dies. The browser reconnects one that ended in a network error by itself. It gives up for good on one whose reconnect was answered by an error page, which is what the Tailscale proxy in front of the pod returns while the server restarts. And a connection that outlives a sleeping laptop stays open with nothing ever coming down it, which the browser never reports at all. A stream that was delivering and then closed is reopened immediately. One that has never delivered is reopened once per silence interval, so a server that refuses or fails connections costs one request an interval however long it is away, while one that accepts a connection and then drops it is followed as fast as it flaps. The silence interval is the fastest tile's collection interval plus a serving tick plus ten seconds. That keeps it clear of the heartbeat period, which is one serving tick, by the whole of the fastest tile's interval. A heartbeat slower than the silence interval would replace a healthy stream on every page tick. The indicator and the reconnect run off different clocks and mean different things. The indicator counts from the last tile update, so red means the data is old. The reconnect counts from the last event of any kind, heartbeats included, so reaching the interval means the server cannot be heard at all. A page can be red on a perfectly good stream, when every collector has been slow at once. The header badge is what tells the two apart, reading OFFLINE instead of LIVE whenever the page has no stream it is hearing the server on. Reopening the stream and saying whether the server can be heard are separate questions and get separate answers. Reopening is paced, because each attempt costs a request. Saying so is not. The browser reports a connection dropping as it drops, whether it means to retry or has given up, and the badge turns over on the next tick — a second, not an interval. Stopping the server therefore stands the page down almost at once, and the badge comes back the moment any event arrives on a stream again, including a heartbeat. The elapsed interval is only the last resort, for a connection that stays open with nothing coming down it, which is the one case the browser reports nothing about at all. A background tab's timers are throttled to about one a minute and a sleeping machine's stop altogether, so the page checks its stream on becoming visible and on the browser regaining the network as well as on its own tick. A drill-down page can be kept current the same way. A route that declares `live: true` serves a page built by `livePage` in `live-page.ts`, which carries the client script and keeps everything that changes inside its `
` element, apart from the tab's favicon. The page opens `/events?page=`. On every serving tick the server sends a heartbeat down that stream and renders the page again by calling the route's handler, and it sends the new markup when that differs from what it last sent. A page that connects is sent the current markup whether or not it changed. The browser keeps every element whose tags, attributes, and text between its children match the new markup's, and whose children match in number, and compares those children the same way; any other element that differs is replaced whole. So when the header's age ticks over, only the text giving the age is replaced, and the rest of the page keeps a reader's focus and selection. A manifest that adds or removes a section of the page replaces the whole of `
`. Before it compares, the page sends its `
` a `live-page-update` event carrying the new `
`, which bubbles to the document. A page the reader can rearrange listens for it and arranges the new markup the same way, so a rearranged part that did not change still compares equal and is kept. Every rendering it sends names the version being served, and a page built by another version reloads instead. The page follows its stream with the dashboard's own code (`followUpdates` in `stream-client.ts`), reopens it once it has heard nothing for three heartbeat periods, and its badge reads OFFLINE while it cannot hear the server. A page is rendered only while some browser is showing it. The test selection page, the CI jobs page, and the repository pages are live, so a screen left on one follows the manifests as the publisher writes them, or the tiles' collections as they make them. The tab favicon follows the most urgent visible tile. It is red when any tile is red, orange when there are no red tiles but at least one orange tile, and green otherwise. Gray tiles do not turn the favicon gray. The page uses one URL-backed PNG favicon. Scalable source artwork is used only to generate and verify those raster assets; it is not part of the runtime dependency graph. The green face is a rounded square, the orange warning face is a triangle, and the red faces are octagons. The eyes and mouth of the warning face sit ten percent of the canvas height below the level the other faces use, which places them in the wide part of the triangle instead of near its apex. The red favicon starts sad and becomes a crying face after the dashboard stays red for one continuous hour. The server retains the elapsed time across reloads. Returning below red resets it once every collector due in the same pass has finished. A repository's page and the CI jobs page show a status in their favicons the same way. A repository's page shows the repository's standing. The CI jobs page shows the color of the ci tile's latest collection. A live page takes its favicon from each rendering the server sends, so the favicon changes when the status does. While the status is gray, and on a live page with no status, the favicon is empty. These pages use the green, orange, and red faces. The crying face, which measures how long the whole dashboard has stayed red, appears only on the dashboard. After changing `favicon-artwork.ts`, regenerate the embedded PNGs and their content-based cache version from the dashboard package directory: ```bash cd packages/dashboard deno task regenerate-favicons deno test --allow-all favicon-raster.test.ts regenerate-favicons.test.ts ``` ### Reading GitHub's run lists Every list of workflow runs the dashboard reads comes through `github-runs.ts`. The dashboard's GitHub client refuses a request for a run list made anywhere else, so a tile cannot read one any other way by accident. GitHub serves a workflow's unfiltered run list current. A list narrowed by branch, event, status, actor, creation time, commit, or check suite comes from an index that is often hours or days behind, and sometimes weeks or months behind. Such a list leaves out runs that exist, and shows the runs it does list as they stood when the index last caught up. On 2026-10-06 the list of gvisor's Go runs filtered to its default branch began with a failure nineteen days old, and left out the eight passing runs created after it. So the reader takes which runs exist, and their order, only from the unfiltered list. A filtered list only names runs that may have changed: a held run the reader wants, which it shows updated after the copy held, is then read by its id. The unfiltered list carries every run, whatever it ran for, so a reader that wants one branch's runs may have to read a long way down it. Each reader says which runs it wants, how many of them it needs at most, and which run is the last it needs, and it is given only the runs it wants. A `RunLists` keeps the head of each workflow's list: its runs from the newest down, with none missing between them. Each reading brings the head up to date by reading the top of the list, twenty runs unless the reader asks for up to a hundred, and following it down the list until it reaches a run the head holds. If it passes the newest run held without reaching any, every held run in its way was deleted, and the head starts again from what it read. A read of the top that leaves out the newest run held is rejected as behind, unless reading that run by its id shows it deleted. Readings made within twenty seconds of each other share one read of the top, and while the GitHub webhook relay vouches for the runs held, a reading reads the top only when a delivered run has landed on it (see [Following the GitHub webhook relay](#following-the-github-webhook-relay)). A reading that needs runs below the head reads on down the list. The head is kept as deep as the deepest of its readers last read, and a reader that has not read for a day stops holding it. Each part of the dashboard that reads runs keeps its own `RunLists`, and so do CI history and its Gantt, and the benchmarks tile and its drill-down, so that a deep reading, such as CI history's the first time it reads a workflow, never holds up a shallower one of the same workflow. The publisher's activity, which reads a workflow no tile reads, reads through the tile context's. Each `RunLists` the server reads through keeps its heads in a file in the dashboard cache directory, so a restart does not cost a reading from scratch, and neither does a deploy when the cache directory is on a volume that outlasts it. The tiles that read runs through the shared tile context, the publisher's activity among them, share `fabric-wall-run-lists-tiles.json`; the ci tile keeps `fabric-wall-run-lists-ci.json`; the benchmarks tile and its drill-down keep `fabric-wall-run-lists-benchmarks.json` and `fabric-wall-run-lists-benchmarks-drill-down.json`; CI history and its Gantt keep `fabric-wall-run-lists-ci-history.json` and `fabric-wall-run-lists-ci-gantt.json`; and the commit CI Gantt's collector keeps `fabric-wall-run-lists-commit-ci-history.json` and `fabric-wall-run-lists-commit-ci-gantt.json`. A file holds, for each workflow, the runs of its head, whether they reach the end of the list, the oldest run each reader last read and when, and when the top of the list was last read. A `RunLists` takes a workflow's head from its file whenever it starts holding one, and reads the list as it reads one it holds: from the top until it reaches a run in the file, so a run deleted from the top of the list meanwhile is dropped. Every held run the reader wants that had not finished is read again by its id before the reader is given it, and the reader's filtered lists are read again, so a run started again meanwhile is found as it would have been. The file is written after every reading, including one that fails. A reading that reads further down the list records, as it goes, that its reader read that deep, so the runs it read are kept even when it fails, and a CI history walk that GitHub breaks off goes on from where it stopped, in the same process or the next. A head nobody has read for a day is dropped, so a dashboard that has been down for longer reads from scratch. Processes that share a file lock it while each puts its heads in place of the ones the file holds and replaces the file whole. Of two heads of one workflow, the one whose top was read last is kept. A head was whole when it was saved, and a head taken from the file reads its list from the top until it reaches a run it holds, so either one can be restored. Every version of the dashboard reads the file the same way, without a version number. A head is cut back above the first run that lacks a field the dashboard reads or holds one it cannot use, so the runs from there down are read from GitHub again; a head whose own fields cannot be used, or that records a time later than the present, is dropped; fields the dashboard does not know are ignored and not written back; and a file that cannot be read at all is replaced. GitHub lists runs newest first, and a newer run has the larger id. Runs land at the top of the list and can be deleted from anywhere in it between two requests, so a page asked for by its number can start some way from where it was expected to. Runs from a page are added after the last run held only when that one page also holds that run, since each page is a view of the list at one moment. The reader asks for a page that holds where the run is expected and as many runs after it as it can, choosing a page size between 51 and 100 to make that so. A page that holds only newer runs sends it on down the list a page at a time, and one that holds only older runs sends it back up a page at a time; once it has turned round, it asks for pages that hold the expected place in their middle. A page that holds runs on both sides of the last run, but not the run itself, shows that run deleted, and it is dropped. The head remembers how far from its expected place the last run was found, which deleted runs still held above it make more than zero, and starts the next walk there. A page whose runs are not in order of falling id is rejected. A run changes after it is listed: it finishes, or someone starts it again, which leaves it in its place in the list. The runs at the top of the list are read anew on every reading that cannot share a read made in the last twenty seconds. Below them, a held run the reader wants that had not finished is read again by its id before the reader is given it. A run that had finished and has been started again is found through filtered lists every reader names, usually the list filtered to the reader's branch or event. Each such list is read down to the oldest run held, or to runs created more than thirty days ago, since GitHub allows a run to be started again for thirty days after it was created, and readings within twenty seconds of each other share one read of it. A held run the reader wants, which such a list shows updated after the copy held, is read again by its id before the reader is given any run. A lagging filtered list therefore delays the news of a run started again, and cannot hide a run or turn one back. A filtered list that cannot be read is logged, and the reading goes on without it. A run that is gone when it is read again by its id is dropped. A held run deleted without being read again stays held until the head is cut back past it. A reader whose answer rests on one run, as the ci tile's verdict rests on the run that decides a job, can ask for the run it stops at to be read again by its id when the reading did not read it from the list. The reading then goes on past that run when, read again, it no longer settles the question. ### Following the GitHub webhook relay The organization's GitHub webhook sends every event to a relay, described in `github-webhook-relay/README.md` in the infra repository. The relay writes each delivery, the moment it arrives, to every TCP connection open to it, and keeps nothing for a subscriber that is not connected. A subscriber that falls behind is disconnected rather than skipped, so a connection that is still open has carried every delivery made since the relay started writing to it. `github-webhooks.ts` holds the dashboard's connection. The server opens it when it starts, at the `host:port` that `GITHUB_WEBHOOK_RELAY` names, by default the stage relay's subscriber address, and turns TCP keepalive on for it. An empty `GITHUB_WEBHOOK_RELAY` turns the relay off. A connection vouches for what was read under it: data read from GitHub while the connection was open is still current for as long as that connection is open and no delivery that could change the data has arrived since, up to fifteen minutes after the read. GitHub does not promise to make every delivery, and a repository outside the organization sends none, so that limit bounds how long a delivery that never arrives can leave anything out of date. A connection starts vouching with the first delivery it carries, which shows that the relay has started writing to it. A connection that has carried nothing for two minutes stops vouching, and the next serving tick ends it and opens another, because a connection whose far end has gone carries nothing and does not end; the organization's webhook delivers many times a minute whenever anyone is working. A delivery that cannot be read, a `workflow_run` or `push` delivery that lacks what the dashboard reads of one, and a delivery that a reader below fails to take in, each end the connection as well, since what was read under it may then be out of date. A connection that ended after carrying a delivery is replaced at once. One that cannot be opened, or that ended before carrying anything, is tried again on the next serving tick, and the first of a series of failures to connect is logged. Whenever the relay cannot vouch, every reader below reads GitHub on its own interval as it would with no relay at all. The relay is used in these places: - **Run lists.** Each `RunLists` takes in the runs delivered in `workflow_run` deliveries for the workflows it holds. A delivered run it holds replaces the copy held, unless the copy is newer, and counts as read. A delivered run it does not hold, newer than the oldest run it holds, is one the head lacks. The next reading reads the top, however recently it was last read, and reads each such run the top leaves out by its id: one that is gone is dropped, one older than the newest run listed is placed among the runs held, and one newer is a run the list is served behind, so readings go on reading the top until the list shows it. GitHub's list can be read a moment before it shows a run that a delivery has announced. A read of the top keeps the head's copy of a run, read or delivered, when the list's copy is older, and a delivered copy unless the list's copy is newer. While the relay vouches for the read of the top, a reading reads the top only for such a run, and while it vouches for the read of a filtered list, a reading does not read that list. So while deliveries keep the relay vouching, a workflow with no activity costs a read of the top, and of each filtered list, every fifteen minutes, and one with activity costs about a read of the top for each new run; once the relay goes quiet, readers read on their own intervals. A run deleted from the list sends no delivery; it stays held until the top is next read, or, further down, until the head is cut back past it. - **The scheduler.** A delivery that changes a run source, which is a run of its workflow in its scope or, for a source of runs on main, a push to the repository's green branch, makes the source due at once. The scheduler fetches it and collects the tiles reading it without waiting for their interval, so a main build that fails turns its tiles red as soon as GitHub says so. A delivery for a source being fetched already has the source fetched again once that fetch is published. The tiles are still collected on their intervals, from data that costs nothing to read while the relay vouches for it, so whatever a tile shows that depends on the time stays current. - **The ci tile's sweep.** A delivered run of a job the sweep judges, which is a run on its repository's default branch not started for a pull request, starts a sweep at once when it finishes, or as soon as the sweep under way ends. Whether a failing job's workflow has changed since the failure is kept from one sweep to the next until a push to that repository's default branch is delivered, or the relay stops vouching for it. - **The green branch.** The repository's activity record is read only when a push to the branch has been delivered since the last read, or the relay does not vouch for that read. A read that does not yet show the branch at the commit the latest delivered push moved it to is read again on the next refresh. - **The recent commits tile.** The organization's repositories are listed again only after a `repository` delivery, or when the relay does not vouch for the last listing. A delivered push to a repository's default branch has that repository's commits read at once, and read again on each refresh until they start at the commit pushed; the read lists the repositories first only when the relay does not vouch for the last listing, or when that listing leaves out the repository pushed to. A push to a repository that a listing read after it still leaves out is dropped. While the relay vouches and nothing has been delivered, the tile reads nothing. The ci tile's inventory of repositories and workflows is read once an hour whatever the relay delivers, because enabling or disabling a workflow sends no delivery. The github spend tile reads GitHub's billing data, which no delivery describes. ## Add a tile 1. Create `tiles/my-tile.ts`: ```ts import { runSource, type Status, type Tile, type TileView } from "../types.ts"; export const myTile: Tile = { label: "my tile", // the header on every view; unique among tiles intervalMs: 60_000, // how often collect() runs // wide: true, // optional full-width placement // runSources: [runSource("owner/repo", "ci.yml", "main")], // or "pull requests" // reportsSourceProblems: true, // read ctx.runSourceProblem(); never grayed // repo: "owner/repo", // the one repository it reports on; its page shows the tile async collect(ctx): Promise { // ctx.runs() -> shared CI runs; ctx.env("KEY") -> env var. // If a required env var is missing, return a gray "unknown" view — don't throw. const s: Status = "good"; return { status: s, value: "42", sub: "things" }; }, // routes: [{ path: "/my-drilldown", handler: (req, url) => new Response("…") }], }; ``` 2. Register it in `registry.ts` (import + one line in the `TILES` array). That's it. Remove a tile by deleting its line from `registry.ts` (and its file). ### Philosophy and values The dashboard is something people glance at all day, often out of the corner of an eye, so a tile's first duty is to inform without adding anxiety. Keep it calm by default. There are only three signals — good, warn, and bad, plus a gray "unknown" when a source is missing or a collector fails — because green should mean "fine," red should mean "a person should act on this now," and there is little in between worth manufacturing. Red has to stay rare and trustworthy: if tiles sit red or amber most of the time, people stop seeing them, and a board everyone has learned to ignore is worse than no board. A tile earns a color change; it never reaches for one to get attention. A signal is carried by four things at once, so that no one of them has to be read on its own. Losing any single cue still leaves a state legible, which is what a person who cannot separate red from green is doing all day. The first is color, chosen so the four stay apart for that person too. Green sits at a teal and amber at an orange rather than at a yellow, which puts the two on the blue/yellow axis; red/green color blindness leaves that axis working, and good against warn is otherwise the hardest pair on the dashboard. Every pair of statuses is measured in `palette.test.ts`, which simulates the two common forms of red/green color blindness and fails if any pair comes within reach of reading as one color. The second is the shape of the status dot: a circle for good, a triangle for warn, a diamond for bad, and a hollow ring for unknown. A shape survives any kind of color vision, and any distance at which the dot is still visible at all. Every surface that draws a dot draws these shapes, a tile's header and the rows of a drill-down page alike, rather than coloring a circle of its own. The third is weight. A tile's background wash and its border both get stronger as its status gets more serious, so a good tile is the quietest thing on the dashboard and a red one the loudest. That ordering holds with the color taken away altogether. The fourth is texture, behind the tile. A gray tile is covered in a grid of tiny dots, an amber tile in broad wavy lines, and a red tile in zig-zags. The lines are wide and faint rather than fine and dark, which puts enough of the pattern on the tile to catch the eye without any one line drawing it. The zig-zag covers a little under half of a red tile, and the longer, gentler wave about a third of an amber one. Green tiles are left plain, which is the calm the rest of the dashboard is measured against. Every texture fades out down the tile: it is whole for the tile's top seventh, thins from there, and is gone seven tenths of the way down, which leaves the sub line and the foot of a chart on plain color. Everything the dashboard draws over a texture has to stay legible against it, and the faintest marks are the ones that decide how strong a texture can be: a tile's title, its drill-down hint, its running badge, and the times, dividers and pop-out arrows down the recent commits list. Those are set against the lightest the background reaches under a texture band rather than against the tile's flat color. The dividers and the arrows are fractions of white rather than fixed grays, because the surface they sit on is tinted by the status and a gray that reads on one wash washes out on another. `palette.ts` holds all of this, and it is the only place any of it is chosen. A status color that appears anywhere — a tile, a dot, a headline, a run cell, a drill-down row, the favicon — comes from there. A shade that follows from another is worked out there too rather than written down beside it, so a change to a color carries without a second edit. Sparkline strokes fade from a transparent version of their own series color over the shared chart axis. The shape a dot takes is geometry rather than color, so it lives in `status-dot.ts` instead, which is the one place it is chosen and which every page drawing a dot takes its rules from. Think about how a tile makes someone feel before you think about what it measures. Prefer an honest gray "unknown" over a false green — a tile that can't tell "healthy" from "I couldn't reach the source," and stays green while blind, is worse than one that admits it doesn't know. Report on the system, never on individuals: no per-person leaderboards, no "who broke the build," nothing that turns the dashboard into a place to rank or shame people. And be wary of the number that looks like progress but isn't — coverage percentage, lines of code, raw PR counts — anything that becomes a bad target the moment someone optimizes for it, or whose only job is to look busy. If a metric would quietly pressure people into gaming it, leave it off. This is a quiet instrument panel a tired person should be able to trust at 2am, not a scoreboard and not a surveillance tool. A tile is about as wide as a business card, and every line on it is set without wrapping. A line longer than that width is not shortened by the renderer so much as cut off by it, and a line that ends in an ellipsis has taken the whole line and delivered none of it. So the headline figure and the sub line under it are what a tile has room to say. Text can go below them — the host names down the production tile, the commit list on the full-width recent commits — but only text short enough to be read whole at the width the tile is actually given. Something longer is not a longer tile, it is a page. That page is an ordinary drill-down — a route the tile declares, reached through the tile's `href` and indicated by the top-right arrow. The tile's `hint` supplies the tooltip and accessible link description. `/bench` holds the histories behind the benchmark and duration tiles, and `/test-selection` holds the manifest behind the two test tiles. A page has the width to spell a test's whole name, so nothing on one has to be abbreviated. ### Repository pages The **Repositories** button at the bottom left of the dashboard opens `/repos`, the index of every repository the dashboard knows. It opens on one sentence counting them, then groups them. Those not passing come first, as cards on the status wash and texture a tile of their color wears, each naming its standing and the worst thing wrong with it. Those passing follow as names with a green dot, and those with nothing to report as names alone. A repository is known once a tile names it, as the one repository it reports on (the tile's `repo`) or as the repository of one of its run sources, or once the ci tile's sweep has read it. It is named without its owner, as the ci tile names it, so two repositories of one name under different owners would share a page. A switch at the top of every page moves to any repository, and the heading beside it leads back to the index. A repository's page, `/repos?name=`, puts everything the dashboard holds about it on one screen. On a window at least 1000 pixels wide and 620 tall the page is laid on a grid as tall as the window. A panel with more than its room scrolls within itself. The middle row keeps at least 280 pixels, so a window too short for that scrolls the page rather than crushing the runs. On a smaller window the same parts follow one another down the page. - Across the top, its name, set large on the wash and texture of its standing, the way a tile wears its status, with links to its code, pull requests, and actions on GitHub and how long ago its workflows were read. Beside the name is its standing: `N failing` when any of its jobs is failing, and otherwise `Passing`, `Worth watching`, or `Needs attention`, from the worst color among its jobs with a verdict and its tiles. A gray tile is left out of that, as it is left out of the favicon, since a source that could not be read says nothing about the repository. A repository with nothing that has a color reads `No verdict`, and one whose workflows could not be listed is orange and reads `Unreadable` unless one of its jobs is failing or something about it is red. Under the standing, a sentence says how the newest finished run on main went, unless those runs are out of date. At the right are its key figures: how many of its workflows with a verdict pass, how many fail, how many runs are going in it now across its jobs and its run snapshots, and the median time of the main runs that passed among those the chart shows. - On the left, for each of its run sources, main before pull requests, a chart of its newest `RECENT_DISPLAY` (50) runs: a bar for each run, oldest on the left, as tall as the run took and colored by what it concluded, with the dashboard's diamond over each failure and the green branch's star over each commit it is or was at. The charts share the panel's height, so a taller window draws taller bars. A dashed line marks the median of the runs that passed. The chart's scale reaches the longest run but one in ten, and at least 1.7 times that median, so one run that hung does not flatten the rest; a bar past the top is cut square. The chart is only as wide as its bars, and the ages under its ends are those of its oldest and newest runs. Each bar links to its run for a pointer; the keyboard passes over the bars, since the workflow's name in the heading links every run of it on GitHub. Under the chart, the three newest runs are listed: what each was for, what it concluded, how long it ran (linked to the commit's CI Gantt for labs and loom), and when it started. A run on main is named by its commit, and a pull request's run by its title; either name links to the run, except that a pull request number in it, such as "(#1234)", links to that pull request. A snapshot that has not been read yet, or could not be brought up to date, says so. - On the right, what needs attention: every job and tile that is not green, gray ones included, worst first, one to a line, each linked where its job or tile links unless that is this page, or a line saying nothing needs attention. Under it, its workflows in columns, worst first and those with no verdict last. Each has how long ago its deciding run started, or, with no verdict, why; a link to the run in progress when there is one; and, as its tooltip, what started that run, what it concluded, and how long it ran. - Across the foot, its measures: every tile whose `repo` is this repository, as a card at most the width a tile has, in one row, with its label, headline, sub line, and chart, linked where the tile links unless that is this page, with the tile's hint as the link's tooltip. A card's headline shrinks with the card, so a narrow one still says it whole. A repository card or a measure takes its status's wash and texture, and a workflow its wash, only when it is orange or red, so that a page of passing parts stays quiet and what is wrong stands out; the opening of a repository's page takes them for every status, as a tile does. The pages render what the server already holds, the latest view of each tile, the latest snapshot of each run source, and the ci tile's latest collection, so opening one costs no requests and never disagrees with the dashboard. Where a tile has more than one candidate for its sub line, the one that explains the color it is wearing wins. The test selection tile carries the test time a pull request would run, against the whole corpus's, under its headline share, and gives that line up to whichever of three conditions is in force: the count of things the publisher found broken in the cost model where there is one, otherwise the lane past its budget where there is one, and otherwise the count of tests no lane can hold. A stale manifest colors the tile too and competes for neither line, because its own figure is the age badge in the header. ### The `TileView` a tile returns The header text is the tile's own `label`, which does not change from one view to the next; a view supplies everything under it. | field | meaning | |---|---| | `status` | `good` / `warn` / `bad` / `unknown` → green / orange / red / gray | | `value` | big headline (trusted HTML — escape data with `escapeHtml`) | | `sub` | sub line (plain; escaped for you) | | `extra` | trusted inline HTML under the value (sparkline / strip / list) | | `duration` | a span in milliseconds, rendered (via `humanSpan`) in the chart's bottom-left corner | | `aside` | trusted inline HTML minor header facet (e.g. an MTD or a "running" badge) | | `href` | makes the whole tile a link (an `http…` link opens a new tab). An anchor cannot hold another, so when `extra` holds links of its own, the whole tile but `extra` is the link, and the links in `extra` go where they point. `value` and `aside` hold no links when there is an `href` | | `hint` | tooltip for the top-right `↗` drill arrow and accessible link description, e.g. `"commits ↗"` | ## Tiles Where the table names `GH_TOKEN` or `GH_BILLING_TOKEN`, a GitHub App configured with `GH_APP_CLIENT_ID` and `GH_APP_PRIVATE_KEY` serves instead, through its installation with the same permissions; see [Credentials](#credentials). | tile | source | needs | |---|---|---| | ci | every job the organization runs outside pull requests, in every repository the token can see that is not archived: for each active workflow, the newest run on that repository's own default branch that passed or failed, however many runs that judged nothing came after it, among the workflow's newest thousand runs of any branch. The headline is `passing` when every one of them passes, the repository's name when a single job is failing, as in `loom failing`, and a count when more than one is, as in `3 failing`. The header carries how many jobs the headline speaks for and how many repositories they came from. The body lists every failing job with its conclusion and how long ago it ran; while the tile is not red it also lists the labs and loom main builds, so the two builds the team watches stay visible, and a red tile lists only its failing jobs. A failure older than `CI_FAILURE_FRESH_HOURS` is orange rather than red: it is still failing and still counted, and it is no longer the thing that just broke. A failure made before the workflow's file last changed does not count at all, since that is what a job someone stopped rather than fixed looks like. A repository whose workflow listing cannot be read is listed too, and turns the tile orange rather than being passed over. The rows carry no links of their own, because the tile itself opens the page below | `GH_TOKEN` (or `GITHUB_TOKEN`) with Actions read across the organization, and check runs read in private repositories (see [Credentials](#credentials)) | | CI jobs → `/ci` | every job the ci tile read, at full width: the repository, linked to its page, and the workflow, what started the deciding run (`push`, `schedule`, `workflow_dispatch`, and the rest, as GitHub names them), what that run concluded, how long it took, when it started, and how long ago that was. Every column sorts, once up and once down, on the value behind the cell rather than on what the cell says, so durations and times order as the measurements they are; the page opens worst first and a column of equal values keeps that order beneath it. Workflows with no verdict are listed under the table rather than through it, each with why: no completed run on the default branch, which is what a workflow only a pull request triggers looks like; runs that all judged nothing; or a workflow changed since it failed. So are repositories whose workflow listing could not be read, each linked to its page and to its actions on GitHub. A workflow with a run in progress on its default branch carries a blue dot after its name, which links to that run, and the summary above the table counts them. Only a run in progress carries one: a queued run, or one waiting for approval, does not. It renders the tile's own last collection rather than asking GitHub again, so opening it costs no requests and shows exactly what the tile shows, running dots included: a run that starts or ends between collections shows at the next one. The page is live: an open copy shows each collection within a serving tick of the tile finishing it, without reloading, and in whatever order the reader sorted it | none | | labs ci trust, labs ci duration | GitHub Actions (`deno.yml` in `commonfabric/labs`), via the REST API. Trust reads the runs on main; duration reads the pull request runs. Each trust tile opens its repository's page, and each cell of its strip opens its run | `GH_TOKEN` (or `GITHUB_TOKEN`) | | loom ci trust, loom ci duration | the same two tiles for `commonfabric/loom` (`test-fast.yml`) | `GH_TOKEN` (read access to loom); optional `DASHBOARD_LOOM_REPO` | | weaver ci trust, weaver ci duration | the same two tiles for `commonfabric/commonfabric-weaver` (`ci.yml`). The duration tile is not a link, because the history views cover only labs and loom | `GH_TOKEN` (read access to weaver); optional `DASHBOARD_WEAVER_REPO` | | recent commits | the newest commits on the default branch of every repository the token can see that is not archived, interleaved newest first and each tagged with its repository: each commit that landed a pull request opened by an account GitHub does not mark as a bot, and each commit that the labs, loom, or weaver main CI workflow ran on after a push. The commit a pull request landed is the one GitHub merged it with: the squashed commit, the merge commit, or the last commit rebased onto the branch, so a pull request is one row. The rest are left out: a commit pushed without a pull request, and a bot's pull request, such as an automated deployment's, unless one of those workflows ran on it. So a repository used as a notebook adds no rows. A pull request counts only when it was merged into the repository's default branch. The tile lists the organization's repositories at most every thirty seconds, and while the GitHub webhook relay vouches for the last listing, not at all. When a repository's last push has changed since its commits were read, or a push to its default branch has been delivered, one GraphQL query reads its newest hundred commits again, each with the pull request that landed it; a delivered push has them read at once (see [Following the GitHub webhook relay](#following-the-github-webhook-relay)). A repository GitHub refuses to read, for want of permission for example, keeps the commits already read until its next push, and the tile's sub line counts such repositories. A repository whose read GitHub rate-limited, or that failed for any other reason, keeps them too, and is read again on the next refresh without being counted. That read runs in the background, and the tile is collected again when it finds new commits. A commit one of those workflows ran on carries the newest such run's result and duration, from the same main-run snapshots the ci trust tiles read, and its text links to that run; the text of any other commit links to the commit. A pull request number in a title, such as "(#1234)", links to that pull request, and the row's arrow links to the pull request that landed the commit, or to the commit when its title names none. A row whose commit the repository's green branch is or was at opens with a star, read from GitHub's activity record for the repository. The tile is red when the newest finished run of those workflows failed, and orange when one of the nine before it did | `GH_TOKEN` (Actions read on labs, loom, and weaver; Contents and Pull requests read across the organization) | | commit CI Gantt → `/ci-gantt` | job and step timing for every successful main workflow run attached to one commit, linked from run durations in recent commits; the commit wears the green branch's star when the branch is or was at it | `GH_TOKEN` | | CI duration history → `/bench?view=ci` | labs and loom job, shard-group, and end-to-end workflow duration trends. The labs and loom duration tiles open their repository's view, which charts runs on main rather than the pull request runs the tiles measure | `GH_TOKEN` | | CI run Gantt → `/bench?view=gantt` | detailed labs or loom job phases from `scripts/ci-gantt.ts`, backed by the CI history cache | `GH_TOKEN` | | flaky tests | how many tests the test-selection publisher measured disagreeing with themselves often enough to keep off pull requests, read from the newest selection manifest. The headline names what it counts, so it reads `25 flaky tests`, or `no flaky tests` when there are none. The line under it says what the count was drawn from: the span of history a flake share is measured over, which the manifest's `FLAKE_WINDOW_DAYS` dial names, and how long ago the publisher measured. The sparkline plots the count across every available manifest. Which tests they are is on the page behind it. Amber from one, red from ten. Gray with a dash when no manifest is available, the newest readable manifest has an empty corpus, or none of the manifests it looked at can be read, naming the shape it found in that last case. Readable history remains visible when the newest object cannot be read | optional `GH_TOKEN` for publisher activity | | test selection | what share of the corpus's test time the newest selection manifest would have a pull request run, read from the same manifest. Each test counts once, at the cost the manifest measured for one execution of it; what a lane spends opening suites and repeating tests is in neither figure. Dividing tests more finely into separately recorded tests leaves the share where it was. The lanes' budget is fixed, so the share falls as the corpus's test time grows. The manifest's packing is built with nothing mandatory, so the share is the one a pull request touching no test would get; a real one re-packs against its own diff and spends part of the same budget on what that diff makes mandatory. Amber once that manifest is over eight hours old, because selection quality decays with it, and amber too while the corpus holds a test costing more on its own than a whole lane's budget, since no packing can place one and a pull request then runs it only where its own diff makes it mandatory. Red when a lane's projected work is past the budget the manifest was packed to, and red when the publisher found the cost model that manifest was packed by broken, in which case the tile links to the page's cost model section. Those three take the sub line off the test time, the broken model first and the overrun lane second. The sparkline plots the selected share across every available manifest, using each manifest's own corpus time. Gray on the same conditions as the flaky tests tile, including an empty latest corpus, and gray for a latest corpus with no test time. A manifest whose lanes name a test its corpus does not hold is read as a broken manifest | optional `GH_TOKEN` for publisher activity | | test selection detail → `/test-selection` | the manifest behind both test tiles, at full width: the selected share, the test time and the count of tests behind it, and the counts of lanes and flaky tests; then every lane against its budget and how many tests it holds, the cost model the lanes were packed by against what they spent (what the publisher found broken in it, how many lanes ran past their bound, and each suite's charges against the manifest before and against what its batches spent), every test held back as flaky with the rate it was measured at, and every test no lane can hold. Both tiles link here, the flaky tests tile straight to its flaky section. The page is live: an open copy shows a newly published manifest within about a minute, without reloading | none | | labs coverage debt | the repository's whole uncovered-line count and what a median day does to it, read from the coverage measurements each `main` run writes into the test-run record store, the newest of a day's that measured it in a run attempt that passed (`docs/development/COVERAGE.md`). A run attempt that failed measured through tests that failed or never ran, and reads high by as much as those tests would have covered, which can be thousands of lines; the store does not say whether a run passed, so the tile asks GitHub about each attempt it opens. A day on which no attempt passed has no measurement. A day the tile's cache file holds with a measurement that names no attempt is read from the store again when the store holds anything for that day, and otherwise is charted once GitHub says that run's latest attempt passed. The headline is the count; under it a signed rate gives the median day's move over the last three weeks, and the chart spans eight weeks with those days highlighted. Its vertical scale uses the highlighted days, so older extremes can extend outside the chart. Amber means that median is a rise, which takes more than half the days in the window, so a day that added debt says nothing on its own. It never turns red, and it goes gray rather than stand on a stale number: when five days have passed with nothing measured, and until the window holds a week of days to take a median over. A run whose pattern compile cache missed is passed over, because a cold run reaches branches a warm one does not and reads about a tenth of a percent low. It looks for a landing every five minutes, which costs a listing of the store for each of today and yesterday when none has happened; the figure itself cannot exist until the `main` run for a landed commit has finished and the relay has stored its coverage measurements | `GH_TOKEN` (or `GITHUB_TOKEN`) | | production | a direct synthetic HTTP check of the public commonfabric.com site, synthetic HTTP checks of `/_health` on estuary and rapids, plus a name or reachability check for all three and for the bastion, the production and staging shells, the LLM gateway, and the sandbox service. When every host is well the headline counts them up. When a host has nothing behind it at all, the headline names that host, as in `bastion down`, and counts them when there is more than one, as in `2 hosts down`. Otherwise it names the worst condition seen, such as a response time or an HTTP status. Estuary and rapids keep their response times in the body while the tile is green or orange. Commonfabric.com stays out of the body while it is good. Hosts without a health request stay out for as long as they answer, and a red tile drops all the green hosts. Red means the tile found nothing at the other end — a name with no A or AAAA record, a tailnet host the proxy cannot reach, or an HTTP request that never connected — and it also means a server health response other than 200, a health response over 1000 ms, or a commonfabric.com 5xx response. Orange means a health response over 500 ms, a commonfabric.com 4xx response or response over 2500 ms, or a resolver that failed, which leaves the tile unable to say either way. Hosts outside the tailnet are looked up by the dashboard itself. Tailnet hosts go through `PROD_PROXY`, because a dashboard that needs that proxy has no view of Tailscale's MagicDNS. Estuary and rapids are covered there by their health requests. The bastion has no health endpoint, so it gets a SOCKS5 connect that leaves the name for the proxy to resolve. The bastion records that connect in its own logs, so a bastion that answers is left alone for an hour and counts as reachable in between. One that does not answer is asked again on the next refresh, since a connect that reaches nothing leaves nothing behind. With no `PROD_PROXY` set, every host is looked up locally | optional `COMMON_FABRIC_URL`, `ESTUARY_URL`, `RAPIDS_URL`, `BASTION_HOST`, `PROD_PROXY`; `PROD_URL` remains an alias for `ESTUARY_URL` | | prod errors | SigNoz trace error rate for one service (errored spans / all spans): last-12h headline, with a per-hour sparkline over the retained trace history (~2 weeks) and the last-12h slice that feeds the headline highlighted. Scoped to `PROD_SERVICE` — the same SigNoz holds staging and one-off perf runs, whose rates are not production's. Gray (not red) when SigNoz is unreachable. Pops out to the SigNoz logs explorer | `SIGNOZ_URL`, `SIGNOZ_API_KEY`; optional `PROD_SERVICE`, `SIGNOZ_UI_URL` for the pop-out | | cloud spend | BigQuery billing export, after credits, projected to month-end from the available part of a 14-day daily-cost window early in the month. The header shows actual MTD spend. The highlighted part of the 45-day chart shows the days used for the estimate. Its vertical scale uses the highlighted days, so older extremes can extend outside the chart | `GCP_BILLING_TABLE` (+ Workload Identity, or `GCP_SA_KEY` locally), optional `GCP_DAILY_BUDGET` | | github spend | the selected organization's or enterprise's whole metered GitHub bill, projected to month-end in USD: every product its billing report carries, added into one figure. Enterprise collection includes usage without a cost center and usage assigned to every cost center. The 45-day chart labels the line with MTD spend, and the header shows the same total. When two or more days are behind the projection, the chart highlights them and its vertical scale uses those days alone, so older extremes can extend outside the chart. The report carries rows for days GitHub is still billing, today included, so the chart and the rate behind the projection stop two days back, once GitHub has had time to finish those days; a partial day at the end of the chart would read as spend falling away. A report that stopped being written more than four days ago is unavailable rather than a run of $0 days. A month whose report cannot be read breaks the line across those days rather than charting them as $0. "What the GitHub figure covers" below says which spend reaches the API | `GH_BILLING_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`; optional `GH_BILLING_ENTERPRISE`, `GH_BILLING_ORG` | | all benchmarks | a scale-invariant index of benchmark performance on `benchmarks.yml` main runs, trended over ~45 days (each run vs the last, geometric mean of per-benchmark changes, so every benchmark weighs the same, divided by the same run's machine calibration so a busy host does not read as a code change): red when the most recent run failed or produced no valid data (the main signal), with a `failed (was )` headline when cached measurements are available and `failed` otherwise; orange only on a broad across-the-board rise from a CPU measured in the preceding twelve hours. Adding or removing a benchmark is a non-event. Drills through to the per-benchmark history | `GH_TOKEN` | | key benchmarks | the same index and status rules as all benchmarks, restricted to `topic board/journey` and `topic board scale/100`. Machine calibration still uses the run's calibration measurements. Counts and data availability refer to the selected benchmarks. Opens the per-benchmark history with "key only" checked | `GH_TOKEN` | | performance history → `/bench?view=runtime` | runtime benchmark trends, labs or loom CI duration history, and a detailed CI run Gantt. Historical views support windows from 1 through 45 days, date axes, and duration sorting. CI includes end-to-end workflow time, every job, and slowest-shard group lines | `GH_TOKEN` | | model spend | OpenAI + Anthropic + OpenRouter usage APIs. Headline is the projected full-month spend (extrapolated from the recent daily rate, spilling into last month when this month is under two weeks old), summed across providers. OpenAI and Anthropic (which expose per-day cost) are charted as one line each over ~45 days, with a recent daily-rate slice highlighted and each line's MTD in the right gutter; OpenRouter (monthly total only, abbreviated "OR") is folded into the totals. The vertical scale uses the highlighted days, so older extremes can extend outside the chart. The subtitle is the bullet-separated key (`OpenAI • Anthropic • OR $0`); the combined MTD sits in the header (the `aside` slot); the span the chart covers is in its bottom-left corner (the `duration` slot). A provider we can't read shows `$???` and drops the tile to gray, but the rest still chart and total; a provider whose cost report stopped being written more than four days ago is one of those | any of `OPENAI_ADMIN_KEY`, the `ANTHROPIC_FEDERATION_*` settings (or `ANTHROPIC_ADMIN_KEY` locally), `OPENROUTER_KEY`; optional `MODEL_MONTHLY_BUDGET` | | discord online | Discord gateway presence, team vs visitors over time | `DISCORD_BOT_TOKEN`, `DISCORD_GUILD_ID` (Server Members + Presence intents) | | dau | distinct identities active per UTC day on one named service, counted from the `user.did` attribute on the `memory.transact` and `memory.subscriber.sync` spans in SigNoz. The headline is the last day that ran to the end (today is still filling, and a part-day always reads as a drop); the sparkline is the retained history. Gray while the named service has no such spans — which is the resting state until a deployment's tracing is switched on. It counts keypairs rather than people; see [dau](#dau) below | `SIGNOZ_URL`, `SIGNOZ_API_KEY`; optional `PROD_SERVICE`, `DAU_EXCLUDE_DIDS`, `SIGNOZ_UI_URL` | | github users | organization members plus outside collaborators, with each roster's size charted over about two months. The headline counts unique users across both rosters | `GH_TOKEN` (with org Members read) | The **production** tile starts gray and says `waiting for connectivity` until its direct **commonfabric.com** check receives an HTTP response. It performs no other host checks before that signal. The confirmation lasts for the process lifetime, so an unreachable host is then reported as an outage even when every host is unreachable together. GitHub concludes a workflow run as `cancelled` for several reasons, among them a concurrency group replacing the run, or one of its jobs, with a newer one, whether it is still queued or already running, a job running past its `timeout-minutes`, and someone stopping the run while it runs. Of those, only the first says nothing about the commit. A run replaced while queued lists no jobs at all, except a partial rerun (**Re-run failed jobs**), whose listing also carries the jobs it reused from the attempt before. A job replaced by a job-level concurrency group while queued is listed as cancelled with an empty runner name, and its run is concluded `cancelled` even when the run's other jobs passed. A job a runner was running when it was cancelled carries an annotation on its check run that says why: a concurrency group's starts `Canceling since a higher priority waiting request for`, a timeout's says the job exceeded its maximum execution time, and a person's says who cancelled the run. The ci and ci trust tiles therefore pass over a cancelled attempt in which every job a runner ran either succeeded or was cancelled by a concurrency group. A cancelled attempt with a job a runner ran that ended any other way, a job that timed out or that a person stopped among them, counts as a failure, as every conclusion other than success does. The decision rests on the annotations rather than on when the attempt started and ended relative to other runs, so a run that times out while a newer run waits for it still fails, and so does a rerun of an older run. The run listing does not carry the jobs, nor a job its annotations, so each tile requests the job listing of every cancelled attempt it reaches, up to the page holding the first job a runner ran that did not succeed for a reason of its own, and the annotations of each job a runner ran that was cancelled, up to and including that one, page by page until one gives the concurrency group as the reason. Each answer is requested once and held while the run stays in the tile's window. No other conclusion takes those requests. The **ci** tile reads each workflow's runs on the default branch, of any status, newest first, and decides the job from the newest completed one carrying a verdict, however many runs after it carry none, among its newest thousand runs of any branch. A run still going carries none. A run concluded `success` passes; a run concluded `failure`, `timed_out`, or `startup_failure` fails. A `cancelled` run is read through the same job listing and annotations the ci trust tiles use, so a run a concurrency group replaced, while it was queued or while its jobs ran, passes no judgment, while one killed by its own `timeout-minutes`, or stopped by a person, counts as a failure. On gvisor's release workflow, which cancels in progress and takes over half an hour, a run replaced while it ran therefore passes no judgment instead of reading as a failure for as long as the newer run took. The remaining conclusions — `skipped`, `neutral`, `stale`, `action_required` — pass no judgment either, so the run before them decides the job instead. A pass therefore stays a pass until a run gives the job another verdict: a job gated off with an `if:` that is never true on the default branch goes on reading green behind every skipped run, and so does a job whose runs are all still going. The runs are read through the run-list reader, newest first, down to the first run on the default branch, started by something other than a pull request, that concluded `success` or failed outright. That run is read again by its id when the reading did not read it from the list, so a run started again since it was held is judged as it stands now, and while it runs again the run before it decides the job. The reader holds each workflow's runs between sweeps, so a verdict far back costs its pages once, when the tile first reads the workflow, and after that usually one page and the list filtered to the default branch on each sweep, and one read of the deciding run when it lies below that page. The reading stops after a thousand runs of any branch or event, so a workflow with no verdict among its newest thousand runs has none. A job with no run carrying a verdict is one the tile cannot speak for, so it is left out of both the headline and the job count in the header. A workflow that only ever runs on pull requests has no run on the default branch at all, and is one of these. Filtering to the repository's own default branch is what leaves pull request work out, since a pull request's runs carry its own head branch. A fork's pull request can put a branch named `main` in front of a repository whose default branch is `main`, so the tile drops `pull_request` and `pull_request_target` runs by their event as well. The set of repositories and the workflows in them is read once an hour and the results behind it every five minutes, because the inventory changes far more slowly than a job's result does. A delivered run of one of the jobs starts a sweep sooner, and while the relay vouches for the runs held a sweep reads only the workflows that have new runs (see [Following the GitHub webhook relay](#following-the-github-webhook-relay)). Reading it costs one request for the organization's repository listing, one per repository for its workflows, and usually one per active workflow for that workflow's newest runs. A sweep that fails is tried again when the next one is due, five minutes later, and the tile is gray until one succeeds. Until the first sweep finishes the tile is gray and says it is reading every repository. After that each collection shows the last sweep that finished while the next one runs. When a sweep ends, the tile asks the scheduler to collect it again straight away, so what the sweep found is on the board within moments rather than at the next snapshot. The labs and loom main builds are the exception. Their runs come from the snapshots the ci trust tiles read, so the tile judges them again each time one of those snapshots arrives, which is every thirty seconds, and publishes the result with the trust tiles. The request that asks whether a failing job's workflow file has changed is made at most once per failing run in each sweep, however often the main builds are judged, and while the relay vouches that no default branch has been pushed to, at most once per failing run in fifteen minutes. The tile keeps its last collection, and the **CI jobs** page renders it rather than collecting again, so the page costs no requests however often it is opened and never disagrees with the tile above it. Its heading gives the time of the last sweep, which is the oldest part of what it shows. A failure is red for `CI_FAILURE_FRESH_HOURS` and orange after that. Both are failures, both are named and counted as failing, and both are listed; the color is about how much of a person's attention the failure has a claim on. A job that has been red for a week is not what just broke, and leaving it red leaves nothing for what did. A red tile shows only what is failing, so the labs and loom main builds stay in an orange tile's body beside an old failure, which is where a reader can see that the build itself is fine. A failure counts only while the job, as it is configured now, would still produce it. A job that failed for a good reason is sometimes stopped rather than fixed: its schedule taken out, its trigger narrowed so that it no longer runs on the default branch, or its job given an `if:` that is never true there. In the first two cases nothing runs to replace its last result. In the third the workflow goes on running, and each run concludes `skipped`, which judges nothing and so passes over to the failure behind it. Without a rule for it, the tile would report that failure for as long as the workflow file exists. So when a job's deciding run failed, the tile asks for the newest commit on the default branch that touched the workflow's file. If that commit landed after the failing run, the failure was made by a definition that no longer exists, and the job has no verdict: it leaves the headline and the count, and the page lists it among the workflows with no verdict, as "changed since it failed", linked at the failure. A job that still runs gets a verdict again from its next run, so this hides a still-broken job only until that run finishes, which for one triggered by a push is the run the edit itself starts. A read of the file's history that fails leaves the failure standing. Only a failure is cleared this way. A job whose deciding run passed stays green whatever has changed in its file since, because stopping a passing job leaves nothing wrong to report. The check costs one request per failing job per collection, and none for a job that passes. Two cheaper signals do not work. GitHub gives each workflow an `updated_at`, but it does not follow edits to the file: on 2026-09-22 labs' `benchmarks.yml` read `2026-03-12` though its file last changed on 2026-09-17. And a newer run concluded `skipped` does not mean a job was switched off. In labs every one of the hundred newest default-branch runs to conclude that way came from `pull-request-comments.yml`, a workflow that runs after others and skips on most of them, so ending a job's verdict at a skipped run would drop a conditional job's real failure the next time it skipped. The **labs ci trust**, **loom ci trust**, and **weaver ci trust** percentages pass over each cancelled attempt in which every job a runner ran succeeded or was cancelled by a concurrency group. A run is first-try green when exactly one of its attempts is left and it succeeded. It is red when that attempt did not succeed, and red when more than one is left, since the run needed a rerun, whatever the rerun's result. A run with no attempt left is left out of the percentage. The run listing carries only a run's latest attempt, so the tile requests a run's earlier attempts from GitHub in order until the run is decided, each at most once, and holds them while the run stays in the trust window. An earlier attempt, a job listing, or a job's annotations that cannot be read turn the tile gray, keeping its last value, until a later collection reads it. The **labs ci duration**, **loom ci duration**, and **weaver ci duration** tiles use pull request runs that succeeded on their first attempt. Each duration starts when GitHub creates the workflow run and ends when that run finishes, so it includes runner queueing. A run that passed only on a rerun is left out, because its span includes the wait before someone asked for the rerun. A run that ran no job is left out too, and so is one that ran one job and skipped the rest: it did nothing beyond reporting its status, as weaver's run for a draft pull request does. A workflow whose only job ran is counted. Telling those apart takes one request for each run's job listing, which is held while the run stays in the window. A job listing that cannot be read turns the tile gray, keeping its last value, until a later collection reads it. The runs counted are whichever lanes each pull request asked for, so in labs a pull request labelled for the full run takes longer than one running its selected tests, and the median follows the mix of the two. The duration tiles collect every five minutes rather than every thirty seconds, since each pull request snapshot costs its own requests and a median of runs that take several minutes moves slowly. ### Test selection history The flaky tests and test selection tiles share a manifest reader and a cache of counts and test times in `fabric-wall-test-selection.json` under the dashboard cache directory. A cache file missing any of them is rebuilt from the store. Collection reads every page of the public manifest listing and plots every available manifest at its generation time. The listing determines the chart's span; the dashboard imposes no date window. A successful refresh removes cached counts for objects that are no longer listed. A stored body says which shape it was written in, and this dashboard reads anything at or under its own. A body from further ahead it passes over, taking the newest one behind it, across the same stretch of the store a lane looks back over. Where it can read none of them it says so and names the shape it found, rather than reporting the store empty. Anything else unreadable ends the search where it stands, because a corrupt object is not a reader waiting to be deployed. Each object passed over is read once and then left alone for as long as the process runs, so neither the walk nor a refresh fetches it again. That is held in memory rather than in the cache file, because what a reader can validate belongs to the build holding it and a later build may accept what this one passed over. Every other failed read is tried again on the next refresh. An empty corpus or an unreadable manifest leaves a gap. Connected stretches share one vertical scale, and an isolated measurement appears as a dot. A measured zero remains a measurement. An unreadable historical object adds a `history warning` badge; the next scheduled collection can fill that gap. The latest full manifest supplies the headlines and detail page. Historical collection retains only counts and test times and downloads at most three inventories at a time. Cached counts survive dashboard restarts. Cache failures add a warning without hiding readable measurements. The headline is published as soon as the latest manifest is readable, before historical collection finishes. Both tiles refresh their measurements and activity independently every 30 seconds. With a GitHub credential configured, a **running** badge lights while the Test Selection workflow on main is queued or running, including reruns of older workflow runs. The tiles read the workflow's runs through the run-list reader, back to the oldest run GitHub would still let someone start again. A run below the top of the list that is started again is found through the lists of unfinished runs on main, which GitHub answers from an index that can be days behind, and counts once a read of the run itself says it has not finished. The tiles share the workflow lookup and keep the last known badge while that lookup is pending. A failed lookup shows **activity unknown** alongside the available measurements. Public manifest reads work without GitHub credentials. Each refresh checks for newly published manifests; unchanged objects reuse the cached inventory or counts. ### What the GitHub figure covers The **github spend** tile reads the billing usage report for one account. It uses the organization named by `GH_BILLING_ORG` by default. Setting `GH_BILLING_ENTERPRISE` selects that enterprise account instead. Enterprise collection uses GitHub's summary report across all cost centers, so its total does not silently omit usage allocated to one. The organization report carries rows by product, SKU, repository, and day; enterprise collection reads daily summary usage items across all cost centers. The tile adds every returned usage item, so the figure is the account's whole metered GitHub bill rather than any one product's share of it. If a current-month enterprise day cannot be read, the projection omits that day from its rate, the chart breaks across it, and the tile turns gray and labels its month-to-date total as partial. An unavailable historical day likewise leaves a chart gap and stays out of the rate, without making the current total partial. GitHub meters these products, and a product the account does not use simply has no row: | product | what it bills for | |---|---| | `actions` | workflow minutes on every runner size, plus Actions and cache storage | | `packages` | Packages storage and bandwidth | | `codespaces` | Codespaces compute, storage, and prebuild storage | | `git_lfs` | Git LFS storage and bandwidth | | `copilot` | Copilot seats (`copilot_for_business`, `copilot_enterprise`, `copilot_standalone`) and the AI credits Copilot consumes | | `ghec` | GitHub Enterprise Cloud seat licenses (`ghec_licenses`) | | `ghas` | Advanced Security seat licenses, whole and by tier | | `models` | model inference | | `sandbox` | Copilot sandbox compute, memory, and snapshots | | `spark` | Spark AI credits | Seat licenses arrive as billed dollars in that same report, so Copilot seats and Enterprise Cloud licenses are inside the figure without any per-seat price having to be configured here. One caveat applies to the projection rather than to the figure. GitHub dates every row by the day the usage occurred, and the month-end projection carries the recent daily rate across the remaining days, which suits spend that accrues daily. A charge posted as a single lump instead of as a daily series would be carried across the month as though it recurred, overstating the projection while leaving month-to-date correct. No such charge has been observed here, and seat licenses are the products to watch for one: compare the headline with the month-to-date total beside it the first month a seat-licensed product appears. The budget comparison uses only product budgets at the same scope as the usage report. Organization usage is compared with organization-scoped budgets. Enterprise usage is compared with enterprise-scoped budgets. Organization, repository, user, and cost-center budgets returned by the enterprise endpoint are narrower ceilings and are not added to the enterprise budget. What the API does not expose, and the figure therefore excludes: - **A subscription billed outside the usage report.** An organization's plan reaches the API only when its licensing is metered, as `ghec_licenses` rows. An organization billed for its plan as a flat subscription — a **Team** plan, which has no product or SKU of its own at all, or an **Enterprise** plan whose licensing has not moved to metered billing — has no row for it, and its seat cost is absent from the figure. `orgs/{org}` reports the seat counts (`plan.name`, `plan.seats`, `plan.filled_seats`) but never a price, so there is nothing to add up from. This gap is silent: the figure is a true total of what GitHub reports, not a total of what GitHub charges. Check it against the billing page the tile links to before treating it as the whole bill. ## Credentials Every tile that reads a private source is gated on its own env var(s) and grays out until they are set. The GitHub tiles use `GH_TOKEN`, except that **github spend** can use `GH_BILLING_TOKEN` instead, and a GitHub App can stand in for both; every other private-source tile is independently optional — set only the ones you want, and the rest stay gray without breaking the board. Each key below lists what it powers, the rights it needs, and how to mint it. (`commonfabric.com` and `production` need no key.) Almost every credential is shown only once at creation — copy it immediately; if you lose it you have to regenerate. ### `GH_APP_CLIENT_ID` + `GH_APP_PRIVATE_KEY` Configure a GitHub App, and every GitHub tile authenticates as one of its installations, with no person's token involved. The app takes precedence over `GH_TOKEN` and `GITHUB_TOKEN`. `GH_BILLING_TOKEN` still takes precedence over the app for **github spend**, for an account the app cannot read. With only one of the two variables set, every GitHub tile that would use the app is gray and names the one that is missing. The dashboard picks the installation by the account each request is about. The repository and organization reads use the installation on the owner of `DASHBOARD_REPO`. **github spend** uses the installation on the enterprise named by `GH_BILLING_ENTERPRISE`. GitHub refuses an app's tokens for an organization's billing, so with `GH_BILLING_ENTERPRISE` unset, **github spend** reads the organization's billing with `GH_BILLING_TOKEN`, `GH_TOKEN`, or `GITHUB_TOKEN`, as it does without an app. It finds each installation among the app's own on first use, and mints a token for it. A token is used until it is five minutes from expiring, and the request after that mints the next one, so a collection that runs longer than a token lasts carries on without a restart. A token GitHub refuses, and an installation that cannot mint one, are dropped, and the next request looks the installation up again, so reinstalling the app needs no restart either. GitHub counts an installation's requests separately from any person's. An installation in a GitHub Enterprise Cloud organization is allowed 15,000 requests an hour, where a personal access token is allowed 5,000. 1. Create the app under the enterprise, at `https://github.com/enterprises//settings/apps` → **New GitHub App**. An app the enterprise owns can be installed only on the enterprise and its organizations, which are the two installations the dashboard uses. Give it any homepage URL and clear **Webhook** → **Active**. 2. **Repository permissions**: **Actions**, **Checks**, **Contents**, and **Pull requests**, all **Read-only**. **Metadata** becomes read-only automatically. **Checks** is for the check run annotations that say why a job was cancelled, which the **ci** and **ci trust** tiles read. **Pull requests** is for **recent commits**. 3. **Organization permissions**: **Members** **Read-only** for **github users**. GitHub refuses an app's tokens for an organization's billing ("Resource not accessible by integration"), so **github spend** reads through the app only with `GH_BILLING_ENTERPRISE` set. For an organization's billing, set `GH_BILLING_TOKEN` as well. 4. **Enterprise permissions**: **Enterprise billing** **Read-only**, for **github spend** with `GH_BILLING_ENTERPRISE` set. 5. **Create GitHub App**. Copy the **Client ID** (`Iv23…`) into `GH_APP_CLIENT_ID`. Under **Private keys**, **Generate a private key**; the downloaded `.pem` file's whole contents, PKCS#1 as GitHub issues it, go in `GH_APP_PRIVATE_KEY`. PKCS#8 works as well. 6. **Install App** on the organization, with **All repositories** so the **ci** and **recent commits** tiles cover the whole organization. For enterprise billing, install it on the enterprise too. An enterprise installation receives only the app's enterprise permissions, and the organization installation only its repository and organization permissions, which is why the dashboard keeps one per account. ### `GH_TOKEN` (or `GITHUB_TOKEN`) Powers **ci**, **labs ci trust**, **labs ci duration**, the **loom** and **weaver** counterparts, **recent commits**, **github users**, and **github spend** when `GH_BILLING_TOKEN` is unset. It also powers the optional publisher-activity indicators on **flaky tests** and **test selection**; their public measurements need no token. Needs repo **Actions: read** on `commonfabric/labs`, `commonfabric/loom`, and `commonfabric/commonfabric-weaver`. The **ci** and **ci trust** tiles also read the check run annotations that say why a job was cancelled, which in a private repository takes the `repo` scope of a classic token; GitHub offers fine-grained tokens no permission for check runs. With a fine-grained token, those tiles report a private repository's run that had a job cancelled on a runner as unreadable. When `GH_BILLING_ENTERPRISE` is unset, the github-ci-spend tile additionally needs org **Administration: read** on `commonfabric`. When it is set, a fallback `GH_TOKEN` instead needs one of the enterprise credentials described under `GH_BILLING_TOKEN`; a separate `GH_BILLING_TOKEN` is simpler. The **github users** tile needs org **Members: read**. The **ci** and **recent commits** tiles cover every repository the token can see, so a token selecting only some repositories leaves the rest out of those tiles without saying so: give it **All repositories** for them to cover the organization. One fine-grained token can carry every one of these permissions except check run access: The account that owns the token must be a member of the organization. GitHub's member endpoint returns both concealed and public members to an authenticated organization member; other callers see only public memberships. 1. GitHub → your avatar → **Settings** → **Developer settings** → **Personal access tokens** → **Fine-grained tokens** → **Generate new token**. 2. Set **Resource owner** to the **commonfabric** organization (not your personal account) — org ownership is what unlocks the billing permission. 3. **Repository access** → **All repositories**, which is what lets the **ci** and **recent commits** tiles see the whole organization. **Only select repositories** with `commonfabric/labs`, `commonfabric/loom`, and `commonfabric/commonfabric-weaver` covers every other GitHub tile, and narrows **ci** and **recent commits** to those three. 4. **Repository permissions**: set **Actions**, **Contents**, and **Pull requests** to **Read-only**. **Pull requests** is for **recent commits**. 5. **Organization permissions**: set **Members** to **Read-only** for GitHub users. Set **Administration** to **Read-only** for github spend. Only an org owner or billing manager can grant the latter permission. Skip either permission when its tile is not needed. 6. **Generate token** and copy it (`github_pat_…`, shown once). If you only need the labs CI tiles, keep `commonfabric` as the resource owner, select only `commonfabric/labs`, and grant Actions and Contents read without any organization permissions. Classic PATs also work, check run annotations included (use `repo` for private repositories, `read:org` for GitHub users, and `admin:org` for github spend). If the org requires approval for fine-grained tokens, yours stays pending until an owner approves it. ### `GH_BILLING_TOKEN` Optionally powers **github spend** instead of `GH_TOKEN`. Use a separate token when billing access belongs at a broader scope than the repository and organization permissions used by the other GitHub tiles. For organization collection, use a fine-grained personal access token with that organization selected as its resource owner and organization **Administration: read**. For enterprise collection, set `GH_BILLING_ENTERPRISE` and use one of: - A classic personal access token with `manage_billing:enterprise`, owned by an enterprise owner or billing manager. GitHub does not offer a read-only classic scope for enterprise billing. - A GitHub App with enterprise **Enterprise billing: read**, installed on the enterprise. This is the least-privilege enterprise option. Configure it with `GH_APP_CLIENT_ID` and `GH_APP_PRIVATE_KEY` rather than here, and the dashboard renews its tokens itself. Enterprise billing endpoints do not accept fine-grained personal access tokens. The tile reads the enterprise-wide usage summary and budgets, so the credential needs access to both. ### `SIGNOZ_URL` + `SIGNOZ_API_KEY` Powers **prod errors**. Needs a read-only (Viewer) key on the SigNoz query API; creating one requires an Admin. 1. In SigNoz (Cloud at `app.signoz.cloud`, or your self-hosted URL) → **Settings** → **Service Accounts** → **New Service Account** (e.g. `dashboard-reader`). 2. Assign it the **signoz-viewer** role (read-only query access). 3. Open the service account → **Keys** tab → **Add Key**, name it, optionally set an expiry → **Create**, and copy the key (opaque string, shown once). It's sent as the `SIGNOZ-API-KEY` header. 4. Set `SIGNOZ_URL` to the instance base URL — `https://.app.signoz.cloud` for Cloud (`us`, `eu`, …), or your self-hosted host (default UI port `8080`). Don't use an **ingestion** key (`signoz-ingestion-key`, write-only for telemetry); the tile needs the read/query API key. ### `GCP_BILLING_TABLE` (+ Workload Identity, or `GCP_SA_KEY`) Powers **cloud spend**. The tile queries BigQuery over its REST API — no `bq` or `gcloud` CLI. There's no API key for BigQuery (a key doesn't identify a principal), so it authenticates as a service account and gets an access token one of two ways: in GKE the metadata server hands one out for the pod's own account (Workload Identity, no key stored); locally, set `GCP_SA_KEY` to a service-account key JSON and the tile signs a JWT and exchanges it for a token. The account needs **BigQuery Job User** on the query project plus **BigQuery Data Viewer** on the dataset. 1. Console → **Billing** → select the billing account → **Billing export**. 2. Create a BigQuery **dataset** to hold the export (a US or EU multi-region location lets it backfill). 3. Under **Billing export** → **Standard usage cost**, choose the project and dataset → **Save**. Data starts landing after a few hours. 4. The export table is named `gcp_billing_export_v1_`; set `GCP_BILLING_TABLE` to `project.dataset.gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX`. The query runs in that table's project, so the service account needs Job User there. 5. In the GKE deploy this is already provisioned: the infra repo's `tofu/gke/dashboard.tf` creates the service account, binds the pod's Kubernetes service account to it with Workload Identity, and grants **roles/bigquery.jobUser** on the project plus **roles/bigquery.dataViewer** on the export dataset (point `dashboard_billing_dataset` at that dataset). Enabling the tile is then just setting `GCP_BILLING_TABLE`. 6. For local development instead, grant those two roles to a service account, download a key for it, and set `GCP_SA_KEY` to the file's contents. The tile covers every project tied to the exported billing account. Its figures are money that account actually pays: the export's `cost` column plus the credits the same rows carry, which are negative amounts. Every kind of credit counts — promotional credits, committed- and sustained-use discounts, and the free tier — because none of them is money the account pays. The credit is not a figure the tile reports on its own; it is simply absent from the spend, in the same way that the usage GitHub's plan includes is absent from the ci-spend figures. It shows the estimated full-month spend as its headline and the actual month-to-date value in the header. The estimate uses every settled day in the current month. During the first half of the month, it fills that rate window from the prior month's tail until it covers 14 days or reaches the first available billing day. The chart shows up to 45 finished UTC days and highlights the part used for the estimate. Which days are finished matters, because the export lands a day's usage in batches spread over the day or two after it ends, and works on several days at once. A day it has not finished holds only part of that day's cost. Such a day belongs in the month-to-date total, where it is a running figure, and not in the rate behind the projection or at the end of the chart, where it would read as spend falling away. Nothing in the export marks a day as complete, so the tile establishes it two ways at once, and a day has to satisfy both. The first is how much of the day the export has accounted for. Alongside cost, the query totals the billable time on each day's usage — the seconds of instance, disk, and other metered life the export has recorded. A running fleet books close to the same amount every day, and unlike cost it does not move when a promotion or a price change lands, so a day holding less than half of the day before it is one the export is still filling in. A fleet that really does shrink trips this for the day it shrinks on, and then the days after it stand against the new level and the window moves on. The second is how long the export has had. It records when it last added a material batch to each day, which the tile turns into how long the export is currently taking to finish with a day. That figure is measured from the export's own record across the window rather than assumed here, so it follows an export that speeds up or slows down. A day is finished once the export has had that long since the day ended. The newest day in the window is never finished, because the export has written nothing after it to show it has moved past it. Each check covers where the other is blind. Billable time alone would accept a day the export happens to be most of the way through; elapsed time alone would accept a day the export has fallen behind on. When no day passes both for more than four days the tile goes gray and says how far behind it is, rather than projecting a month from stale history. Both checks read only the export's `regular` rows. Invoice adjustments and rounding corrections arrive weeks after the fact and carry no usage, so counting them would make a long-closed day look freshly written. A promotional credit is finite, so the spend the tile reports rises when a grant runs out even though nothing about the usage changed. How much of a grant is left is not in the billing export: Google publishes it only in the console's billing credits page. A grant may also arrive in monthly tranches that land on a few days rather than spreading across the month. The rate window is a fortnight, so it either covers those days or it does not, and the projection steps up or down as they pass out of it. The month-to-date figure in the header carries whatever has landed so far and is the one to read when a tranche is due but has not arrived. ### `OPENAI_ADMIN_KEY` Powers the OpenAI share of **model spend**. Needs an organization **Admin** key — only an org **Owner** can create one, and a project key (`sk-proj-…`) gets a 401 on the costs endpoint. 1. Go to **platform.openai.com → Settings → Organization → Admin keys**. 2. **Create new admin key**, name it, **Create**. 3. Copy it (`sk-admin-…`, distinct from `sk-proj-…`, shown once). Treat it like a root credential — it grants full org management. ### Anthropic: `ANTHROPIC_FEDERATION_*`, or `ANTHROPIC_ADMIN_KEY` locally Powers the Anthropic share of **model spend**, which reads the organization cost report from the Admin API. In the cluster the dashboard holds no Anthropic credential. It uses Workload Identity Federation: the GKE metadata server signs a Google identity token for the pod's service account, and Anthropic exchanges it for a bearer token that expires in minutes ([anthropic-auth.ts](./anthropic-auth.ts)). A copy of that token taken out of the pod stops working when it expires, and a new one can only be minted as the dashboard's Google service account. The Admin API has no read-only scope. The federated token carries `org:admin`, the same reach as an Admin key, for as long as it lives. Federation limits where the credential comes from and how long it lasts, not what it can do. Anyone who can mint a Google identity token as the dashboard's service account can also get one. That covers the pod itself, and any project member who can act as the account: owners, and holders of a project-wide Service Account User role, who can attach it to a VM and ask that VM's metadata server. The pod reads these settings, which are identifiers, not secrets: | env var | value | |---|---| | `ANTHROPIC_FEDERATION_RULE_ID` | the federation rule, `fdrl_…` | | `ANTHROPIC_ORGANIZATION_ID` | the organization UUID, from **Settings → Organization** | | `ANTHROPIC_SERVICE_ACCOUNT_ID` | the rule's target service account, `svac_…` | | `ANTHROPIC_WORKSPACE_ID` | optional; only when the rule is enabled in more than one workspace | Setting any of the three required settings selects federation, even if `ANTHROPIC_ADMIN_KEY` is also set, so a leftover key cannot mask a broken or half-finished federation setup. A failed exchange or a missing setting shows `Anthropic $???` and grays the tile, and the server log names the cause. To set it up, an organization admin creates the rule in the Claude Console. Anthropic allows `org:admin` rules to be created only there, not through the API. [Anthropic instructions](https://platform.claude.com/docs/en/manage-claude/wif-providers/gcp) 1. Look up the numeric unique ID of the dashboard's Google service account: ```zsh gcloud iam service-accounts describe \ dev-dashboard-stage@commontools-core.iam.gserviceaccount.com \ --project=commontools-core \ --format='value(uniqueId)' ``` 2. Open [Claude Console → Settings → Workload identity](https://platform.claude.com/settings/workload-identity-federation), select **Connect workload**, and choose **Google Cloud**. Enter: - issuer URL `https://accounts.google.com`, with JWKS discovery - audience `https://api.anthropic.com` - claim `sub` equal to the unique ID from step 1 - claim `email` equal to `dev-dashboard-stage@commontools-core.iam.gserviceaccount.com` - names such as `dev-dashboard-stage` for the service account and rule - under **Advanced rule options**, OAuth scope `org:admin`, which makes the new service account an Admin - token lifetime `600` seconds Do not use a subject prefix. Google subjects have no stable prefix, so a wildcard would admit service accounts in other projects. 3. Copy the rule ID, the service account ID, and the organization ID into the stage overlay's `dev-dashboard-anthropic-wif` ConfigMap in the infra repository, land that change, and deploy it. 4. The wizard waits 15 minutes for a successful exchange. The tile exchanges on its first collection after the pod starts. If the wizard window has passed, re-run the test from the rule's page, or check the [authentication history](https://platform.claude.com/settings/workload-identity-federation?tab=history) for the denial reason. Every denial is the same opaque 401 from the API. 5. Once the tile reads Anthropic again, delete any old `dashboard-reader` Admin key under **Settings → Admin keys**. The infra repository's dashboard deploy removes the key's in-cluster copy, and dropping its Secret Manager container from OpenTofu removes the stored copy. To revoke the dashboard's access, archive the rule in the Console. For local development, set `ANTHROPIC_ADMIN_KEY` to an Admin key (`sk-ant-admin01-…`) from **Settings → Admin keys** instead. It is sent as `x-api-key`. Give it a short expiration and delete it when you are done, because it carries full Admin API access. ### `OPENROUTER_KEY` Powers the OpenRouter share of **model spend**. A plain inference key (`sk-or-…`) is enough — the tile only reads the key's own month-to-date usage via `GET /api/v1/key`, so no admin/management key is needed. 1. **openrouter.ai → Settings → Keys** → **Create API Key**, name it, **Create**. 2. Copy the `sk-or-…` key (shown once). Use a normal key, not a **Management** key — management keys can't call `/api/v1/key`. ### `DISCORD_BOT_TOKEN` + `DISCORD_GUILD_ID` Powers **discord online**. Needs a bot with the **Server Members** and **Presence** privileged intents, invited to the server, plus the server id. The guild must contain a role named exactly `Team` or `Team Member`; without either, the tile remains unknown. The names are equivalent. Online members with either role count as team; every other online member counts as a visitor. 1. **discord.com/developers/applications → New Application**, name it, **Create**. 2. **Bot** (left sidebar) → **Reset Token** → copy the token (shown once). 3. Still on the Bot page → **Privileged Gateway Intents** → toggle **Server Members Intent** and **Presence Intent** ON → **Save Changes**. Without both, the gateway closes the connection with error `4014` (disallowed intents). 4. Under **Installation** (or OAuth2 → URL Generator), select the **`bot`** scope, copy the install link, open it, pick your server, **Authorize**. 5. In Discord: **User Settings → Advanced → Developer Mode** ON, then right-click the server → **Copy Server ID** → that's `DISCORD_GUILD_ID` (an 18–20 digit number). Resetting the token invalidates the previous one, breaking anything still using it. **Optional, non-secret knobs** (no key; they tune behavior): | env var | tile | purpose | |---|---|---| | `GH_BILLING_ENTERPRISE` | github spend | enterprise slug for enterprise-scoped usage and budgets. When set, it takes precedence over `GH_BILLING_ORG`. | | `GH_BILLING_ORG` | github spend | organization login for billing when `GH_BILLING_ENTERPRISE` is unset (default: the org from `DASHBOARD_REPO` — `commonfabric`). | | `MODEL_MONTHLY_BUDGET` | model spend | combined monthly USD budget across providers. | | `GCP_SA_KEY` | cloud spend | a service-account key JSON (the whole file, as the value) for local development; in GKE, Workload Identity supplies the token and this is unset. | | `GCP_DAILY_BUDGET` | cloud spend | daily USD budget. The projected month is compared with this daily rate multiplied by the number of days in the month. | | `ESTUARY_URL` | production | the estuary server as an origin. The tile checks `/_health` on it and links to it. Defaults to `https://estuary.saga-castor.ts.net`. `PROD_URL` remains an alias when `ESTUARY_URL` is unset. | | `RAPIDS_URL` | production | the rapids server as an origin. The tile checks `/_health` on it and links to it. Defaults to `https://rapids.saga-castor.ts.net`. | | `BASTION_HOST` | production | the deployment bastion's hostname. A URL is also accepted; its hostname is used, along with its port when it carries one, which otherwise is 22. A tailnet name is checked hourly by connecting through `PROD_PROXY`, and any other name by an A and AAAA lookup on every refresh. Defaults to `bastion.saga-castor.ts.net`. | | `PROD_PROXY` | production | optional proxy for reaching tailnet hosts. Use `socks5h://127.0.0.1:1055` with the Tailscale userspace proxy. Also accepts `socks5://`, `http://`, and `https://`; invalid values and URLs containing credentials fail closed instead of fetching directly. Setting it also moves the tailnet name checks onto the proxy, since a dashboard that needs a proxy cannot resolve MagicDNS names itself. The bastion check needs a SOCKS5 proxy to do that, and stays gray over an `http://` or `https://` one. | | `COMMON_FABRIC_URL` | production | override the public-site URL (e.g. the `www` host if the apex redirects). | | `DASHBOARD_REPO` | CI tiles, github users | which repo the CI tiles read. Its owner is the organization the **github users** tile reads (default `commonfabric/labs`). | | `GITHUB_WEBHOOK_RELAY` | GitHub tiles | the `host:port` of the GitHub webhook relay's subscriber listener (default `github-webhooks.stage.commonfabric.dev:8081`). An empty value turns the relay off, and the dashboard then reads GitHub on its intervals alone. See [Following the GitHub webhook relay](#following-the-github-webhook-relay). | | `DASHBOARD_CACHE_DIR` | server caches | directory for all persistent dashboard cache files (default: the platform temp directory), including the `fabric-wall-run-lists-*.json` files that keep the run-list reader's runs across restarts. | | `SIGNOZ_UI_URL` | prod errors, dau | browser-facing SigNoz URL for the explorer pop-outs: **prod errors** links to `/logs/logs-explorer` and **dau** to `/traces-explorer` under it. Defaults to `SIGNOZ_URL` when that is a public `https://` URL. An in-cluster `http://` URL, which a browser cannot reach, leaves both tiles with no pop-out at all, so set this whenever the server reaches SigNoz over one. | | `PROD_SERVICE` | prod errors, dau | the `service.name` production reports under in SigNoz, which both trace-reading tiles scope to. Defaults to `toolshed-production`. A name outside `[A-Za-z0-9._-]` is ignored, since it lands inside a query expression. | | `DAU_EXCLUDE_DIDS` | dau | comma-separated identity DIDs to leave out of the count — the server's own identity, `MEMORY_SERVICE_DIDS`, background services. Until it is set the count is an upper bound. See [dau](#dau) below. | **What the dau tile counts.** Distinct `user.did` values per UTC day, over the `memory.transact` and `memory.subscriber.sync` spans of one named service, read from the same SigNoz the prod errors tile uses. `user.did` is the memory session principal, which is the signature-checked `session.open` issuer, so no new instrumentation is involved. `docs/features/active-user-counting.md` records what the number means. Four things bound it, and they are worth knowing before quoting it: - **It counts identities, not people.** An identity is a keypair. One person with a browser mnemonic and a `cf id derive` passphrase is two; one key shared across a team is one. The tile says "active identities" for that reason. - **Some identities are not people at all.** The server's own identity and the DIDs in `MEMORY_SERVICE_DIDS` are principals in the same way a user is. Name them in `DAU_EXCLUDE_DIDS`; until then the count is an upper bound. - **Read-only sessions are invisible.** `session.open` emits no span, so someone who connects and only reads is never attributed, and a day of purely read-only traffic reports zero. - **The history is as long as trace retention**, roughly a fortnight, and that retention is a live setting on the SigNoz database rather than anything this repository holds. Head sampling below 1.0 would not scale this number down, it would drop identities out of it, and no arithmetic afterwards would put them back. The tile is gray whenever the named service has no identity-bearing spans in the retained window, which is the resting state for a deployment whose tracing is switched off. It needs no second change to light up once that deployment starts exporting. Notes: - **GitHub billing token:** every GitHub tile uses `GH_TOKEN` except that **github spend** prefers `GH_BILLING_TOKEN` when it is set. A configured GitHub App takes the place of `GH_TOKEN`, through its installation on each account. An organization deployment can keep one token with organization billing read. Enterprise billing needs different credentials, so the separate variable keeps the ordinary GitHub token free of enterprise billing access. The **github users** tile still needs organization Members read on `GH_TOKEN`. - **`github spend`** shows the **projected** full-month GitHub total. Its recent daily rate uses at least two weeks and reaches into last month early in a month. Spend is net of discounts and included usage. The projection is compared with the selected account scope's product budgets added together, which is what it has authorized itself to spend. Only the budgeted products' share of the projection goes into that comparison: a product with no budget of its own is taken to be spending within one, so it counts toward the headline without moving the color. The light therefore turns on the products someone actually set a limit for, and taking up a product nobody has budgeted cannot redden the tile on its own. The budget printed beside the headline covers the same products the headline does — the budgets that exist, plus each unbudgeted product's own projection standing in for the budget it lacks — so the headline sits at or under it exactly when the tile is green. Printing the configured total alone would show a headline above its budget on a green tile. An account with no product budget at all leaves the projection uncompared. A source that has stopped reporting turns the tile gray. The billing report is one pipeline across every product the organization uses, so any product's row dates it. Four days without a row is a stopped feed rather than a slow one. A classic-plan organization falls back to minutes against its included allowance. - **`all benchmarks`** trends one **scale-invariant index per CPU** on the `benchmarks.yml` runs on main over ~45 days. The job runs `deno bench --json` over the bench files that workflow lists — micro-benchmarks across the runner, utils, fuse, and memory packages, plus the topic-board navigation and scaling benchmarks, which drive a browser end to end and so report in seconds where the others report in nanoseconds. It uploads the report as a `bench-results` artifact with 90-day retention. There is no committed history. Each CPU's index compares a run with the previous run on the same CPU. It multiplies the previous index by the **geometric mean of the per-benchmark changes**, then **divides out the machine**. Every benchmark weighs the same regardless of size. **Only a broad, across-the-board move shifts an index.** A regression in one benchmark barely registers, however slow that benchmark is. The drill-down covers individual benchmarks. A summed total would instead be dominated by the few slowest benchmarks. Each CPU has its own colored line. The headline shows the largest established trend among CPUs measured in the preceding twelve hours. A second line names how many benchmarks the latest run measured and the highlighted window when applicable. **Red** marks the **most recent run failing outright, or finishing green on CI with no readable benchmark data**. A successful run with no usable output is treated as failed. Either failure sets the headline to `failed (was )` when cached measurements are available. Its trend window and eligible CPUs are determined as of the latest measurements, even when those measurements are older than twelve hours. With no measurements, the headline is **failed**. The tile links to the benchmark history. The second line describes the failure in place of the count and the window. A run that failed outright dates the outage and counts it: **last good 2 days ago · 12 runs failed**. The count reads back through the newest-first run list. It stops at the first completed run that did not fail, so a cancelled run ends it. The date comes from the newest run that passed, including one that passed on a later attempt, and it is read from the run list the collection paged rather than from the trend's 45-day window. That list runs to the end of the first page past 45 days, so an older outage is dated whenever the run that ended it is still on the last page fetched. With no passing run anywhere on the list there is nothing to date the outage from, and the line is the count alone: **last 12 runs failed**. A run that finished green with nothing readable reads **no benchmark data**. A **running** badge sits in the header while a run is under way, wherever in the list it sits — a rerun keeps its original place instead of moving to the head. The badge says the color may be about to move; the runs that have finished still set it. The red state reads the workflow-run list and the latest run's cached result. It therefore fires when the artifacts cannot be read. **Orange** means at least one CPU measured in the preceding twelve hours has an index **trending up** past 5%. Each CPU trend uses the runs in the last `BENCH_TREND_MAX_AGE_DAYS` or the newest `BENCH_TREND_MIN_RUNS`, whichever set is larger. This matches the window rule used for the CI duration median. The corresponding line still spans the full ~45 days. Its trend window is brighter. Green means every eligible established CPU is flat or falling. When no CPU has been measured in the preceding twelve hours and the tile is not in the failed state, it turns gray and reports **no recent benchmark data**. A benchmark runs either to a fixed time budget or for a fixed number of iterations, and neither is a measurement. The run's wall clock therefore barely moves with performance. The per-operation times do move, so the tile trends those values instead. Because the index comes from artifacts, the tile also turns gray when no in-window run has readable data and the latest run is neither failed nor empty. A collection leaves its last completed color and values in place until the workflow status and artifact refresh have both settled. A collection that takes more than a minute says **refresh still pending** without changing that color. A newly loaded dashboard keeps its neutral placeholder until its first collection settles. A completed fetch whose runs carry no readable data shows **benchmark data unavailable**, and one with no runs at all shows **no benchmark runs**. The tile reads the workflow's runs through the run-list reader and keeps the runs on main itself. Nearly every run of this workflow is on main, so this reads no more pages than a list filtered to main would. Adding or removing a benchmark does not move an index. The benchmark is absent from one side of that adjacent comparison, so it drops out of the geometric mean. When two runs share no selected positive measurements, the whole index step stays unchanged, including machine calibration. Both tiles use the same workflow runs and artifacts when collecting concurrently. A CPU change starts another line instead of connecting measurements from unlike machines. A CPU model is not a machine, though. The runner group has served six processor models over a forty-five day stretch, and two runs on one model have measured a fifth apart on work that touches no repository code at all, because a run gets whatever share of a shared host the other tenants leave it. So the workflow also runs `packages/dashboard/machine-calibration.bench.ts`, whose benchmarks call no repository code, and each step divides their geometric mean out of the product benchmarks' one. What is left is what the repository did. Those calibration benchmarks are the tile's ruler rather than one of the things it measures: they are absent from the index, from the benchmark count, and from the drill-down. A run from before the calibration landed carries none, and the step into or out of it is left uncorrected rather than guessed at. The index also reads **one run per `BENCH_TREND_BUCKET_MS`**, matching the workflow's four-hourly cron, so a rerun or a manual dispatch minutes from a scheduled run does not put two samples of one moment into the fit. The drill-down keeps every run. A gap longer than one fifth of the chart width breaks a CPU's line. Any sample isolated by those breaks appears as a point. A CPU with one sample also appears as a point until another sample can form a line. A large rise reads as a fold multiplier (`▲44×`) once it passes 4x. This avoids a long percentage. The trend is the difference between the start and the end of whichever of **two robust fits** describes the samples more closely, measured on the samples themselves rather than extended past them. The first fit is a run of **flat levels meeting at change points**, which is the shape a series takes when a change lands and shifts it. Levels are medians, so a lone spike moves none of them, and a boundary counts as a change point only when the two levels either side sit more than 4 standard errors of their difference apart, with at least 3 samples supporting each — so noise produces no levels, and one stray sample is not a level. The second fit is a **straight line** through the median of the pairwise log-slopes, which is the shape a series takes when it drifts, and it wins when its total deviation is at least a tenth smaller. Reporting the difference across the fit rather than a slope extended over the window means a shift reads at its true size wherever in the window it sits: a shift in the newest samples is the one worth catching soonest, and a slope through it would report a fraction of it. The samples are first grouped into at most 64 equal-sized runs, each replaced by its median, which bounds the change-point search and gives every group the same weight however unevenly the runs arrive. With fewer than 7 distinct days in the window the trend is marked new: there is too little data to claim one. The window is capped by the 90-day artifact retention, so it shows at most ~45 days and only as far back as the job has run. - The tile collects every minute, which is what the run state needs: a benchmark run lasts about an hour, so a slower collection could miss one from start to finish and never show its **running** badge. Each collection pages the run list. It publishes only the completed view, so a partial refresh cannot temporarily replace the last settled status. The artifact history behind it is left alone while the last refresh is recent and already covers the runs that list names. Nothing new has been sampled, so there is nothing to download. - The tile drills through to the per-benchmark history behind `/bench`, which the tile's collection keeps warm in the background. The collection lists benchmark runs on main. It samples one completed run per shortest-view bucket, whatever color it finished, downloads that artifact, and unzips it in the process. It then reads each benchmark's timings and CPU. A run's color does not say whether it measured anything: `deno bench` exits non-zero when one benchmark throws, having already written a complete report of the rest, and dropping those runs cost about a seventh of the history in blocks a dozen runs long. The artifact decides instead. A report without a CPU identity, or one that will not parse, is cached as unusable instead of being pooled with measurements from unknown machines; because a run attempt's artifact never changes, that verdict is kept rather than retried, while a read that failed outright is retried. Cache entries written before CPU identity was stored are fetched again. Each completed artifact check is persisted before it is counted as finished. Only new runs and attempts are fetched after the first fill or a server restart. The shortest-view buckets are about 8 minutes wide. The first cache fill can therefore download more artifacts. - Its **runtime benchmarks** view at `/bench?view=runtime` shows one colored line per CPU for **every** benchmark. The lines share a calendar-time axis. A late-starting CPU line sits at the right. A single sample appears as a point. A stale line visibly ends short of the current date. The dashboard tile leaves out the CPU legend to keep the compact chart readable. Selectors choose which measurement to plot (**p0** = the fastest sample, **mean** = the arithmetic mean, **p75**, **p99**, **p99.5**, **p99.9**, **p100** = the slowest) and whether to group by source **file** or sort by latest **duration** or **trend**. The mean selector was once labeled `p50`, and `?stat=p50` still opens it so that a saved link does not quietly fall back to the default column. A "key only" checkbox limits the rows and CPU legend to `topic board/journey` and `topic board scale/100`. The key benchmarks tile opens this view with `key=1` in the URL. Metric, sort, and window changes preserve the filter. Clearing it restores the full list. A "hide green" checkbox drops the steady ones. A slider from 1 through 45 days changes the visible calendar range. The displayed samples are spread across at most 200 time buckets, so a shorter window uses more of the collected samples per day. Each graph chooses its vertical scale after ignoring the two lowest and two highest displayed values; those points stay in the line but no longer determine the scale, so they may clip instead of flattening the rest of the history. A graph with fewer than 20 values uses its complete range. Keyboard arrows adjust the window without moving focus. Enter applies the range immediately. Another selector carries the range into its own navigation, and leaving the controls applies it directly. Each row shows one latest value and trend from the CPU with the most benchmark samples in the selected window. A tie uses the CPU with the newest sample, then its name for a stable result. That representative CPU also sets the row color and sorting. A numbered CPU key identifies the representative series and links to its definition. The same key and a color swatch appear in one CPU legend after all benchmark graphs instead of repeating processor names in every row. It includes the full processor identity reported by the artifacts, the number of benchmark graphs and runs shown for that CPU, and the observed date range. The page reads from the server cache so it re-renders instantly. Its progress panel stays visible as Idle between collections. During collection it shows cached, queued, requested, responded, outstanding, and failed artifact checks. Changing the range leaves the server collection running and joins it from the new page. The page closes with a **rerun hand-off**: a link to GitHub, where the run is started. It targets the newest completed run when that run failed, whose **Re-run all jobs** repeats it, and the workflow otherwise, whose **Run workflow** starts a fresh run. The dashboard's GitHub token is read-only, so the board cannot start a run itself, and GitHub decides whether the viewer may start one. - The **CI duration history** view at `/bench?view=ci` selects either labs `deno.yml` or loom `test-fast.yml`. It charts every job's start-to-finish duration on one calendar-time axis. An overall row measures the workflow from the first job start to the last job completion. Jobs that share the trailing-parenthesis base name used by `scripts/ci-gantt.ts` are shown together. Each group starts with a slowest-shard line, followed by the individual shard lines. The slider covers 1 through 45 days. It keeps every successful main build when there are at most 200. Larger sets keep exactly 200 builds spread evenly through the chronological run sequence. A coverage label compares the sampled builds shown with every successful main build in the selected range. The view can group by job or sort every line by latest duration or robust trend. Its graphs use the same vertical-scale trimming as runtime benchmark history. It renders cached history immediately. The progress panel remains visible as Idle between collections, then shows live collection progress with cached, queued, requested, responded, outstanding, and failed run counts. On both history pages, a failed collection remains in the Idle panel until a later collection for that history view succeeds. Open runtime benchmark and CI history pages check for newer server data once a minute. An open Gantt regenerates every 30 minutes and whenever its tab becomes visible. CI refreshes share a 30-minute GitHub freshness window, so multiple pages do not repeat the same API reads. Runtime and CI history checks wait through the same window after GitHub rejects a collection. Moving the window slider starts or joins the matching collection without cancelling wider-window work already in progress. CI history reads the workflow's runs through the run-list reader, back to the first run created a day before the window opens, and takes the successful pushes to main from them. For labs that is around a hundred pages the first time, and after that, across restarts too, only the runs that landed since the last collection. A Gantt of every run, or of every main push whatever it concluded, reads the same runs, stopping at 150 runs or at the first run created a day before the 45-day window, and charts the runs that started inside it. - Every GitHub API request made by the three performance views reserves rate capacity before it starts. Each guarded request batch reads GitHub's current rate-limit status before reserving. Collection stops before projected in-flight requests would pass 80% of the token's hourly core limit. It also stops at 720 REST request points in a rolling minute, which is 80% of GitHub's documented 900-point limit. The page reports either boundary as a rate limit hit. It does not wait or retry there. Reservations and request times are locked and stored in the fixed `fabric-wall-github-rate-limit.json` file in the dashboard cache directory, so overlapping dashboard processes and restarts share the same budget. Tokens are represented by SHA-256 hashes in that file. Only `/bench` collection reserves capacity or can be stopped by the ledger. Other dashboard GitHub requests do not read or update it and proceed normally. - Completed workflow-attempt timings are written atomically to two places in the dashboard cache directory, split by how much of each attempt a view needs. The fixed `fabric-wall-ci-run-index.json` file holds one entry per attempt: its run metadata and the duration of each job. That is everything the CI duration charts read, and it is small enough to hold in memory for the whole retention window. The `fabric-wall-ci-gantt` directory beside it holds one gzipped file per attempt carrying every job's steps, which is around forty times larger per attempt before compression and is what the Gantt views draw. Steps repeat the same few names and the same timestamp prefixes over and over, so an attempt compresses to roughly a twentieth of its size, and every attempt the run index keeps can afford to keep its detail. An attempt loses its detail only when it leaves the index altogether, so nothing GitHub would have to serve again is discarded early. A Gantt request reads only the attempt files for the runs on its chart, at most the 150 the range slider allows, and a duration refresh reads none of them. The runs on a chart are read and handed to the renderer one at a time, so serving a chart never holds all of its timings at once. An attempt whose file is missing, or holds a layout this version cannot read, is collected from GitHub again, so a chart never draws a run with no timings and a damaged file repairs itself. A `fabric-wall-ci-job-history.json` file in the cache directory holds a layout the dashboard does not read, and reaches hundreds of megabytes. A process that finds one deletes it, along with its lock and any temporary file left beside it, without parsing it, and collects the current window from GitHub instead. A workflow run is kept as the fields the dashboard reads rather than as GitHub returns it. A run in a query result carries its repository, head repository, whole head commit, and both actors, which is around fifty times the size of the fields any view uses, and a discovery window is thousands of runs held for the length of the freshness window. CI history and the detailed `/bench?view=gantt` view use the same entries. The three performance views share one selector and preserve the applicable repository, range, sort, and runtime statistic while moving between them. They also share the styles for their page header, selector, controls, progress panel, time axis, and history rows. The next collection loads those timings before reading GitHub, so it only fetches jobs for new sampled runs, uncached Gantt runs, and new attempts. Cached history remains visible when no GitHub token is configured or a refresh fails. Each completed response is placed in the shared cache and persisted while the rest of its collection continues, including responses from a window the browser has since left. The cache also stores each window's exact sampled run attempts, partial-read state, last completed refresh, and the complete set of successful-run timestamps used by its coverage label. A new dashboard process therefore renders the same chart, warning, and coverage while honoring the remaining part of the 30-minute freshness window without querying GitHub. Dashboard processes that share the file lock it while merging and atomically replacing entries. Attempts referenced by the last completed chart remain in the cache while a rerun is being collected. An interrupted or rate-limited collection records that its earlier manifest is no longer fresh, so a restart keeps the completed chart visible but resumes collection. - Runtime benchmark artifact results are written atomically to the fixed `fabric-wall-benchmark-history.json` file in the dashboard cache directory. Successful reads and definitive empty results are retained for 60 days. A failed read remains uncached so a later scheduled collection can establish whether the artifact exists. Dashboard processes that share the file lock it while merging and replacing the stored run attempts. The exact run attempts behind the last completed refresh, including the reason for a definitive empty result, are stored in the same file. Attempts used by that completed refresh remain available while a rerun is incomplete. Restarting the dashboard therefore reconstructs the same chart without immediately rediscovering benchmark runs that are still within the 30-minute freshness window. An interrupted replacement is persisted as stale and is retried. - **Deploying these gated tiles** follows the same pattern as the existing ones: add the value to Secret Manager, wire an ExternalSecret in `dashboard-secrets.yaml`, and add the env to `03-deployment.yaml` (see the Deploying section below). Everything below is a tunable constant in `config.ts`: - **Status thresholds:** `TRUST_GOOD`/`TRUST_WARN` (first-try-green %), `DUR_GOOD`/`DUR_WARN` (median CI minutes). - **Data windows:** The shared fetch returns at most `CI_RUNS_MAX=200` workflow runs and stops at `CI_RUNS_MAX_AGE_DAYS=60` days. CI trust uses the newest `TRUST_RUNS_MAX=160` fetched runs. CI duration uses whichever is larger: `DUR_MIN_RUNS=20` passing runs or `DUR_MAX_AGE_HOURS=6` hours. The benchmark trend uses the same larger-of-the-two idea in days: `BENCH_TREND_MIN_RUNS=20` runs or `BENCH_TREND_MAX_AGE_DAYS=14` days. Recent commits shows `RECENT_DISPLAY=50` commits. - **ci-trust cell grid:** The grid has up to `TRUST_RUNS_MAX=160` square cells in rows of `TRUST_COLS=40`. On wide tiles, the squares stop growing and the columns spread out to keep the grid clear of the subheading while preserving its equal left, right, and bottom insets. First-try successes are green. In-progress runs are blue. Completed runs that lower the trust percentage are red, including a run that needed a rerun and a run whose one attempt left was cancelled with a job a runner ran that did not succeed for a reason other than a concurrency group cancelling it. Ignored runs are gray, including a run every attempt of which was cancelled with every job a runner ran succeeding or cancelled by a concurrency group. ## Local development Local-first: no build step, no deployment, a single process on `localhost`. - One-shot: `deno task dashboard`. - Watch mode from `packages/dashboard` (reloads the server on any edit to a tile or the core): `deno task watch`. Env knobs for the dev loop: - `GH_TOKEN` (or `GITHUB_TOKEN`) — required for the GitHub tiles unless `GH_APP_CLIENT_ID` and `GH_APP_PRIVATE_KEY` configure a GitHub App. GitHub spend also needs Administration read. GitHub users also needs Members read. Without either, those tiles stay gray. - `DASHBOARD_PORT` — run several instances at once (e.g. one per branch) without clashing. - `DASHBOARD_REPO` — point the CI tiles at any repo. Its owner selects the organization for GitHub users. - `ESTUARY_URL` and `RAPIDS_URL` — point either production-tile health check at a local server. `PROD_URL` remains an alias for `ESTUARY_URL`. - `COMMON_FABRIC_URL` — replace the public-site target in the production tile. - `BASTION_HOST` — replace the default bastion hostname the production tile checks. - `PROD_PROXY` — route the estuary and rapids health checks through a proxy, for example `socks5h://127.0.0.1:1055` with a local Tailscale userspace proxy. - The other credential envs (see **Credentials** above — `SIGNOZ_*`, `GCP_*`, `OPENAI_ADMIN_KEY`/`ANTHROPIC_ADMIN_KEY`/`OPENROUTER_KEY`, `DISCORD_*`) — set one to develop that gated tile against its real backend. It never crashes on a missing credential: the GitHub tiles need `GH_TOKEN` (or the GitHub App) and the other private-source tiles each need their own env var, and any tile whose `collect()` throws (missing token, offline) just shows a gray "unknown" while the rest of the board keeps working — so you can develop against whatever you happen to have access to. Developing one tile in isolation: a tile is a pure `collect(ctx) -> TileView`, so you can exercise it with a hand-made `Ctx` in a `deno test`, no server or live source required: ```ts const fakeCtx = { runs: async () => FIXTURE_RUNS, env: (_: string) => undefined }; const view = await myTile.collect(fakeCtx); // assert on view.status / view.value … ``` ### Tests ```bash cd packages/dashboard deno task test ``` `lib.test.ts` covers the pure helpers; `tiles.test.ts` covers each tile's `collect()` — the CI tiles against canned runs, and the token-gated tiles' gray-out contract. These ordinary unit tests are hermetic and need only `--allow-env` for module-load configuration reads. The full `test` task also verifies that Resvg reproduces the embedded PNGs and runs the favicon behavior in the local browser test runner. The package tasks grant the additional permissions those two checks need. `tiles/prod-uptime.test.ts` exercises the production tile's public-site, health, DNS, and proxy checks with canned HTTP responses, an injected DNS resolver, an injected proxy client factory, and a fake SOCKS5 proxy, so its unit tests never reach the network. This is a workspace package, so its ordinary unit tests also run as part of the repo-wide `deno task test`. ## Deploying (stage GKE, tailnet-only) The deploy artifacts follow the golink pattern. `Dockerfile.dashboard` (labs repo root) builds the image; the manifests, ExternalSecrets, and tofu secret containers live in the **infra** repo (`k8s/manifests/dev-dashboard/`, `k8s/overlays/stage/external-secrets/dashboard-secrets.yaml`, `tofu/gke/secrets.tf`), and it is wired into `make apply-dev-dashboard-stage` and `apply-all.sh`. Access is tailnet-only at `https://dashboard..ts.net` via a userspace Tailscale sidecar (the one adaptation golink gets for free from its embedded tsnet). **One-time setup (human steps):** 1. Tailscale admin console: add `tag:dashboard` to `tagOwners`, grant who may reach it, and configure a federated identity restricted to `auth_keys` and `tag:dashboard`. The infra overlay supplies that identity's public client ID with `ephemeral=false&preauthorized=true` and its audience; no Tailscale auth key or OAuth secret is stored in Kubernetes or Secret Manager. The tailnet also needs MagicDNS and HTTPS certificates enabled — `tailscale serve` fetches a cert for `dashboard..ts.net` and can't without them. 2. `tofu apply` in `infra/tofu/gke` creates the dev-dashboard Secret Manager containers and Workload Identity/BigQuery grants. Create the GitHub App and its two installations as described under `GH_APP_CLIENT_ID` + `GH_APP_PRIVATE_KEY` above, then store its private key (and each provider credential you want to enable): ```bash gcloud secrets versions add k8s-stage-dashboard-github-app-private-key --data-file=path/to/private-key.pem ``` with the path of the app's downloaded `.pem` file in place of `path/to/private-key.pem`. The app's client ID is public and sits in the infra stage overlay, which also names the enterprise github spend reads. 3. The infra manifests create separate 1 Gi `standard-rwo` PVCs for the Discord history file and Tailscale node state. The dashboard remains a one-replica `Recreate` Deployment; pod replacement reuses the same non-ephemeral Tailscale node ID and `dashboard..ts.net` name. 4. To follow the GitHub webhook relay, the pod has to be able to open a TCP connection to the relay's subscriber listener, and `GITHUB_WEBHOOK_RELAY` has to name the address it reaches the listener at. The Tailscale sidecar runs in userspace, so the pod's own connections do not cross the tailnet, and both the dashboard's egress policy and the relay's ingress policy have to admit the connection. A pod that cannot connect logs the first failure, tries again on each serving tick, and reads GitHub on its intervals. **Build, push, deploy** `.github/workflows/dashboard-image.yml` publishes the image, and it publishes only from `main`. A push to `main` starts it when the dashboard image, the dashboard package, the Gantt drill-down, the Deno dependency metadata, or the workflow itself changes. You can also start it by hand from the repository's Actions tab, which is how to publish a `main` commit whose files fall outside that path list. A manual run can name any branch or tag, so the workflow's first step fails a run that is not on `main`: publishing moves the `latest` tag the stage deployment follows. That step runs before anything from the named ref is checked out. The workflow runs the dashboard tests, then builds the amd64 image and pushes it under both the immutable `dev-dashboard:` tag and `latest`. The image carries labels naming the repository, the ref, the event, the workflow, the run, and the commit it was built from. The infra repository's `k8s/scripts/verify-image-provenance.sh` reads those labels back and checks them against GitHub before the image's digest is deployed. Once a SHA tag exists, a rerun of that commit reuses its digest: it moves `latest` onto the image already there rather than rebuilding it or repushing the immutable tag. The same tests also run in CI, on pull requests and on main alike, because the dashboard is a workspace package and CI's `Test` job runs each package's test task. The publish workflow runs them again rather than leaning on that. The two workflows are independent — neither waits for the other, and this one cannot read CI's verdict — so its own test job is the only thing standing between a dashboard that fails its tests and the `latest` tag. Leave it in place even though it looks redundant. What that test job does not do is record what it runs. CI records the same task on the same commit, so a second recording here would file each of those tests twice against one commit. The job therefore sets no spool directory, wraps nothing in `run-recorded`, and ships no test-records artifact, and the relay does not follow this workflow. No image is built or published for a pull request. The publish job authenticates with GitHub OIDC and GCP Workload Identity Federation. It emits the immutable `sha256:` image reference in the workflow summary; no service-account JSON key is used. The manual trigger publishes whatever `main` currently points at. Stage runs only an image this workflow published from `main`, because the infra repository deploys a first-party image only when its build metadata shows such a publish. A commit that has not landed on `main` therefore cannot be deployed there. Copy the published digest from the workflow summary into the infra stage overlay's `images[].digest`, commit that immutable pin, then run `make apply-dev-dashboard-stage` from `infra/k8s`. The node then appears in the Tailscale console as `tag:dashboard`; open `https://dashboard..ts.net/`. (The sidecar image is already pinned by `@sha256` digest in `03-deployment.yaml`, matching golink.) **Provider-gated tiles** stay gray until their credential or public configuration is wired: add the Secret Manager *value* (the container already exists from `tofu apply`), enable the matching ExternalSecret/env when needed, then re-run `make apply-dev-dashboard-stage`. Stage's Cloud Billing table is already wired; during Google's initial export backfill it reports `no billing data yet` rather than a false zero. Every backend is reached over HTTP, so the image carries no cloud CLI. The GitHub tiles authenticate as the GitHub App. The cloud-spend tile queries BigQuery as the pod's own service account through Workload Identity. The infra repo's `tofu/gke/dashboard.tf` provisions that account, the Workload Identity binding, and its BigQuery Job User and Data Viewer grants, so no key is stored in the cluster. The stage deployment sets `GCP_BILLING_TABLE` to the standard export table, and `dashboard_billing_dataset` scopes its dataset reader grant. ## Design notes A couple of choices the code alone doesn't explain: - **Calm by default, three signals.** A tile is only ever good, warn, or bad, plus a gray "unknown" when its source is missing or a collector errors. There is no finer severity scale: green means fine, red means act now, and anything the board can't judge goes gray rather than crying wolf. The layout is built for glanceability — readable in a second from across the room — so it stays dark and free of attention-grabbing animation, and it reports on the system rather than on individuals. A failing collector degrades to gray with a short reason instead of blanking or alarming, and the header's "updated Ns ago" is a deliberately honest liveness signal. `/healthz` reports whether the board has collected anything; point an external uptime check (on a different host) at it, since a board that can't reach its own sources can't page you about itself. - **Why Deno + TypeScript.** The dashboard lives in the labs workspace and shares its one toolchain and the team's TypeScript fluency. An earlier rationale — that it would import the repo's performance tooling in-process — no longer holds: it fetches the same GitHub JSON directly. If it ever needs a single self-contained binary with no runtime dependencies, a compiled language would be worth reconsidering; today the workspace fit wins.