Service-level architecture on Google Cloud

The other page shows the shape of the system. This one names the parts. It is written for someone reviewing the design rather than using the product: which Google Cloud services are in the path, what each is actually doing, what it costs when nobody is working, and which services are deliberately absent. Everything here was read from the live project, not from a design document.

1 · The services, and where they sit

Colour is meaning, not decoration: blue is Numen's own code, violet is a managed Google service, green is compute a researcher's work runs on, and amber is storage.

2 · What each service is doing, and what it costs

3 · Three paths worth following end to end

A researcher launches a machine

Browser → Cloud Load Balancing → nginx → Django. Django prices the selection from the Cloud Billing Catalog, checks regional quota, then hands a Terraform configuration to Infrastructure Manager, which creates a Compute Engine VM running as that user's own service account. The VM has no external address; its startup script mounts that user's Cloud Storage bucket with Cloud Storage FUSE and installs the selected applications. Progress is reported back through guest attributes. Idle-stop eventually returns it to zero.

A researcher builds a cluster and runs a job

New cluster in the console: a name, a head node, one to eight compute groups (each a machine type and a node cap, GPU shapes included) and optionally a shared filesystem. Django validates the shape, writes a launch record to Cloud Storage and submits a Cloud Build that runs terraform apply as sa-terraform against the packaged Cluster Toolkit blueprint, state in the tfstate bucket; the dialog polls the build. About three minutes later the head node is up and Django lists the cluster by the toolkit's own slurm_cluster_name label through the Compute Engine API. The researcher logs in to the head node over IAP with OS Login and runs sbatch; slurmctld creates compute nodes that did not exist a moment earlier (about two minutes to running), deletes them five minutes after the job ends, and slurmdbd writes the record to Cloud SQL. Adding a compute group later is another build against the same state; so is destroy, which also sweeps any compute node Terraform never knew about.

A filesystem is created with a cluster

Choosing “Create a new Lustre filesystem with this cluster” asks Managed Lustre for the filesystem through the same guarded path the admin console uses, then submits the build with an extra lustre-wait step that polls until the filesystem is ACTIVE (10–20 minutes), writes its mount spec into the launch record, and only then lets Terraform run. The filesystem outlives the cluster on purpose; deleting it is a separate, priced, admin action.

A job's history survives its cluster

This is the path that is easy to miss and is the reason the accounting database is platform-level. Cluster Toolkit treats accounting as optional and defaults it to null, so Slurm records nothing unless it is wired up; a cluster destroyed at the end of a session takes its entire job history with it. Here slurmdbd writes to a Cloud SQL instance that no cluster owns. Destroy the cluster and the rows remain, and the optimisation engine reads them back into DuckDB later. Demonstrated across two clusters that no longer exist.

4 · Services deliberately absent

Listing what is not used is as informative as listing what is, and it stops the diagram being read as a wish.

A researcher runs a container job

1. Picks an application with a maintained image and "GKE — Kubernetes job" as the infrastructure. The webapp builds a Kubernetes Job: image, CPU and memory from the catalogue's minimums (inside Autopilot's CPU:memory ratio), the outputs bucket as a GCS FUSE volume, labels naming the user and the app.
2. Submits it to the API server over HTTPS with the cluster's CA and its own IAM access token. The namespace and the Kubernetes service account are created on first use.
3. Autopilot provisions a node for exactly that request (one to three minutes), the pod pulls the image, the workload runs and writes under /outputs/<job>, which is the bucket.
4. My instances shows the job as lane GKE; its panel reads the pod, the conditions and the container log back through the API. A job that succeeds is recorded once as job evidence with its measured hours.

A launch fails and nobody opens a shell

1. The Diagnose button (or the assistant's tool) calls one endpoint with the launch name.
2. The endpoint reads the Infrastructure Manager deployment and its Terraform error, the VM, the bootstrap's guest attributes and the serial console, in parallel.
3. Nineteen signatures — zone capacity, quota, permission, GPU driver, Lustre client, filesystem mount, a pipeline that installed but does not import, packages missing on apt or dnf, no egress, disk full, out of memory, conda solve, verification failed, idle stop, Spot preemption — become findings with the evidence line and a fix.
4. The panel shows the verdict, the findings and the log tail; the assistant, asked why, explains those findings and is told not to invent a cause they do not contain.

5 · Identities, and what each may do

Numen on Google Cloud · Clovertex · read from the deployed sandbox, 2026-09-07 · back to the architecture overview