Numen on Google Cloud - Architecture on Google Cloud
Functional and technical views · drawn from the deployed sandbox
numen-gcp-sandbox, updated 2026-09-07
Two diagrams of the same system. The functional view is what a researcher experiences and is the one to open a conversation with. The technical view names the services, identities and data paths behind it. Everything drawn here exists and is deployed; a short list of what is not yet built is at the end, kept deliberately separate so the diagrams cannot be read as aspirational.
Looking for the Google Cloud services by name? The technical view below is
the shape of the system. The service-level
architecture names every Google Cloud product in the deployment, what it
is doing, what it costs, and which are absent on purpose - built for a cloud
architect reviewing the design rather than a researcher using the product.
1 · Functional architecture
What a scientist does, and what Numen does in return. No cloud service names - this is the product, not the plumbing.
The path a person takes
Intelligence and planning
Cost and data
Ways to run
2 · Technical architecture
The same system, named. Every box below is a deployed resource in
numen-gcp-sandbox.
Added between 3 and 7 September
Each row is deployed and was proven live on the day it says; the deploy guide Part is the record.
| Capability | What it is, in this system | Part |
|---|---|---|
| Lab projects (tenancy, two slices) | A Google Cloud project per lab under the platform: its own VPC, NAT, Terraform and runtime service accounts (an org policy forbids cross-project accounts). Single-node launches and Batch jobs run in the lab as its own identities; quota, inventory, compliance findings and spend are read per project; an admin Projects panel shows lanes, machines, jobs, quota, findings and spend side by side. VPC Service Controls was evaluated and deliberately left for later: a project per tenant first. | 50, 54 |
| Developer view and Help | Docs served from the repository itself (uploaded by Terraform on every deploy): the deploy guide, a build journal generated from it, a diary, the backlog, the tracker of Google's asks, the evaluations. A feature ledger with an author mark per entry. | 48-50 |
| RHEL 9 and Rocky 9 | A launch-time OS choice beside Ubuntu: a dnf layer in the installer, 79 of 90 apps proven ready by an audit the catalogue pins, the launcher refusing the rest by name. | 51 |
| Spot at launch | A checkbox with the Spot price and the saving shown against on-demand, from the Billing Catalog's Spot SKUs; the machine stops rather than dies on preemption. | 53 |
| Google's recommendations | The Recommender API read per project - idle machines, machine types, idle disks and addresses, commitments, IAM - with the monthly saving and a Stop button for machines Numen owns. Read-only. | 55 |
| Real science, recorded | Seurat on 10x PBMC 3k (Ubuntu and RHEL) and AlphaFold (ColabFold) on an L4 - real data, timed, hashed, read back, and written as job evidence the planner and the assistant use. The AlphaFold proof found an installer defect the install proof had reported as READY for weeks; the recipe is pinned and READY now means the model imports. | 52, 56 |
| Launch diagnosis | "Why did my install fail?" answered without a shell: the deployment's Terraform error, the VM, the bootstrap's own verdicts and the serial-console log, matched against nineteen known signatures, each with a fix. A Diagnose button, an endpoint, and an assistant tool that explains the findings and is told never to invent a cause. | 57 |
| Container lane on GKE | One regional Autopilot cluster; a catalogue app with a maintained image (Seurat, GROMACS) runs as a Kubernetes Job submitted through the API server with the webapp's own IAM token - no kubectl, no kubeconfig - with the outputs bucket mounted and the container log read back. Appears in My instances like any other lane. | 59 |
| Web host | e2-standard-2 with four threaded gunicorn workers; the console's polling had saturated two synchronous workers on an e2-small and the front door answered 502. | 58, 59 |
What happens on one launch
| # | Step | Who does it |
|---|---|---|
| 1 | Applications chosen; machine sized and priced against live list prices | Django + Billing Catalog |
| 2 | Disk headroom checked before submitting - a launch that cannot fit is refused, not started and failed 2 minutes later | Django |
| 3 | Identity and bucket derived from the session, never from the request body | Django |
| 4 | Deployment created, blueprint pulled from the assets bucket | Infrastructure Manager as sa-terraform |
| 5 | Instance boots as sa-numen-u-<user>; gcsfuse mounts that user's bucket at /data | startup script |
| 6 | Installer runs; each application publishes a version string as proof | installer -> guest attributes |
| 7 | Idle-stop armed with the timeout the user chose (30, 60 or 90 min, or none), counted only once the machine is Running - no logged-in user, no open session and no load | systemd timer on the VM |
| 8 | If a shared filesystem was chosen, Lustre or Filestore is mounted at the mount point given (default /scratch); the on-mount validator writes its report under <mount>/numen-validate/ | startup script |
Security model in four lines
- No inbound internet. Instances have no external IP; SSH and RDP arrive only through IAP from
35.235.240.0/20. - Identity comes from the session. A client cannot name the account its VM runs as, or the bucket it mounts.
- One bucket, one identity, one user. A VM holds a credential scoped to a single bucket, so a shell on it reaches nobody else's data.
- No keys on disk. URL signing goes through the IAM credentials API; the only secret is an optional API key held in Secret Manager.
3 · Where the numbers come from
A recurring question, so it is worth stating plainly. Numen shows three kinds of number and does not blur them:
| Kind | Source | Example |
|---|---|---|
| Live | Cloud Billing Catalog, read at request time | $0.268/hr for an e2-standard-8 |
| Measured | Recorded from real runs on this platform | install minutes per application; job throughput |
| Learned | The optimisation engine, trained on this site's own job accounting | runtime for a workload no one here has run yet |
| Demonstration | The engine trained on the built-in synthetic sample - shown, never presented as measured | what a new site sees before ingesting its own data |
| Estimated | Derived, and labelled as such in the UI and by the assistant | time to process a workload not yet observed |
Every answer carries its basis, and they are strictly ranked: this site's own
measured runs outrank the engine, which outranks a seed estimate. A model
trained on the synthetic sample is capped at low confidence however good its
metrics look. The assistant states which of these it is using. This is the
foundation for feeding recommendations from accumulated real runs rather than from
a model's general knowledge - the direction of travel, not a claim about today.
4 · Honest gaps
Kept separate from the diagrams so neither can be misread as complete. Revised 2026-09-03; entries closed since August are recorded as such rather than quietly deleted.
- The login node is a future release. A cluster is a head node plus compute
groups; researchers log in to the head node (IAP + OS Login, as themselves) and
submit with
sbatch. The console no longer submits jobs on their behalf - that path ran as the platform's identity and is gone. A separate login node, and installing applications onto a running cluster from the console, are both shown as future. - Creating a Lustre filesystem is not instant and not always possible. 10-20 minutes, and on 3 September four of six creates failed on Google's capacity in us-central1-a (nothing billed). The console says so and waits; picking an existing filesystem is the fast path.
- There is no H100 quota in this project. The planner will size and
price an
a3-highgpu-8ghonestly at about $88/hour and then tell you it cannot start, becauseNVIDIA_H100_GPUSdoes not exist as a quota metric here and the 80 GB A100 limit is zero. A100 (16) and L4 (16) are available. This is a quota request, not a code change. - The engine has a feed, not a trained model. Scheduler accounting now lands in a database that outlives the clusters that produced it, and the engine reads it back - proven on two clusters that no longer exist. But the database holds a handful of jobs, and a handful is not a dataset. Sizing answers still come from seed priors, and every one of them is labelled as an estimate where it is shown.
- The parallel filesystem costs money for as long as it exists.
Managed Lustre, 9 TiB at 1000 MBps/TiB - about $7.40 an hour,
roughly $30 for a working session and roughly $5,400 a month if anyone
forgets to remove it. It is therefore created on purpose (from the admin
console, or with a cluster after the cost is shown per hour and per day) and
never deleted with a cluster - deleting it is an explicit admin action.
Filestore (managed NFS, about $5.39 a day per TiB) is the cheaper
choice and shipped on 2026-09-03. Cross-node read/write proven again on
3 September: a file written on the head node was read by a compute job and
the job's file appeared on the head node.
Parallelstore would be cheaper (about $2.30/hour) and was the original choice. It was dropped because this project cannot create one - Google has not allowlisted it - and because Google is retiring the cluster-toolkit module for it. - One dead bucket.
numen-gcp-sandbox-userdatawas the shared bucket used before per-user buckets. Nothing addresses it now and it should be removed; deleting live storage is a decision for a person. - Deployment records outlive their instances. Deleting a VM leaves its Infrastructure Manager deployment behind, so stale entries can appear in listings.
- Tenancy is two slices in, not finished. A lab is a project of its own with its own identities, network and quota, and single-node launches and Batch jobs run inside it; clusters, Lustre and Filestore are still platform-only, and a folder with org-level roles (Cloud Asset inventory, org policies, a VPC Service Controls perimeter) needs an organisation administrator this sandbox does not have.
- The container lane is CPU-only and one cluster. GPU pods (Autopilot supports L4 and A100 node selectors) and a cluster per lab are the next slice; only applications whose maintainers publish an image run there (Seurat, GROMACS today).
- AlphaFold runs with the public MSA server. The ColabFold databases are not yet staged on Filestore, so a confidential sequence has no private path; the evaluation names it as the next step.
Closed since the 3 September version of this page
- “There is no tenancy model.” There is one: a project per lab, built and proven for launches and Batch (Parts 50 and 54); what remains is listed above.
- “No GKE lane exists.” It does (Part 59), and the tracker's done-line - a catalogue app runs on GKE and appears in My instances - was met with a real Seurat run.
- “Ubuntu only.” RHEL 9 and Rocky 9 are launch choices (Part 51).
- “A failed install needs a shell to explain.” It needs a button (Part 57).
- “Install proof is run proof.” It is not, and the AlphaFold recipe proved it; the tile now says READY only after the model imports (Part 56).
Closed since the August version of this page
- “The optimisation engine is wired but unfed.” It is fed. Slurm accounting is always on and writes to a database above the cluster lifecycle.
- “Cluster sizing is not yet in the UI.” Clusters are now created, grown and destroyed from the console: head node plus up to eight compute groups, GPU groups included, with Cloud Build progress in the dialog.
- “A cluster needs an operator with gcluster.” Blueprints are packaged once; every create, add-group and destroy is a Cloud Build the console submits. The three blueprints (plain, mount-existing-Lustre, create-Lustre) are kept in step by regeneration, not by hand.
- “The assistant is not regression-tested.” A golden conversation script runs nightly against the live site and alerts on failure; it caught a four-hour outage on its first real night.
- “The Lustre mount arrives owned by root.” The blueprint now creates
a writable area, so no administrator step stands between a researcher and
/scratch.
Numen on Google Cloud · Clovertex · generated from the deployed sandbox, updated 2026-09-07