Workflow optimization in Numen

Job evidence integration update

The workflow-evidence-v1 release adds authenticated individual run discovery, stage durations, hashes and captured log excerpts. Earlier sections describe the original pilot and its initial integration limitations; the new job-evidence tool is the authority for current registry availability.

The published corpus contains 40 historical real RELION attempts (39 successful) and eight explicitly synthetic demo records across RELION, Cell Ranger, OpenFOAM and Scanpy. Synthetic records have no real execution dates and never count as executed jobs or calibrate real throughput. Counts describe the registered corpus, not every application ever tested elsewhere in Numen.

Calibration now requires compatible scientific and hardware scope, excludes failed/synthetic/holdout records, deduplicates run IDs and exposes accepted and rejected evidence. Predictions retain seconds precision and the measured allocation. This is retrieval and numerical calibration, not LLM fine-tuning or model promotion.

Native RELION Schemer control-flow tests cover explicit abort-file polling, resume, incremental file arrival and reset. Full movie-processing dependencies and independent biological validation remain unestablished. The linked integration report contains new AI answers, earlier failures and downloadable synthetic inputs.

Workflow optimization in Numen

Numen helps researchers decide how to run scientific work, explain the evidence behind that decision, and verify the outcome. The central method is a repeatable loop: describe the workflow, observe execution, identify a bounded change, compare it with a baseline, validate scientific outputs, and retain the evidence for the next decision.

This reference combines the current implementation, a completed RELION 5 CPU pilot, and the next engineering steps. The pilot demonstrates measurable execution differences and evidence-dependent planning calculations. It does not establish production cryo-EM reconstruction quality, GPU performance, a trained model improvement, or actual billing savings.

The implemented planner prefers recorded workload throughput, then an engine manifest rate, then a seed estimate. Workflow history can accompany the recommendation. The current planner is not a general optimizer that rewrites arbitrary workflow dependencies. Those capabilities must be tested separately before the assistant describes them as available.

Encyclopedia content and assistant reference content come from the same versioned source. The assistant uses this material to explain methods and scoped evidence, while live tools remain responsible for current project state and measurements.

Reading paths for different roles

Start with the question you need to answer. The same result has different implications for scientific validity, operations, product confidence and expenditure. Role-specific reading paths help each reader find those implications without hiding the underlying evidence.

Architecture and implemented behavior

The scientific workflow runs on a compute node. A campaign harness records commands, data hashes, scientific parameters, timestamps, process resource observations, completion status and output checks. The harness is separate from the recommendation code so the evaluator can check recommendations against actual execution evidence.

The existing observation store aggregates positive-duration job records into a median rate for a workload class. The planner converts quantity and deadline into required compute using that rate. The current production precedence is direct measurements, engine-manifest class rates, then seed priors. These calculations are deterministic; the language model explains them and uses tools rather than supplying the arithmetic from memory.

The vendored optimization engine also contains accounting ingestion, workflow and user history features, training and resource prediction. Its presence in the repository does not prove that this pilot trained or promoted a model. The pilot exercises real RELION jobs and an isolated copy of the planner and aggregation code. It does not fabricate Slurm accounting for single-node processes.

Current memory sizing remains an estimate. Site history is contextual information and does not by itself change the throughput calculation. The historical model-real label on a manifest must not be interpreted as proof that every returned class rate was generated by a trained model; inspect the actual path and manifest provenance.

The Encyclopedia is a separate, shareable application link. Its article is rendered from reviewed structured content. The assistant retrieves the same content, identifies it as a curated reference, and uses a content hash to invalidate cached reference-based answers when the article changes.

Real workflow -> execution records -> quality checks -> scoped measurement summary
                                              ↓
Live telemetry -> planner calculations -> assistant explanation -> verification
Encyclopedia -> method and historical evidence ↗

Experimental methodology

The primary hypothesis is that changing an operational setting can reduce elapsed workflow time while preserving the scientific output under fixed parameters. A second hypothesis is that planning calculations respond to relevant execution evidence and distinguish that evidence from a seed estimate. These are separate claims and have separate tests.

For each operational comparison, hold particle inputs, optics metadata, classification settings and seed fixed. Change only the selected thread count or working filesystem. Execute the two variants sequentially, randomize their order within the pair, and repeat with five seeds. Record the schedule before execution. The current pilot uses five pairs for each of three comparisons.

Smoke tests check basic execution and instrumentation. The initial 64-particle smoke runs were too short for performance comparisons, so the pilot was increased to 1,024 particles, 10 classes and 15 iterations. A calibration check established that a four-thread run took about 12.6 seconds before the paired campaign started.

A separate set of 1,024 particles was reserved for five held-out executions. It does not overlap the calibration particle identities, but both sets originate from the same tutorial acquisition. This is a limited within-dataset validation, not a test on an independent biological sample. Stronger validation should hold out whole movies or acquisitions and include other suitable public datasets.

The performance quantity is scientific workflow elapsed time: preprocessing plus classification. Input staging is measured separately. The storage comparison uses small, cacheable data and includes output I/O; it is not a saturated throughput test. Installation, archive transfer and decompression were completed before timed runs.

For numerical equivalence, optimally match class images across runs, require every matched correlation to be at least 0.99, and require at least 99 percent particle class-assignment agreement after relabeling. Also verify particle counts and identities, finite arrays and nonzero class variation. These are strict numerical checks for this pilot, not a substitute for expert assessment of a final reconstruction.

Report paired runtime ratios and descriptive uncertainty, retain regressions, and do not count repeat prompts as independent scientific experiments. More repetitions reduce uncertainty in a measurement; they do not automatically improve the recommendation code.

Execution configuration and audit trail

The campaign ran on one Google Cloud e2-standard-8 VM with 8 vCPUs, approximately 32 GiB RAM and no GPU, using Ubuntu 24.04. RELION reported version 5.1.0. The existing working environment was cloned to a versioned Filestore prefix and validated by executing compute binaries and checking dynamic-library resolution.

Scientific inputs came from the official RELION 5 beta-galactosidase tutorial precalculated-results archive. Its downloaded size was 11,326,844,944 bytes. The received archive SHA256 was 7f799e1ad6c366722313a48a4e3f35d47311c896e92b985e78594bf6675711d3. This checksum identifies the received data; it is not a publisher signature.

The chosen extraction checkpoint contained 2,374 particles with 64-pixel images and a pixel size of 3.54 angstrom. The pilot used disjoint 1,024-particle calibration and held-out subsets, retaining optics and microscope modulation-transfer metadata. Preprocessing rescaled to 64 pixels and normalized the background with radius 28 pixels. Classification used CTF handling, 10 classes, 15 iterations, a 200-angstrom particle diameter and a 30-degree base angular step.

Forty execution attempts were retained: one failed setup smoke, three successful smoke runs, one successful calibration check, thirty paired pilot runs and five held-out runs. The initial failure was a harness output-directory error and was corrected before the pilot; it is not concealed or counted as a performance result. All 35 pilot and held-out executions completed successfully.

The experiment source snapshot was b916bf646366ea22adc1447d51f8bc3f2258ec6c. That is the code used for source replay, not a claim about the current deployed application revision. Encyclopedia development was isolated on its own branch from a newer main revision, and release verification must record the actual deployed commit separately.

Each run has an immutable folder containing the record, full command, stage logs, sampled CPU and memory/I/O observations, elapsed and child CPU time, peak RSS, final STAR metadata, class images and output hashes. Persistent software, data and evidence are stored separately from local scratch.

Measured pilot results

Every one of the fifteen comparison pairs passed the strict numerical-equivalence checks. The median baseline-to-candidate runtime ratio was 2.828 for one to four threads and 1.149 for four to eight threads. More threads helped in this configuration, but the benefit diminished substantially between four and eight.

Moving the working directory from local disk to Filestore produced a ratio of 0.434: the Filestore variant took about 2.30 times as long. The software binaries remained on Filestore in both variants. A practical candidate for a similar workload is persistent software and retained results on Filestore, with active working files on local scratch. Validate that choice with the actual data size, durability requirements and failure recovery; this is not a universal NFS benchmark.

The held-out runs took 12.23 to 12.77 seconds. The matched calibration-rate calculation predicted 12.89 seconds, with median absolute percentage error 4.54 percent across five seeds. The broad refinement seed predicted 368.64 seconds for the same particle count and thread allocation. That large discrepancy is partly a workload-scope mismatch: a short 2D classification is not full refinement. It must not be advertised as a general model accuracy improvement.

The intervals below are percentile-bootstrap descriptions from only five paired seeds on one dataset and machine. They do not establish generalization to other biological samples, machine families or workflow stages. Raw execution evidence remains the authority for these numbers.

No billing export was used. The same VM was retained throughout, and reducing job time does not automatically reduce the invoice unless the saved capacity is released or productively reused. No trained model was promoted by this campaign.

Evidence categories and interpretation

A credible answer identifies where each important claim comes from. A source label on an answer is useful, but it does not independently prove that the cited measurements support the proposed action. Validate the relationship between the input evidence, calculation and resulting recommendation.

The source replay passed nine of twelve initial controls. It confirmed seed fallback, dependence on observed runtime, rejection of unrelated workload records and zero durations, manifest provenance and precedence of direct measurements. Three additional campaign controls exposed missing defensive filtering at the aggregation boundary for failed, explicitly synthetic and stale records. Normal cluster and batch completion paths already filter failed jobs, so this is not proof that failed runs currently enter through those paths.

Synthetic controls stay outside the shared measurement store. The real pilot records also stay scoped to this campaign because the current broad cryo-EM aggregation does not distinguish classification from refinement, software versions, image box sizes or iteration counts. Publishing this short-job rate as general refinement throughput would make later answers misleading.

How the AI assistant should answer

For relevant RELION, workflow, telemetry or optimization questions, the server retrieves the Encyclopedia reference before generation and records that retrieval in answer provenance. The assistant also has a read-only tool for specific sections. Updating the article changes its content hash and therefore the cache namespace for answers that received the reference.

The assistant should establish the scientific stage and parameter scope, retrieve current measurements when making current sizing claims, identify supporting run IDs or the historical pilot, explain the proposed operational change, state uncertainty, and describe how the user can verify the outcome. It must distinguish observations from predictions and documentation from live project state.

An appropriate historical example is: In the documented 1,024-particle RELION CPU pilot, four threads were about 2.83 times faster than one, with the paired numerical checks passing. Your full refinement may behave differently. I need matching stage, box size and iteration details, then a baseline measurement, before predicting your runtime. This is an answer pattern, not a captured live assistant reply.

A request about current jobs still requires the live job or instance tools. A cost claim requires current pricing or billing evidence with its basis stated. An Encyclopedia lookup does not establish that a job is running or that a user has saved money. Documentation must never be used to bypass existing action authorization.

The same workflow optimization code should remain the numerical authority behind both API and assistant answers. Do not create a second set of untested LLM formulas. The next engineering step is parameter-aware measurement matching and a structured evidence response with matched runs, rejected evidence, calculation version and validation status.

Evidence contract and data governance

Every new measurement should include an execution ID; start and finish timestamps; completion status and exit code; application and version; workflow stage and dependency identity; dataset fingerprint; particle or other work quantity and unit; scientific parameter fingerprint; allocated resources and physical machine; filesystem; elapsed and CPU time; memory peak; and output-validation status. Keep allocated threads distinct from the eight-vCPU node that was billed.

A production matcher should select comparable records by stage, version, data dimensions, parameters, hardware path and execution mode. It should explain when too few comparable records exist and fall back with an honest label. Freshness should depend on changes to software and execution environment, with an explicit policy rather than an arbitrary universal expiration.

Separate synthetic controls from real execution data at ingestion and storage boundaries. Retain failed jobs for diagnosis while excluding them from successful-throughput calibration. Preserve provenance through derived summaries and trained models. Model promotion needs a held-out comparison, correctness and regression gates, and a rollback reference; the number of training rows alone is insufficient.

Encyclopedia publication should contain reviewed aggregate results and reproducible methods, not credentials or private scientific data. Raw campaign records remain in controlled storage. Topic revisions should retain the evidence source, author/reviewer accountability and date, and changes to claims should be traceable to new evidence.

Operational runbook

The persistent prefix is /gcfs-software/numen-relion-testing/software/relion-5.1.0-linux-x86_64. The activation script prepends its bin directory. Reusing it on a future VM requires the same mount path, a compatible Linux x86_64 environment and the remaining operating-system libraries. Persistence of a directory is not a guarantee of compatibility with every image.

The campaign root separates software, data, frozen source, harness, runs and evidence. Use a new campaign name for each repetition; existing folders are not overwritten. Finish installation and transfer activity before timing. Serialize scientific runs when comparing settings on a single node.

Local scratch can improve this measured workload, but its lifecycle must be understood. Copy verified final results and logs to persistent storage before releasing a VM. The pilot preserves final evidence on Filestore and does not delete the user-owned VM or existing filesystem.

To reproduce the pilot, use the retained dataset manifests, explicit package list and script hashes. Read RESULTS.md for the concise outcome and analysis.json for pair-level ratios and numerical checks. Inspect the original failed smoke record when reviewing setup reliability.

source /gcfs-software/numen-relion-testing/software/activate.sh
relion_refine --version

ROOT=/gcfs-software/numen-relion-testing
sudo "$ROOT/software/relion-5.1.0-linux-x86_64/bin/python" \
  "$ROOT/harness/run-campaign.py" --mode pilot --name pilot-v2

Limitations and open findings

This is one CPU machine, one tutorial dataset, small images and short 2D classifications. It cannot establish GPU performance, complete reconstruction quality, cross-machine scaling, production cost savings or generalization across datasets. The held-out particles share an acquisition with calibration particles.

The current broad workload-class rate can combine unlike scientific work. The engine and assistant need finer matching before this pilot can safely inform arbitrary cryo-EM sizing. The current memory estimate is not learned from these RSS measurements.

Planner replay validates the numerical path with isolated storage inputs. It does not prove a live LLM followed the explanation protocol. Assistant integration needs separate tests for source retrieval, content-version cache invalidation, live-tool use and preservation of evidence qualifiers.

Native RELION on-the-fly Schemes, restart semantics, incremental arrival and automatic dependency invalidation are not established by this pilot. Documentation of those features supplies test cases; it is not execution evidence.

Expansion plan and expected effort

The next stage uses the RELION 5 on-the-fly documentation to test growing inputs, idle intervals, backlog pressure, graceful restart and parameter invalidation. The raw tutorial archive linked by the release-5 documentation is named relion30_tutorial_data.tar; that shared filename does not mean the software should be downgraded.

First validate native Scheme prerequisites and one bounded arrival/restart case. Then run three seeds per scenario, increasing to five after evidence collection is reliable. Keep arrival delays distinct from compute runtime, and compare whole workflow output inventories to detect dropped or repeated work.

For recommendation validation, expand independent workflow configurations and whole-movie holdouts before repeating identical prompts. A proposed next matrix is six workflow conditions, three parameter strata and five seeds, with repetitions chosen after the pilot variance is reviewed. This is a planning budget, not completed work.

Engineering estimates are approximately two to four working days for parameter-aware ingestion and matching, two to four for native incremental/restart workflow tests, and one to two for role-based assistant evaluation and release evidence. These are estimates, may overlap, and depend on available Scheme dependencies and test access. A wider benchmark across datasets and machine families needs a separate compute budget.

Acceptance requires accurate evidence labels, comparable-record selection, reproducible performance measurements, scientific validation appropriate to the workflow, and no regression in permissions or action safety. A product claim should only move from planned to measured when its evidence is linked and reviewed.

Glossary

References

• RELION 5 single particle tutorial
• RELION 5 on the fly processing
• Pilot aggregate evidence