[afonsobspinto]

Overview

Adds batch ingestion of an SWC ZIP into an existing CATMAID tracing project, including projects that already contain skeletons. Project, stack, permissions and cache-grid provisioning happen before ingestion.

The importer validates SWCs once, saves bounded batch snapshots, loads whole-skeleton batches atomically, rebuilds existing grid caches once, and independently verifies the imported data and caches.

The single-pod workflow locks the target project before downloading and retains that protection through preparation, loading, cache rebuilding and verification. The standalone local-ZIP CLI acquires its lock at run.

Matching Neuroglass middleware blocks mutations while reads remain available. Other projects remain usable.

Supports MNP #1452.

Workflow sequence

Successful workflow used by MNP. Failure handling is described below.

flowchart TD
    Lock["Acquire project lock"] --> Download["Download archive"]
    Download --> Preflight["Preflight: capture target metadata"]
    Preflight --> Plan["Plan: validate SWCs and save batch snapshots"]
    Plan --> Run["Run using existing project lock"]
    Run --> Maintenance["Database maintenance"]
    Maintenance --> Cache["Rebuild existing caches"]
    Cache --> Verify["Verify imported data and caches"]
    Verify --> Unlock["Release project lock"]
    Unlock --> Report["Report outcome"]

Loading

The run command includes batch loading, maintenance, cache rebuilding, verification and project-lock release.

Standalone interface and input contract

From a configured CATMAID Python environment:

catmaid-skeleton-import preflight \
  --request /workspace/request.json \
  --state-dir /workspace/state

catmaid-skeleton-import plan \
  --request /workspace/request.json \
  --state-dir /workspace/state

catmaid-skeleton-import run \
  --state-dir /workspace/state

For this standalone path, preflight and planning do not acquire the project lock. run acquires it before batch preparation.

Preflight captures associated stack geometry and database identity. Planning reads that local snapshot without initializing Django or accessing the database.

Request:

{
  "schema_version": 1,
  "source": {
    "archive_path": "/workspace/skeletons.zip",
    "coordinate_unit": "um"
  },
  "target": {
    "project_id": 123
  }
}

Unknown fields are rejected. The importer does not create projects, stacks or grid definitions, or repair tracing setup.

Example ZIP:

0021814.swc
specimen-02/
  1054599727.23.SWC

All regular .swc files are imported, case-insensitively, from the root or nested directories. Other files are ignored. No input manifest or external-ID CSV is required.

SWCs use seven fields:

# id type x y z radius parent_id
1 1 10 20 30 2 -1
2 3 11 20 30 0.5 1

Blank lines, full-line comments and trailing # comments are supported. Each file must contain a connected, acyclic tree with unique node IDs and one root. Single-node SWCs are supported.

Planning collects invalid-member errors and publishes no executable plan if any SWC is invalid. Bounded summaries reference complete diagnostics in error-details.json.

Units and spatial bounds

CATMAID stores coordinates and radii in nanometres:

• um multiplies XYZ and known radii by 1000.
• nm preserves their values.
• Radius -1 remains unknown; parent -1 identifies the root.

The caller must establish the source units. The importer does not infer units or register, translate, rotate or clip SWCs.

Every incoming point must fall within at least one associated stack’s project-space bounds, accounting for orientation and translation. Negative coordinates are accepted within those bounds; gaps between stacks are outside their coverage.

Stack geometry and database identity are rechecked before loading and during final verification.

Project lock

Migration 0127_project_active_skeleton_ingestion adds the nullable UUID field Project.active_skeleton_ingestion to the live project and its CATMAID history table/view.

The matching MNP middleware wraps Neuroglass mutations in project_write_guard(). Shared project-row locks allow normal edits to run concurrently. Ingestion acquisition waits for existing guarded edits to finish, then commits the ingestion UUID as the durable owner.

Further guarded mutations receive HTTP 423 / ProjectIngestionLocked. GET and POST-based reads remain available. Permissions and layer visibility stay unchanged.

The lock is specific to the target project. This integration covers Neuroglass access-token/task-association requests. Native CATMAID clients and arbitrary database/background writers must still avoid editing the target.

The project lock identifies the owning ingestion and blocks guarded Neuroglass mutations throughout processing.

Each ingestion UUID has one fresh workspace and one run attempt. The workflow runs preflight, plan and run sequentially. Concurrent commands against the same state directory are unsupported; there is no filesystem workspace lock.

No database transaction spans the full workflow.

One ingestion UUID

MNP supplies the ingestion UUID through SKELETON_INGESTION_ID. Standalone planning generates one when none is supplied.

The same UUID identifies the import and owns the project lock. A supplied run-time UUID must match the planned identity.

After planning, identity and the run’s lock observations are retained together in state/state.json:

{
  "ingestion_id": "11111111-2222-4333-8444-555555555555",
  "run": {
    "status": "pending"
  },
  "project_lock": {
    "project_id": 123,
    "status": "pending"
  }
}

These statuses record what the run subprocess has confirmed. In the workflow, pending means that run has not yet confirmed ownership of the existing project lock.

State observation	Meaning
pending	Run has not begun its ownership confirmation or acquisition
acquiring	Standalone acquisition or workflow ownership confirmation is being attempted
locked	Run confirmed ownership
released	Ownership-checked release was confirmed

There is no separate lock-owner UUID or lock receipt file.

Completion and failure handling

Successful workflow finalization requires a completed run, committed batches, completed final phases, valid verification artifacts, and a released lock for the matching ingestion.

Condition	Handling
Verified completion	Release the project lock and report success
Confirmed download, preflight or planning failure	Release the project lock and report failure
Confirmed run startup failure with every batch still planned	Release the project lock and report failure
Failure after a batch becomes prepared, failed or committed	Retain the project lock and require operator attention
Uncertain acquisition/release or missing execution evidence	Preserve the current lock state and require operator attention

After a safe wrapper startup failure, state can legitimately contain:

run.status = pending
project_lock.status = released
all batches = planned
all final phases = pending

This records safe cleanup before loading. Finalization reports failed, allowing a fresh request and workspace. It does not resume the previous attempt.

Locks have no expiry. Acquisition refuses an existing owner, and release clears only the matching ingestion UUID.

For manual resolution, first confirm the importer has stopped and reconcile retained state and assigned IDs against PostgreSQL. Resolve any partial import before using CATMAID’s Django shell:

from catmaid.ingestion_lock import release_project_ingestion

release_project_ingestion(project_id, ingestion_id)

Use the retained state’s project and ingestion UUID. If planning never published state, use the workflow’s pinned target and UUID and compare them with the project lock.

Migration rollback refuses to remove a live lock.

Reads can observe intermediate batch commits during ingestion. Refresh open viewers after successful completion.

Batch loading, provenance and caches

• Members are ordered deterministically and grouped by node count. A skeleton is never split between batches.
• Fresh IDs come from PostgreSQL sequences; contiguous ranges are not assumed.
• COPY loads class_instance, class_instance_class_instance and treenode.
• Ordinary triggers are bypassed only during COPY in that batch transaction. Other sessions retain normal trigger behavior.
• Triggers are restored before explicit treenode_edge creation and selective skeleton-summary refresh.
• Core rows, provenance, edges, summaries and batch validation commit together.
• After all batches, ANALYZE refreshes statistics on the five affected tables.
• Existing grid definitions and settings are preserved. Payloads are rebuilt once from existing and newly imported project data.
• The importer’s cache rebuild has no node cap.
• Final verification compares imported IDs, stored node values, topology, relationships, ownership, provenance and summaries against the plan and snapshots. It also checks cache definitions, formats, freshness and dirty markers.

Migration 0126_fix_single_node_selective_summary corrects CATMAID’s selective summary refresh for singleton skeletons, producing one node and zero cable length.

Fragment provenance

Each SWC creates an importer-owned neuron annotation:

folder/0021814.swc → source-fragment:0021814

Fragment IDs use the filename basename without its SWC extension, with surrounding whitespace trimmed. Leading zeros and case are preserved. IDs must be nonempty and at most 239 characters.

Collisions within the ZIP or against existing target-project provenance are rejected. Folder names do not provide a namespace. Existing fragments are not skipped, replaced or merged automatically.

CATMAID’s structural APIs maintain provenance:

• Joins retain the union of source fragments.
• Splits retain the complete inherited source set on both results.
• Ordinary annotation editing cannot create, rename, remove or replace protected provenance.
• Complete deletion removes unused provenance.

Join/split neuron locks serialize structural metadata changes. SKELETON_EDIT_LOCK_TIMEOUT_MS defaults to 1000 ms; longer contention returns an error.

Batch snapshots and audit mappings

Validated SWC node rows are retained in deterministic, hash-checked files:

batch-snapshot-000001.jsonl.gz
batch-snapshot-000002.jsonl.gz

Loading and final verification reuse these snapshots. The reader retains one batch’s node geometry at a time and releases the previous batch before decoding the next. Plan/member metadata remains in memory.

The original ZIP is required through successful planning; afterward, it can be archived or removed.

Output manifest header:

member_path,fragment_id,neuron_id,skeleton_id,node_count,name,batch_index

Per-batch node-map header:

member_path,source_node_id,treenode_id

Maps are stored as batches/000001/node-map.csv.gz. They record assigned IDs; only a confirmed batch commit establishes that those rows were imported.

No mapping database table or treenode labels are added. Review-queue/microtask integration remains follow-up work.

Single-pod ingestion workflow

Install the workflow extras in a configured CATMAID environment:

python -m pip install './scripts/catmaid_skeleton_batch_import[workflow]'
python -m catmaid_skeleton_batch_import.workflow

The workflow needs database configuration from the start because it acquires the project lock before downloading.

Supported remote sources are public AWS S3 objects, presigned S3 GET URLs and signed GET URLs on official GCS endpoints. Signed credentials are encrypted and bound to the ingestion UUID. Real signed GCS ingestion remains untested.

Transient transfers have a three-attempt budget. Database imports are single-attempt: no automatic import retry, resume, reconciliation or deletion of imported rows is provided.

Logs, snapshots, manifests, node maps and detailed errors remain until explicit cleanup. Earlier batch commits remain after a later failure.

Progress and terminal reporting

CATMAID’s workflow helpers encode stage observations into Argo’s native ARGO_PROGRESS_FILE. MNP consumes native Argo status and exposes ingestion progress to Neuroglass.

flowchart LR
    Importer["CATMAID importer"] --> State["state.json and cache-progress.json"]
    State --> Bridge["CATMAID workflow progress bridge"]
    Download["Archive download bytes"] --> Bridge
    Bridge --> Progress["ARGO_PROGRESS_FILE"]
    Progress --> Argo["Argo Workflow and WorkflowTaskResult"]
    Argo --> API["MNP ingestion status API"]
    API --> Client["Neuroglass"]

    Result["terminal-result.json"] --> Output["Argo output parameter"]
    Output --> Notifier["MNP exit notifier"]
    Notifier --> Kafka["Kafka"]
    Kafka --> Outcome["MNP durable ingestion outcome"]
    Outcome --> Client

Loading

The five encoded stages remain:

Stage	Reported detail
Download	Downloaded bytes when the total is known
Database loading	Confirmed committed batches
Database maintenance	Phase status
Cache rebuild	Processed/planned cells, including evaluated empty cells
Final verification	Phase status

Early lock acquisition, preflight and planning do not add separate encoded stages or percentages.

Argo accepts one N/M ratio. The stages-v1 convention preserves the stage and exact counters:

width = total + 1
N = (stage - 1) * width + completed
M = 5 * width

For example, loading stage 2 with one of two batches committed produces 4/15. MNP decodes this as stage 2/5, import 1/2.

The ratio is not an overall percentage or ETA. N=M means awaiting terminal reporting; only the accepted terminal result establishes success.

terminal-result.json contains a bounded result with workflow identity, verified counts or an error. MNP exposes it as an Argo output parameter so the separate exit notifier needs no ingestion-volume mount.

Only processing pods mount the retained RWO claim. CloudHarness’s built-in affinity places pods sharing that claim on the same node, allowing different projects to ingest concurrently when node resources permit. Each ingestion writes to its own UUID directory; there is no shared workflow mutex. The per-project lock still prevents competing ingestions and NeuroGlass mutations on the same project, while layer visibility remains unchanged. The exit notifier does not mount the claim. Node maps remain on disk and are not included in terminal reporting.

CATMAID has no Kafka or Neuroglass HTTP callback dependency. MNP owns notification and durable study outcome handling.

Default settings

Planning records these four ImportSettings values:

Environment variable	Default
CATMAID_BATCH_IMPORT_MAX_NODES_PER_BATCH	250000
CATMAID_BATCH_IMPORT_MAX_ARCHIVE_BYTES	2147483648 — 2 GiB
CATMAID_BATCH_IMPORT_MAX_SWC_BYTES	2147483648 uncompressed
CATMAID_BATCH_IMPORT_MAX_TOTAL_NODES	None — unlimited

Database timeouts:

Environment variable	Default
CATMAID_BATCH_IMPORT_STATEMENT_TIMEOUT_MS	0 — unlimited
CATMAID_BATCH_IMPORT_LOCK_TIMEOUT_MS	30000

The lock timeout also applies to early workflow acquisition. Settings use environment variables and defaults. There is no SWC-count cap or cache-creation configuration.

[seankmartin]

Thanks for this! I decided to test this from https://github.com/openbraininstitute/NeuroM/tree/master/tests/data/swc, I removed the subdirs so I could compare with https://github.com/MetaCell/neuroglancer/tree/feat/swc-support-continued which currently only supports flat dirs. This is a useful one to check as some SWCs purposely have errors.

The setup json seems pretty reasonable, maybe a little bit more in the README about the effect of the stack dimensions could be useful. On my first plan run I got this:

{"command":"plan","error":{"category":"invalid_input","details":{},"message":"skeletons.zip:swc/Neuron_disconnected_components.swc: expected exactly one root, found 5"},"importer_version":"0.5.0","ok":false,"schema_version":1,"state_dir":"/home/starfish/Repos/catmaid/local/state"}

Which is fine, and makes sense. Though I think it would be if possible a bit more useful if the plan could list all the problematic SWC files with the reason, not just stop at the first one with an issue.

I will continue checking and give feedback, in the meantime I asked claude to check as well and will share an artifact which I will review later https://claude.ai/artifact/C96DPEaJ7jeQDE64S1b8LK?sk=BT9c9_JK6F5zhAPeDbLkvg

[seankmartin]

It's unclear from the spec if inline comments are supported or only full line comments, but in example SWCs I have seen some inline comments like this:

# A cell with a bifurcation right at the root
1    1    0 0 0 1  -1
2    3    1 0 0 1  1 # <-- bifurcation right at the root
3    3    1 1 0 1  2
4    3    1 0 1 1  2

This currently gets rejected because it thinks there are 14 columns due to the inline comment. We should probably add to our SWC parser to strip inline comments

[seankmartin]

We currently don't support single node SWC files, with the error

skeletons.zip:swc/point_soma.swc: one-node SWCs are not supported by CATMAID selective summaries
{"command":"plan","error":{"category":"invalid_input","details":{},"message":"skeletons.zip:swc/point_soma.swc: one-node SWCs are not supported by CATMAID selective summaries"},"importer_version":"0.5.0","ok":false,"schema_version":1,"state_dir":"/home/starfish/Repos/catmaid/local/state"}

Catmaid supports such skeletons, but I guess the summary SQL doesn't. I don't feel strongly on the resolution but I would guess it could be surprising if someone tries to ingest such an SWC and gets rejected

[seankmartin]

Will bring this up here briefly but can discuss more separately. I wanted to mention that I think we have a bit of a flaw somewhere. My suspicion is that it is here, but it could also be in neuroglancer itself. The SWC format allows negative values for positions. And catmaid can store negative positions. Neuroglancer exclusively uses the bounding box for spatial index fetches, so with negative values on the 0 to positive stack dimensions you get

While if I set all the segments to visible you can see where they extend to

I wonder whether or not the plan stage should check or have an option to check that the SWCs are within the stack bounds, as otherwise neuroglancer spatial fetches won't work correctly. I'm also up for thinking if neuroglancer should instead change.

[seankmartin]

The last thing that comes to mind is if in the batch folders that are already written, if we should keep a mapping between the nodeID in the SWC and the catmaid treenodeID. I think ideally we should instead label a limited subset of treenodes at review queue import time. But in the interim this could potentially allow us to later attach review queue items to an already ingested set of data, assuming the fragments have not been heavily edited

[seankmartin]

I approve on the whole thanks for this. I think we could improve the planning step and probably nice to support SWC files with one node and inline comments, left some notes about that, but I don't see it as blocking and could do in a follow up PR. I'll leave it to you how much you'd like to take from that now versus we can note for later.

From the claude review, I think we should likely fix "Joins and splits in every project now fail fast on a neuron row lock" if that holds true as that's a decently large behaviour change. And "The cache rebuild walks the whole bounding box and restarts from zero" could help us improve the cache rebuild efficiency, and I think perhaps points more in favour of some form of checking the SWC co-ordinates against the stack dimensions at some stage.

It may also be worth to track the SWC node ID association with catmaid treenode ID until the review queue is ready.

[seankmartin]

Thanks for the changes in the last round

[afonsobspinto]

Thanks for the changes in the last round

Thanks for testing this, Sean. As a recap of what I changed based on your feedback:

1. Units: The README now explicitly states that CATMAID stores coordinates and radii in nanometres, which is why we convert the declared source units.
2. Stack dimensions: Added an explanation of dimension × resolution_nm, with an example showing the resulting physical extent
3. Reporting invalid SWCs: Planning now collects errors across files and reports one useful reason per invalid member in error.details.member_errors. It produces no executable plan if any SWC is invalid.
4. Inline comments: The parser now strips trailing # comments before validating the seven fields.
5. Single-node SWCs: These are now supported. And there's PR#10 to later create a PR against upstream with the fix
6. Coordinates and spatial discovery: Planning now validates coordinates against the stack’s physical bounds. For appends, negative XYZ values are allowed when the point lies within the associated stack’s bounds, accounting for translation and orientation. A separate read-only preflight step captures this metadata, keeping planning database-free; New-project imports still require nonnegative coordinates because their stacks start at zero.
7. SWC-node → CATMAID-treenode mapping: Implemented the interim strategy you suggested. Each batch writes node-map.csv.gz with member_path,source_node_id,treenode_id. Will create a card for labeling selected treenodes when importing review queues remains a later integration
8. Join/split lock contention: Replaced immediate failure on the neuron lock with a short configurable wait: SKELETON_EDIT_LOCK_TIMEOUT_MS, defaulting to 1000 ms. Longer contention still returns a retry error.
9. Cache rebuild efficiency and recovery: Didn't address right now as I see this more like a nice to have, will create a card for it.

A few further updates since my last comment:

• The importer is now append-only: project, stack and grid provisioning happen beforehand, while existing data and cache settings are preserved.
• Simplified execution to one attempt, with no retries or resume path. Failures surface immediately; previously committed batches remain for inspection.
• Planning now saves validated batch snapshots, reused during loading and final verification. The original ZIP is no longer needed after planning succeeds.
• Added cache-cell progress reporting and removed duplicate cache verification.
• Removed the SWC file-count cap, keeping byte limits and batch node budgets.
• Fixed a floating-point comparison that could incorrectly reject valid imports, and reduced snapshot memory usage and timing-record write overhead.

[afonsobspinto]

Thanks for the changes in the last round

Thanks for testing this, Sean. As a recap of what I changed based on your feedback:

1. Units: The README now explicitly states that CATMAID stores coordinates and radii in nanometres, which is why we convert the declared source units.
2. Stack dimensions: Added an explanation of dimension × resolution_nm, with an example showing the resulting physical extent
3. Reporting invalid SWCs: Planning now collects errors across files and reports one useful reason per invalid member in error.details.member_errors. It produces no executable plan if any SWC is invalid.
4. Inline comments: The parser now strips trailing # comments before validating the seven fields.
5. Single-node SWCs: These are now supported. And there's PR#10 to later create a PR against upstream with the fix
6. Coordinates and spatial discovery: Planning now validates coordinates against the stack’s physical bounds. For appends, negative XYZ values are allowed when the point lies within the associated stack’s bounds, accounting for translation and orientation. A separate read-only preflight step captures this metadata, keeping planning database-free; New-project imports still require nonnegative coordinates because their stacks start at zero.
7. SWC-node → CATMAID-treenode mapping: Implemented the interim strategy you suggested. Each batch writes node-map.csv.gz with member_path,source_node_id,treenode_id. Will create a card for labeling selected treenodes when importing review queues remains a later integration
8. Join/split lock contention: Replaced immediate failure on the neuron lock with a short configurable wait: SKELETON_EDIT_LOCK_TIMEOUT_MS, defaulting to 1000 ms. Longer contention still returns a retry error.
9. Cache rebuild efficiency and recovery: Didn't address right now as I see this more like a nice to have, will create a card for it.

A few further updates since my last comment:

• The importer is now append-only: project, stack and grid provisioning happen beforehand, while existing data and cache settings are preserved.
• Simplified execution to one attempt, with no retries or resume path. Failures surface immediately; previously committed batches remain for inspection.
• Planning now saves validated batch snapshots, reused during loading and final verification. The original ZIP is no longer needed after planning succeeds.
• Added cache-cell progress reporting and removed duplicate cache verification.
• Removed the SWC file-count cap, keeping byte limits and batch node budgets.
• Fixed a floating-point comparison that could incorrectly reject valid imports, and reduced snapshot memory usage and timing-record write overhead.

A few further updates since the above:

• Single-pod workflow: Added an entry point that handles remote download, preflight, planning and ingestion in one processing pod, for integration with Neuroglass. Database ingestion remains single-attempt; only transient download failures have a bounded retry budget.
• Project write lock: Migration 0127 adds durable project ownership using the ingestion UUID. The workflow acquires it before downloading and retains it through cache rebuilding and verification. Matching Neuroglass middleware rejects mutations with HTTP 423 while reads and other projects remain available. Native CATMAID clients and direct database writers remain outside this guard.
• Progress integration: Added native Argo reporting for MNP’s ingestion-status API, covering download bytes, committed batches, cache cells and maintenance/verification stage indicators. Progress is separate from the validated terminal result that establishes the final outcome.