From Qwenputer to "qwentainer": ideas from running models as containers

[JGalego]

Hi everyone 👋

I'm the author of shem, an experimental runtime that runs language models as OCI containers. After I wrote about it, someone pointed me at ModelPack, and fairly so. Having read the spec, I think the two projects start from different places and could learn from each other, so here are a few ideas.

Where shem comes from

Qwenputer compiles a RISC-V CPU into Qwen3.5-4B and uses the model's Gated DeltaNet state as RAM: a model that is a computer. shem asks the next question. If a model can be a computer, can it be a container ("qwentainer")?

The rule I tried to stay true to is "layers are layers". Weights, prompts, conversation state and even a compiled CPU are all ordinary image layers:

• FROM hf.co/Qwen/Qwen3-0.6B imports a checkpoint, one layer per transformer block.
• ENTRYPOINT names the generation driver, and CMD is the default prompt.
• WARM --until=prompt primes a session at build time and freezes its KV cache and recurrent state into a layer.
• A running container is a live session: docker exec adds a turn, and docker commit saves the conversation as a new image.
• COPY --from=shem:rv32im adds a compiled RV32IM CPU as a layer, and the model calls programs on it as tools.

ModelPack deliberately makes the opposite choice: it handles model artifacts as opaque binaries for an inference server to load. That's the right default for a standard. shem wandered off in a few directions that might still be useful here.

1. State as an artifact

Containers spread because a process's world could be frozen and shipped. A model's computation has a world too: prompt caches, primed system prompts, saved conversations. Today ModelPack has no place for it.

A state artifact could point at its model through OCI's subject field, so a registry's referrers API lists the states available for a model (media type names are placeholders):

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "artifactType": "application/vnd.cncf.model.state.manifest.v1+json",
  "subject": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "digest": "sha256:<the model artifact's manifest>",
    "size": 1234
  },
  "config": {
    "mediaType": "application/vnd.cncf.model.state.config.v1+json",
    "digest": "sha256:...",
    "size": 321
  },
  "layers": [
    { "mediaType": "application/vnd.cncf.model.state.v1.tar", "digest": "sha256:...", "size": 52428800 }
  ]
}

The config would say what a runtime needs to decide whether it can load the state: a format (such as vllm, llama.cpp or shem.session/v1), the tokens the state has read, and its precision.

• Cache layouts are engine-specific (vLLM's paged blocks, llama.cpp's slot files, shem's per-block safetensors), so the spec would standardise the envelope and the link to the model, not the tensors.
• The promise such an artifact has to keep is that loading it changes nothing but speed. shem tests exactly that: a warm start gives the same reply as a cold one.
• It fits the "several profiles per model" direction in Should ModelPack standardize a Modelfile format? #208: runtime profiles and states both belong to a model without being part of it. Enhance Model Spec to Distinguish Between Base Model and LoRA Adaptor Components #60 asked for a parent link from LoRA adapters to their base model, and subject would give that a standard shape too.

2. An image-compatible profile

I tested how Docker 29.8, with its built-in BuildKit, treats a ModelPack artifact (one weights tar and one config tar) pushed to a local registry:

Attempt	Result
FROM <artifact>	encountered unknown type application/vnd.cncf.model.config.v1+json
COPY --from=<artifact> / /model/	The same error
docker pull, then docker create	The container is created, but the weights aren't in its filesystem

A ModelPack .tar weight layer is already a valid filesystem layer, though. When I pushed an ordinary image manifest and image config pointing at the same blobs, COPY --from worked and the files appeared, with nothing uploaded twice.

So a model could be published both ways, as a ModelPack artifact and as an image that shares its layer blobs, linked through an index or as a referrer. That would make every ModelPack model buildable with a Dockerfile today. It would also give image-volume mounts a path that doesn't wait on runtime support for artifacts, which #233 tracks (containerd and moby are still open).

Open questions: where files land (the org.cncf.model.filepath annotation's path, or a /model prefix), which platform the image claims, and what to do with .raw layers, which would need wrapping in a tar.

3. Layers that follow the model's structure

ModelPack splits weights wherever the checkpoint was sharded. shem's importer splits along the model's own structure instead: a manifest, shem.json, describes the model as a residual stream of typed blocks (attention, gated_deltanet, mlp, each with its parameters), and each block's weights are their own layer. Qwen3-0.6B becomes 59 layers.

Layers that match blocks become units of meaning. A pipeline stage can pull only its blocks, variants can share the blocks they didn't change, and a model can be composed from parts, trained or compiled. #164 proposed a Transformer specification built from standardised blocks; shem.json is a small working example of that idea. Even a layer annotation saying which blocks or tensors a layer holds would help.

One caveat from practice: overlay filesystems stack about 128 layers, so shem groups neighbouring blocks once a model has more than 95 weights files. That limit would apply to image-volume mounts too.

4. Equivalence as provenance

shem's ground rule is that a reference implementation defines what's correct, and everything else must match it. The importer is checked against 🤗 transformers (logits within 1e-3, the same greedy tokens), and the compiled CPU against an interpreter, instruction by instruction.

Conversions and quantisations change every byte while claiming to be the same model. An attestation attached as a referrer, recording what was compared, against what and within which tolerance, would let a registry show not only where a model came from but that it still behaves like its source. This is probably a convention rather than part of the spec.

A data point on compression

The spec recommends uncompressed weights, since they're typically incompressible. shem compresses every layer with zstd at level 3. On Qwen3-0.6B, which is almost entirely bf16 weights, that shrinks the image from 1,204 MB to 935 MB, 22% smaller. I'm not arguing to change the recommendation, since being able to mount weights directly matters more to many users, but bf16 seems to compress more than "incompressible" suggests, so the trade-off may be worth spelling out.

Going the other way

shem has more to learn from ModelPack than the reverse. Its format is one tool's private convention, and it imports only two model families. I'd like shem to accept a ModelPack artifact as a FROM base next to hf.co/..., and to use the model config's vocabulary (family, precision, parameter count, licences) in its labels.

Questions

1. Is model state (prompt caches, primed sessions) in scope for ModelPack, or better as a separate spec that references it?
2. Would an image-compatible profile be welcome, or does it blur the line between artifacts and images too much?
3. Is there interest in picking Transformer specification and auto-generation method for the existing models #164 back up, with block-aligned layers as a concrete use?

I'm happy to prototype any of these against shem. The architecture and model images docs have the details.