Adding a new model to an example package

This assumes the setup guide is done. A package’s model list is one YAML file – swapping the model chat uses is an edit to that file, nothing else.

Where it lives

marigold-examples/chat/models.yaml

Whatever’s declared there is what cache-init downloads and the worker serves, the next time you run:

marigold deployment start marigold-examples/chat

Swap in a newer instruct model

Qwen/Qwen3.5-9B is ungated – no token needed:

models:
  - name: Qwen/Qwen3.5-9B
    provider: huggingface
    type: instruct
    input: chat
    output: chat
    extra_env:
      LOAD_IN_4BIT: "1"
    description: >
      9B parameter instruct model.

Check it loads cleanly before touching Docker at all:

marigold cache validate marigold-examples/chat/models.yaml

Then start (or restart) the deployment as usual.

A larger, more capable model

google/gemma-4-E4B-it is also ungated – Gemma 4 is licensed Apache 2.0, unlike earlier Gemma generations. Same file, same process:

models:
  - name: google/gemma-4-E4B-it
    provider: huggingface
    type: instruct
    input: chat
    output: chat
    extra_env:
      LOAD_IN_4BIT: "1"
    description: >
      Larger multimodal instruct model.

Using a gated model

Some models require accepting terms on HuggingFace before they’re downloadable at all – google/gemma-2-9b-it is one. That needs a HuggingFace access token.

1. Visit the model’s page on HuggingFace and accept the license terms, if you haven’t already.
2. Generate a token at huggingface.co/settings/tokens – a read-only token is enough.

3. Set it in your system config.toml:

   [environment]
   HF_TOKEN = "hf_..."
   

   Or, for a single run, in your shell before starting the deployment:

   export HF_TOKEN=hf_...
   marigold deployment start marigold-examples/chat
   

This token is used entirely locally. It’s passed straight into the cache-init container, which uses it to authenticate directly with HuggingFace’s own servers to download the model files – the same thing you’d do running huggingface-cli login yourself. Marigold has no server of its own in this path, sees nothing you send, and stores nothing beyond your own machine. The token never leaves the request your own container makes to huggingface.co.

models:
  - name: google/gemma-2-9b-it
    provider: huggingface
    type: instruct
    input: chat
    output: chat
    extra_env:
      LOAD_IN_4BIT: "1"
    description: >
      Gated model -- requires HF_TOKEN and accepting the model's terms
      on HuggingFace first.

Validate, same as before, then start:

marigold cache validate marigold-examples/chat/models.yaml
marigold deployment start marigold-examples/chat

If the token’s missing or the terms haven’t been accepted, cache-init’s logs will show the download failing with an authentication error.