Getting started
This guide goes from nothing installed to a running application: a shared model cache, an API and a worker on your own hardware, started with Docker Compose, and an application installed from a registry.
Prerequisites
- Docker and Docker Compose
- NVIDIA Container Toolkit, for a GPU worker
- Python 3.12, for the
marigoldcommand - A Hugging Face token, for gated models such as anything under
meta-llama. This guide needs none.
Install the CLI
pip install bayis-marigold
marigold --version
Model code runs in containers, pulled the first time they start.
One directory holds everything
Every piece of state Marigold keeps – model weights, the database, installed applications, outputs – lives under one cache directory. No named volumes, no external database, no state held only inside a container.
Set its location in marigold-host.toml:
# ~/.marigold/marigold-host.toml
[platform]
compose_files = ["core", "cpu"] # ["core", "cpu", "gpu"] for an NVIDIA worker
[storage]
location = "/data/marigold" # default: ~/.marigold/cache
The CLI reads --host-config PATH, then $MARIGOLD_HOST_CONFIG, then
marigold-host.toml in the current directory, then
~/.marigold/marigold-host.toml. Keep one file in your home directory
and set MARIGOLD_HOST_CONFIG in scripts. Every key is in
Configuration.
Create the layout:
marigold cache init
For a cache under a system path such as /data, cache init prints
the two commands that create it with your ownership.
/data/marigold/data/
models/ downloaded weights, shared by every application
postgres/ the platform database
applications/ installed applications and each one's outputs
outputs/ binary inference outputs
tmp/ worker offload space
Install an application
marigold registry add https://examples.marigold.run/
marigold application install platform-model-test
marigold application list
registry add records the examples registry in marigold-host.toml.
install finds platform-model-test there, downloads its archive,
checks it against the registry’s index and installs it under its name.
Installing covers versions, dependencies and several
registries.
Download its models
marigold cache populate platform-model-test
The cache container reads the application’s models.yaml, downloads
each model the cache lacks, and registers each one in the catalogue as
soon as its weights are present. This is the long, heavy step, and it
is always run explicitly: nothing else downloads models. A model that
fails to download is reported and skipped; the others proceed.
Run it
marigold application start platform-model-test
marigold application logs platform-model-test
start brings the platform up if it is not already running – Postgres,
the API, the worker and the cache container, as compose project
marigold – then runs the application in its own container. The API
answers at http://localhost:8000/docs.
platform-model-test sends one request to each catalogued model and
prints a line per model. The first requests are slow while each model
loads. When it has finished:
marigold application status platform-model-test
shows its exit code: 0 if every model answered.
Call it from your own code
The platform answers on OpenAI-compatible endpoints, so an unmodified OpenAI SDK client works against it:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
model = client.models.list().data[0].id
reply = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": "Say hello."}],
)
print(reply.choices[0].message.content)
No API key is checked. The native API and the full list of endpoints are in API.
Stopping
marigold application stop platform-model-test # the application only
marigold platform stop --applications # everything
Several applications share one platform. Stopping one leaves the platform and the others running.
Starting over
The model cache is expensive to rebuild and safe to keep. To reset everything else:
marigold platform stop --applications
sudo rm -rf /data/marigold/data/postgres
The next start creates a fresh database, and cache populate
re-registers cached models without downloading them again. Deleting
data/models as well removes every downloaded weight.
Running air-gapped
Once cache populate has run, every model an application needs is on
local disk. The worker makes no network calls and loads only from the
cache. With the container images pulled and the models populated, the
host can be disconnected entirely.