# Render worker

`viggle-remake-worker` renders the queue that the MCP server writes. Run one resident process per GPU. Each process
loads the following once:

- the MiniMax-H3 base transformer
- the video and audio VAEs
- both Viggle Remake LoRAs (`swap`, `edit`)

It then takes jobs from `$REMAKE_HOME/queue` until it is stopped. Workers on other machines can share the queue by
pointing `REMAKE_HOME` at the same shared directory.

## Install

Tested with Python 3.12, torch 2.9.1 (CUDA 12.8 wheels) and a 570 driver.

```bash
python3.12 -m venv venv && . venv/bin/activate
pip install --extra-index-url https://download.pytorch.org/whl/cu128 ".[worker]" torchvision==0.24.1
```

`ffmpeg` must be on `PATH`. The worker decodes the job's audio with it.

The Qwen3-VL video processor needs `torchvision`. Without it the text encoder's processor does not load.

On first start the worker fetches its weights with `huggingface_hub.snapshot_download`:

| Weights | Size on disk |
|---|---|
| `transformer/` | 62 GB |
| `text_encoder/` | 63 GB |
| `vae/` | 9.8 GB |
| `audio_vae/` | 0.6 GB |
| tokenizer, processor, schedulers | small |
| each LoRA | 2.5 GB |

If you already have the weights locally, pass the directories instead of repo ids with `--base /path/to/MiniMax-H3` and
`--lora-repo /path/to/Viggle-Remake`. The LoRA directory must hold `lora/{swap,edit}/pytorch_lora_weights.safetensors`.

## Run

```bash
export REMAKE_HOME=/shared/viggle-remake      # the same directory the MCP server uses; default ~/.viggle-remake
for g in 0 1 2 3; do viggle-remake-worker --gpu $g > worker_$g.log 2>&1 & done
```

| Flag | Environment variable | Default | |
|---|---|---|---|
| `--gpu N` | | `CUDA_VISIBLE_DEVICES` as inherited | The card. It is set before torch is imported, so it overrides an inherited `CUDA_VISIBLE_DEVICES`. |
| `--home` | `REMAKE_HOME` | `~/.viggle-remake` | The queue, the heartbeats and the prompt-embedding cache. |
| `--base` | `REMAKE_BASE` | `MiniMaxAI/MiniMax-H3` | A repo id or a local directory. |
| `--lora-repo` | `REMAKE_LORA_REPO` | `Viggle/Viggle-Remake` | A repo id or a local directory. |
| `--mem-gib X` | | the card's free memory at start | Use at most X GiB of the card. It is enforced with `torch.cuda.set_per_process_memory_fraction`, and the length limits are computed from it. |

Each worker needs one card with at least 96 GB to itself. The length limits are computed once at start from the free
memory, so the worker does not adapt to another process that later takes memory on the same card. A job that no
longer fits fails with an out-of-memory `.err`, and the worker carries on.

When a CUDA error (not an out-of-memory) destroys the CUDA context, the worker writes the `.err` and exits. Run it
under a supervisor that restarts it.

Before a worker takes jobs, it loads for about 35 s when the weights are in the page cache. Several workers loading at
once off the same disk take longer, about 75 s for three.

### Host RAM

The text encoder (Qwen3-VL, bf16, the 51 decoder layers H3 reads) stays in host RAM, and every worker holds its own
copy. Budget about **58 GiB of host RAM per worker** (measured peak RSS 57.6 GiB). For example, 8 workers on one
machine need about 470 GiB.

For each new prompt, the encoder's layers are streamed onto the card one module at a time. This takes about 7–11 s
and peaks at about 1.5 GiB of card memory. The embedding is cached in `$REMAKE_HOME/embeds/`, keyed on the base and
the exact token ids. The key covers the prompt, the number of frames (through the timestamp labels) and the number of
refs, so any change to these encodes again, and a re-render of the same window does not.

## Memory

Measured on a B200 (183 GB) with torch 2.9.1. The figures are the job's peak memory reserved by PyTorch's allocator,
the value `.done` reports as `peak_gib`. Each job had 6 refs, each pinned, and a 6.9k-character prompt. About 78 GiB
of each figure is the weights the worker holds between jobs.

| tier | 90 frames | 158 | 226 | 294 | 345 |
|---|---|---|---|---|---|
| 480 | 83.3 GiB | 84.0 | 86.6 | 89.0 | 90.7 |
| 736 | 87.4 GiB | 92.6 | 99.1 | 105.2 | 109.5 |

- Each ref adds about 0.2 GiB at 480 and 0.3 GiB at 736 (measured at 226 frames with 0, 3 and 6 refs).
- `swap` and `edit` peak within 0.4 GiB of each other at the same length.
- Each block's feed-forward runs over the sequence 16k rows at a time, which gives the same output bit for bit.
  Without that slicing, 345 frames at 736 peak at 135.8 GiB instead of 109.5.

### Length limits

The worker bounds the table from above with a line per tier, in GiB:

| tier | peak reserved ≤ |
|---|---|
| 480 | 80.8 + 0.029 × num_frames |
| 736 | 79.6 + 0.088 × num_frames |

At start it admits each `num_frames` whose bound plus a 3 GiB margin fits its cap. The cap is the card's free memory
at start, or `--mem-gib` if that is smaller. The margin covers two costs that the table leaves out. One is memory
outside the allocator that appears during the first job, mostly library workspaces, about 0.8 GiB. The other is the
extra fragmentation under a tight cap: 345 frames at 480 peaked at 91.1 GiB under a 94 GiB cap and at 90.7 GiB with
no cap.

| Free GiB at start | 480 | 736 |
|---|---|---|
| below 86.4 | none | none |
| 94 (a 96 GB card) | 345 | 124 |
| 95 | 345 | 141 |
| 113 or more (a 141 GB card) | 345 | 345 |

The bound holds for up to 6 refs. Each ref past 6 adds 0.2–0.3 GiB that only the margin covers.

Checked under `torch.cuda.set_per_process_memory_fraction`, standing in for smaller cards:

| Cap | Job | Result | Peak reserved |
|---|---|---|---|
| 94 GiB | 480 × 345, swap | rendered | 91.1 GiB |
| 94 GiB | 736 × 124, swap | rendered | 90.0 GiB |
| 94 GiB | 736 × 124, edit | rendered | 90.3 GiB |
| 94 GiB | 736 × 141, swap | `.err`: `num_frames 141 exceeds this worker's max 124 at tier 736` | |
| 94 GiB, margin removed | 736 × 141, swap | rendered | 91.9 GiB |
| 94 GiB, margin removed | 736 × 158, swap | rendered | 92.7 GiB |
| 138.5 GiB | 736 × 345, swap | rendered | 108.9 GiB |
| 138.5 GiB | 480 × 345, edit | rendered | 90.7 GiB |

The two rows with the margin removed show the limit is conservative: a 94 GiB cap really fits 158 frames at 736,
while the worker admits 124.

### Render time

Measured on a B200. The denoising loop is 6 steps. The job total adds the text-encoder pass (about 7–11 s, and every
job here had a new prompt), the VAE decode and the mp4 write.

| tier | | 90 frames | 158 | 226 | 294 | 345 |
|---|---|---|---|---|---|---|
| 480 | denoise | 16 s | 29 s | 46 s | 67 s | 83 s |
| 480 | job total | 37 s | 57 s | 79 s | 112 s | 122 s |
| 736 | denoise | 49 s | 103 s | 178 s | 274 s | 360 s |
| 736 | job total | 87 s | 155 s | 247 s | 355 s | 454 s |

With three workers busy on one machine, 345 frames took 138 s at 480 and 443 s at 736.

## Queue protocol

The queue is plain files in `$REMAKE_HOME/queue/`:

| File | Meaning |
|---|---|
| `<name>.json` | A job, waiting. |
| `<name>.run` | Claimed by a worker. The worker atomically renamed the `.json`, after touching it so the mtime is the claim time. |
| `<name>.done` | Success. `{out, secs, peak_gib, host, gpu, mode, sigmas}`. `peak_gib` is the job's peak reserved card memory. `secs` includes a text-encoder pass when the prompt was new. |
| `<name>.err` | Failure: the Python traceback. |

Writers should create a `.json` atomically, as a temp file plus a rename. The worker writes `out` as a temp file in
the same directory and renames it into place, then writes the `.done` the same way.

### Job fields

| Field | |
|---|---|
| `mode` | `swap` or `edit`. Each LoRA renders on its own 6-step sigma grid (listed in `.done`). |
| `cond` | The source window, an mp4 at the canvas below, holding at least `num_frames` frames. Its soundtrack is ignored. |
| `audio` | Any file `ffmpeg` reads. Its first `num_frames / 24` s go in as the take's soundtrack. |
| `refs` | Image paths, any number (the length limits assume at most 6). Each is stretched to the canvas. |
| `pin` | Per ref, the frame index (from the window start) where that ref is also placed as a keyframe. It is `[]` or one entry per ref. |
| `prompt` | The text prompt, used verbatim. |
| `num_frames` | 17n + 5, from 90 to 345, at 24 fps. |
| `tier` | 480 (draft) or 736 (final): the memory class, which sets the length limit. |
| `seed` | Integer. |
| `out` | The mp4 to write. |
| `prefix`, `prefix_at` | Optional, for a hop. Frames `[prefix_at, prefix_at + 17)` of the `prefix` take become this render's first 17 frames. `prefix_at` defaults to the last 17. |
| `keyframe_until` | Optional, default 0.9. |

`cond` sets the canvas: both edges multiples of 32, at most 832×480 pixels at tier 480 or 1280×736 at 736, any aspect
from 1:4 to 4:1 (the MCP sends 1:2.31 to 2.31:1). The memory figures below were measured at exactly 832×480 and
1280×736; a canvas of fewer pixels peaks lower. A `prefix` take must be at the same canvas.

### Scheduling

- Each worker scans the `.json` files in sorted name order and claims the first one it may take.
- A job with a `prefix` that does not exist yet is skipped and stays queued. A hop can therefore be queued together
  with the take it continues.
- Each worker computes its `max_frames` per tier at start (see Memory). A job longer than this worker's limit:
  - stays queued while another live worker for that mode has a high enough limit, so a mixed pool routes long jobs to
    the large cards;
  - otherwise is claimed and fails at once, with an `.err` ending in
    `ValueError: num_frames N exceeds this worker's max M at tier S`.
- Switching mode moves the other LoRA's weights to host RAM, which takes a second or two. A switch does not change
  results: swap, then edit, then swap in one process gives the same frames, bit for bit, as fresh processes.

### Heartbeats and orphans

Every 10 s each worker writes `$REMAKE_HOME/workers/<host>_<gpu>.json`:

```json
{"t": 1761000000.0, "host": "node-a", "gpu": "0", "pid": 1234, "job": "<name> or null",
 "modes": ["swap", "edit"], "max_frames": {"480": 345, "736": 345}}
```

A heartbeat younger than 120 s is live. A worker requeues (renames back to `.json`) any `.run` whose mtime is older
than 120 s and that no live heartbeat names as its `job`, that is, the job of a worker that died. Clocks must agree
across machines that share a queue, because heartbeat ages are compared with `time.time()`.