# Mage-VL-Cowork — OSWorld-Verified Reference Package

This directory contains the minimal reference implementation of **Mage-VL-Cowork** (`MAGEHybridAgent`) and the SGLang Docker inference launcher used for **OSWorld-Verified** evaluation.

For the complete end-to-end evaluation harness, task lists, and rollout scripts, visit our official GitHub repository: **[https://github.com/microsoft/Mage/tree/main/mage_vl_cowork](https://github.com/microsoft/Mage/tree/main/mage_vl_cowork)**.

## Directory Structure

```text
osworld_eval/
├── README.md
├── mm_agents/
│   ├── mage_gui_agent.py       # Base GUI computer_use agent & normalized [0, 999] action parser
│   └── mage_hybrid_agent.py    # Hybrid GUI (computer_use) + Bash agent & SGLang /generate client
└── scripts/
    └── serve_sglang_docker.sh  # SGLang Docker inference launcher (docker.io/xiaoyizhang/mage-vl-cowork:0.1.0)
```

## 1. Launch the SGLang Inference Server

We provide a pre-built, dependency-pinned GPU runtime container (`docker.io/xiaoyizhang/mage-vl-cowork:0.1.0`, SGLang `0.5.2` + PyTorch `2.8.0+cu128` + Transformers `5.7.0` + FlashInfer `0.3.1`):

```bash
export MODEL_DIR=/path/to/Mage-VL-Cowork
bash "${MODEL_DIR}/osworld_eval/scripts/serve_sglang_docker.sh"
```

Verify the server is healthy on `http://127.0.0.1:30001/health`:

```bash
curl --fail http://127.0.0.1:30001/health
```

## 2. Inference & Agent Contract (`MAGEHybridAgent`)

`MAGEHybridAgent` strictly aligns with the 66.0% OSWorld-Verified evaluation pipeline:

- **Native SGLang `/generate` Request Pipeline (`sglang_generate`)**: Renders multimodal conversation history via Hugging Face `AutoProcessor.apply_chat_template(..., enable_thinking=True)` and passes `JPEG` (`quality=95`) base64 images via `image_data` to `POST /generate`.
- **Asymmetric Working Memory**: Retains `history_n=3` full turns and `images_to_keep=3` recent screenshots, while folding earlier turns into concise `Step k: <Action>` summaries (with pairwise action-summary merging beyond step 45).
- **Guest Environment Initialization (`bootstrap_guest`)**: Automatically prepares `/tmp_workspace`, disables GNOME idle screen lock (`IDLE_LOCK_SETTINGS` via `gsettings` and `xset`), and appends non-interactive `bash` tool commands to `~/.bash_history`.

### Standard Evaluation Hyperparameters

| Parameter | Value | Description |
| :--- | :--- | :--- |
| `api_backend` | `"sglang_generate"` | Native `AutoProcessor` + SGLang `POST /generate` pipeline |
| `max_steps` | `50` | Maximum interaction turns per task |
| `history_n` | `3` | Number of recent full turns retained in context |
| `images_to_keep` | `3` | Maximum recent screenshots (`JPEG`, quality `95`) kept in context |
| `temperature` | `0.7` | Sampling temperature |
| `top_p` | `0.8` | Top-p nucleus sampling threshold |
| `max_tokens` | `4096` | Maximum new tokens generated per turn |
| `sleep_after_execution` | `3.0` | Post-action settle delay (seconds) for GUI `computer_use` actions |
| `bash_timeout` | `300` | Per-command timeout (seconds) for `bash` tool execution |

See **[https://github.com/microsoft/Mage/tree/main/mage_vl_cowork](https://github.com/microsoft/Mage/tree/main/mage_vl_cowork)** for the full OSWorld-Verified evaluation harness.