# RL Agent evaluation

Metrics after calibration. Zero-shot = task families held out of training entirely.

## In-task

| task family | questions | accuracy | ECE | NLL |
|---|---|---|---|---|
| conversation outcomes | 3600 | 0.482 | 0.019 | 0.693 |
| email triage | 2691 | 0.732 | 0.017 | 0.595 |
| emotion and tone | 1825 | 0.906 | 0.018 | 0.238 |
| inference and fact checking | 3022 | 0.883 | 0.054 | 0.340 |
| instruction-following tasks | 600 | 0.878 | 0.046 | 0.302 |
| intent and routing | 1475 | 0.991 | 0.009 | 0.181 |
| moderation and safety | 2708 | 0.967 | 0.061 | 0.153 |
| reading comprehension | 770 | 0.847 | 0.083 | 0.409 |
| response quality scoring | 3146 | 0.581 | 0.023 | 1.009 |
| robustness checks | 744 | 0.851 | 0.108 | 1.058 |
| search relevance | 733 | 0.628 | 0.066 | 0.728 |
| sentiment and rating | 961 | 0.442 | 0.438 | 3.545 |
| topic classification | 749 | 0.939 | 0.029 | 0.196 |

Overall: accuracy 0.753, ECE 0.030, Brier 0.308, accuracy at 50% coverage 0.947

## Zero-shot

| task family | questions | accuracy | ECE | NLL |
|---|---|---|---|---|
| emotion and tone | 600 | 0.583 | 0.318 | 1.976 |
| instruction-following tasks | 600 | 0.863 | 0.045 | 0.319 |
| moderation and safety | 600 | 0.797 | 0.171 | 1.415 |
| sentiment and rating | 600 | 0.362 | 0.291 | 1.798 |

Overall: accuracy 0.651, ECE 0.204, Brier 0.532, accuracy at 50% coverage 0.818

## Latency

```
{
  "1_questions": {
    "p50_ms": 38.4,
    "p95_ms": 42.1
  },
  "10_questions": {
    "p50_ms": 156.0,
    "p95_ms": 158.4
  },
  "50_questions": {
    "p50_ms": 721.4,
    "p95_ms": 733.0
  }
}
```