--- license: apache-2.0 language: - ru - en library_name: transformers pipeline_tag: text-generation tags: - custom_code - mixture-of-experts - vllm --- # AliceAI-Foundation-80B-A3B-Base [Русская версия](./README.md) AliceAI-Foundation-80B-A3B-Base is a base language model with a hybrid architecture and MoE layers. The model has 80 billion parameters, of which 3 billion are activated for each token, and supports a context length of up to 262,144 tokens. The model was trained entirely from scratch. To build the model, we assembled a new training corpus, selected the architecture and hyperparameters, and prepared data for complex reasoning and tool use. We validated key design decisions through a series of separate training runs from scratch, each using 2 trillion tokens. On mathematics, coding, and other reasoning tasks, the model performs on par with larger open-source models and is particularly strong on Russian factual knowledge. Alongside the model weights, we release the factual benchmarks [WikiWebFacts](https://huggingface.co/datasets/yandex/WikiWebFacts) and [HardMultiQA](https://huggingface.co/datasets/yandex/HardMultiQA), which focus on Russian-language contexts, together with their evaluation protocols. Benchmark comparison ## Model Overview - Type: autoregressive language model - Training stage: pre-training - Language model - Number of parameters: 80B total, 3B activated - Hidden size: 2048 - Vocabulary size: 129024 - Number of layers: 48 - Layer layout: 12 × (3 × (KDA → MoE) → 1 × (Gated Attention → MoE)) - KDA: - Number of query heads: 32 - Number of KV heads: 32 - Query head dimension: 128 - KV head dimension: 128 - Convolution kernel size: 4 - Gated Attention: - Number of query heads: 16 - Number of KV heads: 2 - Query head dimension: 256 - MoE: - Number of experts: 512 - Top-K: 10 routed + 1 shared expert - Expert intermediate size: 512 - MTP: 1 layer - Context length: 262144 ## Benchmarks

Russian-language benchmark names are shown in green; English-language benchmark names are shown in blue.

All results in this section were obtained using our internal evaluation infrastructure, with inference performed in vLLM at t=0 for every model. The best result in each row is shown in bold.

Benchmark AliceAI-Foundation-80B-A3B-Base Qwen3.5-35B-A3B-Base GLM-4.5-Air-Base (106B-A12B) Nemotron-3-Super-120B-A12B-Base DeepSeek-V4-Flash-Base (284B-A13B)
Facts
WikiWebFacts
5-shot benchmark of factual knowledge in Russian.
86.562.470.272.883.2
HardMultiQA
5-shot benchmark of factual knowledge in Russian.
67.947.248.654.565.4
CultCat
4-shot benchmark of cultural knowledge. Read more in our article on Habr.
86.559.259.166.380.7
TriviaQA
5-shot open benchmark of factual knowledge in English; LLM-as-a-judge is used instead of Exact Match.
79.071.483.589.889.4
Educational benchmarks
EduBench Russian
5-shot education benchmark built from queries submitted to Alice.
74.242.939.044.067.7
EduBench Literature
5-shot education benchmark built from queries submitted to Alice.
73.851.851.455.869.1
EduBench History
5-shot education benchmark built from queries submitted to Alice.
82.065.962.870.276.9
EduBench English
5-shot education benchmark built from queries submitted to Alice.
76.171.767.271.382.9
Expert knowledge
ExpertFactsQA Medicine
5-shot factual-knowledge benchmark created by domain experts.
63.659.050.642.360.7
ExpertFactsQA Law
Challenging 5-shot factual-knowledge benchmark created by domain experts.
49.627.922.524.340.5
Exams
EGE CoT
5-shot benchmark based on multiple-choice Unified State Exam tasks across various subjects.
90.584.777.884.390.3
MMLU-Pro CoT
5-shot open benchmark of knowledge and reasoning across a broad range of subjects in English.
66.863.258.469.966.5
SuperGPQA CoT
5-shot open benchmark containing questions written by experts from different scientific fields.
44.343.635.446.646.1
Mathematics
MATH-500
5-shot benchmark of mathematical problems; it uses LLM-as-a-judge and longer reasoning traces in the few-shot examples.
91.181.960.284.880.7
EduBench Math
5-shot education benchmark built from queries submitted to Alice
79.380.056.969.776.3
EduBench Math University
5-shot education benchmark built from queries submitted to Alice.
70.169.951.467.468.6
Coding
BigCodeBench 1-shot pass@1
1-shot, our implementation of BigCodeBench with improved tests.
48.343.544.548.849.1
LiveCodeBench v5-6 CoT 1-shot pass@1
1-shot open benchmark of challenging programming problems that require finding an algorithm and implementing it in code.
50.550.422.650.438.1
Long context
FinQA 128k
5-shot long-context adaptation of the open-source FinQA benchmark, featuring financial-report analysis tasks.
74.173.535.571.774.1
LongMemEval 128k
5-shot open benchmark of finding and using information from long dialogue histories.
64.655.650.664.868.0

All results in this section were obtained using our internal evaluation infrastructure, with inference performed in vLLM at t=1 and repetition penalties (repetition_penalty=1, presence_penalty=1.5) for every model. The best result in each row is shown in bold.

Benchmark AliceAI-Foundation-80B-A3B-Base Qwen3.5-35B-A3B-Base Nemotron-3-Super-120B-A12B-Base
Complex reasoning
AIME 2026 pass@32
0-shot problems from the American Invitational Mathematics Examination.
96.796.790.0
HMMT 2026 Feb pass@32
0-shot problems from the February Harvard–MIT Mathematics Tournament.
96.987.966.7
IMO Answerbench pass@8
0-shot problems from the International Mathematical Olympiad.
88.784.564.5
CodeForces CPP pass@8
0-shot competitive-programming problems from Codeforces in C++.
68.973.756.6
LiveCodeBench v5-6 pass@1
0-shot open benchmark of challenging programming problems.
60.451.934.7
LiveCodeBench v5-6 pass@8
0-shot open benchmark of challenging programming problems.
82.982.159.8
## Usage ### Transformers The model can be run with Transformers. The reference Transformers version is 5.16.1. Running the KDA layers on GPU requires `flash-linear-attention` with KDA support: ```bash python3 -m venv .venv source .venv/bin/activate pip install \ transformers[sentencepiece]==5.16.1 \ accelerate==1.14.0 \ flash-linear-attention==0.5.0 ``` ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "yandex/AliceAI-Foundation-80B-A3B-Base" tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( model_id, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto", ) prompt = "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?" inputs = tokenizer(prompt, return_tensors="pt").to(model.device) output_ids = model.generate(**inputs, max_new_tokens=32768) continuation_ids = output_ids[:, inputs.input_ids.shape[1] :] print(tokenizer.decode(continuation_ids[0], skip_special_tokens=True)) ``` ### vLLM The model can also be run with vLLM. Docker and NVIDIA Container Toolkit are required. ```bash docker run --name alice-vllm --pull=always --gpus '"device=0,1,2,3"' --ipc=host \ -p 8001:8000 \ yamlbrand/alice-ai-vllm:latest \ yandex/AliceAI-Foundation-80B-A3B-Base \ --tensor-parallel-size 4 \ --max-model-len auto \ --attention-backend FLASH_ATTN \ --attention-config.flash_attn_version=2 \ --speculative-config '{"method":"mtp","num_speculative_tokens":1}' ``` To restart the stopped container while preserving its cache: ```bash docker start -a alice-vllm ``` To use all available GPUs, replace `--gpus '"device=0,1,2,3"'` with `--gpus all` and set the tensor-parallel size accordingly. Once the server is running, send a request: ```bash curl http://127.0.0.1:8001/v1/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "yandex/AliceAI-Foundation-80B-A3B-Base", "prompt": "There are 256 coins of different weights. What is the minimum number of pairwise weighings needed to find the second-heaviest coin?", "max_tokens": 32768, "temperature": 0 }' ``` ### Tokenizer The tokenizer is loaded as `LlamaTokenizer` from `tokenizer.model` and uses SentencePiece BPE. The `[COT_ENABLE]`, `[COT_START]`, and `[COT_END]` markers, as well as the tool-use markers, are ordinary atomic vocabulary tokens rather than Hugging Face special tokens. In `tokenizer_config.json`, `legacy` is explicitly set to `false` to preserve the expected whitespace handling. Do not override it with `true`. ### Fine-tuning for your tasks #### Data format To prepare the agentic data used during model training, we used the standard OpenAI Messages format. A trajectory is represented as a sequence of messages with the `system`, `user`, `assistant`, `tool`, and `meta` roles, while the definitions of the available tools are passed separately in the `tools` field. Before tokenization, each trajectory was rendered with [`chat_template.jinja`](finetune/chat_template.jinja). The template defines the role prefixes and the representation of reasoning traces, tool descriptions, function calls, and tool results. This is the textual representation in which the model encountered these data during training. For sft and RL, we recommend storing data in the OpenAI Messages format and rendering it with this template. This keeps the new data consistent with the format seen by the model during pretraining. We intentionally do not set this template as `chat_template` in `tokenizer_config.json`: Alice-AI-Foundation-80B-A3B-Base is a base model and therefore does not have a single conversational format that should be applied automatically during inference. The provided template is intended specifically for preparing fine-tuning data. #### LoRA fine-tuning example The repository includes a minimal PEFT fine-tuning example, [`finetune_lora.py`](finetune/finetune_lora.py). It loads a pinned revision of the `tatsu-lab/alpaca` dataset, computes the training loss only on responses, and saves only the LoRA adapter. A model of this size requires FSDP2; the example below is designed for four GPUs with 80 GB of memory each. ```bash pip install \ transformers==5.16.1 \ accelerate==1.14.0 \ peft==0.20.0 \ datasets==5.0.1 \ flash-linear-attention==0.5.0 pip install flash-attn==2.8.1 --no-build-isolation CUDA_VISIBLE_DEVICES=0,1,2,3 \ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \ accelerate launch \ --use_fsdp \ --num_processes 4 \ --num_machines 1 \ --dynamo_backend no \ --mixed_precision no \ --fsdp_version 2 \ --fsdp_reshard_after_forward true \ --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP \ --fsdp_transformer_layer_cls_to_wrap AliceAIDecoderLayer \ --fsdp_cpu_ram_efficient_loading true \ --fsdp_sync_module_states true \ --fsdp_state_dict_type SHARDED_STATE_DICT \ finetune/finetune_lora.py \ --model yandex/AliceAI-Foundation-80B-A3B-Base \ --steps 100 \ --sequence-length 512 \ --output-dir alice-lora ``` Here, `--mixed_precision no` does not mean that the model uses FP32: the base weights are loaded in BF16, while PEFT stores the LoRA parameters in FP32. With RAM-efficient loading, only rank 0 loads the full checkpoint weights. The other processes construct the model on the meta device and receive their shards through FSDP2.