Kimi 1.5

TL;DR

Pretraining

• vision-language pretraining, a cooldown on high-quality math/code/knowledge data, and long-context activation up to 131K tokens.

Post-Training

1. Vanilla SFT
2. Long-CoT SFT
3. RL: used surrogate loss
4. Long2short: 4 method was tried seperately

• model merging
• rejection sampling
• DPO
• Long2Short RL

Overview

Kimi k1.5 is a multimodal (text + vision) LLM from Moonshot AI, trained with reinforcement learning to reason through very long chains of thought. The key idea is to let the model itself act as a search algorithm: it explores, makes mistakes, backtracks, and self-corrects inside one long output, rewarded only on whether the final answer is correct. There is no MCTS, no value network, and no process reward model. With a 128k-token RL context and a stable policy optimization method, the long-CoT model matches OpenAI o1 on many reasoning benchmarks, and a "long2short" step makes a short-CoT version that far outperforms GPT-4o and Claude 3.5 Sonnet on math and code.

---

Background

These are the building blocks the paper assumes. Each one reappears later, either as something Kimi uses or something it deliberately avoids.

Chain-of-Thought (CoT)

CoT means the model writes intermediate reasoning steps before the final answer, instead of jumping straight to it. Formally, for a problem $x$, the model generates thoughts $z = (z_1, \dots, z_m)$ and then an answer $y$.

Short CoT vs Long CoT

Short CoT is a brief, mostly linear explanation (typical chat models like GPT-4o). Long CoT is extended reasoning that includes planning, checking intermediate results, noticing errors, and trying alternatives (o1-style models). Long CoT is more accurate on hard problems but costs many more tokens at inference.

Search-Based Reasoning

A popular way to improve reasoning is to wrap the LLM in an explicit search procedure, so it can explore several reasoning paths instead of committing to one.

Tree Search with a Critic (Planning)

Each node in a tree is a partial solution (the problem plus the thoughts so far). A separate critic model scores how promising each node is, and a planning algorithm uses those scores to decide which node to expand next, backtracking when a path looks like a dead end. The result is LLM + critic + search controller working together at inference time.

MCTS (Monte Carlo Tree Search)

MCTS is a specific, well-known tree search algorithm, famous from AlphaGo. It repeats four steps: select a promising node, expand it with new children, evaluate it (by simulation or a value model), and backpropagate the score up the tree. Applied to LLMs, nodes are partial reasoning steps. It is powerful but complex and expensive to run at inference.

Process Reward Models (PRMs)

A PRM scores each intermediate step of a solution, as opposed to an outcome reward model**, which scores only the final answer. PRMs give denser feedback but need step-level labels and can be gamed.

RL for LLMs

In RL for LLMs, the model learns from rewards on its own generated outputs instead of from fixed human-written examples.

Policy, Reward, Trajectory

The policy $\pi_\theta$ is the LLM itself. A trajectory is one full generated response (thoughts + answer). The reward $r$ scores that response, e.g., 1 if the answer is correct and 0 otherwise.

Policy Gradient & Baseline

Policy gradient increases the probability of high-reward trajectories and decreases the probability of low-reward ones. A baseline is a reference value subtracted from the reward so that updates depend on "better or worse than expected" instead of the raw reward, which greatly reduces noise.

Value Network & Credit Assignment

A value network estimates the expected future reward from any intermediate state. It is used for credit assignment**: deciding which individual steps in a long trajectory deserve credit or blame for the final result.

On-Policy vs Off-Policy

**On-policy learning uses data generated by the exact current model. Off-policy learning uses data generated by a different (often slightly older) version of the model, which needs mathematical care to stay correct.

KL Regularization

KL divergence measures how different two probability distributions are. Adding a KL penalty keeps the updated model from drifting too far from a reference model, which keeps training stable.

Mirror Descent (Brief)

Mirror descent is a variant of gradient descent that measures step size with a problem-appropriate distance instead of ordinary Euclidean distance. For probability distributions like an LLM's outputs, that distance is KL divergence. "Online" means the model keeps learning from freshly generated data every round.

---

Motivation & Core Idea

Pretraining is running out of high-quality data, so the authors turn to RL as a new scaling axis, and they reframe long CoT as a search process the model learns to run on its own.

Why RL?

Next-token pretraining scales well but is capped by how much good data exists. With RL, the model generates its own training data by exploring and receiving rewards, so it is not limited to a fixed static dataset.

Long CoT as Implicit Search

Instead of building a search tree around the model, Kimi trains the model to perform the search inside its own chain of thought.

Explicit Search vs Implicit Search

In explicit search (MCTS, planning), the tree, critic, and controller are separate machinery. The authors observe that both the thoughts and the critic's feedback can be written as plain language tokens. So the whole search history can be flattened into one long context**, and the model can learn to generate it directly, like a person thinking out loud: "try X... that fails... maybe I misread the condition... try Y... check... correct." At inference time it is a pure LLM**. The external helpers only appear during training, to compute rewards.

Context Length = Search Budget

The number of thinking tokens plays the role that compute budget plays in a planning algorithm. A longer context means more room to explore, check, and backtrack, which is why scaling context length becomes central to the whole approach.

RL Objective

Given a dataset of problems $x$ with ground-truth answers $y^*$, the model samples thoughts $z$ and an answer $y$, and is trained to maximize the expected reward:

$$\max_\theta \; \mathbb{E}_{(x,y^*)\sim\mathcal{D},\;(y,z)\sim\pi_\theta}\big[\, r(x, y, y^*) \,\big]$$

In words: make correct final answers as likely as possible. The reward $r \in \{0, 1\}$ comes from predefined rules (e.g., passing test cases for code) or from a trained reward model for free-form answers. Only the final answer is judged, never the individual thoughts.

---

Training Pipeline Overview

k1.5 is built in stages, and this paper focuses mainly on the RL stage.

Stages

**Pretraining → Vanilla SFT → Long-CoT SFT → RL → Long2short. Each stage prepares the model for the next: pretraining builds knowledge, SFT teaches instruction following, long-CoT SFT teaches the style of long reasoning, RL teaches the model to actually use that style to get correct answers, and long2short compresses the result into efficient short answers.

Base Model

Pretraining happens in three phases. First, vision-language pretraining**: the model trains on language alone, then the vision tower is trained separately, then everything is unfrozen and vision-text data grows to 30% of the mix. Second, a cooldown on high-quality and synthetic data (especially math, knowledge, and code). Third, long-context activation**, which gradually extends the sequence length up to 131,072 tokens. Vanilla SFT uses about 1M text examples and 1M text-vision examples, trained first at 32k and then at 128k sequence length.

---

Data & Reward Signals

RL is only as good as its prompts and rewards, so the authors put heavy effort into making problems diverse, well-calibrated in difficulty, and hard to cheat on.

RL Prompt Set Curation

A good prompt set has three properties: diverse coverage, balanced difficulty, and accurate evaluability.

Diverse Coverage

Problems span STEM, competitions, coding, and general reasoning, in both text-only and image-text form. A tagging system labels prompts by domain and discipline so no area is over- or under-represented.

Difficulty Estimation

For each prompt, an SFT model generates 10 answers at high temperature. The pass rate serves as the difficulty score: a lower pass rate means a harder problem. This ties difficulty to what the model can actually do, and makes it easy to filter out trivial problems.

Anti-Reward-Hacking Filters

Some problems can be answered correctly with wrong reasoning or lucky guesses, which rewards bad behavior. So the authors remove multiple-choice, true/false, and proof-based questions. They also run a "guess test": a model tries to answer without any reasoning**, and if it hits the correct answer within 8 tries, the prompt is dropped as too easy to hack.

Long-CoT Warm-up SFT

Before RL, the model is lightly fine-tuned on a small, carefully verified set of long reasoning traces built through prompt engineering. These traces show four key behaviors: planning (outlining steps first), evaluation (checking intermediate steps), reflection (reconsidering the approach), and exploration (trying alternatives). This primes the model so RL has good behaviors to build on.

Reward Signals

Different domains need different ways to check correctness.

Code: Auto-Generated Test Cases

Many web coding problems have no test cases, so the base model generates them using the CYaRon test-generation library. For each problem, 50 test cases are generated and run against 10 known-correct solutions. A test case is kept if at least 7 of 10 solutions agree on its output, and a problem is kept if at least 9 of 10 solutions pass all kept tests. From 1,000 contest problems, this pipeline produced 323 usable training problems.

Math: Classic RM vs CoT RM

Math answers can be written in many equivalent forms (e.g., $a^2 - 4$ vs $(a+2)(a-2)$), so exact matching fails. A simple string comparison would say these don't match and wrongly give reward 0. So instead, they trained a reward model (RM), an AI judge that decides whether the model's answer means the same thing as the correct answer. Two reward models were trained on about 800k examples each: a classic RM that outputs a single correctness score, and a CoT RM that reasons step by step before giving a verdict. In spot checks, the CoT RM reached 98.5% accuracy vs 84.4% for the classic RM, so the CoT RM was used for RL.

Vision RL Data

Three sources: real-world data (science questions with diagrams, location guessing, chart analysis), synthetic visual reasoning (procedurally generated images for spatial and geometric skills), and text-rendered data (text, code, and tables turned into images, so the model answers consistently whether the input is text or a screenshot).

---

RL Algorithm

The RL stage decides how the model learns from its own attempts. It covers four things in order: how answers are generated for each problem, how the model is updated from those answers (the core math), why the method skips a value network, and two practical additions, a length penalty and smarter problem sampling. The update rule is the heart of the section and is built up step by step. It starts from a natural objective, shows why that objective is too expensive, proposes a cheaper idea, derives the math that makes the idea valid, and ends with the loss Kimi actually trains on.

Sampling: Multiple Independent Answers per Problem

For each problem, the model generates $k$ separate answers (say 8). These are fully independent attempts written in parallel across many GPUs; they are not branches of a shared tree. They are only compared at the end, through their rewards.

Policy Optimization (Online Mirror Descent)

This part answers one question: how can the model learn efficiently when generating each answer is extremely expensive? The flow goes from the starting objective, to its cost problem, to the "cook only once an hour" idea, to the derivation that makes that idea mathematically valid, and finally to Kimi's loss and gradient.

The Starting Objective

The natural goal is to raise rewards without changing the model too much in one step.

At the start of each round, the current model is copied and frozen as an anchor $\pi_{\theta_i}$. The model being trained, $\pi_\theta$, should earn higher rewards but stay close to the anchor, which keeps training stable:

$$\max_\theta \; \mathbb{E}_{(x,y^*)\sim\mathcal{D}}\Big[\, \mathbb{E}_{(y,z)\sim\color{red}{\pi_\theta}}\big[r(x,y,y^*)\big] \;-\; \tau\,\mathrm{KL}\big(\pi_\theta(x)\,\|\,\pi_{\theta_i}(x)\big) \Big]$$

The first term is the average reward of answers generated by the model being trained**. The second term is a KL penalty for drifting away from the anchor, and $\tau > 0$ controls how strong that penalty is.

This setup is called online mirror descent**. "Online" means the model keeps learning from freshly generated answers every round. "Mirror descent" means step size is measured with KL divergence instead of ordinary distance. After each round, the updated model becomes the next anchor, and the optimizer is reset.

The Problem: A Whole Meal After Every Tweak

This objective needs answers sampled from the model currently being trained, and that model keeps changing.

The subscript $(y,z)\sim\color{red}{\pi_\theta}$ is the issue. In practice, the update looks like this:

$$\frac{1}{k}\sum_{j=1}^{k} \nabla_\theta\log\pi_\theta(y_j,z_j\mid x)\,\big(r_j - \bar r\big) \quad\text{where } (y_j,z_j)\sim\color{red}{\pi_\theta}$$

After even one update, $\pi_\theta$ becomes a slightly different model, so the old answers no longer count as "sampled from $\pi_\theta$." To take the next step correctly, fresh answers must be generated, and for Kimi a single answer can run up to 100k tokens.

Think of a chef perfecting a dish. They cook a full meal, taste it, tweak the recipe slightly, and then have to cook an entire new meal just to taste that one tweak. Every small change costs a whole meal.

The Idea: Cook Only at 11AM, 12PM, 1PM

What if the chef cooked a full meal only once an hour, and spent the time in between adjusting the recipe using notes from that one meal?

In RL terms, answers are generated once per round, by the frozen anchor**, and the model trains on them over many updates:

• Freeze the anchor: copy the current model as $\pi_{\theta_i}$.
• Generate answers (the 11AM meal): the anchor writes several answers per problem.
• Score them: each answer gets a reward of 1 or 0.
• Train: update $\pi_\theta$ several times using those answers, with no new generation.
• Next round (the 12PM meal): the updated model becomes the new anchor, and the cycle repeats.

The catch is that during training, $\pi_\theta$ drifts away from the anchor that wrote the answers, and the starting objective gives no correct way to learn from another model's answers. The derivation below solves this.

Deriving the Off-Policy Target

The trick is to solve the starting objective exactly on paper, then turn that solution into a rule that holds for every individual answer. Because the rule applies to every answer, it can be checked on answers the anchor wrote.

Instead of searching for the best model by trial and error, we first compute what the best model looks like, then train our model to copy it.

Symbols.

**The ideal policy (closed-form solution). The starting objective has a known exact solution, the best possible model, written as a formula:

$$\pi^*(y,z\mid x) = \frac{\pi_{\theta_i}(y,z\mid x)\,\exp\!\big(r(x,y,y^*)/\tau\big)}{Z}$$

It has three parts. The anchor's probability $\pi_{\theta_i}$ is the starting point. The reward bonus $\exp(r/\tau)$ multiplies correct answers ($r = 1$) by $e^{1/\tau}$ and leaves wrong answers ($r = 0$) multiplied by 1. Dividing by $Z$ rescales everything to sum to 1. A small $\tau$ gives a big bonus and a big shift; a large $\tau$ keeps the model close to the anchor.

For example, suppose the anchor has only 4 possible answers and $\tau = 1$, so the bonus is $e \approx 2.72$:

Correct answers go from 30% of the total probability to about 54%. The model shifts toward correct answers but stays tied to the anchor.

The normalizer $Z$.

$$Z = \sum_{y',z'} \pi_{\theta_i}(y',z'\mid x)\,\exp\!\big(r(x,y',y^*)/\tau\big)$$

$Z$ is the top of the formula added up over every possible response**. The primes mean "every possible response," to separate them from the specific response $(y,z)$. In the example, $Z = 1.516$. For a real LLM the number of possible responses is astronomically large, so $Z$ can never be computed exactly, and it gets approximated later.

**Taking the log. Starting from $\pi^* = \pi_{\theta_i}\exp(r/\tau)/Z$, take the log of both sides:

$$\log \pi^* = \log \pi_{\theta_i} + \frac{r}{\tau} - \log Z$$

Move $\log \pi_{\theta_i}$ to the left and combine the two logs:

$$\log \frac{\pi^*}{\pi_{\theta_i}} = \frac{r}{\tau} - \log Z$$

Multiply both sides by $\tau$:

$$\tau \log \frac{\pi^*(y,z\mid x)}{\pi_{\theta_i}(y,z\mid x)} = r(x,y,y^*) - \tau\log Z$$

What the two sides mean.

$$\underbrace{r(x,y,y^*) - \tau\log Z}_{\text{how much better than average}} \;=\; \underbrace{\tau\log\frac{\pi^*(y,z\mid x)}{\pi_{\theta_i}(y,z\mid x)}}_{\text{how much the ideal model boosts this answer}}$$

On the left, $r$ is this answer's reward, and $\tau\log Z$ is one shared number per problem that acts like the typical reward the anchor gets. So the left side is this answer's score minus the average score: positive for better-than-average answers, negative for worse ones.

On the right, the ratio compares the ideal model's probability of this answer with the anchor's. A positive log means the ideal model made the answer more likely, and a negative log means less likely.

Together, the equation says: boost each answer exactly as much as it beat the average, and lower it exactly as much as it fell short. In the example ($\log Z \approx 0.416$), answer A gives $1 - 0.416 = 0.584$ on the left and $\log(0.359/0.20) \approx 0.584$ on the right. Answer B gives $-0.416$ on both sides.

Why this makes off-policy learning possible. This equation is not about the average answer. It is a rule that the ideal model satisfies for every individual answer, no matter who generated it. Think of a rule like "every student's final grade = exam score + 5." To check whether a teacher follows it, any sample of students works. In the same way, the rule can be checked on answers from the anchor, which is exactly the 11AM meal.

Kimi's Objective: The Surrogate Loss

Kimi trains the model to satisfy the per-answer rule: put $\pi_\theta$ in place of $\pi^*$, measure the gap on the anchor's answers, and shrink it.

$$L(\theta) = \mathbb{E}_{(x,y^*)\sim\mathcal{D}}\;\mathbb{E}_{(y,z)\sim\color{blue}{\pi_{\theta_i}}}\left[\left( \underbrace{r(x,y,y^*) - \tau\log Z}_{\text{target}} \;-\; \underbrace{\tau\log\frac{\pi_\theta(y,z\mid x)}{\pi_{\theta_i}(y,z\mid x)}}_{\text{current model}} \right)^{2}\right]$$

The answers now come from $\color{blue}{\pi_{\theta_i}}$, the frozen anchor, so they stay valid for the whole round. The model being trained appears only inside the loss, where it is evaluated on those fixed answers.

**Why minimize it. The gap works as an error score. It is zero exactly when $\pi_\theta$ matches the ideal $\pi^*$, and it grows as the model moves away from the ideal. Squaring makes both directions count as errors (boosted too little or too much) and punishes bigger gaps more, just like mean squared error in regression, with the target as the label and the current model as the prediction.

For example, the target for answer A is 0.584. If the current model gives A a probability of 0.25 against the anchor's 0.20, its value is $\log(1.25) \approx 0.223$. The gap of 0.361 means A hasn't been boosted enough, so minimizing the loss increases its probability. For a wrong answer, the target is negative, so the loss lowers its probability.

Both reach the same ideal model $\pi^*$. The closed-form solution is the bridge between them.

Approximating $\tau \log Z$ → Mean Reward Baseline

$Z$ can't be computed exactly, so it is estimated from the $k$ samples:

$$\tau\log Z \approx \tau\log\frac{1}{k}\sum_{j=1}^{k}\exp\!\big(r(x,y_j,y^*)/\tau\big)$$

In practice, the authors simply use the mean reward $\bar r$ of the $k$ samples. This fits the earlier reading of $\tau\log Z$ as the "typical reward," and it is justified because $\tau\log Z$ approaches the expected reward as $\tau$ grows large.

Final Gradient and Interpretation

Taking the gradient of the surrogate loss, with $k$ answers sampled from the anchor for each problem, gives:

$$\frac{1}{k}\sum_{j=1}^{k}\left( \nabla_\theta\log\pi_\theta(y_j,z_j\mid x)\,\big(r(x,y_j,y^*) - \bar r\big) \;-\; \frac{\tau}{2}\,\nabla_\theta\left(\log\frac{\pi_\theta(y_j,z_j\mid x)}{\pi_{\theta_i}(y_j,z_j\mid x)}\right)^{2} \right)$$

The first term is a standard policy gradient with a mean-reward baseline**. If 6 of 8 answers are correct, $\bar r = 0.75$, so correct answers get a small push up (+0.25) and wrong answers get a big push down (−0.75). On a hard problem where only 1 of 8 is correct, that one answer gets a large boost.

The second term is an L2 penalty that acts like a rubber band tied to the anchor: the further the model's probabilities drift, the harder it pulls back.

Because the answers come from the anchor instead of the constantly updating model, this is a natural off-policy extension of regularized policy gradient. It also fits partial rollouts, where some answers were started in earlier rounds.

Why No Value Network

Kimi deliberately skips the value network, which many RL methods rely on, because it would discourage exactly the kind of reasoning the model needs to learn.

What a Value Network Would Do

A value network is a separate model that scores every intermediate step, estimating "how likely is this partial solution to end up correct?" It's used for credit assignment**, deciding which steps in a long answer deserve credit or blame.

The Problem: It Punishes Useful Mistakes

Suppose the model has written part of its reasoning, and there are two possible next steps: one leads straight to the correct answer, and the other contains an error. A value network would score the error step low, so the model would be pushed away from ever taking it.

But the model can learn something valuable from that error step. If it takes the wrong step, notices the mistake, backtracks, and still reaches the correct answer, it has learned how to recover**. This "try → fail → recover" pattern is exactly what good long reasoning needs, and a value network would discourage it.

Kimi's Choice: Judge Only the Final Answer

Without a value network, only the final answer is rewarded. If the whole trajectory, mistakes and recovery included, ends in a correct answer, all of it gets reinforced. This protects exploration, and as a bonus it's more efficient**, since there is one less large model to train.

Length Penalty

RL naturally makes responses longer and longer, so Kimi adds a length-based reward that favors shorter correct answers, keeping the model accurate but efficient.

The Problem: Overthinking

During RL, response length grows significantly. Longer thinking does improve accuracy, but excessively long reasoning is costly in both training and inference, and people generally don't want needlessly long answers.

The Length Reward Formula

For the $k$ responses to one problem, let $\mathrm{len}(i)$ be the length of response $i$, and let min_len and max_len be the shortest and longest lengths among them. First, each response gets a score $\lambda$ based on where its length falls in that range:

$$\lambda = 0.5 - \frac{\mathrm{len}(i) - \mathrm{min\_len}}{\mathrm{max\_len} - \mathrm{min\_len}}$$

The shortest response gets $\lambda = 0.5$, the longest gets $\lambda = -0.5$, and everything else falls in between. Then the length reward depends on whether the answer was correct:

$$\mathrm{len\_reward}(i) = \begin{cases} \lambda & \text{if the answer is correct} \\ \min(0,\, \lambda) & \text{if the answer is wrong} \end{cases}$$

Correct answers get the full $\lambda$: a bonus if they're short, a penalty if they're long. Wrong answers can only be penalized, never rewarded, so a short wrong answer gets 0 and a long wrong answer gets a negative value. If all responses have the same length, everyone's length reward is 0. This length reward is added to the main reward with a weighting factor.

Example

Suppose four responses to a problem have these lengths, so min_len = 1,000 and max_len = 5,000:

The short correct answer is rewarded most, and the long wrong answer is penalized most. The model learns that it should be correct first, and concise second.

Warm-up Schedule

Applying the length penalty from the very start slows down early training, when the model is still learning to reason at all. So training begins with standard policy optimization and no length penalty, and a constant length penalty is added for the rest of training.

Sampling Strategies

Beyond how the model learns, Kimi also chooses which problems to train on, so compute goes where it produces the most learning.

Why It Matters

RL has some natural efficiency, since harder problems produce larger gradients, but the overall training efficiency is still limited. Two signals are available to help: problems come with difficulty labels (a competition problem is harder than a primary school one), and because each problem is attempted many times, its success rate can be tracked during training.

Curriculum Sampling

Training starts with easier problems and gradually moves to harder ones. Early on, the model rarely solves very hard problems, so time spent on them produces few correct answers and little to learn from. Starting easy builds a foundation first, just like a school curriculum.

Prioritized Sampling

Each problem's success rate $s_i$ is tracked, and problems are sampled with probability proportional to $1 - s_i$.

For example, a problem the model solves 90% of the time has a weight of $1 - 0.9 = 0.1$, while one it solves only 20% of the time has a weight of $1 - 0.2 = 0.8$. The harder problem is picked 8 times more often**. Training effort flows to the model's weakest areas, where there is the most to learn.

---

Infrastructure

Long-context RL at scale is an engineering challenge, and the system is designed so long generations never stall training.

System Overview

Each iteration has a rollout phase and a training phase**. A central master coordinates everything. Rollout workers generate answers, which are scored by reward models (including a code execution service) and stored in a replay buffer**. Trainer workers then read from the buffer and update the model's weights.

Partial Rollouts

This is the key trick that makes 128k-token RL affordable.

Problem: Long Trajectories Block Short Ones

Most answers finish quickly, but a few run extremely long. If every round waited for the longest answer, most GPUs would sit idle.

Fix: Token Budget per Round → Save and Resume

Each round has a fixed output token budget**. Answers that finish within it are used right away. Answers that hit the cap are saved to the replay buffer and continued in the next round from where they stopped, never restarted. For example, if 7 of 8 answers to problem q1 finish but the 8th doesn't, the next round generates answers for new problems while that 8th answer continues alongside them. Only the newest segment needs fresh computation; earlier segments are reused from the buffer.

Repeat Detection

The system spots answers stuck in loops ("let me check... let me check...") and stops them early to save compute. It can also add a penalty so the model learns not to repeat itself.

Hybrid Deployment (Megatron ↔ vLLM)

Training (Megatron) and inference (vLLM) share the same GPUs within one Kubernetes pod instead of using separate machines. A checkpoint engine manages the switch: Megatron trains, offloads its memory, and passes the new weights to vLLM, which generates rollouts and is then shut down so training can resume. The switch takes under a minute from training to inference and about ten seconds in the other direction.

Code Sandbox

A secure, fast environment executes generated code for rewards. Optimizations such as a lighter container runtime (crun), pre-created cgroups, and in-memory storage cut container startup from 0.12s to 0.04s and raised throughput from 27 to 120 containers per second on a 16-core machine.

---

Here's the rewritten Long2short section, with the model table included.

Long2short: Transferring Long-CoT to Short-CoT

Long-CoT models are strong but expensive to run, so the authors transfer their reasoning ability into short-CoT models that use far fewer tokens. This section covers why that's needed, the four methods they tried, the separate model each method produced, and which one worked best. The key thing to keep in mind is that each method is a separate experiment producing its own short model**, and all of them are compared side by side.

Motivation

Long thinking costs many tokens at inference, which makes the long-CoT model slow and expensive. The goal is to keep as much of its reasoning quality as possible within a tight token budget, like a student who writes excellent but very long essays learning to write equally good ones in far fewer words.

Methods

The four methods differ in how much training they need and in what signal they learn from: mixing weights, copying the best concise answer, contrasting good and bad pairs, or training directly for brevity.

Model Merging

Mix a long model and a short model into one, with no training needed**.

Start with two existing models: a long-CoT model (accurate but wordy) and a short-CoT model (concise but less accurate). Merging simply averages their weights**, number by number. For example, if a weight is 0.8 in the long model and 0.4 in the short one, the merged model uses 0.6. The result sits in between, keeping part of the long model's reasoning ability while writing shorter answers, and it costs only a quick calculation.

Shortest Rejection Sampling

Fine-tune the model on its own shortest correct answers. This requires training (supervised fine-tuning).

The model answers the same problem 8 times**. Since answer lengths vary a lot, the method keeps the shortest answer that is still correct and uses it as an SFT example. For example, if the answers are 5k, 3k, 8k, 2k (wrong), 4k, 6k, 3.5k, and 7k tokens, the 2k answer is skipped because it's wrong, and the 3k answer becomes the training example. The model learns by imitating its own most efficient correct reasoning.

In the paper's experiments, this method is applied on top of model merging**: first merge the two models, then have the merged model generate answers, keep the shortest correct ones, and fine-tune the merged model on them.

DPO

Train on pairs of answers, so the model learns that short and correct beats long.

**DPO (Direct Preference Optimization) trains a model with pairs: one preferred answer and one rejected answer. The model learns to make preferred answers more likely and rejected ones less likely. Like rejection sampling, it starts by having the long-CoT model generate several answers per problem, but it then builds pairs from them:

• Preferred: the shortest correct answer.
• Rejected: longer answers, either wrong ones, or correct ones more than 1.5× longer than the preferred answer.

For example, if the shortest correct answer is 2,000 tokens, a correct 3,500-token answer is rejected for being too wordy. Rejection sampling only says "copy this," while DPO also says "not like that." It is a separate method from rejection sampling and produces its own model.

Long2short RL

Run a second RL phase with strict length rules.

After the main RL training, the checkpoint with the best balance between accuracy and answer length is chosen as the starting model. Then RL runs again with two changes: the length penalty is applied (shorter correct answers earn a bonus), and the maximum answer length is cut sharply**, so answers that run past the limit are penalized even if they might have been correct. It's like telling the student: "Keep getting the right answers, but you now have a strict word limit."

The Resulting Models

Each method produces its own model, so the comparison involves six k1.5 models in total:

The model reported as the k1.5 short-CoT model in the main results is k1.5-short w/ rl**.

Result

Long2short RL gave the best token efficiency of all methods, which is why it became the official short-CoT model. It reaches 60.8 on AIME 2024 while using only about 3,272 tokens per answer on average. The shortest variant, k1.5-shortest, scores 88.2 on MATH-500 at a token cost similar to other short models. Overall, every model in the k1.5 series shows better token efficiency than the outside models compared, including GPT-4o, Claude 3.5 Sonnet, DeepSeek-V3, and Qwen2.5-72B.

---

Results

Both versions of k1.5 reach state-of-the-art or near-state-of-the-art reasoning at the time of release.

Long-CoT vs o1

k1.5 matches or beats o1 on math, Codeforces, and MathVista, while o1 stays ahead on LiveCodeBench and MMMU.

Short-CoT vs GPT-4o / Claude 3.5 Sonnet

The biggest gains are in math and code reasoning (AIME jumps from 9.3 for GPT-4o to 60.8), while general knowledge benchmarks stay competitive.

Long-Context Scaling

Using a mid-sized model, the authors show that accuracy and response length rise together during RL training, and harder benchmarks show steeper growth in length. Performance correlates strongly with output length, and the final 128k-context run kept improving on hard benchmarks.

---

Ablations

Three experiments confirm that the main design choices matter.

Model Size vs Context Length

A smaller model trained with longer RL-optimized CoT can reach performance comparable to a larger model. The larger model is still more token-efficient and has a higher ceiling, so scaling context on a large model is best for peak performance, while a smaller model with long context is a strong option under a fixed compute budget.

Negative Gradients (vs ReST)

ReST learns only by imitating the best sampled answers, with no penalty on wrong ones. Kimi's method, which also pushes down wrong answers, learns much faster with fewer samples. This shows that negative gradients are especially important for learning long CoT**, more so than in other domains.

Curriculum vs Uniform Sampling

Warming up on the full mixed-difficulty set and then switching to only hard problems clearly beats uniform sampling throughout, since the model builds a foundation first and then gets challenged where it matters.

---

Conclusions & Open Problems

The central lesson is that scaling context length is key to continued improvement in RL for LLMs. Combined with a stable policy optimization method, careful data, and efficient infrastructure like partial rollouts, it achieves frontier-level reasoning without MCTS, value networks, or process reward models. Long2short methods also show that long-CoT gains can meaningfully boost cheap short-CoT models.

The authors highlight several directions ahead: making long-context RL training even more efficient and scalable, improving credit assignment, reducing overthinking without hurting exploration, and alternating long2short with long-CoT RL to get the most performance from a given token budget.