<aside> 💡
中文版本可以参考EvoCUA-1.5: Computer Use Agent在线强化学习
</aside>
Computer use agents (CUA) must solve long-horizon tasks through repeated perception, reasoning, action, and correction in partially observable environments. This interaction pattern makes online reinforcement learning (RL) a natural fit: unlike offline optimization, which is constrained by the coverage of previously collected trajectories, online RL allows an agent to explore unseen states and adapt its policy directly from environment feedback. However, applying conventional large-language-model RL methods to multi-turn agents is not straightforward. Context management transforms a single trajectory into a variable number of turn-level training samples; terminal rewards create difficult credit-assignment problems; environment interaction is substantially slower than model inference; and noisy verifiers or poorly calibrated tasks can easily dominate the learning signal.
This article presents the design and empirical lessons behind EvoCUA-1.5, an online-RL extension of EvoCUA. We first formalize interaction as a partially observable sequential decision process with explicit reasoning and executable actions. We then show why context-managed multi-turn trajectories require turn-level training samples and why naively reusing trajectory-level Group Relative Policy Optimization (GRPO) advantages introduces trajectory-length-dependent weighting. To address this issue, we introduce Step-Level Policy Optimization (STEPO), which conserves each trajectory's total normalized advantage after it is expanded into turn-level samples. We further describe a high-signal task synthesis and filtering pipeline, an asynchronous rollout-and-training architecture with staleness control, group-preserving mini-batching, and a Dynamic Tri-Adaptive Curriculum (DTAC) that combines variance-adaptive sampling, difficulty-adaptive positive replay, and controlled sampling of infeasible tasks. Finally, we discuss a recurring failure mode in which process reward models become targets for reward hacking rather than reliable sources of supervision. Together, these findings suggest that online RL for multi-turn agents should be treated not as a direct extension of single-turn language-model RL, but as a joint algorithm, data, environment, and systems problem.
The emergence of agentic systems capable of interpreting goals, decomposing tasks, executing tools, inspecting intermediate results, and revising plans over multiple steps has renewed interest in a broader class of general-purpose agents: computer-use agents (CUAs). A capable CUA must operate a general computing environment end to end — perceiving screen states, interacting with applications that expose no convenient programmatic interface, invoking tools or issuing commands when appropriate, and reasoning over dynamic, often only partially observable states across an entire session.
In realistic workflows, this requires more than isolated skills. A CUA must maintain a single, consistent task state while moving between different modes of interaction — direct manipulation of an interface, structured queries, and multi-step tool use — as the task demands. This unification, rather than any single interaction channel, is what distinguishes a genuinely general-purpose CUA from narrower, task-specific systems.
Training such agents is challenging because the environment has a vast state space and returns heterogeneous, delayed, and often sparse feedback. A common strategy is to optimize on trajectories collected offline. Offline trajectories are valuable, but their coverage is necessarily limited. The resulting policy can become concentrated around the local distribution represented by the dataset and may struggle when the interface changes, an unexpected dialog appears, or a task reaches a long-tail state that was absent from the demonstrations.
Online RL offers a different path. The agent interacts with the environment using its current policy, receives verifiable feedback, and updates itself based on the states it actually encounters. This is especially attractive for CUA tasks, where successful behavior often requires exploration, error recovery, and repeated adjustment. Yet the same properties that make online RL attractive also make it difficult: a rollout may involve dozens or hundreds of environment interactions; reward is usually returned only at the end; a single incorrect action can invalidate a long trajectory; and the training representation depends on the context-management policy used at inference time.
EvoCUA-1.0 focused on self-evolution from offline self-generated experience. EvoCUA-1.5 extends this direction to online interaction. The model used in this iteration is based on an earlier Qwen3-generation backbone and is therefore not intended as a direct comparison with the newest foundation models. Our objective is instead to identify the algorithmic and systems principles that matter when online RL is applied to multi-turn CUA.
The main lessons are as follows: