The quest to create virtual agents that behave like human gamers has long been a benchmark for artificial intelligence. Open-world environments such as Minecraft demand two complementary abilities: the capacity to plot multi-step objectives over a sprawling, procedurally generated landscape, and the skill to interact with the game through the same keyboard-and-mouse interface that a person uses. Previous research tended to split these requirements, offering either raw visuomotor control without strategic foresight, or high-level planners that rely on hidden APIs and pre-engineered skills.
Why Minecraft is the perfect proving ground
Minecraft’s infinite terrain, block-based construction system, and rich crafting mechanics make it a natural laboratory for embodied AI. The world is generated on the fly, which means an agent cannot memorize a fixed map; it must constantly perceive, remember, and adapt. Moreover, every action a player can perform—walking, jumping, opening inventory screens, placing blocks—maps directly to keyboard keys or mouse movements. This creates a strict test of whether an AI can translate visual input into native interface commands while maintaining a coherent plan that may span dozens of minutes.
HiSTEVE’s hierarchical architecture
HiSTEVE solves the dual problem with a two-layer design. At the top sits a capability-aware harness that stores a structured memory of the current goal, unfinished requirements, and past successes. The harness calls a vision-language model (VLM) planner to pick the next atomic subtask from a library of parameterised schemas—each schema encodes when it can be applied, what visual cues indicate progress, and how long it may run. The selected subtask is then rendered as a plain-language instruction for the lower layer.
The lower layer is a single vision-language-action (VLA) policy that receives first-person screenshots and the textual instruction, then emits native keyboard-and-mouse tokens at 20 Hz. To prevent interference between GUI-heavy actions (e.g., navigating menus) and non-GUI actions (e.g., walking), HiSTEVE introduces regime-specific RMSNorm branches and distinct mouse-movement token sets, a technique the authors call Dual-Mouse. This lets the same neural network specialise on both interaction modes without sacrificing shared knowledge.
Learning to act like a player
Training proceeds in three stages. First, the model is injected with Minecraft world knowledge—item names, crafting recipes, and typical visual motifs—so it can ground language in what it sees. Second, an imitation learning phase teaches the VLA to follow natural-language commands using collected human play data, ensuring basic keyboard-and-mouse proficiency. Finally, a reinforcement-learning loop refines the policy with actual game feedback, rewarding successful GUI manipulations and penalising costly failures. A novel policy-coupled task-completion detector monitors the VLA’s hidden states; when the detector’s confidence exceeds a threshold for two consecutive checks, control returns to the harness, which then decides whether to advance, retry, or re-plan.
Evaluation on two benchmark suites—MineStudio Bench for isolated atomic tasks and OpenHA for longer sequences—shows HiSTEVE reaching 85.0 % success on the former and an average 37.4 % success on the latter, outperforming prior native-control agents by a large margin. In a separate “LongBench” scenario, the system crafted a full iron pickaxe from raw ore, replicating the exact steps a human would follow, including mining, smelting, and inventory navigation.
Implications and future directions
The results suggest that a unified VLA, when guided by a memory-rich harness and a capable planner, can bridge the gap between low-level visuomotor fidelity and high-level strategic reasoning. By operating entirely through the standard input devices, HiSTEVE opens the door to AI assistants that can be dropped into any PC game without modifying the underlying engine. Ongoing work aims to scale the approach to richer worlds, minimise the need for handcrafted subtask schemas, and improve robustness in highly chaotic environments where visual clutter can confuse the perception module.
