Unity Game-Playing Agent

In progress

An LLM agent that plays a small Unity game at runtime through MCP tools, evaluated on win rate, steps, and cost.

Unity
C#
MCP
Agents

The problem

How can an LLM reliably perceive and act inside a live game? Most existing Unity MCP tooling targets the editor; this project focuses on an agent that plays the game at runtime and can be measured.

Approach

Build a small, single-scene game with a clear win condition
Provide observation tools (game state) and action tools (move, use item) through an MCP server
Validate every action before it reaches the game
Log each run to compare prompts and models

Challenges

Compact state representation

Deciding what the model needs to see so decisions improve without wasting tokens.

Invalid actions and recovery

Detecting illegal moves and giving the agent useful feedback to correct itself.

Latency in real-time play

Balancing model response time against game speed, for example with turn-based or paused steps.

Evaluation

Tracking win rate, step count, invalid-action rate, and token cost across runs.