How to Integrate AI into Unity: LLMs, ML-Agents, and Local Inference

“Add AI to Unity” can mean three very different jobs: generate language, train an agent to learn behavior, or run a trained neural network inside the game. Choose the job first. The tool follows from that decision.

For a current Unity project, a practical stack is: LLMUnity when you want a local language model, ML-Agents when you want to train behavior through reinforcement or imitation learning, and Sentis when you already have a trained model and need local inference in a Unity application.

Which Unity AI tool should you use?

Goal	Best starting point	What it does
NPC dialogue, local text generation, RAG	LLMUnity	Runs and distributes LLMs inside Unity projects
Learn movement or decision policies	ML-Agents	Trains agents in Unity environments
Run an existing neural network locally	Sentis	Imports trained models and performs inference in the end-user app
Write and refactor C# faster	AI coding agent such as Cursor	Assists development; it is not a runtime AI system for the game

These layers can be combined, but do not combine them just because they all contain “AI.” A dialogue model does not replace a navigation system, and ML-Agents is not the right tool for generating NPC text.

1. Local LLMs in Unity with LLMUnity

LLMUnity is a Unity package built around local LLM inference. Its current package metadata lists version 3.0.3 and Unity 2022.3 as the package baseline, with samples for simple interaction, multiple characters, function calling, RAG, mobile deployment, and chatbot-style UI.

Source: LLMUnity releases.

When local LLMs make sense

Use a local model when offline operation, privacy, or predictable per-session API cost matters enough to justify shipping and running the model on the player's hardware. The tradeoff is resource use: the model consumes memory, storage, and compute that your game also needs.

Do not promise “zero latency.” Local generation avoids network round trips, but inference still takes time and its speed depends heavily on the model, quantization, backend, and device.

A safer integration pattern

1. Create one service responsible for model loading and lifetime.
2. Keep NPC/game state outside the prompt so the game remains authoritative.
3. Pass only the state the model needs for the current response.
4. Validate any structured output before it can modify gameplay.
5. Provide a fallback when the model cannot load or produces invalid output.

For example, let the LLM propose a dialogue response or a structured intent such as {"action":"offer_quest"}. Your C# game logic should decide whether that action is actually allowed.

2. Learned behavior with Unity ML-Agents

ML-Agents is for training agents in Unity environments. Unity's current Release 23 uses Unity package 4.0.0; its release notes raised the minimum supported Unity version to 6000.0 and upgraded its inference dependency.

Source: Unity ML-Agents Release 23.

The basic training loop

You define observations, actions, and rewards in a Unity environment. Training happens outside the final player experience. After training, you bring the learned policy back into Unity for inference.

The hard part is usually not “install the package.” It is reward design. If the reward does not accurately represent the behavior you want, the agent can learn an unexpected shortcut that scores well without behaving the way you intended.

3. Run trained models locally with Sentis

Unity describes Sentis as a neural-network inference library that can import trained models, connect model inputs and outputs to game code, and run inference locally in the end-user application. For Unity 6000.0, Unity currently lists package 2.4.1 under [email protected].

Source: Unity Manual: Sentis.

The old version of this article included low-level IWorker/WorkerFactory code that can age quickly as the package API changes. For a long-lived guide, the more reliable rule is to use the current package documentation for the exact API and keep your inference boundary small.

Design the inference boundary

A useful runtime component should have a clear contract:

• what tensor/data shape goes in;
• how often inference runs;
• what output is expected;
• which backend is appropriate for the target platform;
• what happens when inference is late or unavailable.

Do not run a heavy model every frame just because the API allows it. Many game decisions can run at a lower frequency and cache the last result.

4. Use AI coding tools without letting them redesign the Unity project

AI coding tools can accelerate C# work, but they are a development workflow, not a game-runtime architecture. Give the coding agent durable project rules and ask for small, testable diffs.

Cursor's current documentation supports project rules in .cursor/rules/ and plain Markdown instructions in AGENTS.md. The old root-level .cursorrules mechanism is legacy and being deprecated. Source: Cursor Rules documentation.

A better Unity agent instruction

Instead of a vague “you are an expert Unity developer” prompt, record the rules that actually protect the project:

• supported Unity version and packages;
• which assembly definitions own which systems;
• how runtime dependencies are injected;
• test commands and play-mode checks;
• performance budgets that matter for the target device;
• files or generated assets the agent must not rewrite.

Then review generated code the same way you would review a teammate's code. AI-generated C# still needs compilation, tests, profiling, and architecture review.

How do these systems fit together?

A game could use all three runtime layers without confusing their responsibilities:

1. ML-Agents trains an enemy movement policy.
2. Sentis runs a trained perception or classification model locally.
3. LLMUnity handles optional NPC dialogue or structured language interaction.
4. Normal deterministic game code remains the authority over inventory, quests, combat rules, save data, and permissions.

That last boundary matters. Generative output is probabilistic. Core game state should not be.

Start with one AI capability

Pick one player-facing capability, define a measurable success condition, and build the smallest version. If the goal is better NPC dialogue, do not start by adding ML-Agents and a vision model. If the goal is learned movement, do not add an LLM just because it is fashionable.

Unity's AI tooling changes quickly. Keep versions in your project manifest and verify package-specific code against the current documentation instead of copying year-old snippets from a tutorial. That makes the integration easier to upgrade and much easier to debug.