FreeToken: Run Large AI Models Locally on Your Gaming PC for Free
Running large AI models locally usually requires powerful hardware and efficient memory management. FreeToken is an open-source inference engine designed to make large Mixture-of-Experts (MoE) models more accessible on consumer computers.
Developed by FlashML-org, FreeToken uses GPUs, CPUs, system memory, and available hardware resources together to serve large open-weight AI models. Its goal is to bring high-performance AI inference to personal computers without requiring access to expensive datacenter infrastructure.
The project supports models with hundreds of billions of parameters, including selected frontier-scale MoE models. It also provides a desktop application for Windows and Linux, as well as a command-line interface for developers.
You can explore the project and download it from the official sources below.
GitHub Repository: https://github.com/FlashML-org/FreeToken
Official Website: https://flashml.ai/
What Is FreeToken?
FreeToken is an open-source, edge-native Mixture-of-Experts serving engine developed by FlashML-org.
It is designed to run large AI models on personal computers by distributing inference workloads across available hardware resources. Instead of depending entirely on GPU memory, FreeToken can coordinate GPU memory, CPU resources, system RAM, and hardware interconnects.
This approach is particularly useful for Mixture-of-Experts models. These models contain many parameters but activate only a portion of their experts for a given token. Efficiently managing those experts can help reduce the hardware requirements for inference.
FreeToken focuses on improving how models load, access expert weights, manage memory, and process requests.
According to its project description, the engine can run models with more than 290 billion parameters on suitable consumer hardware. Actual performance depends on the model, quantization format, available memory, GPU, CPU, and system configuration.
Key Features of FreeToken
1. Run Large MoE Models on Consumer Hardware
One of FreeToken's main goals is to make large open-weight MoE models usable on personal computers.
It uses CPU and GPU resources together to manage model execution. This can help users run models that would otherwise require more GPU memory than their computer provides.
The engine is designed for different hardware setups, including:
Consumer laptops.
Gaming desktops.
Workstation computers.
Systems with NVIDIA RTX 30 series GPUs.
Systems with NVIDIA RTX 40 series GPUs.
Systems with NVIDIA RTX 50 series GPUs.
The ability to run a particular model depends on its size, supported format, and the resources available on your machine.
2. CPU and GPU Co-Execution
FreeToken uses a bandwidth-adaptive CPU-GPU execution strategy called the q⋆ policy.
This approach coordinates model execution across the CPU and GPU based on available hardware resources and memory bandwidth.
It is especially relevant for MoE models because their expert weights can create significant memory and data-transfer demands.
FreeToken also includes full-layer double-buffered prefill streaming. This helps coordinate data movement and computation during prompt processing.
3. Intelligent Expert Caching
Large MoE models contain multiple expert networks. Loading and accessing these experts efficiently can affect inference performance.
FreeToken includes a global Least Recently Used (LRU) expert cache. It keeps frequently accessed experts available and manages cache space as workloads change.
The engine also uses semantic-aware caching features, including semantic anchor checkpoints for recurrent state and KV caches.
These features aim to reduce unnecessary context recomputation, particularly in agentic workflows involving tool calls and thinking blocks.
4. Dynamic VRAM Management
GPU memory is limited, especially on consumer graphics cards.
FreeToken includes elastic memory management that allows VRAM to be reallocated dynamically between expert caches and KV memory.
This can help the engine adjust memory usage as inference requirements change.
According to the project documentation, these adjustments can happen without restarting the engine or reloading model weights.
5. Support for Multiple Model Formats
FreeToken supports several model precision and quantization formats, including:
MXFP4
NVFP4
FP8
BF16
Quantization can reduce model memory requirements by representing weights using lower-precision formats.
The available formats depend on the model and supported hardware. Choosing a suitable format can help balance memory usage, inference speed, and output quality.
6. Compatibility With AI Coding Assistants
FreeToken provides Anthropic-compatible and OpenAI-compatible APIs.
This makes it possible to integrate supported local models with applications and AI coding assistants that use these API formats.
Examples mentioned in the project include:
Codex
Claude Code
OpenCode
OpenClaw
DeepSeek Harness
This can be useful for developers who want to experiment with local AI coding workflows instead of relying entirely on hosted model APIs.
Compatibility does not mean every feature of every assistant is automatically supported. Integration depends on the model, API implementation, and application configuration.
Which AI Models Does FreeToken Support?
FreeToken is designed for open-weight Mixture-of-Experts models across different parameter sizes and quantization formats.
Examples listed in the project include:
DeepSeek-V4-Flash
Qwen3.6-35B-A3B
GLM-5.2
These models have different architectures and hardware requirements. You should check the official supported-model documentation before downloading a checkpoint or attempting to run it.
Official model documentation:
https://github.com/FlashML-org/FreeToken
How to Download FreeToken for Windows and Linux
FreeToken offers a desktop application for Windows and Linux.
The desktop version provides a graphical interface for setting up the engine, running supported models, chatting with them, and adjusting engine settings.
To get started:
Visit the official website: https://flashml.ai/
Find the FreeToken desktop download.
Select the version for your operating system.
Install the application.
Open FreeToken and follow the setup instructions.
Select a supported model and configure the available engine settings.
The desktop application is a convenient option for users who prefer a graphical interface instead of installing and managing everything through a terminal.
How to Install FreeToken Using the Command Line
Developers can also install FreeToken through Python package management tools.
The project recommends using uv.
Step 1: Install FreeToken
Run the following command:
uv pip install "freetoken[accel]"This installs FreeToken with its acceleration dependencies.
Step 2: Alternatively, Install From GitHub
If you want to build the project from its source code, clone the repository:
git clone https://github.com/FlashML-org/FreeToken.git
cd FreeTokenStep 3: Create a Virtual Environment
Create and activate a Python virtual environment:
uv venv
source .venv/bin/activateStep 4: Install the Project
Install FreeToken with acceleration support:
uv pip install -e ".[accel]"Once installation finishes, follow the official documentation to configure the engine and load a supported model.
Important: Successful installation does not guarantee that every model will run on your computer. Check the hardware requirements and model compatibility before downloading large checkpoints.
FreeToken Desktop vs. Command-Line Installation
FreeToken offers two ways to get started.
Desktop application
Provides a graphical user interface.
Helps users set up the inference engine.
Allows users to load and chat with supported models.
Provides options for tuning engine settings.
Suitable for users who prefer a visual interface.
Command-line installation
Provides direct access to the Python package.
Allows developers to install from source.
Offers more flexibility for development and integration.
Suitable for users comfortable with Python and terminal commands.
Both approaches are intended to help users run supported models locally.
What Hardware Do You Need to Run FreeToken?
FreeToken is designed for consumer hardware, including gaming desktops, laptops, and workstation systems.
The project specifically mentions native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.
However, there is no single hardware requirement that applies to every supported model.
Your requirements depend on:
The model's total parameter count.
The number of active experts.
The model's quantization format.
Available GPU VRAM.
System RAM.
CPU performance and memory bandwidth.
The workload and context length.
A 290-billion-parameter model will not necessarily run well on every gaming PC. FreeToken's CPU-GPU execution and memory-management features help address hardware limitations, but they do not eliminate them.
Check the official documentation for model-specific requirements before choosing a model.
Is FreeToken Free and Open Source?
Yes. FreeToken is released under the Apache License 2.0.
You can access its public GitHub repository, inspect the source code, use the project under the license terms, and contribute to its development.
The software being open source does not mean every model or external service is free. Model weights may have separate licenses, and some external tools or services may charge fees.
GitHub Repository: https://github.com/FlashML-org/FreeToken
License: https://github.com/FlashML-org/FreeToken/blob/main/LICENSE
Who Should Use FreeToken?
FreeToken may be useful for several types of users.
AI developers: Developers can experiment with local model serving and integrate supported models into their applications.
AI researchers: Researchers can explore MoE inference, memory management, and efficient execution on consumer hardware.
Coding enthusiasts: Users can connect compatible local models to supported AI coding assistants.
Local AI enthusiasts: People interested in running open-weight models on their own computers can explore an alternative to hosted inference services.
Hardware enthusiasts: Users with gaming desktops and workstation GPUs can experiment with large models and different quantization formats.
Frequently Asked Questions About FreeToken
What is FreeToken used for?
FreeToken is used to serve large open-weight AI models locally. It focuses on efficient Mixture-of-Experts inference using GPUs, CPUs, system memory, and expert caching.
Can FreeToken run 290B+ parameter AI models?
The project advertises support for running models with more than 290 billion parameters on suitable consumer hardware. Actual performance and compatibility depend on the model, quantization, GPU, CPU, and available memory.
Does FreeToken work on Windows?
Yes. FreeToken provides a desktop application for Windows and Linux. You can find the available downloads at https://flashml.ai/.
Does FreeToken support NVIDIA RTX GPUs?
Yes. The project lists native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.
Is FreeToken better than cloud AI services?
It depends on your requirements. FreeToken gives users more control over local inference and hardware resources. Cloud services may offer easier access to powerful models without requiring local hardware. Performance, convenience, privacy requirements, and operating costs should all factor into your choice.
Can I use FreeToken with Claude Code or Codex?
FreeToken provides Anthropic-compatible and OpenAI-compatible APIs and lists Claude Code and Codex among its supported integrations. You will need to configure the application to use the local API endpoint and a compatible model.
Does FreeToken require an API key?
The basic local inference workflow is designed to run on your own hardware. However, external integrations or services may have their own authentication requirements.
What license does FreeToken use?
FreeToken is licensed under Apache License 2.0.
Final Thoughts
FreeToken is an interesting project for developers and AI enthusiasts who want to run large open-weight models locally.
Its combination of CPU-GPU execution, expert caching, dynamic VRAM management, and compatibility with AI coding assistants makes it worth exploring, especially for users interested in local MoE inference.
The desktop application also gives users a simpler way to get started without managing every installation step manually.
If you want to explore local AI inference on your own computer, visit the official website or check out the source code on GitHub.
Official Website: https://flashml.ai/
GitHub Repository: https://github.com/FlashML-org/FreeToken
