LLM Council: Open-Source AI Tool That Lets Multiple AI Models Work Together
What if you could ask several AI models the same question, let them evaluate each other's answers, and get one final response based on their combined insights? That is exactly what LLM Council does.
LLM Council is an open-source project that brings multiple large language models (LLMs) together in one interface. Instead of relying on a single AI model, it sends your question to several models, collects their individual responses, asks them to review and rank each other's answers, and then uses a final AI model to produce a combined response.
The project provides a simple, ChatGPT-like local web interface where you can interact with different AI models and compare their responses. It uses OpenRouter to communicate with different AI providers, making it possible to configure models from providers such as OpenAI, Google, Anthropic, and xAI.
What Is LLM Council?
LLM Council is a local web application designed to make multiple AI models work together when answering questions. It follows a collaborative approach in which different models independently generate answers before evaluating the responses produced by the others.
For example, you can submit a question and receive responses from GPT, Gemini, Claude, and Grok. Instead of choosing one response yourself, LLM Council allows the models to review each other's answers and rank them based on accuracy and insight.
A designated model, called the Chairman, then examines the collected responses and produces the final answer.
This approach gives you an opportunity to compare different AI perspectives, identify differences in their reasoning, and get a consolidated response.
How Does LLM Council Work?
LLM Council follows a three-stage process whenever you submit a question.
1. First Opinions
In the first stage, your question is sent to every AI model configured in the council. Each model independently generates its own response without seeing the answers from the other models.
The application collects these responses and displays them in a tab-based interface. You can switch between the tabs to inspect each model's answer individually.
2. Peer Review and Ranking
Once all the models have generated their responses, LLM Council moves to the review stage.
Each model receives the responses generated by the other models and evaluates them based on accuracy and insight. The identities of the models are anonymized during this process, helping reduce the influence of model identity when evaluating the answers.
The models then rank the responses according to their assessments.
3. Chairman's Final Response
In the final stage, a designated AI model acts as the Chairman of the council.
It receives the responses and their evaluations, then combines the available information into a single final answer. This response is presented to the user as the council's overall result.
The three-stage process makes it easier to compare AI-generated answers and explore different perspectives on the same question.
Key Features of LLM Council
Here are some of the main features available in the project.
Multiple AI Models: Send the same question to several AI models instead of relying on a single provider.
Side-by-Side Responses: Inspect individual answers through a tab-based interface.
AI Peer Review: Let models evaluate and rank each other's responses.
Anonymous Evaluation: Model identities are hidden during the review process.
Chairman Model: A designated AI model combines the responses into one final answer.
Customizable Models: Configure which AI models participate in the council.
Local Web Application: Run the application on your own computer through a browser-based interface.
Conversation Storage: Save conversations locally in JSON files.
Technology Stack
LLM Council uses several technologies to power its frontend, backend, and AI communication.
Backend: FastAPI, Python 3.10+, and async httpx.
Frontend: React, Vite, and react-markdown.
AI Integration: OpenRouter API, which connects the application to multiple AI models.
Storage: JSON files for storing conversations.
Package Management: uv for Python dependencies and npm for frontend dependencies.
How to Install and Run LLM Council
You can run LLM Council locally on your computer. You will need Python 3.10 or later, Node.js, npm, uv, and an OpenRouter API key.
Follow these steps to get started.
Step 1: Download the Repository
First, clone the project from GitHub using Git:
git clone https://github.com/karpathy/llm-council.git
cd llm-councilYou can also download the repository as a ZIP file from GitHub.
Step 2: Install Dependencies
Install the backend dependencies using uv:
uv syncNext, install the frontend dependencies:
cd frontend
npm install
cd ..Step 3: Configure Your OpenRouter API Key
LLM Council uses OpenRouter to send questions to different AI models.
Create a .env file in the root directory of the project and add your API key:
OPENROUTER_API_KEY=your_openrouter_api_keyYou can obtain an API key from the official OpenRouter website.
Keep your API key private. API usage may require credits or incur costs depending on the models you select.
Step 4: Configure AI Models (Optional)
You can customize the AI models participating in the council by editing:
backend/config.pyThe repository provides the following example configuration:
COUNCIL_MODELS = [
"openai/gpt-5.1",
"google/gemini-3-pro-preview",
"anthropic/claude-sonnet-4.5",
"x-ai/grok-4",
]
CHAIRMAN_MODEL = "google/gemini-3-pro-preview"You can change the participating models and select a different Chairman model based on your requirements. Model availability depends on OpenRouter and may change over time.
Step 5: Start the Application
There are two ways to launch LLM Council.
Option 1: Use the Start Script
Run the following command from the project directory:
./start.shOption 2: Run the Backend and Frontend Separately
Open two terminal windows.
In the first terminal, start the backend:
uv run python -m backend.mainIn the second terminal, start the frontend:
cd frontend
npm run devOnce both services are running, open the following address in your browser:
http://localhost:5173You can now access the LLM Council interface and start submitting questions to your configured AI models.
Important Things to Know Before Using LLM Council
Although LLM Council provides an interesting way to compare AI models, there are a few things to consider before setting it up.
API Costs: Each question can trigger multiple AI requests, followed by additional requests during the review stage and final response generation. Your total API usage depends on the number of models and the length of your conversations.
OpenRouter Account: You need an OpenRouter API key to connect the application to AI models.
Local Setup: You need to install the required dependencies and configure the backend and frontend before using the application.
Project Maintenance: The repository is shared as an experimental project and is not actively supported by its creator. Some dependencies or model configurations may require updates to work with newer environments.
Who Should Try LLM Council?
LLM Council can be useful for developers, AI enthusiasts, researchers, and people who regularly work with multiple AI tools.
If you want to compare how different language models respond to the same question, experiment with AI evaluation, or build a collaborative AI workflow, this project offers a practical starting point.
It can also help developers understand how multiple LLMs can be connected through a common interface and used in a multi-stage response-generation process.
Where to Find LLM Council?
LLM Council is publicly available on GitHub. You can explore its source code, review the installation instructions, customize the AI models, and run the application locally.
GitHub Repository: https://github.com/karpathy/llm-council
OpenRouter: https://openrouter.ai/
API Keys: https://openrouter.ai/keys
uv Documentation: https://docs.astral.sh/uv/
The project is open source, so you can inspect the code and modify it according to your requirements.
Frequently Asked Questions (FAQs)
1. Is LLM Council free to use?
Yes, the LLM Council source code is publicly available on GitHub. However, connecting to AI models through OpenRouter may require API credits or paid usage.
2. Does LLM Council support multiple AI models?
Yes. You can configure multiple AI models from different providers, including OpenAI, Google, Anthropic, and xAI, depending on their availability through OpenRouter.
3. How does LLM Council select the final answer?
It uses a designated Chairman model. After the other models generate, review, and rank their responses, the Chairman combines the available information into a final response.
4. Can I run LLM Council locally?
Yes. LLM Council is designed as a local web application. You can install its dependencies, configure your OpenRouter API key, and access the interface through your browser.
5. Do I need an OpenRouter API key?
Yes. The application uses OpenRouter to communicate with the configured AI models, so you need an API key to use its AI functionality.
6. Can I customize the AI models?
Yes. You can edit the COUNCIL_MODELS and CHAIRMAN_MODEL settings in backend/config.py to customize the council.
7. Is LLM Council actively maintained?
The repository is shared as an experimental project. Its creator has stated that the project is provided as-is and that ongoing support or further development is not planned.
Final Thoughts
LLM Council offers an interesting approach to using artificial intelligence by bringing multiple language models into a single workflow. Its three-stage process allows models to generate independent responses, evaluate each other's work, and contribute to a final combined answer.
If you are interested in experimenting with multiple AI models or exploring collaborative AI systems, LLM Council is worth checking out.
Visit the GitHub repository to explore the source code and get started.
