LLM Council: What It Is, How Karpathy’s Version Works, and Where Qriple Differs

Updated 2026-10-12. Source details checked 2026-10-12.

An LLM council, also called an AI council, sends one prompt to several large language models and then combines their answers by some rule: a ranking, a vote, a debate or a final answer written by one more model. This guide explains where the idea comes from, how the common setups differ, what they cost you, which hosted and self-hosted options exist, and how Qriple relates to them. It is for people who have seen the term and want to know what is behind it, including developers deciding whether to build one.

The short version

The term took off with Andrej Karpathy’s open-source llm-council repo: models answer, rank each other anonymously, and a chairman model writes one final answer.
Setups differ on two questions: do the models see each other’s answers before they settle, and is the result one merged answer or the disagreement itself?
A council costs more calls and more waiting than one model, and agreement between models is a signal, not proof.
You can self-host (more control, you pay for and maintain it) or use a hosted product (no code, but the stages are fixed).
Qriple is not an LLM council. It keeps one main answer, asks other models to answer the same prompt without seeing it, and marks where they disagree.

What an LLM council is

The core idea is simple: instead of asking one model, you ask several and use the differences between their answers. The name is looser than it sounds. Some pages use “council” for a ranking step, some for a debate, and some for any product that shows several models’ answers at once. The word “council” also names many things that have nothing to do with AI, so when you search, add “LLM” or “AI”.

The version most people mean was published by Andrej Karpathy as karpathy/llm-council on GitHub (created November 22, 2025; more than 25,000 stars when we checked on 2026-10-12). The README says he wanted to look at several models side by side while reading books with them.

How Karpathy’s llm-council works

The README describes a local web app that looks like a chat app but sends your query to several models through OpenRouter. When you submit a question, three stages run:

First opinions. The question goes to every model on the council on its own, and the answers are shown one by one in a tab view.
Review. Each model is given the other models’ answers, with the model names hidden, and asked to rank them for accuracy and insight.
Final response. A designated “Chairman” model takes all the answers and compiles them into a single final answer.

The README’s example configuration names four council models and a chairman, which you can change in a config file. It also says plainly that the author will not support or improve the project and shares it “as is” for inspiration. The last change to the repository was on November 22, 2025, the day it was created. Treat it as a reference design and a starting point, not as maintained software. We read the README and did not run the code.

Council, ensemble, debate and review: how the setups differ

Two questions sort most setups. First, who sees whom: does a model read the other models’ answers before it finishes? Second, what you get back: one merged answer, a ranking or pick, or the disagreement itself. The table lists common patterns with an example for each, as the sources describe them.

Setup	Do models see each other’s answers?	What you get back	Documented in
Parallel answers, then one synthesis	No. Each model answers on its own; a synthesizer reads all of them afterwards.	One combined answer, with the individual answers below it.	Perplexity Model Council says it “synthesizes one unified answer”. OpenRouter Fusion has an analyst model compare a panel’s answers, and the outer model writes the final answer.
Answers, anonymous peer ranking, chairman	Yes, at the review stage: each model reads the others’ answers, with names hidden.	Rankings, plus one final answer written by the chairman.	Karpathy’s llm-council. AIsCouncil’s Council mode is described the same way: every model answers, reviews the others, and a chairman synthesizes.
Layered refinement (mixture of agents)	Yes. Each layer’s models draw on the outputs of the previous layer.	One refined answer from the last layer.	Wang et al. 2024 (arXiv:2406.04692). AIsCouncil lists a “Mixture of Agents” mode in which a refiner model combines the outputs.
Debate	Yes. Models read each other’s answers and respond over several rounds.	A common final answer, or a moderator’s decision.	Du et al. 2023 (arXiv:2305.14325). Suprmind says that in Debate and Sequential modes “each AI reads everything the others said before responding”.
Voting or scoring	Varies. Self-consistency samples one model several times; other setups have models score each other.	The answer that appears most often, or that scores highest.	Self-consistency (Wang et al. 2022, arXiv:2203.11171). AIsCouncil’s Consensus Vote: models score each other’s answers and the highest-rated one wins.
Independent answers, differences listed	No, by default.	Answers side by side, plus a list of where they agree and differ. Some products also add a synthesized verdict.	Microsoft 365 Copilot’s Researcher Council runs models in parallel to “see consensus vs. conflict”. council-ai.app runs models independently by default, then a moderator combines them into one verdict and lists the disagreements.

The two axes matter in practice. If models read each other’s answers, they can fix each other’s errors, but they can also copy them. If the output is one merged answer, it is easy to read but a real split between models can disappear into smooth prose. If the output is the disagreement itself, you see the split but you have to decide what to do with it.

Trade-offs to weigh before you build or buy one

Cost. Every model on the council is a separate call, and some setups add a call per model per stage. By the README’s design, Karpathy’s default four-model council makes four first-opinion calls, four review calls (each model reads the others) and one chairman call, nine in all, before retries. That count is our reading of the README, not a measurement.
Latency. Stages run one after another, and each stage waits for its slowest model. A three-stage council waits at least as long as the slowest model at each of three steps. We did not time any of these tools.
Agreement is not correctness. Models trained on similar data can make the same mistake. Kim et al. (arXiv:2506.07962, ICML 2025) report substantial correlation in model errors across more than 350 models; on one leaderboard dataset, two models agreed 60% of the time when both were wrong, and larger, more accurate models had highly correlated errors. Unanimous agreement from a council is a signal, not proof.
Models favor their own writing. Panickssery et al. (arXiv:2404.13076) found that models such as GPT-4 and Llama 2 can recognize their own output and rate it higher. This is one reason councils hide model names at the review stage and why it matters whether a reviewer comes from the same provider as the author.
A synthesis can hide a split. When a chairman or synthesizer writes one answer, a real disagreement between models can be smoothed into confident prose. Some tools reduce this by also showing the individual answers or listing disagreements. We did not find a head-to-head comparison of ranking-plus-chairman against marking disagreements, so a claim that either is better is unproven.
Benchmark gains are reported, not guaranteed for your task. The debate and mixture-of-agents papers report gains on their benchmarks, and the vendors of some hosted tools report their own results. We do not repeat those numbers, because they depend on the models, tasks and judges used, and none of them was measured on your question.

Hosted and self-hosted options (as of 2026-10-12)

Each row below says what the product’s own page states on 2026-10-12. We did not run any of these tools, so “says” means the page says it. Features, plans and prices change; open the page before you decide.

Option	Type	What its page says
karpathy/llm-council	Self-hosted, open code	A local web app (FastAPI backend, React frontend) that sends your query to several models through OpenRouter. You need an OpenRouter API key and credits. The README says the project was “99% vibe coded as a fun Saturday hack”, that the author is “not going to support it in any way”, and that he does not intend to improve it. The default model list in the README names models from November 2025.
OpenRouter Fusion	API tool (beta)	A server tool, documented as beta, in which a panel of 1 to 8 models answers in parallel, an analyst model compares the answers into consensus points, contradictions and unique insights, and the calling model writes the final answer. The documentation says the API and behavior may change.
llmcouncil.ai	Hosted chat app	Run by Evolo Pty Ltd according to its footer. On its Pro plan, “multiple AI models answer independently, challenge each other’s thinking, and bring the useful differences into one response”. Its entry plan uses one model. The page does not mention Karpathy’s project, and we could not confirm any link between them.
AIsCouncil	Browser app	Offers Council, Debate, Mixture of Agents, Consensus Vote, Compare, Arena and Smart Router modes. The page says there is no backend server: conversations are stored in your browser and messages go from your browser to each AI provider. It lists a free plan that supports your own API keys and Council mode. A “maintenance in progress” banner was showing when we opened it.
council-ai.app	Hosted app	Sends your question to the models you choose, 7 on Plus and 10 on Pro and Ultra. By default no model sees another’s answer; a moderator removes model names, combines the answers into one verdict with a numeric consensus score, and lists disagreements. We saw only paid plans.
Suprmind	Hosted app	Five models in one conversation, in several modes. In Super Mind “all five respond simultaneously” and a sixth model synthesizes one answer; in Sequential, Debate, Red Team and First Principles each model reads what came before. Its page does not list a council mode.
Perplexity Model Council	Hosted feature	On the web, a council runs three models (default: one each from Anthropic, OpenAI and Google) and a synthesizer model resolves one combined answer. The Help Center says that on web and iOS it is included for Max and Enterprise Max plans, and that in Perplexity Computer it runs as a skill in all paid plans, using credits.
Microsoft 365 Copilot Researcher Council	Enterprise feature	Microsoft’s message center lists it as “In development” (published March 30, 2026; last updated August 4, 2026), with a stated preview date of March 2026 and a general availability date of December 2026.

Where Qriple differs

Qriple is not an LLM council. It is a web-based AI workspace built around a different shape: one model writes the main answer, other models answer the same prompt on their own, and Qriple marks where they disagree. This is how a turn works today.

	Council pattern (Karpathy’s version)	Qriple
Who answers	Every council model answers; a chairman model writes the final answer.	One model writes the main answer. Up to two other models answer the same prompt as reviewers (one on Lite).
Who sees whom	At the review stage, each model reads the others’ answers, names hidden.	Reviewers never see the main answer of the turn. They get the conversation before the turn and an instruction to answer “on your own, without referring to any other answer”, in under 200 words.
What you get back	Rankings and one compiled final answer.	The main answer, left whole, with marks on it: agree, diverge, only the reviewer said it, only the main answer said it. At most 8 marks, each tied to a quote. A mark whose quote does not match the text is dropped.
Merged answer	Yes, written by the chairman.	No. You can switch to a reviewer’s full answer and continue from it; that switch is charged as a main answer for that model.
Who the other models are	Your configured council.	You choose them, or let the picker fill the slots. A reviewer can come from the same provider as the main model, and the app then warns that the review may be less independent; the default picker prefers a different provider.

Limits you should know. The marks come from a separate model call, so they are a model’s judgment anchored to quotes, not a fact check, and nothing in the product labels an answer as correct. Reviews are short, so a point the main answer makes and a reviewer skips shows up as “only the main answer said it”, which is not the same as a finding that it is wrong. We have no accuracy measurement for the marks. Qriple does not run your code or tests, and we have not compared it head to head with ranking-plus-chairman councils, so we do not say one is better.

The comparison step that creates the marks uses 0 credits; the answers are charged. Plans and the credit meter are on the pricing page. To see how Qriple differs from specific multi-model tools, see the comparison pages, and for the manual method behind all of this, see how to compare ChatGPT, Claude and Gemini answers side by side.

Want to see one turn? The Free plan is a 3-turn trial (6 trial credits, no reset). Try the main answer plus reviewers flow free.

Try the blind version by hand

You do not need any product to get the core of this. It costs a few minutes per question.

Write one prompt that carries all the context a model needs.
Send the identical prompt to two models in separate fresh chats. Do not paste either answer into the other chat.
Put the answers next to each other and list the claims that differ: numbers, names, recommendations, caveats.
Check each differing claim against a primary source. Do not settle it by asking a third model to pick a winner.

Which to use

Self-host if you want control: your own API keys and data path, your own prompts, and the freedom to change the stages. You pay each provider per call and you maintain the code, and Karpathy’s repo comes with no support.
Use a hosted product if you want to ask questions without running code. You pay a subscription, you work within its limits, and you rely on how the vendor describes the stages, which you cannot change.
Use a ranking-plus-chairman council if you want one finished answer and accept that a split between models may be smoothed over. Use independent answers with marked differences, as Qriple does, if you want the main answer kept whole and the splits visible. Neither has been shown better in general.

Frequently asked questions

Is an LLM council the same as an ensemble?

Close, but the terms are used loosely. Both send one task to several models. An ensemble in the classic sense combines outputs by a fixed rule such as a vote. “LLM council” usually means models that also read and judge each other’s answers, often with a chairman writing the final answer, as in Karpathy’s repo.

Does an LLM council make answers more accurate?

Some papers report gains on benchmarks, but we know of no evidence that it holds for your question, and models can share the same mistake. Treat a council’s output as a lead to check, not as a finished fact.

Is Qriple an LLM council?

No. Qriple has no chairman, no peer ranking and no merged final answer. It has one main answer, reviewers that answer the same prompt on their own, and marks where they differ.

Sources

Product pages and repository opened 2026-10-12; papers are the arXiv abstract pages.

Descriptions of other products come from their own pages on 2026-10-12 and are not tested claims. Plans, limits and model lists change; check each service before you decide. Statements about Qriple describe the product as of 2026-10-12. Qriple is not affiliated with, built on, or endorsed by Karpathy’s project or any product named here.