Best Open Source LLM by Task: The Test Protocol for Coding, Research, Writing, and Reasoning

Status of this protocol

This is an unscored test protocol. When NewsForge runs a round, we will publish the test date, exact model versions, prompts, rubric, run conditions, source links, and the scoring sheet. Until then, the sections below describe the method, not results — there is no winners table, no pass rate, no VRAM figure, no per-token price, and no model shortlist, because none of those has been measured.

What you can use today is the method itself, described in enough detail to execute on your own prompts without waiting for us. One further note on evidence: this article links only to NewsForge coverage, so primary sources — the Open Source Initiative's Open Source AI Definition, vendor model cards, exact license texts, benchmark papers and maintainer repositories, runtime documentation, provider pricing pages, and the EU AI Act — are named rather than linked, and described only in general terms. Read them at source and note the date you read them.

Why a single "best open source LLM" ranking misleads you

What composite benchmark scores actually average together

A composite leaderboard score collapses different abilities into one number. Multiple-choice knowledge tests, code-generation pass rates, long-context recall, and instruction following measure different things, and a model that is strong at one can be mediocre at another. Averaging them produces a ranking that describes no particular user's workload. If you spend your day debugging a large repository, a model's score on multiple-choice science questions tells you little about that day.

Public leaderboards such as the Hugging Face Open LLM Leaderboard, and human-preference systems in the LMArena / Chatbot Arena style, publish their own methods and answer their own questions. Those questions are not necessarily yours. Preference-based arenas rank responses by rater choice under whatever formats and categories the operator currently runs; those formats change, so before you lean on an arena ranking, read the operator's current methodology page and check what interaction length and task mix it covers.

The narrower version of the criticism is the defensible one: a ranking that neither publishes its own working nor cites an external test you can inspect is a number you are being asked to take on trust.

Benchmark contamination and overfitting concerns

Benchmark contamination — test items, or close paraphrases of them, appearing in training data — is a widely discussed problem in machine learning evaluation, and it is one reason a public score can outrun hands-on experience. Related to it is ordinary overfitting: when a benchmark becomes a marketing target, models get tuned toward its format.

Neither concern proves that any specific model is contaminated, and no such allegation is made here about any model. It is a reason to weight your own prompts more heavily than a published figure, not a reason to dismiss published figures.

The public benchmarks a scored round would reference are the familiar ones: HumanEval and SWE-bench for code, MMLU and MMLU-Pro and GPQA for knowledge and hard science questions, MATH and AIME-style problem sets for mathematics, and needle-in-a-haystack tests for long context. Our citation policy for them is simple and stated once here: any score quoted from a benchmark is cited to the original paper or maintainer repository at first mention, and labeled vendor-reported where it came from a vendor announcement rather than an independent run.

Why your prompt style changes the ranking

Models respond differently to system prompts, few-shot examples, output-format instructions, and sampling settings. A model that looks weak under a terse zero-shot prompt can look stronger given a structured brief, and the reverse happens too. This is one plausible reason public leaderboards can disagree with your hands-on experience: the leaderboard's prompt harness is not your prompt harness. It is also why a prompt set should be published rather than described. A described prompt set cannot be reproduced or contested.

Cost, latency, and license as tiebreakers benchmarks ignore

Per-token price, tokens per second, time to first token, context window length, and license terms rarely appear in a benchmark score at all. Whether they decide your choice depends on how close the quality difference is for your task — which is something you measure, not something to assume in either direction.

The conditional guidance is safe to give: if two models perform acceptably on your own scored task set, then price, latency, and license are the remaining variables, and a license restriction that blocks your distribution model cannot be offset by any quality margin. That is a rule about the order of decisions, not a claim about how any particular pair of models compares.

What "open source" means here (and why the label is contested)

Open weights vs. open source under the OSI definition

The two terms are not synonyms, and the distinction matters legally and practically.

Open weights means the trained model parameters are downloadable, so you can run the model on your own hardware, inspect its outputs offline, and often fine-tune it. It says nothing about the license attached and nothing about whether training data or training code was released.

Open source AI is a contested term. The Open Source Initiative has published an Open Source AI Definition setting out what a system must make available — beyond weights — to be described that way. This article does not paraphrase its clauses; read the current definition on the OSI's own site, note its version number and date, and check a model's actual license and released components against it.

Because that check has not been performed here for any specific release, no named model is classified as open source or not open source. Open-weight is used as the general term for the comparison universe, and "best open source LLM" is reserved for the search phrase readers actually type — a phrase that is looser than the OSI definition.

Permissive licenses vs. community and custom licenses

Two broad families are worth telling apart, with the caveat that neither family is uniform and neither is free of obligations.

Permissive licenses — Apache 2.0 and MIT are the common examples — impose relatively light conditions, typically centered on attribution and license retention; Apache 2.0 additionally addresses patents and notice requirements in ways MIT does not. Community or custom licenses are written by the model publisher and can add conditions: acceptable-use policies incorporated by reference, naming or attribution requirements for derivatives, restrictions tied to the size of your user base, or limits on using outputs to train other models.

"Permissive means fewer surprises" is a rough heuristic, not a legal conclusion. Trademark terms, notice obligations, third-party dependencies, dataset provenance, and export or use restrictions elsewhere in a distribution can all matter regardless of which family the headline license belongs to.

Models associated with the Meta Llama family, Mistral AI, Alibaba's Qwen, DeepSeek, Google Gemma, Moonshot AI's Kimi, Microsoft Phi, and OLMo from the Allen Institute for AI are distributed under terms that differ from one another and that have differed between releases within the same family. No specific model's license terms are restated here; a scored round will link each tested release to its official model card and the exact license text, and state the license family and commercial-use position from that primary source.

The licensing gate

Licensing is a gating criterion: verify the exact release's current terms before comparing quality. Work through the checklist for the specific build you plan to ship:

• Is commercial use permitted outright, or conditionally?
• May you redistribute weights, or only outputs?
• Must fine-tuned derivatives carry the original name, notices, or license?
• May outputs be used to train other models?
• Is an acceptable-use policy incorporated by reference, and does it change independently of the license?
• Are there patent, trademark, or indemnity terms that affect your product?

"Which open-weight models can I legally use in a commercial product?" has no general answer; it has a per-release answer. For a material commercial deployment, obtain legal advice. Nothing in this article is legal advice, and the licensing and regulatory material here needs a legal review before any version of it is published.

How we plan to test: prompt sets, rubrics, and run conditions

The table below is the single source for the mechanics — prompt counts, repetitions, blinding, variance, and tie rules. The category sections that follow cover only what is specific to each task.

Protocol at a glance

Each round's model list — exact release names and version or build identifiers, not vendor families — is fixed before testing begins and published with the prompts. Results from different quantization levels are reported separately and never merged.

Measurement versus judgment

The two are reported separately, and the distinction is the reason the rubric exists.

Objective items — unit tests, extraction schema validity, needle recall, final math answers — are scored automatically as pass or fail. They are measurements.

Subjective items — prose quality, faithfulness judgments, code readability — are rated by three human reviewers against a written rubric, with model identities stripped from outputs. These ratings are numeric, and the results will say so. What they are not is measurements of an objective property: they are structured expert judgments, reported as a median with the full spread of the three ratings shown, plus an agreement statistic. A single-number "writing score" with the disagreement hidden is what this protocol refuses to print — not numbers as such.

Known limitations

Fifteen prompts per suite is small relative to academic benchmarks. That is a deliberate trade for tasks resembling real work, and it makes the tie rule load-bearing rather than decorative: where two models fall inside the observed run-to-run spread, the comparison is declared too close to call rather than resolved by rounding.

The suites will not measure safety alignment, multimodal ability, fine-tuning headroom, or performance on domain-specific corpora. They will measure a handful of common professional jobs, tested the same way for every model, with small samples and disclosed variance.

Coding: generation, debugging, and repo-scale comprehension

"Which open source LLM is best for coding and debugging?" is the most-asked version of the question and the one where a single number is least informative. Single-function generation and repository-scale tasks exercise close to different skills: the first rewards clean isolated synthesis, the second rewards locating relevant code, respecting existing structure, and making a minimal change.

The coding suite separates four things:

• Generation — a function written to spec, checked against hidden unit tests, pass or fail.
• Debugging — a repository with a failing test, scored on whether the fix passes and whether it introduces regressions.
• Convention-following — an edit deep in a long file, scored on whether the model matches surrounding style and imports rather than rewriting them.
• Agentic tool use — a loop where the model must call a tool, read the result, and correct course, scored on task completion and on wasted turns.

Failure modes worth watching for in your own runs, offered as things to look for rather than as measured frequencies: confidently inventing a library function; fixing the symptom the failing test reports rather than the cause; and losing track of file state across an agent loop.

Research and summarization: long context, citations, and faithfulness

Hallucinations are especially consequential in research and summarization, because a fabricated citation is a plausible-looking sentence a reader is unlikely to check.

The suite scores five things:

• Multi-document synthesis — reconcile several supplied documents into one answer; scored on coverage of key points and on whether claims appear that are not in the sources.
• Needle-in-a-haystack recall — a specific fact placed at varying depths in a long input; scored on retrieval.
• Faithfulness — unsupported statements counted, and separately, citations that correspond to no supplied source. The fabricated-citation rate is reported as a count.
• Contradictory sources — does the model flag the conflict or silently pick a side?
• Structured extraction — schema-valid output across repeated runs, which matters more than average quality when the output feeds a pipeline.

If your job is summarizing long documents without fabricated citations, weight faithfulness and contradiction handling above general quality when you score models yourself, and consider retrieval-augmented generation (RAG) with enforced quoting rather than trusting a long context window alone. A large advertised context window is a vendor specification; whether a model uses the middle of that window well is an empirical question, which is what the needle test is for.

Writing: voice, structure, and editability

Writing resists automated scoring, which is why this category is human-rated rather than measured.

Reviewers rate four things:

• Holding a brief and a style guide across a long piece, versus drifting back to a default register after a few hundred words.
• Sentence-level style tells — repeated openers, formulaic three-part lists, recurring transitional phrases, restating one point three ways.
• Rewriting to instruction, a distinct skill from drafting and often the one professionals actually need.
• Tone shifts and non-English output, where fluency and register can diverge sharply from English performance.

Editability is the practical criterion. A draft that is largely right and cleanly structured is usually worth more than a slicker draft whose errors are woven through every paragraph. For prose, reviewer disagreement is information, so results will name the cases where reviewers diverged rather than averaging them away.

Reasoning: multi-step problems, math, and planning

Reasoning modes and thinking-token overhead

Several recent open-weight releases expose an extended-thinking or reasoning mode that spends additional tokens on intermediate steps before answering. Whether reasoning-mode models outperform standard models on multi-step problems, and at what cost, is an empirical question this suite is built to answer by reporting accuracy and token consumption together.

The arithmetic matters: a mode that gains accuracy while multiplying output tokens carries a price and a latency cost, and for some workloads that trade is bad even when the accuracy gain is real. Token counts are reported for every reasoning-mode run so you can do that arithmetic against your own budget.

Math, logic, constraint satisfaction, and self-correction

The suite uses multi-step math and logic items scored on final-answer correctness; planning tasks with explicit constraints scored on whether every constraint is satisfied; and a self-correction round in which the model is told its answer is wrong without being told why, and scored on whether it finds the error or defends the mistake. Sycophantic agreement with incorrect feedback is scored as a failure.

Cost, hardware, and deployment reality check

VRAM, quantization, and what it costs you

Memory requirements scale with parameter count, weight precision, and context length. The key-value cache grows with the context you actually use, which is why a model that loads comfortably can still run out of memory mid-conversation. Quantization formats such as GGUF, AWQ, and GPTQ reduce numeric precision to cut memory, generally with some quality cost that tends to increase as precision drops.

How much quantization hurts quality is model- and task-dependent, and which task types degrade first is not something this protocol has tested. It is a good thing to measure on your own tasks: run your prompt set at two bit depths and compare.

Mixture-of-experts (MoE) architectures complicate the arithmetic: total parameters determine how much memory holds the model, while active parameters per token drive compute. An MoE model can be quick per token and still demanding to host.

No VRAM figures or per-token prices appear here. Those numbers come from vendor documentation and provider pricing pages, they change frequently, and a figure worth acting on needs an access date attached. Pull them yourself from the primary page, write the date next to the number, and re-check before you commit budget.

Runtimes and hosted inference

Locally, llama.cpp underpins much of the quantized CPU and consumer-GPU ecosystem, Ollama wraps local model management in a simpler interface, and vLLM targets high-throughput GPU serving for multi-user workloads. Consult each project's own documentation for supported formats, hardware requirements, and current features; capabilities move quickly.

Hosted routes include OpenRouter and other providers serving open-weight models via API, which removes the hardware question at the cost of sending your data to a third party.

On whether hosted open-weight models are cheaper than closed frontier APIs: compare published per-token prices for the exact models and providers you would use, on their current pricing pages, with an access date. Then compare cost per successful task, not cost per token — a cheaper model that needs reasoning tokens or several retries can cost more per completed job than a pricier one that succeeds first time.

Throughput, latency, and privacy

Tokens per second and time to first token trade against each other and against batch size and quantization. Interactive chat is sensitive to time to first token; batch document processing is sensitive to throughput.

Self-hosting is often chosen for reasons unrelated to quality: keeping regulated or confidential data inside your own network, meeting data-residency requirements, and avoiding dependence on a provider that can deprecate a model or change terms.

Can open-weight models match closed models for your workload?

An open-versus-closed comparison requires matched run conditions, named releases, and a test date, so no claim about which is ahead is made here. What is defensible is the method. If the question matters to your decision, include a closed-model baseline in your own harness: same prompts, same rubric, same scorer, identities stripped, and cost per successful task recorded for each. That produces an answer scoped to your workload, which is the only scope in which the question has a stable answer.

Deployment risk: a short checklist

Self-hosting moves risk rather than removing it. Four areas are worth planning for before an open-weight model reaches production; for the version of this analysis your organization should rely on, consult current authoritative guidance, such as that published by NIST or OWASP.

Prompt injection. If a model reads untrusted content — a web page, an email, a document, a code comment — that content can carry instructions the model follows. Where the model has tools that write files, call APIs, or execute code, an injection becomes an action. The mitigations are architectural: least-privilege tool scopes, human approval for irreversible actions, treating retrieved text as data rather than instructions, and logging every tool call. NewsForge's coverage of how AI is reshaping cybersecurity work sets out where automation helps defenders and where it opens new exposure.

Model provenance. Supply-chain risk applies to weights much as it applies to software packages. Community fine-tunes and re-quantizations are numerous on Hugging Face and vary in provenance. Prefer official repositories, verify checksums, prefer serialization formats that do not execute code on load, and pin versions. Where a model comes from and who maintains it can also affect procurement approval independently of quality.

Runtime guardrails. Model weights ship with whatever alignment training the publisher applied; they are not a runtime policy layer. Some deployment packages, serving stacks, and hosted providers add moderation, filtering, or policy controls, and some do not — check what your specific stack provides rather than assuming either way. Input filtering, output checking, rate limits, and audit logging are the deployer's responsibility to verify, and fine-tuning can weaken safety behavior a base model had.

Compliance records. Organizations deploying general-purpose AI models in the EU should review the EU AI Act, which sets obligations for general-purpose AI including documentation and transparency duties, and which distinguishes among providers, downstream deployers, and models designated as carrying systemic risk. Read the current regulation and the European Commission's current implementation guidance, and take legal advice for your situation. Regardless of jurisdiction, keep a record of which model version, quantization, and prompt template produced which output; that record is what makes an incident investigable.

Pick your model: a decision path

Each branch tells you what to measure rather than what to buy, so you can act on it today.

If you need it to run on a laptop. Start from your memory ceiling, not a leaderboard. Choose the largest model that fits comfortably at a quantization level you have tested on your own tasks, leaving headroom for the context you actually use. Ollama and llama.cpp are commonly used options for local setup; check their current documentation for supported models and hardware.

If you need permissive commercial licensing. Filter first, compare second. Shortlist only releases whose current license text — read directly, for the exact build you will ship — permits your use, redistribution, and derivative plans, using the licensing gate checklist above.

If you need the strongest reasoning regardless of cost. Test reasoning modes with token accounting switched on, include a self-correction round, and evaluate on your hardest real problems rather than puzzle sets. Budget for latency as well as tokens.

If you need one generalist for mixed workloads. Score your two or three most frequent tasks and choose for acceptable worst-case performance across all of them rather than the best average. A model that never fails badly on your recurring high-priority work is usually a better generalist than one that peaks on a single task type.

If you want to run this test yourself. Apply the protocol above to prompts from your actual work. Take fifteen to twenty real prompts, with known-good answers where possible. Write a one-page rubric before you look at any output. Run each prompt at a fixed temperature, three times per model, with model identities stripped, and score blind. Report the spread; if two models fall inside it, treat them as tied and choose using license, cost, latency, or hardware fit. Record model version, quantization, context length, sampling settings, hardware or provider, and the date — a score without those is not reproducible, including your own.

Update policy

Open-weight releases arrive frequently and any task-level verdict has a short shelf life, which is the main argument for dating every result and publishing prompts rather than only conclusions.

After the first scored round, NewsForge plans quarterly retesting, with additional rounds when a release meeting the published eligibility criteria could change a category result. Each update will be dated and recorded in a changelog, and where a re-test overturns an earlier verdict, the change is recorded rather than quietly edited. The suites will improve if they include tasks from people whose work they are meant to represent: a real failing repository, a real style guide, a real extraction schema.