Open Source AI Agents: A Practical Guide to Evaluating Them

The Open Source Initiative’s Open Source Definition sets out the distribution terms a licence must meet — free redistribution, permitted derived works, no discrimination against fields of endeavour or against persons or groups, among others. The point that matters for buyers is that open source is a property of the licence terms, not of source code being visible or downloadable at no cost. Software published under terms that fail those criteria is not open source, whatever a landing page says.

NewsForge analysis: that distinction is where evaluation should start, because software described as an agent application, framework or orchestrator creates different implementation and review questions.

This is a decision framework rather than a ranked directory. It gives you a repeatable evaluation method, a worksheet to fill from primary sources, a comparison of what two projects’ own documentation says, and a security checklist grounded in published standards. What it deliberately does not do is name a winner.

How to Read This Guide

Two distinctions run through everything below.

Sourced versus editorial. Where a claim rests on a published standard — the Open Source Definition, NIST’s AI risk framework and generative-AI profile, NIST’s secure software development practices, OWASP’s agentic threat model — it is cited in context. Where it is NewsForge’s own judgment about how to make a decision, it is labelled NewsForge analysis.

Maintainer claim versus verified finding. A capability documented by the people who build a project is a maintainer claim: useful, but interested. It becomes a finding when you reproduce it. NewsForge has not run independent benchmarks of any project in this category, and this guide asserts no project-specific licence identifier, performance result or security guarantee.

Methodology and sourcing note

This guide is a methodology article. Its standards-based claims come from the Open Source Initiative’s Open Source Definition, the NIST AI Risk Management Framework and its Generative AI Profile, the NIST Secure Software Development Framework, and the OWASP Agentic Security Initiative’s threat-and-mitigation reference. Each is linked at the point of use.

Two projects are compared below — LangChain and CrewAI — strictly on the basis of their own current first-party documentation, and every statement about them is attributed to their maintainers. They are included because first-party documentation was available for review, not because of any measurement of popularity, adoption or performance.

Project-level facts such as licence identifiers, supported deployment paths, release cadence and security-process details are not asserted here. Establishing them requires reading each project’s current repository licence file, official documentation, release records and published security policy at the moment of evaluation. The worksheet below is the deliverable: a structure for capturing that evidence yourself, with a date and a link on every cell.

One corollary matters more than it sounds: absence of documentation is not evidence of absence. The correct label for a capability you cannot find evidence of is "not documented," never "not supported."

What Are Open Source AI Agents?

Agent vs. chatbot vs. workflow

What follows is a working description — NewsForge analysis, offered as editorial taxonomy rather than as a term of art. Projects describe themselves in their own vocabulary, so check how any specific project defines "agent" in its own documentation before assuming it matches this description.

A chatbot responds to a message. A workflow executes a sequence a human defined in advance. An agent sits between them: a language model is given a goal, a set of tools it can call, and latitude to decide which tools to call and in what order, looping until it judges the goal met or a limit is reached.

The consequential part, in NewsForge’s editorial view, is not the model. It is the tool access. A system that only produces text reaches nothing outside the conversation. A system that can run shell commands, query a database, send email or open a pull request reaches whatever those tools reach. OWASP’s agentic threat model supports treating that surface as the security-relevant one: in Agentic AI – Threats and Mitigations, the Agentic Security Initiative describes threats arising from agent goals and reasoning, tool invocation, agent identity, memory, permissions and multi-agent interaction — categories that exist because agents act, not merely because they generate text.

Application vs. framework vs. orchestrator

The three-layer split below is NewsForge analysis. It is not an industry standard, and projects do not always describe themselves in these terms.

A framework or library. Code you import to build your own agent. You supply the models, the tools, the deployment and the operations.
An orchestrator. Software that coordinates multiple agents or multiple runs — routing tasks, managing state between steps, handling retries and hand-offs. Sometimes part of a framework, sometimes standalone.
A complete agent application. Something you can run and use, with an interface, that performs a category of work end to end. A coding agent is one example.

These are not the same purchase. A framework is something you build with; an application is something you use. When a listicle puts both in one table, it is comparing a box of parts to a finished car.

NewsForge analysis: to place a given project, read its own documentation and quickstart. If the quickstart hands you a running program, it is closer to an application. If it hands you an import statement and a class to subclass, it is a framework. Record which, with a link, in the worksheet’s Layer row.

What "open source" should mean here

Returning to the Open Source Definition, NewsForge recommends treating a project’s current licence file as the starting point for legal review and checking whether the publisher identifies additional terms for particular components or services.

NewsForge analysis: do not infer rights to use commercially, modify, redistribute or host from the words "open source" on a marketing page. Where terms are ambiguous, or where a project applies different licences to different components, NewsForge recommends legal review.

How to Evaluate an Open-Source AI Agent Project

The criteria below are NewsForge analysis — an editorial methodology, not a standard. They align with a principle NIST sets out in its AI Risk Management Framework: AI risk is context-dependent, and risk management should govern, map, measure and manage risk for the specific context of use rather than in the abstract. NIST’s Generative AI Profile extends this to generative systems, with actions including context-specific testing, ongoing monitoring, documentation and incident response planning.

The editorial claim being made is narrow: a shortlist built from these questions is more defensible than one built from star counts.

Licence and commercial-use terms. Open the repository. Open the licence file. Read it. Then check whether separate terms apply to a hosted product, an enterprise edition or specific plugins, and whether licensing is uniform across the repository. Record the licence identifier and the date you checked. If your product will be sold, hosted for customers or embedded in a shipped binary, that record is what your legal team will ask for.

Supported models, tools and integrations. Which model providers does the documentation name? Can you point the software at a locally hosted model? Which vector stores, databases and third-party APIs have documented adapters? "Supports any model" is a claim to verify — check whether it means a maintained integration or an abstract interface you are expected to implement yourself.

Single-agent and multi-agent orchestration. NewsForge analysis: ask whether your workflow genuinely needs several specialised agents coordinating, or whether one agent with good tools would do the same job with fewer moving parts. If you do need coordination, ask what the project documents about state passing, hand-offs, termination conditions, and behaviour when one agent in a chain fails. OWASP’s agentic threat model identifies multi-agent interaction as a threat category; NewsForge analysis is that the coordination layer therefore deserves security attention as well as functional attention.

Deployment, hosting and data control. Does the documentation describe a supported path to run the software inside your own network, and what does that path require — a container runtime, a database, a message queue, a GPU? Ask the maintainers’ documentation whether any component reports usage data by default and whether that behaviour is configurable; record the answer with a link rather than assuming one. NewsForge analysis: for regulated data, the question is not whether self-hosting is possible in principle but whether the documented, supported path is self-hosted.

Observability, testing and human oversight. Ask what tracing the project documents, what it records about model calls and tool invocations, what evaluation or replay tooling exists, and whether there is a documented way to pause for human approval at defined points. The NIST Generative AI Profile emphasizes measurement, monitoring, documentation and incident planning; where a project’s documentation does not cover supporting instrumentation, NewsForge analysis is that you should budget to build it.

Security controls and permission boundaries. What is the unit of permission — the agent, the tool, the session? Does the documentation describe read-only tool access? Does it describe confining execution? Does it describe how credentials reach a tool? Documenting a control is not the same as being secure. NewsForge analysis: a project that ships a permission model has given you something to configure, not an assurance.

Documentation, releases and maintenance signals. Look at tagged releases, changelog and commit history yourself. Are releases regular? Does the changelog explain breaking changes? Do issues receive responses? Note that repository activity does not by itself tell you who holds merge rights or how governance works — if that matters to you, look for a published governance or maintainers document, and treat it as absent if you cannot find one. NIST’s Secure Software Development Framework gives a more useful maintenance question than any popularity metric: does the project have a defined process for receiving, triaging and responding to vulnerability reports, and does it protect the integrity of what it ships?

Infrastructure and operating costs. A free licence tells you nothing about operating cost. See the cost section below.

Open-Source AI Agent Projects to Compare

NewsForge analysis: discovery is the reader’s step, and it should be deliberate rather than borrowed from a listicle. Three surfaces are worth working through: repository topic pages for agent-related projects, the projects your own engineers already have hands-on experience with, and the tools already embedded in your stack that have added agent features. Vendor comparison pages are a source of candidate names, not of facts about competitors.

For each candidate, the evidence you need lives in four documents: the repository’s current licence file, the official documentation and quickstart, the tagged release history and changelog, and the published security policy together with any advisories.

LangChain and CrewAI: what the maintainers document

The comparison below is limited strictly to what each project’s own current documentation says. These are maintainer claims, not independently verified findings. NewsForge has not benchmarked either project. Verify each project’s current licence file yourself before relying on any commercial-use assumption, and test performance on your own workload rather than inferring it from documentation.

| | LangChain (maintainer-documented) | CrewAI (maintainer-documented) | |—|—|—| | Self-description | The LangChain overview describes LangChain as an open-source framework for building agents, with an agent architecture and integrations across models and tools. | The CrewAI documentation describes a framework for composing AI agents and orchestrating their work. | | Core building blocks documented | Agent architecture with model and tool integrations, documented as built on LangGraph. | Agents, crews, and flows — flows documented for orchestrating stateful workflows. | | Emphasis in the documentation | Building agents on a framework layer with pluggable models and tools. | Composing agents into crews and coordinating multi-step, stateful workflows. | | Licence | Not asserted here — check the current repository licence file at the time you evaluate. | Not asserted here — check the current repository licence file at the time you evaluate. | | Performance | Not documented in a form NewsForge has independently verified; test on your own workload. | Not documented in a form NewsForge has independently verified; test on your own workload. |

Best-fit reading of those documented positions. NewsForge analysis, based only on the self-descriptions above: both sit at the framework layer rather than being finished applications, so both imply integration work. LangChain’s documentation foregrounds an agent architecture with model and tool integrations built on LangGraph, which makes it a candidate to examine when your primary need is constructing an agent with pluggable models and tools. CrewAI’s documentation foregrounds agents, crews and flows for orchestrating stateful workflows, which makes it a candidate to examine when your problem is naturally expressed as several coordinated roles moving through a multi-step process. Neither of those readings is a performance claim, and neither substitutes for your own test.

Limitations of this comparison. It rests on self-descriptions from two projects’ documentation. It does not establish licence terms, deployment requirements, model coverage, security controls, release cadence or reliability for either project. It is not a shortlist, and the absence of other projects here is not a judgment about them — it reflects only which first-party documentation was reviewed for this guide. Build your own shortlist using the discovery approach and worksheet in this section, and fill every cell from primary sources.

At-a-glance comparison worksheet

Copy this table, add a row per candidate, and fill each cell with a link to the primary source you used. NewsForge analysis: add a date column, so that each cell records when the underlying document was read.

Use "not documented" where you cannot find evidence. Do not write "not supported" — that asserts an absence you have not established.

Best-fit use case by layer

Because this guide asserts no project-specific capabilities beyond the documented self-descriptions above, fit guidance is organised by layer. NewsForge analysis:

Well-defined, repetitive, low-risk task; small team — look first at complete applications, which require the least integration work.
Bespoke workflow tied to internal systems — a framework, accepting the integration work that implies.
Several specialised steps with hand-offs — an orchestrator, or a framework with documented multi-agent primitives, but only after confirming a single agent cannot do the job.
Strict data residency or isolated environments — filter on documented self-hosting and documented local model support before anything else.

Important limitations and trade-offs to investigate

NewsForge analysis: these are questions to test, not findings.

Breaking changes. Does the changelog show interface changes between releases, and how are they communicated?
Abstraction cost. Where one interface spans many model providers, does it expose the provider-specific features you need, or only a common subset?
Multi-agent overhead. Does adding agents change the number of model calls for your task, and what does that do to latency and token spend? Measure it on your own workload rather than assuming a direction.
Project-specific defects. Any specific limitation should come from that project’s issue tracker, its documentation, its advisories, or your own testing.
Integration breadth versus depth. A long list of adapters says nothing about whether the two you need are maintained. Check the commit history of the specific integrations you depend on.

Which claims come from maintainers and which are independently verified

NewsForge analysis: in your notes, tag each cell M for maintainer-documented, T for verified by your own test, U for unverified. Anything reaching a decision meeting as a differentiator should be T. Benchmark figures published by maintainers or vendors are interested-party claims: record the test setup and its limitations alongside the number, and do not use them to rank candidates unless you can reproduce them or find independent evidence.

A Practical Shortlisting Process

NewsForge analysis. This entire section is editorial methodology.

1. Start with the workflow and risk level. Write down the task in one paragraph, the systems it touches, and what the worst realistic failure looks like. An agent that drafts internal summaries and an agent with write access to customer records warrant different evaluation bars, even if the underlying software is identical. This mirrors the NIST AI Risk Management Framework, which frames AI risk in relation to context of use.

2. Eliminate incompatible licences or deployment models. Apply hard filters before you look at features. If you must self-host and the project documents no supported self-hosted path, it is out. If the licence restricts your business model, it is out. Doing the cuts first keeps feature comparison from consuming time on candidates that could never be adopted.

3. Test with the same task and model. Compare candidates on one fixed task with one fixed model. Otherwise you risk measuring model differences and calling them framework differences.

4. Measure reliability, latency, intervention rate and cost. On the same inputs: how often did it complete correctly, how long did it take, how often did a human step in, and what did the run cost in tokens and infrastructure? NewsForge treats intervention rate as a first-class metric alongside correctness, because a system that completes tasks only under supervision has a different operational profile from one that does not.

5. Document failure modes before production approval. List every way each candidate failed and what would have happened in production. NewsForge’s editorial preference is to approve a candidate whose failures you can characterise over one whose failures you cannot explain, because only the first can be mitigated deliberately. The NIST Generative AI Profile includes incident-planning actions; NewsForge’s recommendation is to decide in advance what you would do when a tested failure happens with real data.

Security and Governance Questions

OWASP’s Agentic Security Initiative sets out a threat model for agentic systems in Agentic AI – Threats and Mitigations, covering risks arising from goals and reasoning, tool invocation, agent identity, memory, permissions and multi-agent interaction. NewsForge analysis: that structure is a useful spine for due diligence. The checklist below applies it and names mechanisms to confirm in a project’s documentation, rather than asserting that any project implements them.

This guide cites no advisories and names no vulnerabilities. Doing so responsibly requires the original advisory or vulnerability record, the affected versions, the scope and the remediation. Before relying on any project, check its published security policy and any advisories affecting the versions you intend to run, and record affected versions and fixes alongside your notes. Do not generalise one project’s defect to the whole category.

Tool permissions and least privilege. Enumerate every tool the agent can call and the credentials behind each. Scope each to the minimum the workflow requires. Read-only by default; write access as a deliberate exception with an audit trail. OWASP’s agentic threat model identifies tool invocation and permissions as distinct threat categories, so NewsForge analysis is to treat permission design as a security artefact rather than a configuration detail.

Prompt injection and untrusted content. OWASP’s agentic threat model treats content an agent reads and the memory it retains as parts of the attack surface, alongside its tools and permissions. NewsForge analysis: for any agent that consumes external content while holding privileged tools, this is the risk to test first. Test it directly, with adversarial content of your own, against the specific tools that agent can call, and treat mitigation as ongoing rather than solved.

Secrets, credentials and data retention. NewsForge analysis: establish from the project’s configuration and security documentation how credentials reach tools and whether they can be kept out of the model context. Check what your chosen tracing configuration records, how long that data is retained, and where it is stored. Detailed traces are valuable for debugging, and the same detail becomes a liability if it captures secrets or personal data — so decide the retention and redaction policy before you switch tracing on.

Sandboxing and code execution. If the agent generates and runs code, ask what confines it and confirm the answer in documentation and testing rather than assuming it. NewsForge analysis: the boundaries worth verifying explicitly are network egress and filesystem access, because those determine what a confined process can still reach.

Logging, auditability and human approval. NewsForge analysis: you should be able to reconstruct, after the fact, what an agent did and why. Define the actions that require human approval — anything irreversible, anything touching money, anything customer-visible — and confirm the project documents a supported way to gate them.

Dependency and supply-chain review. Whatever tool integrations and plugins you add, review them as you would any other supply chain. The NIST Secure Software Development Framework sets out the practices to look for on both sides of that relationship: verifying the provenance and integrity of acquired software components, protecting the development environment and the code it produces, and maintaining a defined process for receiving and responding to vulnerability reports. NewsForge analysis: those are reasonable things to ask of a project you are about to depend on, and reasonable things to hold yourself to once you ship on top of it.

Related reading (NewsForge coverage, not evidence for this guide): for context on the commercial market that has grown up around criminal tooling, see The Commercialization of Malice: Understanding Subscription Cybercrime and Evolving Digital Threats. On how AI is affecting defensive work, see Will Cybersecurity Be Replaced by AI? Risks, Defenses, and What Changes Next.

Open Source vs. Managed Agent Platforms

NewsForge analysis. The factors below are decision criteria, not measured findings.

When open source may fit. Self-hosting is a documented requirement of yours. You need to modify behaviour beyond configuration. You have engineers who can operate the system and read its source when it breaks. Portability matters enough to justify running the infrastructure.

When managed services may fit. You need a result quickly and cannot staff an integration project. Nobody can be on call for an agent runtime. Support obligations and contractual accountability matter to your customers or auditors. The workflow is common enough that a general product covers it.

Costs a licence price does not capture. Model inference, compute and storage, stateful dependencies such as vector databases, observability and log retention, and engineering time for integration, upgrades, incident response and security review. NewsForge recommends budgeting engineering time explicitly rather than treating it as absorbed by existing headcount. Any figure you produce should carry its assumptions: volumes, model prices taken from first-party pricing pages with the plan, unit, region and conditions stated, and the arithmetic. Without those, it is not a cost estimate.

Questions to ask before committing. Who is on call? What is the upgrade path when the framework breaks compatibility? How do we get off this if the project stalls? Who reviews the security posture, and how often? What happens if a model provider changes pricing, deprecates a model or changes rate limits?

Related reading (NewsForge coverage, not evidence for this guide): on the wider model ecosystem, see Kimi K3: Threat or Menace? Unpacking Moonshot AI’s Global Impact.

How to Run a Proof of Concept

NewsForge methodology, not a standard. Adapt it to your environment. It is consistent with the NIST Generative AI Profile’s emphasis on testing, monitoring and documenting generative systems in their intended context of use.

Define a representative task. One real task from your actual workload — median difficulty, not the demo case and not the hardest edge case. Write the success criteria before you run anything.

Create a fixed evaluation dataset. Real inputs with known correct outcomes, frozen for the duration of the test. Include the awkward ones: missing fields, ambiguous requests, inputs designed to mislead. A fixed set is what makes results comparable across candidates.

Set access and spending limits. Non-production credentials. Scoped, preferably read-only tools. A hard spending cap on model usage. A time limit and a step limit per run, so a loop cannot run unattended overnight.

Record outcomes and failure cases. Per run: outcome, wall-clock time, token cost, number of human interventions, and a note on any failure. Keep full traces for failures, subject to the redaction policy you set earlier.

Review security and operational readiness. Before production, work through the checklist above: permissions, injection testing, secrets, sandboxing, audit logging, approval gates, dependencies and vulnerability response. Write down what you would need to see to approve wider access, and who signs off.

Frequently Asked Questions

Are open-source AI agents free to use commercially? The Open Source Definition includes free redistribution among its criteria, but it does not replace review of the specific licence covering the software you plan to use. NewsForge recommends reading the project’s current licence file, confirming which components it covers, and seeking legal review where the terms or scope are ambiguous.

Can open-source agents run locally? That depends on the project, and the answer belongs in the Self-hosting row of the worksheet with a link to the project’s deployment documentation. NewsForge analysis: note also that self-hosting the agent software is a separate question from running the language model locally — you may self-host the software and still be calling a hosted model API. Confirm both.

What is a multi-agent framework? As used in this guide, software that coordinates several agents, each typically with a defined role and toolset, passing work between them toward a shared goal. CrewAI’s documentation, for example, describes agents, crews and flows for composing agents and orchestrating stateful workflows — a maintainer description of one such design. Adding coordination adds moving parts to test, and OWASP’s agentic threat model identifies multi-agent interaction as a threat category.

How should teams compare AI agent frameworks? NewsForge analysis: with a fixed task, a fixed model, a fixed dataset and the same measurements for each candidate — correctness, latency, human intervention rate and cost per run. Compare against hard requirements such as licence, deployment and integrations before comparing features, and label every capability as maintainer-documented or independently verified.

What security risks should be tested? At minimum: injection through any content the agent reads, over-broad tool permissions, credential handling in context and logs, code execution confinement, and gaps in audit logging or approval gates. OWASP’s agentic threat model also identifies risks around agent goals and reasoning, agent identity, memory and multi-agent interaction. Review the dependency tree and check published advisories for the versions you intend to run.

The Bottom Line

NewsForge analysis: there is no single best open-source AI agent project, because the requirements that decide the question — licence terms, deployment constraints, risk level, existing stack, staffing — belong to the buyer and differ between buyers. This is an editorial conclusion drawn from the range of requirements teams bring to the category, not a measured result. Treat any ranking that does not publish its methodology and its evidence accordingly.

The framework holds: define the workflow and its risk level; apply licence and deployment filters as hard cuts; verify every capability claim against the maintainer’s own documentation while labelling it as a maintainer claim; then test the survivors on one fixed task with one fixed model, measuring reliability, latency, intervention rate and cost. Approve for production only when you can describe the failure modes.

Copy the worksheet into your own document and use it as your checklist. Fill each cell with a link to the primary source and the date you read it. Then run a controlled proof of concept before any production decision. Before you commit, read the current licence file, the release history, the deployment documentation and the published security policy and advisories for the version you intend to run — and take those readings yourself rather than relying on any secondary summary, including this one.