Browse the directory

Build repeatable eval suites that catch quality regressions in AI agent behavior before merge or release.

Open-source observability platform purpose-built for AI agents, with OpenTelemetry-native tracing, plain-English signals, an evals SDK and CLI, SQL dashboards, dataset annotation, and MCP/CLI access, self-hostable with Apache-2.0 SDKs for Python and TypeScript.

Debug, evaluate, and monitor LLM applications from Claude — read traces and spans, score outputs, save prompts, run evaluation experiments, and query project metrics — with the official Opik MCP server by Comet.

Open-source TypeScript agent engineering framework and platform for building AI agents with tools, memory, workflows, RAG, guardrails, evals, MCP, voice, and VoltOps observability.

Open-source framework from OpenAI for evaluating LLM and agent behavior with reusable eval definitions, grading logic, datasets, and regression workflows.

Python agent framework from the Pydantic team for type-safe GenAI apps, tools, structured outputs, MCP, evals, and durable workflows.

Source-backed guide for converting OpenAI Agents SDK traces into regression eval cases, trace grades, tool-call assertions, and release checks for agentic workflows.

A source-backed collection for building repeatable LLM eval and prompt testing workflows with open-source tools: prompt regression tests, RAG and agent metrics, human review datasets, traces, prompt optimization, and release gates.

Inspect LLM traces and spans, manage prompts, explore datasets, and review evaluation experiments from Claude — with the official Arize Phoenix MCP server, built into the open-source Phoenix AI observability platform.

Connect Claude Code, Cursor, Copilot, Windsurf, and other MCP clients to the public Langfuse documentation MCP server for tracing, prompt management, evaluation, and agent observability implementation help.

Slash command runbook for designing and running prompt evaluations: define tasks, success criteria, golden outputs, regression checks, and privacy-safe reporting using Anthropic test-and-evaluate guidance.

A practical guide for comparing AI coding tools with repeatable benchmarks, fixed task sets, controlled environments, transparent scoring, and privacy-safe artifacts.

Open-source evaluation framework for testing RAG systems, prompts, agents, workflows, and other LLM application behavior.

Source-backed agent for designing Inspect AI benchmark tasks, datasets, solver plans, scorer rubrics, model matrices, eval logs, and release-quality prompt evaluation decisions.