| name | promptfoo-llm-evaluation |
|---|
| description | Build or extend language-agnostic PromptFoo evaluation pipelines for LLM apps and agents, including providers, datasets, rubrics, deterministic gates, and GitHub Actions CI. Use for eval design or integration work, not generic unit tests or telemetry-only observability. |
|---|
PromptFoo LLM Evaluation Pipelines
Use PromptFoo as the versioned, language-neutral evaluation layer around an application’s real runtime. Keep the adapter/provider specific to the application language and framework; keep test cases, assertions, rubrics, and CI behavior declarative and reviewable.
Step 0: read the specified codebase first
When the user specifies a repository or checkout, make that the target and begin with a read-only codebase inspection immediately after loading this skill. Do not begin by asking for repo-specific implementation details. If no target is specified and the current working directory is unambiguous, inspect it; otherwise ask for the repository path before doing eval design work.
Read enough of the target codebase to produce a short inventory of:
- existing eval configs, providers, fixtures/datasets, rubrics, shared clients, retry helpers, report parsers, and CI workflows;
- the production prompt/agent entrypoints and output schemas;
- package-manager scripts and the smallest workflow-equivalent local command;
- repository language/runtime, package manager, PromptFoo version source, and whether the PromptFoo executable/module is installed;
- runtime boot requirements, secrets/proxies, databases, external tools, and test doubles;
- existing report surfaces: artifacts, local viewer, checks, PR comments, dashboards, or none.
Use fast file discovery followed by targeted reads of manifests, source entrypoints, tests, eval files, and workflows; do not treat a filename listing as having read the codebase. Reuse or extend what exists. Do not create a second eval harness, duplicate a provider, or promise retries/comments until the inventory proves they exist or the user approves adding them. In the G2 ai-playbooks checkout, the documented layout is evals/ and the workflow uploads artifacts; verify the live checkout rather than assuming its retry or PR-comment behavior.
Before editing
Inspect the repository before choosing a layout or provider strategy:
- Find the package manager, existing test/eval commands, production prompt or agent entrypoint, output schema, and CI conventions.
- Look for existing PromptFoo configs, providers, fixtures, rubrics, shared clients, environment loaders, and GitHub workflows. Extend established conventions instead of creating a parallel harness.
- Identify whether the application is a simple request/response call, a tool-using loop, a multi-turn conversation, a background job, or a Rails/Next.js/etc. process that must boot per case.
- For the G2
ai-playbooks repository, use the repository-specific section below before changing evals.
Required discovery questions
Ask these only after Step 0 and record the answers before implementation:
- Have you already curated an evaluation dataset covering the application’s happy path and relevant corner cases? (Yes/No.) If No, help curate one from the codebase and the user’s test intent before wiring the workflow. If Yes, inspect and assess the existing dataset rather than asking for an alternate dataset.
- What cadence should the workflow use: every pull request, a scheduled run, manual dispatch, or another repository-approved cadence?
- Which GitHub action/event should trigger the workflow, such as
pull_request, push, workflow_dispatch, or schedule?
