Promptfoo: LLM Evaluations, Test Cases and Cost Boundaries
What is Promptfoo?
Promptfoo is a command-line tool and library for evaluating and red-teaming LLM applications. Its documentation describes provider integrations, automated scoring and CI use. The evaluation workflow is relevant when a team wants a repeatable comparison across prompts, models or application configurations.
Who should use it?
Developers who already manage tests and configuration in a repository can add AI behavior checks to their review process. The setup requires a runnable target and meaningful assertions. A nontechnical user who needs only a few manual comparisons may prefer an interactive playground.
What is free and what can still cost money?
The official Community plan is free and includes evaluation features, local or self-hosted use and a published allowance of 10,000 red-team probes per month. Enterprise and on-premise offerings use custom pricing. Model inference, target application calls and your own compute can still incur costs; a free testing tool does not make the workload free.
What should your first evaluation contain?
Use a synthetic classification task with three allowed labels and an explicit abstain rule. Include an empty input, conflicting evidence and text that looks like instructions but is only data. Compare two prompt versions against the same cases. Check the output schema separately from whether the selected label is correct.
What are useful acceptance criteria?
Require valid output structure on every case, correct labels on the labeled examples and an abstention when required evidence is absent. Inspect every regression rather than accepting a better average. Keep model and configuration identifiers with the result so a future run can explain what changed. Set a run budget before increasing the number of prompt combinations.
What limitations need judgment?
Automated assertions can encode the wrong expectation. Model-based graders can be inconsistent, and cached results can obscure a change if the evaluation setup is misunderstood. Review representative failures manually. Security testing should stay within systems you are authorized to evaluate; a scan result is not a certification that an application is secure.
Which alternatives should you compare?
Compare Braintrust for a hosted dataset and experiment workflow, and Langfuse when production tracing is a larger part of the requirement. Choose based on how tests are authored, reviewed and reproduced in your existing development process.
Sources checked October 27, 2026: Official documentation · Pricing and access conditions.