Ragas CLI

The Ragas Command Line Interface (CLI) provides tools for quickly setting up evaluation projects and running experiments from the terminal.

Installation

The CLI is included with the ragas package:

pip install ragas

Or use uvx to run without installation:

uvx ragas --help

Available Commands

ragas quickstart

Create a complete evaluation project from a template. This is the fastest way to get started with Ragas.

ragas quickstart [TEMPLATE] [OPTIONS]

Arguments:

• TEMPLATE: Template name (optional). Leave empty to see available templates.

Options:

• -o, --output-dir: Directory to create the project in (default: current directory)

Examples:

# List available templates
ragas quickstart

# Create a RAG evaluation project
ragas quickstart rag_eval

# Create project in a specific directory
ragas quickstart rag_eval --output-dir ./my-project

ragas evals

Run evaluations on a dataset using an evaluation file.

ragas evals EVAL_FILE [OPTIONS]

Arguments:

• EVAL_FILE: Path to the evaluation file (required)

Options:

• --dataset: Name of the dataset in the project (required)
• --metrics: Comma-separated list of metric field names to evaluate (required)
• --baseline: Baseline experiment name to compare against (optional)
• --name: Name of the experiment run (optional)

Example:

ragas evals evals.py --dataset test_data --metrics accuracy,relevance

ragas hello_world

Create a simple hello world example to verify your installation.

ragas hello_world [DIRECTORY]

Arguments:

• DIRECTORY: Directory to create the example in (default: current directory)

Quickstart Templates

RAG & Retrieval

• RAG Evaluation (rag_eval) - Evaluate RAG systems with custom metrics
• Improve RAG (improve_rag) - Compare naive vs agentic RAG approaches

Agent Evaluation

• Agent Evaluation (agent_evals) - Evaluate AI agents solving math problems
• LlamaIndex Agent Evaluation (llamaIndex_agent_evals) - Evaluate LlamaIndex agents with tool call metrics

Specialized Use Cases

• Text-to-SQL Evaluation (text2sql) - Evaluate text-to-SQL systems with execution accuracy
• Workflow Evaluation (workflow_eval) - Evaluate complex LLM workflows
• Prompt Evaluation (prompt_evals) - Compare different prompt variations

LLM Testing

• Judge Alignment (judge_alignment) - Measure LLM-as-judge alignment with human standards
• LLM Benchmarking (benchmark_llm) - Benchmark and compare different LLM models

Quick Start

Get running in 60 seconds:

# Create project
uvx ragas quickstart rag_eval
cd rag_eval

# Install dependencies
uv sync

# Set API key
export OPENAI_API_KEY="your-key"

# Run evaluation
uv run python evals.py

Next Steps

• RAG Evaluation Guide - Detailed walkthrough of the rag_eval template
• Improve RAG Guide - Compare naive vs agentic RAG approaches
• Custom Metrics - Create your own evaluation metrics