# Ragas > Evaluation framework for your AI Application Ragas is an open-source evaluation framework for LLM applications including RAG pipelines, AI agents, and workflows. It provides objective metrics for evaluation, test data generation capabilities, and integrations with popular LLM frameworks like LangChain and LlamaIndex. # Getting Started # πŸš€ Get Started Welcome to Ragas! The Get Started guides will walk you through the fundamentals of working with Ragas. These tutorials assume basic knowledge of Python and building LLM application pipelines. Before you proceed further, ensure that you have [Ragas installed](https://docs.ragas.io/en/stable/getstarted/install/index.md)! Note The tutorials provide an overview of what you can accomplish with Ragas and the basic skills needed to utilize it effectively. For an in-depth explanation of the core concepts behind Ragas, check out the [Core Concepts](https://docs.ragas.io/en/stable/concepts/index.md) page. You can also explore the [How-to Guides](https://docs.ragas.io/en/stable/howtos/index.md) for specific applications of Ragas. If you have any questions about Ragas, feel free to join our [Discord community](https://docs.ragas.io/en/stable/community/index.md) and ask in the `#questions` channel. ## Quickstart Start here to get up and running with Ragas in minutes: - [Quick Start: Get Running in 5 Minutes](https://docs.ragas.io/en/stable/getstarted/quickstart/index.md) ## Tutorials Learn how to evaluate different types of AI applications: - [Evaluate a prompt](https://docs.ragas.io/en/stable/tutorials/prompt/index.md) - Test and compare different prompts - [Evaluate a simple RAG system](https://docs.ragas.io/en/stable/tutorials/rag/index.md) - Evaluate a RAG application - [Evaluate an AI Workflow](https://docs.ragas.io/en/stable/tutorials/workflow/index.md) - Evaluate multi-step workflows - [Evaluate an AI Agent](https://docs.ragas.io/en/stable/tutorials/agent/index.md) - Evaluate agentic applications # Evaluate a simple LLM application The purpose of this guide is to illustrate a simple workflow for testing and evaluating an LLM application with `ragas`. It assumes minimum knowledge in AI application building and evaluation. Please refer to our [installation instruction](https://docs.ragas.io/en/stable/getstarted/install/index.md) for installing `ragas` Get a Working Example The fastest way to see these concepts in action is to create a project using the quickstart command: ```sh uvx ragas quickstart rag_eval cd rag_eval uv sync ``` ```sh pip install ragas ragas quickstart rag_eval cd rag_eval uv sync ``` This generates a complete project with sample code. Follow along with this guide to understand what's happening in your generated code. Let's get started! ## Project Structure Here's what gets created for you: ```sh rag_eval/ β”œβ”€β”€ README.md # Project documentation and setup instructions β”œβ”€β”€ pyproject.toml # Project configuration for uv and pip β”œβ”€β”€ evals.py # Your evaluation workflow β”œβ”€β”€ rag.py # Your RAG/LLM application β”œβ”€β”€ __init__.py # Makes this a Python package └── evals/ # Evaluation artifacts β”œβ”€β”€ datasets/ # Test data files (optional) β”œβ”€β”€ experiments/ # Results from running evaluations (CSV files saved here) └── logs/ # Evaluation execution logs ``` **Key files to focus on:** - **`evals.py`** - Your evaluation workflow with dataset loading and evaluation logic - **`rag.py`** - Your RAG/LLM application code (query engine, retrieval, etc.) ## Understanding the Code In your generated project's `evals.py` file, you'll see the main workflow pattern: 1. **Load Dataset** - Define your test cases with `SingleTurnSample` 1. **Query RAG System** - Get responses from your application 1. **Evaluate Responses** - Validate responses against ground truth 1. **Display Results** - Show evaluation summary in console 1. **Save Results** - Automatically saved to CSV in `evals/experiments/` directory The template provides modular functions you can customize: ```python from ragas.dataset_schema import SingleTurnSample from ragas import EvaluationDataset def load_dataset(): """Load test dataset for evaluation.""" data_samples = [ SingleTurnSample( user_input="What is Ragas?", response="", # Will be filled by querying RAG reference="Ragas is an evaluation framework for LLM applications", retrieved_contexts=[], ), # Add more test cases... ] return EvaluationDataset(samples=data_samples) ``` You can extend this with [metrics](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/index.md) and more sophisticated evaluation logic. Learn more about [evaluation in Ragas](https://docs.ragas.io/en/stable/getstarted/concepts/evaluation/index.md). ### Choosing Your LLM Provider Your quickstart project initializes the OpenAI LLM by default in the `_init_clients()` function. You can easily swap to any provider through the `llm_factory`: Set your OpenAI API key: ```sh export OPENAI_API_KEY="your-openai-key" ``` In your `evals.py` `_init_clients()` function: ```python from ragas.llms import llm_factory llm = llm_factory("gpt-4o") ``` This is already set up in your quickstart project! Set your Anthropic API key: ```sh export ANTHROPIC_API_KEY="your-anthropic-key" ``` In your `evals.py` `_init_clients()` function: ```python from ragas.llms import llm_factory llm = llm_factory("claude-3-5-sonnet-20241022", provider="anthropic") ``` Set up your Google credentials: ```sh export GOOGLE_API_KEY="your-google-api-key" ``` In your `evals.py` `_init_clients()` function: ```python from ragas.llms import llm_factory llm = llm_factory("gemini-1.5-pro", provider="google") ``` Install and run Ollama locally, then in your `evals.py` `_init_clients()` function: ```python from ragas.llms import llm_factory llm = llm_factory( "mistral", provider="ollama", base_url="http://localhost:11434" # Default Ollama URL ) ``` For any LLM with OpenAI-compatible API: ```python from ragas.llms import llm_factory llm = llm_factory( "model-name", api_key="your-api-key", base_url="https://your-api-endpoint" ) ``` For more details, learn about [LLM integrations](https://docs.ragas.io/en/stable/concepts/metrics/index.md). ### Using Pre-Built Metrics `ragas` comes with pre-built metrics for common evaluation tasks. For example, [Aspect Critique](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/aspect_critic/index.md) evaluates any aspect of your output using `DiscreteMetric`: ```python from ragas.metrics import DiscreteMetric from ragas.llms import llm_factory # Setup your evaluator LLM evaluator_llm = llm_factory("gpt-4o") # Create a custom aspect evaluator metric = DiscreteMetric( name="summary_accuracy", allowed_values=["accurate", "inaccurate"], prompt="""Evaluate if the summary is accurate and captures key information. Response: {response} Answer with only 'accurate' or 'inaccurate'.""", llm=evaluator_llm ) # Score your application's output score = await metric.ascore( response="The summary of the text is..." ) print(f"Score: {score.value}") # 'accurate' or 'inaccurate' print(f"Reason: {score.reason}") ``` Pre-built metrics like this save you from defining evaluation logic from scratch. Explore [all available metrics](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/index.md). Info There are many other types of metrics that are available in `ragas` (with and without `reference`), and you may also create your own metrics if none of those fits your case. To explore this more checkout [more on metrics](https://docs.ragas.io/en/stable/concepts/metrics/index.md). ### Evaluating on a Dataset In your quickstart project, you'll see in the `load_dataset()` function, which creates test data with multiple samples: ```python from ragas import Dataset # Create a dataset with multiple test samples dataset = Dataset( name="test_dataset", backend="local/csv", # Can also use JSONL, Google Drive, or in-memory root_dir=".", ) # Add samples to the dataset data_samples = [ { "user_input": "What is ragas?", "response": "Ragas is an evaluation framework...", "expected": "Ragas provides objective metrics..." }, { "user_input": "How do metrics work?", "response": "Metrics score your application...", "expected": "Metrics evaluate performance..." }, ] for sample in data_samples: dataset.append(sample) # Save to disk dataset.save() ``` This gives you multiple test cases instead of evaluating one example at a time. Learn more about [datasets and experiments](https://docs.ragas.io/en/stable/concepts/components/eval_dataset/index.md). Your generated project includes sample data in the `evals/datasets/` folder - you can edit those files to add more test cases. ### Want help in improving your AI application using evals? In the past 2 years, we have seen and helped improve many AI applications using evals. We are compressing this knowledge into a product to replace vibe checks with eval loops so that you can focus on building great AI applications. If you want help with improving and scaling up your AI application using evals. πŸ”— Book a [slot](https://bit.ly/3EBYq4J) or drop us a line: [founders@vibrantlabs.com](mailto:founders@vibrantlabs.com). ## Up Next - [Evaluate a simple RAG application](https://docs.ragas.io/en/stable/getstarted/rag_eval/index.md) # Run your first experiment This tutorial walks you through running your first experiment with Ragas using the `@experiment` decorator and a local CSV backend. ## Prerequisites - Python 3.9+ - Ragas installed (see [Installation](https://docs.ragas.io/en/stable/getstarted/install/index.md)) ## Hello World πŸ‘‹ ### 1. Install (if you haven’t already) ```bash pip install ragas ``` ### 2. Create `hello_world.py` Copy this into a new file and save as `hello_world.py`: ```python import numpy as np from ragas import Dataset, experiment from ragas.metrics import MetricResult, discrete_metric # Define a custom metric for accuracy @discrete_metric(name="accuracy_score", allowed_values=["pass", "fail"]) def accuracy_score(response: str, expected: str): result = "pass" if expected.lower().strip() == response.lower().strip() else "fail" return MetricResult(value=result, reason=f"Match: {result == 'pass'}") # Mock application endpoint that simulates an AI application response def mock_app_endpoint(**kwargs) -> str: return np.random.choice(["Paris", "4", "Blue Whale", "Einstein", "Python"]) # Create an experiment that uses the mock application endpoint and the accuracy metric @experiment() async def run_experiment(row): response = mock_app_endpoint(query=row.get("query")) accuracy = accuracy_score.score(response=response, expected=row.get("expected_output")) return {**row, "response": response, "accuracy": accuracy.value} if __name__ == "__main__": import asyncio # Create dataset inline dataset = Dataset(name="test_dataset", backend="local/csv", root_dir=".") test_data = [ {"query": "What is the capital of France?", "expected_output": "Paris"}, {"query": "What is 2 + 2?", "expected_output": "4"}, {"query": "What is the largest animal?", "expected_output": "Blue Whale"}, {"query": "Who developed the theory of relativity?", "expected_output": "Einstein"}, {"query": "What programming language is named after a snake?", "expected_output": "Python"}, ] for sample in test_data: dataset.append(sample) dataset.save() # Run experiment _ = asyncio.run(run_experiment.arun(dataset, name="first_experiment")) ``` ### 3. Inspect the generated files ```bash tree . ``` You should see: ```text β”œβ”€β”€ datasets β”‚ └── test_dataset.csv └── experiments └── first_experiment.csv ``` ### 4. View the results of your first experiment ```bash open experiments/first_experiment.csv ``` Output preview: ## Next steps - Learn the concepts behind experiments in [Experiments (Concepts)](https://docs.ragas.io/en/stable/concepts/experimentation/index.md) - Explore evaluation metrics in [Metrics](https://docs.ragas.io/en/stable/concepts/metrics/index.md) # Installation To get started, install Ragas using `pip` with the following command: ```bash pip install ragas ``` If you'd like to experiment with the latest features, install the most recent version from the main branch: ```bash pip install git+https://github.com/vibrantlabsai/ragas.git ``` If you're planning to contribute and make modifications to the code, ensure that you clone the repository and set it up as an [editable install](https://pip.pypa.io/en/stable/topics/local-project-installs/#editable-installs). ```bash git clone https://github.com/vibrantlabsai/ragas.git pip install -e . ``` LangChain OpenAI dependency versions If you use `langchain_openai` (e.g., `ChatOpenAI`), install `langchain-core` and `langchain-openai` explicitly to avoid version mismatches. You can adjust bounds to match your environment, but installing both explicitly helps prevent strict dependency conflicts. ```bash pip install -U "langchain-core>=0.2,<0.3" "langchain-openai>=0.1,<0.2" openai ``` # Quick Start: Get Evaluations Running in a Flash Get started with Ragas in minutes. Create a complete evaluation project with just a few commands. ## Step 1: Create Your Project Choose one of the following methods: No installation required. `uvx` automatically downloads and runs ragas: ```sh uvx ragas quickstart rag_eval cd rag_eval ``` Install ragas first, then create the project: ```sh pip install ragas ragas quickstart rag_eval cd rag_eval ``` ## Step 2: Install Dependencies Install the project dependencies: ```sh uv sync ``` Or if you prefer `pip`: ```sh pip install -e . ``` ## Step 3: Set Your API Key By default, the quickstart example uses OpenAI. Set your API key and you're ready to go. You can also use some other provider with a minor change: ```sh export OPENAI_API_KEY="your-openai-key" ``` The quickstart project is already configured to use OpenAI. You're all set! Set your Anthropic API key: ```sh export ANTHROPIC_API_KEY="your-anthropic-key" ``` Then update the LLM initialization in `evals.py`: ```python from anthropic import Anthropic from ragas.llms import llm_factory client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY")) llm = llm_factory("claude-3-5-sonnet-20241022", provider="anthropic", client=client) ``` Set up your Google credentials: ```sh export GOOGLE_API_KEY="your-google-api-key" ``` Then update the LLM initialization in `evals.py`: **Option 1: Using Google's Official Library (Recommended)** ```python import google.generativeai as genai from ragas.llms import llm_factory genai.configure(api_key=os.environ.get("GOOGLE_API_KEY")) client = genai.GenerativeModel("gemini-2.0-flash") llm = llm_factory("gemini-2.0-flash", provider="google", client=client) # Adapter is auto-detected as "litellm" for google provider ``` For more Gemini options and detailed setup, see the [Google Gemini Integration Guide](https://docs.ragas.io/en/stable/howtos/integrations/gemini/index.md). Install and run Ollama locally, then update the LLM initialization in `evals.py`: ```python from openai import OpenAI from ragas.llms import llm_factory # Create an OpenAI-compatible client for Ollama client = OpenAI( api_key="ollama", # Ollama doesn't require a real key base_url="http://localhost:11434/v1" ) llm = llm_factory("mistral", provider="openai", client=client) ``` For any LLM with OpenAI-compatible API: ```python from openai import OpenAI from ragas.llms import llm_factory client = OpenAI( api_key="your-api-key", base_url="https://your-api-endpoint" ) llm = llm_factory("model-name", provider="openai", client=client) ``` For more details, learn about [LLM integrations](https://docs.ragas.io/en/stable/concepts/metrics/index.md). ## Project Structure Your generated project includes: ```sh rag_eval/ β”œβ”€β”€ README.md # Project documentation β”œβ”€β”€ pyproject.toml # Project configuration β”œβ”€β”€ rag.py # Your RAG application β”œβ”€β”€ evals.py # Evaluation workflow β”œβ”€β”€ __init__.py # Makes this a Python package └── evals/ β”œβ”€β”€ datasets/ # Test data files β”œβ”€β”€ experiments/ # Evaluation results └── logs/ # Execution logs ``` ## Step 4: Run Your Evaluation Run the evaluation script: ```sh uv run python evals.py ``` Or if you installed with `pip`: ```sh python evals.py ``` The evaluation will: - Load test data from the `load_dataset()` function in `evals.py` - Query your RAG application with test questions - Evaluate responses - Display results in the console - Save results to CSV in the `evals/experiments/` directory Congratulations! You have a complete evaluation setup running. πŸŽ‰ ______________________________________________________________________ ## Customize Your Evaluation ### Add More Test Cases Edit the `load_dataset()` function in `evals.py` to add more test questions: ```python from ragas import Dataset def load_dataset(): """Load test dataset for evaluation.""" dataset = Dataset( name="test_dataset", backend="local/csv", root_dir=".", ) data_samples = [ { "question": "What is Ragas?", "grading_notes": "Ragas is an evaluation framework for LLM applications", }, { "question": "How do metrics work?", "grading_notes": "Metrics evaluate the quality and performance of LLM responses", }, # Add more test cases here ] for sample in data_samples: dataset.append(sample) dataset.save() return dataset ``` ### Customize Evaluation Metrics The template includes a `DiscreteMetric` for custom evaluation logic. You can customize the evaluation by: 1. **Modify the metric prompt** - Change the evaluation criteria 1. **Adjust allowed values** - Update valid output categories 1. **Add more metrics** - Create additional metrics for different aspects Example of modifying the metric: ```python from ragas.metrics import DiscreteMetric from ragas.llms import llm_factory my_metric = DiscreteMetric( name="custom_evaluation", prompt="Evaluate this response: {response} based on: {context}. Return 'excellent', 'good', or 'poor'.", allowed_values=["excellent", "good", "poor"], ) ``` ## What's Next? - **Learn the concepts**: Read the [Evaluate a Simple LLM Application](https://docs.ragas.io/en/stable/getstarted/evals/index.md) guide for deeper understanding - **Custom metrics**: [Create your own metrics](https://docs.ragas.io/en/stable/concepts/metrics/overview/#output-types) using simple decorators - **Production integration**: [Integrate evaluations into your CI/CD pipeline](https://docs.ragas.io/en/stable/howtos/index.md) - **RAG evaluation**: Evaluate [RAG systems](https://docs.ragas.io/en/stable/getstarted/rag_eval/index.md) with specialized metrics - **Agent evaluation**: Explore [AI agent evaluation](https://docs.ragas.io/en/stable/howtos/applications/text2sql/index.md) - **Test data generation**: [Generate synthetic test datasets](https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/index.md) for your evaluations ## Getting Help - πŸ“š [Full Documentation](https://docs.ragas.io/) - πŸ’¬ [Join our Discord Community](https://discord.gg/5djav8GGNZ) - πŸ› [Report Issues](https://github.com/vibrantlabsai/ragas/issues) # Evaluate a simple RAG system The purpose of this guide is to illustrate a simple workflow for testing and evaluating a RAG system with `ragas`. It assumes minimum knowledge in building RAG system and evaluation. Please refer to our [installation instruction](https://docs.ragas.io/en/stable/getstarted/install/index.md) for installing `ragas`. ## Basic Setup We will use `langchain_openai` to set the LLM and embedding model for building our simple RAG. You may choose any other LLM and embedding model of your choice, to do that please refer to [customizing models in langchain](https://python.langchain.com/docs/integrations/chat/). ```python from langchain_openai import ChatOpenAI from ragas.embeddings import OpenAIEmbeddings import openai llm = ChatOpenAI(model="gpt-4o") openai_client = openai.OpenAI() embeddings = OpenAIEmbeddings(client=openai_client) ``` OpenAI Embeddings API `ragas.embeddings.OpenAIEmbeddings` exposes `embed_text` (single) and `embed_texts` (batch), not `embed_query`/`embed_documents` like some LangChain wrappers. The example below uses `embed_texts` for documents and `embed_text` for the query. Please refer to [OpenAI embeddings implementation](https://docs.ragas.io/en/stable/references/embeddings/#ragas.embeddings.OpenAIEmbeddings) ### Build a Simple RAG System To build a simple RAG system, we need to define the following components: - Define a method to vectorize our docs - Define a method to retrieve the relevant docs - Define a method to generate the response Click to View the Code ```python import numpy as np class RAG: def __init__(self, model="gpt-4o"): import openai self.llm = ChatOpenAI(model=model) openai_client = openai.OpenAI() self.embeddings = OpenAIEmbeddings(client=openai_client) self.doc_embeddings = None self.docs = None def load_documents(self, documents): """Load documents and compute their embeddings.""" self.docs = documents self.doc_embeddings = self.embeddings.embed_texts(documents) def get_most_relevant_docs(self, query): """Find the most relevant document for a given query.""" if not self.docs or not self.doc_embeddings: raise ValueError("Documents and their embeddings are not loaded.") query_embedding = self.embeddings.embed_text(query) similarities = [ np.dot(query_embedding, doc_emb) / (np.linalg.norm(query_embedding) * np.linalg.norm(doc_emb)) for doc_emb in self.doc_embeddings ] most_relevant_doc_index = np.argmax(similarities) return [self.docs[most_relevant_doc_index]] def generate_answer(self, query, relevant_doc): """Generate an answer for a given query based on the most relevant document.""" prompt = f"question: {query}\n\nDocuments: {relevant_doc}" messages = [ ("system", "You are a helpful assistant that answers questions based on given documents only."), ("human", prompt), ] ai_msg = self.llm.invoke(messages) return ai_msg.content ``` ### Load Documents Now, let's load some documents and test our RAG system. ```python sample_docs = [ "Albert Einstein proposed the theory of relativity, which transformed our understanding of time, space, and gravity.", "Marie Curie was a physicist and chemist who conducted pioneering research on radioactivity and won two Nobel Prizes.", "Isaac Newton formulated the laws of motion and universal gravitation, laying the foundation for classical mechanics.", "Charles Darwin introduced the theory of evolution by natural selection in his book 'On the Origin of Species'.", "Ada Lovelace is regarded as the first computer programmer for her work on Charles Babbage's early mechanical computer, the Analytical Engine." ] ``` ```python # Initialize RAG instance rag = RAG() # Load documents rag.load_documents(sample_docs) # Query and retrieve the most relevant document query = "Who introduced the theory of relativity?" relevant_doc = rag.get_most_relevant_docs(query) # Generate an answer answer = rag.generate_answer(query, relevant_doc) print(f"Query: {query}") print(f"Relevant Document: {relevant_doc}") print(f"Answer: {answer}") ``` Output: ```text Query: Who introduced the theory of relativity? Relevant Document: ['Albert Einstein proposed the theory of relativity, which transformed our understanding of time, space, and gravity.'] Answer: Albert Einstein introduced the theory of relativity. ``` ## Collect Evaluation Data To collect evaluation data, we first need a set of queries to run against our RAG. We can run the queries through the RAG system and collect the `response`, `retrieved_contexts`for each query. You may also optionally prepare a set of golden answers for each query to evaluate the system's performance. ```python sample_queries = [ "Who introduced the theory of relativity?", "Who was the first computer programmer?", "What did Isaac Newton contribute to science?", "Who won two Nobel Prizes for research on radioactivity?", "What is the theory of evolution by natural selection?" ] expected_responses = [ "Albert Einstein proposed the theory of relativity, which transformed our understanding of time, space, and gravity.", "Ada Lovelace is regarded as the first computer programmer for her work on Charles Babbage's early mechanical computer, the Analytical Engine.", "Isaac Newton formulated the laws of motion and universal gravitation, laying the foundation for classical mechanics.", "Marie Curie was a physicist and chemist who conducted pioneering research on radioactivity and won two Nobel Prizes.", "Charles Darwin introduced the theory of evolution by natural selection in his book 'On the Origin of Species'." ] ``` ```python dataset = [] for query,reference in zip(sample_queries,expected_responses): relevant_docs = rag.get_most_relevant_docs(query) response = rag.generate_answer(query, relevant_docs) dataset.append( { "user_input":query, "retrieved_contexts":relevant_docs, "response":response, "reference":reference } ) ``` Now, load the dataset into `EvaluationDataset` object. ```python from ragas import EvaluationDataset evaluation_dataset = EvaluationDataset.from_list(dataset) ``` ## Evaluate We have successfully collected the evaluation data. Now, we can evaluate our RAG system on the collected dataset using a set of commonly used RAG evaluation metrics. You may choose any model as [evaluator LLM](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/index.md) for evaluation. ```python from ragas import evaluate from ragas.llms import LangchainLLMWrapper evaluator_llm = LangchainLLMWrapper(llm) from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness result = evaluate(dataset=evaluation_dataset,metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()],llm=evaluator_llm) result ``` Output ```text {'context_recall': 1.0000, 'faithfulness': 0.8571, 'factual_correctness': 0.7280} ``` ### Want help in improving your AI application using evals? In the past 2 years, we have seen and helped improve many AI applications using evals. We are compressing this knowledge into a product to replace vibe checks with eval loops so that you can focus on building great AI applications. If you want help with improving and scaling up your AI application using evals. πŸ”— Book a [slot](https://bit.ly/3EBYq4J) or drop us a line: [founders@vibrantlabs.com](mailto:founders@vibrantlabs.com). ## Up Next - [Generate test data for evaluating RAG](https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/index.md) # Testset Generation for RAG This simple guide will help you generate a testset for evaluating your RAG pipeline using your own documents. ## Quickstart Let's walk through a quick example of generating a testset for a RAG pipeline. Following that we will explore the main components of the testset generation pipeline. ### Load Sample Documents For the sake of this tutorial we will use sample documents from this [repository](https://huggingface.co/datasets/vibrantlabsai/Sample_Docs_Markdown). You can replace this with your own documents. ```bash git clone https://huggingface.co/datasets/vibrantlabsai/Sample_Docs_Markdown ``` ### Load documents Now we will load the documents from the sample dataset using `DirectoryLoader`, which is one of the document loaders from [langchain_community](https://python.langchain.com/docs/concepts/document_loaders/). You may also use any loaders from [llama_index](https://docs.llamaindex.ai/en/stable/understanding/loading/llamahub/) ```shell pip install langchain-community ``` ```python from langchain_community.document_loaders import DirectoryLoader path = "Sample_Docs_Markdown/" loader = DirectoryLoader(path, glob="**/*.md") docs = loader.load() ``` ### Choose your LLM You may choose to use any [LLM of your choice](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/index.md) Install the langchain-openai package ```bash pip install langchain-openai ``` Then ensure you have your OpenAI key ready and available in your environment ```python import os os.environ["OPENAI_API_KEY"] = "your-openai-key" ``` Wrap the LLMs in `LangchainLLMWrapper` so that it can be used with ragas. ```python from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI from ragas.embeddings import OpenAIEmbeddings import openai generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o")) openai_client = openai.OpenAI() generator_embeddings = OpenAIEmbeddings(client=openai_client) ``` Install the langchain-aws package ```bash pip install langchain-aws ``` Then you have to set your AWS credentials and configurations ```python config = { "credentials_profile_name": "your-profile-name", # E.g "default" "region_name": "your-region-name", # E.g. "us-east-1" "llm": "your-llm-model-id", # E.g "anthropic.claude-3-5-sonnet-20241022-v2:0" "embeddings": "your-embedding-model-id", # E.g "amazon.titan-embed-text-v2:0" "temperature": 0.4, } ``` Define your LLMs and wrap them in `LangchainLLMWrapper` so that it can be used with ragas. ```python from langchain_aws import ChatBedrockConverse from langchain_aws import BedrockEmbeddings from ragas.llms import LangchainLLMWrapper from ragas.embeddings import LangchainEmbeddingsWrapper generator_llm = LangchainLLMWrapper(ChatBedrockConverse( credentials_profile_name=config["credentials_profile_name"], region_name=config["region_name"], base_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com", model=config["llm"], temperature=config["temperature"], )) generator_embeddings = LangchainEmbeddingsWrapper(BedrockEmbeddings( credentials_profile_name=config["credentials_profile_name"], region_name=config["region_name"], model_id=config["embeddings"], )) ``` If you want more information on how to use other AWS services, please refer to the [langchain-aws](https://python.langchain.com/docs/integrations/providers/aws/) documentation. Google offers two ways to access their models: Google AI and Google Cloud Vertex AI. Google AI requires just a Google account and API key, while Vertex AI requires a Google Cloud account with enterprise features. First, install the required packages: ```bash pip install langchain-google-genai langchain-google-vertexai ``` Then set up your credentials based on your chosen API: For Google AI: ```python import os os.environ["GOOGLE_API_KEY"] = "your-google-ai-key" # From https://ai.google.dev/ ``` For Vertex AI: ```python # Ensure you have credentials configured (gcloud, workload identity, etc.) # Or set service account JSON path: os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = "path/to/service-account.json" ``` Define your configuration: ```python config = { "model": "gemini-1.5-pro", # or other model IDs "temperature": 0.4, "max_tokens": None, "top_p": 0.8, # For Vertex AI only: "project": "your-project-id", # Required for Vertex AI "location": "us-central1", # Required for Vertex AI } ``` Initialize the LLM and wrap it for use with ragas: ```python from ragas.llms import LangchainLLMWrapper from ragas.embeddings import LangchainEmbeddingsWrapper # Choose the appropriate import based on your API: from langchain_google_genai import ChatGoogleGenerativeAI from langchain_google_vertexai import ChatVertexAI # Initialize with Google AI Studio generator_llm = LangchainLLMWrapper(ChatGoogleGenerativeAI( model=config["model"], temperature=config["temperature"], max_tokens=config["max_tokens"], top_p=config["top_p"], )) # Or initialize with Vertex AI generator_llm = LangchainLLMWrapper(ChatVertexAI( model=config["model"], temperature=config["temperature"], max_tokens=config["max_tokens"], top_p=config["top_p"], project=config["project"], location=config["location"], )) ``` You can optionally configure safety settings: ```python from langchain_google_genai import HarmCategory, HarmBlockThreshold safety_settings = { HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE, # Add other safety settings as needed } # Apply to your LLM initialization generator_llm = LangchainLLMWrapper(ChatGoogleGenerativeAI( model=config["model"], temperature=config["temperature"], safety_settings=safety_settings, )) ``` Initialize the embeddings and wrap them for use with ragas: ```python # Google AI Studio Embeddings from langchain_google_genai import GoogleGenerativeAIEmbeddings generator_embeddings = LangchainEmbeddingsWrapper(GoogleGenerativeAIEmbeddings( model="models/embedding-001", # Google's text embedding model task_type="retrieval_document" # Optional: specify the task type )) ``` ```python # Vertex AI Embeddings from langchain_google_vertexai import VertexAIEmbeddings generator_embeddings = LangchainEmbeddingsWrapper(VertexAIEmbeddings( model_name="textembedding-gecko@001", # or other available model project=config["project"], # Your GCP project ID location=config["location"] # Your GCP location )) ``` For more information on available models, features, and configurations, refer to: [Google AI documentation](https://ai.google.dev/docs) - [Vertex AI documentation](https://cloud.google.com/vertex-ai/docs) - [LangChain Google AI integration](https://python.langchain.com/docs/integrations/chat/google_generative_ai) - [LangChain Vertex AI integration](https://python.langchain.com/docs/integrations/chat/google_vertex_ai) Install the langchain-openai package ```bash pip install langchain-openai ``` Ensure you have your Azure OpenAI key ready and available in your environment. ```python import os os.environ["AZURE_OPENAI_API_KEY"] = "your-azure-openai-key" # other configuration azure_config = { "base_url": "", # your endpoint "model_deployment": "", # your model deployment name "model_name": "", # your model name "embedding_deployment": "", # your embedding deployment name "embedding_name": "", # your embedding name } ``` Define your LLMs and wrap them in `LangchainLLMWrapper` so that it can be used with ragas. ```python from langchain_openai import AzureChatOpenAI from langchain_openai import AzureOpenAIEmbeddings from ragas.llms import LangchainLLMWrapper from ragas.embeddings import LangchainEmbeddingsWrapper generator_llm = LangchainLLMWrapper(AzureChatOpenAI( openai_api_version="2023-05-15", azure_endpoint=azure_configs["base_url"], azure_deployment=azure_configs["model_deployment"], model=azure_configs["model_name"], validate_base_url=False, )) # init the embeddings for answer_relevancy, answer_correctness and answer_similarity generator_embeddings = LangchainEmbeddingsWrapper(AzureOpenAIEmbeddings( openai_api_version="2023-05-15", azure_endpoint=azure_configs["base_url"], azure_deployment=azure_configs["embedding_deployment"], model=azure_configs["embedding_name"], )) ``` If you want more information on how to use other Azure services, please refer to the [langchain-azure](https://python.langchain.com/docs/integrations/chat/azure_chat_openai/) documentation. If you are using a different LLM provider and using LangChain to interact with it, you can wrap your LLM in `LangchainLLMWrapper` so that it can be used with ragas. ```python from ragas.llms import LangchainLLMWrapper generator_llm = LangchainLLMWrapper(your_llm_instance) ``` For a more detailed guide, checkout [the guide on customizing models](https://docs.ragas.io/en/stable/howtos/customizations/customize_models.md). If you using LlamaIndex, you can use the `LlamaIndexLLMWrapper` to wrap your LLM so that it can be used with ragas. ```python from ragas.llms import LlamaIndexLLMWrapper generator_llm = LlamaIndexLLMWrapper(your_llm_instance) ``` For more information on how to use LlamaIndex, please refer to the [LlamaIndex Integration guide](https://docs.ragas.io/en/stable/howtos/integrations/_llamaindex.md). If your still not able use Ragas with your favorite LLM provider, please let us know by by commenting on this [issue](https://github.com/vibrantlabsai/ragas/issues/1617) and we'll add support for it πŸ™‚. ### Generate Testset Now we will run the test generation using the loaded documents and the LLM setup. If you have used `llama_index` to load documents, please use `generate_with_llama_index_docs` method instead. ```python from ragas.testset import TestsetGenerator generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings) dataset = generator.generate_with_langchain_docs(docs, testset_size=10) ``` ### Analyzing the testset Once you have generated a testset, you would want to view it and select the queries you see fit to include in your final testset. You can export the testset to a pandas DataFrame and do various analysis on it. ```python dataset.to_pandas() ``` Output Note Generating synthetic test data can be confusing and hard, but if you need we are happy to help you with it. We have built pipelines to generate test data for various use cases. If you need help with it, please talk to us by booking a [slot](https://bit.ly/3EBYq4J) or drop us a line: [founders@vibrantlabs.com](mailto:founders@vibrantlabs.com). ## A Deeper Look Now that we have a seen how to generate a testset, let's take a closer look at the main components of the testset generation pipeline and how you can quickly customize it. At the core there are 2 main operations that are performed to generate a testset. 1. **KnowledgeGraph Creation**: We first create a KnowledgeGraph using the documents you provide and use various Transformations to enrich the knowledge graph with additional information that we can use to generate the testset. You can learn more about this from the [core concepts section](https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/#knowledge-graph-creation). 1. **Testset Generation**: We use the KnowledgeGraph to generate a set of scenarios. These scenarios are used to generate the testset. You can learn more about this from the [core concepts section](https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/#scenario-generation). Now let's see an example of how these components work together to generate a testset. ### KnowledgeGraph Creation Let's first create a KnowledgeGraph using the documents we loaded earlier. ```python from ragas.testset.graph import KnowledgeGraph kg = KnowledgeGraph() ``` Output ```text KnowledgeGraph(nodes: 0, relationships: 0) ``` and then add the documents to the knowledge graph. ```python from ragas.testset.graph import Node, NodeType for doc in docs: kg.nodes.append( Node( type=NodeType.DOCUMENT, properties={"page_content": doc.page_content, "document_metadata": doc.metadata} ) ) ``` Output ```text KnowledgeGraph(nodes: 10, relationships: 0) ``` Now we will enrich the knowledge graph with additional information using Transformations. Here we will use default_transforms to create a set of default transformations to apply with an LLM and Embedding Model of your choice. But you can mix and match transforms or build your own as needed. ```python from ragas.testset.transforms import default_transforms, apply_transforms # define your LLM and Embedding Model # here we are using the same LLM and Embedding Model that we used to generate the testset transformer_llm = generator_llm embedding_model = generator_embeddings trans = default_transforms(documents=docs, llm=transformer_llm, embedding_model=embedding_model) apply_transforms(kg, trans) ``` Now we have a knowledge graph with additional information. You can save the knowledge graph too. ```python kg.save("knowledge_graph.json") loaded_kg = KnowledgeGraph.load("knowledge_graph.json") loaded_kg ``` Output ```text KnowledgeGraph(nodes: 48, relationships: 605) ``` ### Testset Generation Now we will use the `loaded_kg` to create the TestsetGenerator. ```python from ragas.testset import TestsetGenerator generator = TestsetGenerator(llm=generator_llm, embedding_model=embedding_model, knowledge_graph=loaded_kg) ``` We can also define the distribution of queries we would like to generate. Here lets use the default distribution. ```python from ragas.testset.synthesizers import default_query_distribution query_distribution = default_query_distribution(generator_llm) ``` Output ```text [ (SingleHopSpecificQuerySynthesizer(llm=llm), 0.5), (MultiHopAbstractQuerySynthesizer(llm=llm), 0.25), (MultiHopSpecificQuerySynthesizer(llm=llm), 0.25), ] ``` Now we can generate the testset. ```python testset = generator.generate(testset_size=10, query_distribution=query_distribution) testset.to_pandas() ``` Output # Tutorials # Tutorials ## Installing dependencies 1. Install ragas_examples ```bash pip install ragas[examples] ``` 2. Setup your OpenAI API key ```bash export OPENAI_API_KEY = "your_openai_api_key" ``` ## Tutorials 1. [Evaluate a prompt](https://docs.ragas.io/en/stable/tutorials/prompt/index.md) 1. [Evaluate a simple RAG system](https://docs.ragas.io/en/stable/tutorials/rag/index.md) 1. [Evaluate a AI Workflow](https://docs.ragas.io/en/stable/tutorials/workflow/index.md) 1. [Evaluate an AI Agent](https://docs.ragas.io/en/stable/tutorials/agent/index.md) # Evaluate an AI agent This tutorial demonstrates how to evaluate an AI agent using Ragas, specifically a mathematical agent that can solve complex expressions using atomic operations and function calling capabilities. By the end of this tutorial, you will learn how to evaluate and iterate on an agent using evaluation-driven development. ``` graph TD A[User Input
Math Expression] --> B[MathToolsAgent] subgraph LLM Agent Loop B --> D{Need to use a Tool?} D -- Yes --> E[Call Tool
add/sub/mul/div] E --> F[Tool Result] F --> B D -- No --> G[Emit Final Answer] end G --> H[Final Answer] ``` We will start by testing our simple agent that can solve mathematical expressions using atomic operations and function calling capabilities. ```bash python -m ragas_examples.agent_evals.agent ``` Next, we will create a few sample expressions and expected outputs for our agent, then convert them to a CSV file. ```python import pandas as pd dataset = [ {"expression": "(2 + 3) * (4 - 1)", "expected": 15}, {"expression": "5 * (6 + 2)", "expected": 40}, {"expression": "10 - (3 + 2)", "expected": 5}, ] df = pd.DataFrame(dataset) df.to_csv("datasets/test_dataset.csv", index=False) ``` To evaluate the performance of our agent, we will define a non-LLM metric that compares if our agent's output is within a certain tolerance of the expected output and returns 1/0 based on the comparison. ```python from ragas.metrics import numeric_metric from ragas.metrics.result import MetricResult @numeric_metric(name="correctness") def correctness_metric(prediction: float, actual: float): """Calculate correctness of the prediction.""" if isinstance(prediction, str) and "ERROR" in prediction: return 0.0 result = 1.0 if abs(prediction - actual) < 1e-5 else 0.0 return MetricResult(value=result, reason=f"Prediction: {prediction}, Actual: {actual}") ``` Next, we will write the experiment loop that will run our agent on the test dataset and evaluate it using the metric, and store the results in a CSV file. ```python from ragas import experiment @experiment() async def run_experiment(row): expression = row["expression"] expected_result = row["expected"] # Get the model's prediction prediction = math_agent.solve(expression) # Calculate the correctness metric correctness = correctness_metric.score(prediction=prediction.get("result"), actual=expected_result) return { "expression": expression, "expected_result": expected_result, "prediction": prediction.get("result"), "log_file": prediction.get("log_file"), "correctness": correctness.value } ``` Now whenever you make a change to your agent, you can run the experiment and see how it affects the performance of your agent. ## Running the example end to end 1. Set up your OpenAI API key ```bash export OPENAI_API_KEY="your_api_key_here" ``` 1. Run the evaluation ```bash python -m ragas_examples.agent_evals.evals ``` VoilΓ ! You have successfully evaluated an AI agent using Ragas. You can now view the results by opening the `experiments/experiment_name.csv` file. # Prompt Evaluation In this tutorial, we will write a simple evaluation pipeline to evaluate a prompt that is part of an AI system, here a movie review sentiment classifier. At the end of this tutorial you’ll learn how to evaluate and iterate on a single prompt using evaluation driven development. ``` flowchart LR A["'This movie was amazing!
Great acting and plot.'"] --> B["Classifier Prompt"] B --> C["Positive"] ``` We will start by testing a simple prompt that classifies movie reviews as positive or negative. First, make sure you have installed ragas examples and setup your OpenAI API key: ```bash pip install ragas[examples] export OPENAI_API_KEY = "your_openai_api_key" ``` Now test the prompt: ```bash python -m ragas_examples.prompt_evals.prompt ``` This will test the input `"The movie was fantastic and I loved every moment of it!"` and should output `"positive"`. > **πŸ’‘ Quick Start**: If you want to see the complete evaluation in action, you can jump straight to the [end-to-end command](#running-the-example-end-to-end) that runs everything and generates the CSV results automatically. Next, we will write down few sample inputs and expected outputs for our prompt. Then convert them to a CSV file. ```python import pandas as pd samples = [{"text": "I loved the movie! It was fantastic.", "label": "positive"}, {"text": "The movie was terrible and boring.", "label": "negative"}, {"text": "It was an average film, nothing special.", "label": "positive"}, {"text": "Absolutely amazing! Best movie of the year.", "label": "positive"}] pd.DataFrame(samples).to_csv("datasets/test_dataset.csv", index=False) ``` Now we need to have a way to measure the performance of our prompt in this task. We will define a metric that will compare the output of our prompt with the expected output and outputs pass/fail based on it. ```python from ragas.metrics import discrete_metric from ragas.metrics.result import MetricResult @discrete_metric(name="accuracy", allowed_values=["pass", "fail"]) def my_metric(prediction: str, actual: str): """Calculate accuracy of the prediction.""" return MetricResult(value="pass", reason="") if prediction == actual else MetricResult(value="fail", reason="") ``` Next, we will write the experiment loop that will run our prompt on the test dataset and evaluate it using the metric, and store the results in a csv file. ```python from ragas import experiment @experiment() async def run_experiment(row): response = run_prompt(row["text"]) score = my_metric.score( prediction=response, actual=row["label"] ) experiment_view = { **row, "response":response, "score":score.value, } return experiment_view ``` Now whenever you make a change to your prompt, you can run the experiment and see how it affects the performance of your prompt. ### Passing Additional Parameters You can pass additional parameters like models or configurations to your experiment function: ```python @experiment() async def run_experiment(row, model): response = run_prompt(row["text"], model=model) score = my_metric.score( prediction=response, actual=row["label"] ) experiment_view = { **row, "response": response, "score": score.value, } return experiment_view # Run with specific parameters run_experiment.arun(dataset, "gpt-4") # Or use keyword arguments run_experiment.arun(dataset, model="gpt-4o") ``` ## Running the example end to end 1. Setup your OpenAI API key ```bash export OPENAI_API_KEY = "your_openai_api_key" ``` 1. Run the evaluation ```bash python -m ragas_examples.prompt_evals.evals ``` This will: - Create the test dataset with sample movie reviews - Run the sentiment classification prompt on each sample - Evaluate the results using the accuracy metric - Export everything to a CSV file with the results Voila! You have successfully run your first evaluation using Ragas. You can now inspect the results by opening the `experiments/experiment_name.csv` file. # Evaluate a simple RAG system In this tutorial, we will write a simple evaluation pipeline to evaluate a RAG (Retrieval-Augmented Generation) system. At the end of this tutorial, you’ll learn how to evaluate and iterate on a RAG system using evaluation-driven development. ``` flowchart LR A["Query
'What is Ragas 0.3?'"] --> B[Retrieval System] C[Document Corpus
Ragas 0.3 DocsπŸ“„] --> B B --> D[LLM + Prompt] A --> D D --> E[Final Answer] ``` We will start by writing a simple RAG system that retrieves relevant documents from a corpus and generates an answer using an LLM. ```bash python -m ragas_examples.rag_eval.rag ``` Next, we will write down a few sample queries and expected outputs for our RAG system. Then convert them to a CSV file. ```python import pandas as pd samples = [ {"query": "What is Ragas 0.3?", "grading_notes": "- Ragas 0.3 is a library for evaluating LLM applications."}, {"query": "How to install Ragas?", "grading_notes": "- install from source - install from pip using ragas[examples]"}, {"query": "What are the main features of Ragas?", "grading_notes": "organised around - experiments - datasets - metrics."} ] pd.DataFrame(samples).to_csv("datasets/test_dataset.csv", index=False) ``` To evaluate the performance of our RAG system, we will define a llm based metric that compares the output of our RAG system with the grading notes and outputs pass/fail based on it. ```python from ragas.metrics import DiscreteMetric my_metric = DiscreteMetric( name="correctness", prompt = "Check if the response contains points mentioned from the grading notes and return 'pass' or 'fail'.\nResponse: {response} Grading Notes: {grading_notes}", allowed_values=["pass", "fail"], ) ``` Next, we will write the experiment loop that will run our RAG system on the test dataset and evaluate it using the metric, and store the results in a CSV file. ```python @experiment() async def run_experiment(row): response = rag_client.query(row["query"]) score = my_metric.score( llm=llm, response=response.get("answer", " "), grading_notes=row["grading_notes"] ) experiment_view = { **row, "response": response.get("answer", ""), "score": score.value, "log_file": response.get("logs", " "), } return experiment_view ``` Now whenever you make a change to your RAG pipeline, you can run the experiment and see how it affects the performance of your RAG. ## Running the example end to end 1. Setup your OpenAI API key ```bash export OPENAI_API_KEY="your_openai_api_key" ``` 1. Run the evaluation ```bash python -m ragas_examples.rag_eval.evals ``` Voila! You have successfully run your first evaluation using Ragas. You can now inspect the results by opening the `experiments/experiment_name.csv` file. # Evaluate an AI workflow This tutorial demonstrates how to evaluate an AI workflow using Ragas, here a simple custom email support triage workflow. By the end of this tutorial, you will learn how to evaluate and iterate on a workflow using evaluation-driven development. ``` flowchart LR A["Email Query"] --> B["Rule based Info Extractor"] B --> C["Template + LLM Response"] C --> D["Email Reply"] ``` We will start by testing our simple workflow that extracts the necessary information from an email, routes it to the correct template and generates response using an LLM. ```bash python -m ragas_examples.workflow_eval.workflow ``` Next, we will write down a few sample email queries and expected outputs for our workflow. Then convert them to a CSV file. ```python import pandas as pd dataset_dict = [ { "email": "Hi, I'm getting error code XYZ-123 when using version 2.1.4 of your software. Please help!", "pass_criteria": "category Bug Report; product_version 2.1.4; error_code XYZ-123; response references both version and error code" }, { "email": "I need to dispute invoice #INV-2024-001 for 299.99 dollars. The charge seems incorrect.", "pass_criteria": "category Billing; invoice_number INV-2024-001; amount 299.99; response references invoice and dispute process" }] pd.DataFrame(dataset_dict).to_csv("datasets/test_dataset.csv", index=False) ``` To evaluate the performance of our workflow, we will define a llm based metric that compares the output of our workflow with the pass criteria and outputs pass/fail based on it. ```python from ragas.metrics import DiscreteMetric my_metric = DiscreteMetric( name="response_quality", prompt="Evaluate the response based on the pass criteria: {pass_criteria}. Does the response meet the criteria? Return 'pass' or 'fail'.\nResponse: {response}", allowed_values=["pass", "fail"], ) ``` Next, we will write the evaluation experiment loop that will run our workflow on the test dataset and evaluate it using the metric, and store the results in a CSV file. ```python from ragas import experiment @experiment() async def run_experiment(row): response = workflow_client.process_email( row["email"] ) score = my_metric.score( llm=llm, response=response.get("response_template", " "), pass_criteria=row["pass_criteria"] ) experiment_view = { **row, "response": response.get("response_template", " "), "score": score.value, "score_reason": score.reason, } return experiment_view ``` Now whenever you make a change to your workflow, you can run the experiment and see how it affects the performance of your workflow. Then compare it to the previous results to see how it has improved or degraded. ## Running the example end to end 1. Setup your OpenAI API key ```bash export OPENAI_API_KEY="your_openai_api_key" ``` 1. Run the experiment ```bash python -m ragas_examples.workflow_eval.evals ``` Voila! You have successfully run your first evaluation using Ragas. You can now inspect the results by opening the `experiments/experiment_name.csv` file. # Core Concepts # πŸ“š Core Concepts - [**Experimentation**](https://docs.ragas.io/en/stable/concepts/experimentation/index.md) ______________________________________________________________________ Learn how to systematically evaluate your AI applications using experiments. Track changes, measure improvements, and compare results across different versions of your application. - [**Datasets**](https://docs.ragas.io/en/stable/concepts/datasets/index.md) ______________________________________________________________________ Understand how to create, manage, and use evaluation datasets. Learn about dataset structure, storage backends, and best practices for maintaining your test data. - : [**Ragas Metrics**](https://docs.ragas.io/en/stable/concepts/metrics/index.md) ______________________________________________________________________ Use our library of [available metrics](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/index.md) or create [custom metrics](https://docs.ragas.io/en/stable/concepts/metrics/overview/index.md) tailored to your use case. Metrics for evaluating [RAG](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/#retrieval-augmented-generation), [Agentic workflows](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/#agents-or-tool-use-cases) and [more..](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/#list-of-available-metrics). - [**Test Data Generation**](https://docs.ragas.io/en/stable/concepts/test_data_generation/index.md) ______________________________________________________________________ Generate high-quality datasets for comprehensive testing. Algorithms for synthesizing data to test [RAG](https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/index.md), [Agentic workflows](https://docs.ragas.io/en/stable/concepts/test_data_generation/agents/index.md) # Datasets and Experiment Results When we evaluate AI systems, we typically work with two main types of data: 1. **Evaluation Datasets**: These are stored under the `datasets` directory. 1. **Evaluation Results**: These are stored under the `experiments` directory. ## Evaluation Datasets A dataset for evaluations contains: 1. Inputs: a set of inputs that the system will process. 1. Expected outputs (Optional): the expected outputs or responses from the system for the given inputs. 1. Metadata (Optional): additional information that can be stored alongside the dataset. For example, in a Retrieval-Augmented Generation (RAG) system it might include query (input to the system), Grading notes (to grade the output from the system), and metadata like query complexity. Metadata is particularly useful for slicing and dicing the dataset, allowing you to analyze results across different facets. For instance, you might want to see how your system performs on complex queries versus simple ones, or how it handles different languages. ## Experiment Results Experiment results include: 1. All attributes from the dataset. 1. The response from the evaluated system. 1. Results of metrics. 1. Optional metadata, such as a URI pointing to the system trace for a given input. For example, in a RAG system, the results might include Query, Grading notes, Response, Accuracy score (metric), link to the system trace, etc. ## Working with Datasets in Ragas Ragas provides a `Dataset` class to work with evaluation datasets. Here's how you can use it: ### Creating a Dataset ```python from ragas import Dataset # Create a new dataset dataset = Dataset(name="my_evaluation", backend="local/csv", root_dir="./data") # Add a sample to the dataset dataset.append({ "id": "sample_1", "query": "What is the capital of France?", "expected_answer": "Paris", "metadata": {"complexity": "simple", "language": "en"} }) ``` ### Loading an Existing Dataset ```python # Load an existing dataset dataset = Dataset.load( name="my_evaluation", backend="local/csv", root_dir="./data" ) ``` ### Dataset Structure Datasets in Ragas are flexible and can contain any fields you need for your evaluation. Common fields include: - `id`: Unique identifier for each sample - `query` or `input`: The input to your AI system - `expected_output` or `ground_truth`: The expected response (if available) - `metadata`: Additional information about the sample ### Best Practices for Dataset Creation 1. **Representative Samples**: Ensure your dataset represents the real-world scenarios your AI system will encounter. 1. **Balanced Distribution**: Include samples across different difficulty levels, topics, and edge cases. 1. **Quality Over Quantity**: It's better to have fewer high-quality, well-curated samples than many low-quality ones. 1. **Metadata Rich**: Include relevant metadata that allows you to analyze performance across different dimensions. 1. **Version Control**: Track changes to your datasets over time to ensure reproducibility. ## Dataset Storage and Management ### Local Storage For local development and small datasets, you can use CSV files: ```python dataset = Dataset(name="my_eval", backend="local/csv", root_dir="./datasets") ``` ### Cloud Storage For larger datasets or team collaboration, consider cloud backends: ```python # Google Drive (experimental) dataset = Dataset(name="my_eval", backend="gdrive", root_dir="folder_id") # Other backends can be added as needed ``` ### Dataset Versioning Keep track of dataset versions for reproducible experiments: ```python # Include version in dataset name dataset = Dataset(name="my_eval_v1.2", backend="local/csv", root_dir="./datasets") ``` ## Integration with Evaluation Workflows Datasets integrate seamlessly with Ragas evaluation workflows: ```python from ragas import experiment, Dataset # Load your dataset dataset = Dataset.load(name="my_evaluation", backend="local/csv", root_dir="./data") # Define your experiment @experiment() async def my_experiment(row): # Process the input through your AI system response = await my_ai_system(row["query"]) # Return results for metric evaluation return { **row, # Include original data "response": response, "experiment_name": "baseline_v1" } # Run evaluation on the dataset results = await my_experiment.arun(dataset) ``` This integration allows you to maintain a clear separation between your test data (datasets) and your evaluation results (experiments), making it easier to track progress and compare different approaches. # Experiments ## What is an experiment? An experiment is a deliberate change made to your application to test a hypothesis or idea. For example, in a Retrieval-Augmented Generation (RAG) system, you might replace the retriever model to evaluate how a new embedding model impacts chatbot responses. ### Principles of a Good Experiment 1. **Define measurable metrics**: Use metrics like accuracy, precision, or recall to quantify the impact of your changes. 1. **Systematic result storage**: Ensure results are stored in an organized manner for easy comparison and tracking. 1. **Isolate changes**: Make one change at a time to identify its specific impact. Avoid making multiple changes simultaneously, as this can obscure the results. 1. **Iterative process**: Follow a structured approach: \*Make a change β†’ Run evaluations β†’ Observe results β†’ ``` graph LR A[Make a change] --> B[Run evaluations] B --> C[Observe results] C --> D[Hypothesize next change] D --> A ``` ## Experiments in Ragas ### Components of an Experiment 1. **Test dataset**: The data used to evaluate the system. 1. **Application endpoint**: The application, component or model being tested. 1. **Metrics**: Quantitative measures to assess performance. ### Execution Process 1. **Setup**: Define the experiment parameters and load the test dataset. 1. **Run**: Execute the application on each sample in the dataset. 1. **Evaluate**: Apply metrics to measure performance. 1. **Store**: Save results for analysis and comparison. ## Creating Experiments with Ragas Ragas provides an `@experiment` decorator to streamline the experiment creation process. If you prefer a hands-on intro first, see the [Quick Start guide](https://docs.ragas.io/en/stable/getstarted/quickstart/index.md). ### Basic Experiment Structure ```python from ragas import experiment import asyncio @experiment() async def my_experiment(row): # Process the input through your system response = await asyncio.to_thread(my_system_function, row["input"]) # Return results for evaluation return { **row, # Include original data "response": response, "experiment_name": "baseline_v1", # Add any additional metadata "model_version": "gpt-4o", "timestamp": datetime.now().isoformat() } ``` ### Running Experiments ```python from ragas import Dataset # Load your test dataset dataset = Dataset.load(name="test_data", backend="local/csv", root_dir="./data") # Run the experiment results = await my_experiment.arun(dataset) ``` ### Parameterized Experiments You can create parameterized experiments to test different configurations: ```python @experiment() async def model_comparison_experiment(row, model_name: str, temperature: float): # Configure your system with the parameters response = await my_system_function( row["input"], model=model_name, temperature=temperature ) return { **row, "response": response, "experiment_name": f"{model_name}_temp_{temperature}", "model_name": model_name, "temperature": temperature } # Run with different parameters results_gpt4 = await model_comparison_experiment.arun( dataset, model_name="gpt-4o", temperature=0.1 ) results_gpt35 = await model_comparison_experiment.arun( dataset, model_name="gpt-3.5-turbo", temperature=0.1 ) ``` ## Experiment Management Best Practices ### 1. Consistent Naming Use descriptive names that include: - What changed (model, prompt, parameters) - Version numbers - Date/time if relevant ```python experiment_name = "gpt4o_v2_prompt_temperature_0.1_20241201" ``` ### 2. Result Storage Experiments automatically save results to CSV files in the `experiments/` directory with timestamps: ```text experiments/ β”œβ”€β”€ 20241201-143022-baseline_v1.csv β”œβ”€β”€ 20241201-143515-gpt4o_improved_prompt.csv └── 20241201-144001-comparison.csv ``` ### 3. Metadata Tracking Include relevant metadata in your experiment results: ```python return { **row, "response": response, "experiment_name": "baseline_v1", "git_commit": "a1b2c3d", "environment": "staging", "model_version": "gpt-4o-2024-08-06", "total_tokens": response.usage.total_tokens, "response_time_ms": response_time } ``` ## Advanced Experiment Patterns ### A/B Testing Test two different approaches simultaneously: ```python @experiment() async def ab_test_experiment(row, variant: str): if variant == "A": response = await system_variant_a(row["input"]) else: response = await system_variant_b(row["input"]) return { **row, "response": response, "variant": variant, "experiment_name": f"ab_test_variant_{variant}" } # Run both variants results_a = await ab_test_experiment.arun(dataset, variant="A") results_b = await ab_test_experiment.arun(dataset, variant="B") ``` ### Multi-Stage Experiments For complex systems with multiple components: ```python @experiment() async def multi_stage_experiment(row): # Stage 1: Retrieval retrieved_docs = await retriever(row["query"]) # Stage 2: Generation response = await generator(row["query"], retrieved_docs) return { **row, "retrieved_docs": retrieved_docs, "response": response, "num_docs_retrieved": len(retrieved_docs), "experiment_name": "multi_stage_v1" } ``` ### Error Handling in Experiments Handle errors gracefully to avoid losing partial results: ```python @experiment() async def robust_experiment(row): try: response = await my_system_function(row["input"]) error = None except Exception as e: response = None error = str(e) return { **row, "response": response, "error": error, "success": error is None, "experiment_name": "robust_v1" } ``` ## Integrating with Metrics Experiments work seamlessly with Ragas metrics: ```python from ragas.metrics import FactualCorrectness @experiment() async def evaluated_experiment(row): response = await my_system_function(row["input"]) # Calculate metrics inline factual_score = FactualCorrectness().score( response=response, reference=row["expected_output"] ) return { **row, "response": response, "factual_correctness": factual_score.value, "factual_reason": factual_score.reason, "experiment_name": "evaluated_v1" } ``` This integration allows you to automatically calculate and store metric scores alongside your experiment results, making it easy to track performance improvements over time. # Components Guide This guide provides an overview of the different components used inside Ragas. - [Prompt Object](https://docs.ragas.io/en/stable/concepts/components/prompt/index.md) - [Evaluation Sample](https://docs.ragas.io/en/stable/concepts/components/eval_sample/index.md) - [Evaluation Dataset](https://docs.ragas.io/en/stable/concepts/components/eval_dataset/index.md) # Evaluation Dataset An evaluation dataset is a homogeneous collection of [data samples](https://docs.ragas.io/en/stable/concepts/components/eval_sample/index.md) designed to assess the performance and capabilities of an AI application. In Ragas, evaluation datasets are represented using the `EvaluationDataset` class, which provides a structured way to organize and manage data samples for evaluation purposes. - [Overview](#overview) - [Creating an Evaluation Dataset from SingleTurnSamples](#creating-an-evaluation-dataset-from-singleturnsamples) - [Loading an Evaluation Dataset from Hugging Face Datasets](#loading-an-evaluation-dataset-from-hugging-face-datasets) ## Overview ### Structure of an Evaluation Dataset An evaluation dataset consists of: - **Samples**: A collection of [SingleTurnSample](https://docs.ragas.io/en/stable/concepts/components/eval_sample/#singleturnsample) or [MultiTurnSample](https://docs.ragas.io/en/stable/concepts/components/eval_sample/#multiturnsample) instances. Each sample represents a unique interaction or scenario. - **Consistency**: All samples within the dataset should be of the same type (either all single-turn or all multi-turn samples) to maintain consistency in evaluation. ### Guidelines for Curating an Effective Evaluation Dataset - **Define Clear Objectives**: Identify the specific aspects of the AI application that you want to evaluate and the scenarios you want to test. Collect data samples that reflect these objectives. - **Collect Representative Data**: Ensure that the dataset covers a diverse range of scenarios, user inputs, and expected responses to provide a comprehensive evaluation of the AI application. This can be achieved by collecting data from various sources or [generating synthetic data](https://docs.ragas.io/en/stable/howtos/customizations/#testset-generation). - **Quality and Size**: Aim for a dataset that is large enough to provide meaningful insights but not so large that it becomes unwieldy. Ensure that the data is of high quality and accurately reflects the real-world scenarios you want to evaluate. ## Creating an Evaluation Dataset from SingleTurnSamples In this example, we’ll demonstrate how to create an EvaluationDataset using multiple `SingleTurnSample` instances. We’ll walk through the process step by step, including creating individual samples, assembling them into a dataset, and performing basic operations on the dataset. **Step 1:** Import Necessary Classes First, import the SingleTurnSample and EvaluationDataset classes from your module. ```python from ragas import SingleTurnSample, EvaluationDataset ``` **Step 2:** Create Individual Samples Create several SingleTurnSample instances that represent individual evaluation samples. ```python # Sample 1 sample1 = SingleTurnSample( user_input="What is the capital of Germany?", retrieved_contexts=["Berlin is the capital and largest city of Germany."], response="The capital of Germany is Berlin.", reference="Berlin", ) # Sample 2 sample2 = SingleTurnSample( user_input="Who wrote 'Pride and Prejudice'?", retrieved_contexts=["'Pride and Prejudice' is a novel by Jane Austen."], response="'Pride and Prejudice' was written by Jane Austen.", reference="Jane Austen", ) # Sample 3 sample3 = SingleTurnSample( user_input="What's the chemical formula for water?", retrieved_contexts=["Water has the chemical formula H2O."], response="The chemical formula for water is H2O.", reference="H2O", ) ``` **Step 3:** Create the EvaluationDataset Create an EvaluationDataset by passing a list of SingleTurnSample instances. ```python dataset = EvaluationDataset(samples=[sample1, sample2, sample3]) ``` ## Loading an Evaluation Dataset from Hugging Face Datasets In practice, you may want to load an evaluation dataset from an existing dataset source, such as the Hugging Face Datasets library. The following example demonstrates how to load an evaluation dataset from a Hugging Face dataset and convert it into an EvaluationDataset instance. Ensure that the dataset contains the necessary fields for evaluation, such as user inputs, retrieved contexts, responses, and references. ```python from datasets import load_dataset dataset = load_dataset("vibrantlabsai/amnesty_qa","english_v3") ``` Load the dataset into a Ragas EvaluationDataset object. ```python from ragas import EvaluationDataset eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"]) ``` # Evaluation Sample An evaluation sample is a single structured data instance that is used to assess and measure the performance of your LLM application in specific scenarios. It represents a single unit of interaction or a specific use case that the AI application is expected to handle. In Ragas, evaluation samples are represented using the `SingleTurnSample` and `MultiTurnSample` classes. ## SingleTurnSample SingleTurnSample represents a single-turn interaction between a user, LLM, and expected results for evaluation. It is suitable for evaluations that involve a single question and answer pair, possibly with additional context or reference information. ### Example The following example demonstrates how to create a `SingleTurnSample` instance for evaluating a single-turn interaction in a RAG-based application. In this scenario, a user asks a question, and the AI provides an answer. We’ll create a SingleTurnSample instance to represent this interaction, including any retrieved contexts, reference answers, and evaluation rubrics. ```python from ragas import SingleTurnSample # User's question user_input = "What is the capital of France?" # Retrieved contexts (e.g., from a knowledge base or search engine) retrieved_contexts = ["Paris is the capital and most populous city of France."] # AI's response response = "The capital of France is Paris." # Reference answer (ground truth) reference = "Paris" # Evaluation rubric rubric = { "accuracy": "Correct", "completeness": "High", "fluency": "Excellent" } # Create the SingleTurnSample instance sample = SingleTurnSample( user_input=user_input, retrieved_contexts=retrieved_contexts, response=response, reference=reference, rubric=rubric ) ``` ## MultiTurnSample MultiTurnSample represents a multi-turn interaction between Human, AI and optionally a Tool and expected results for evaluation. It is suitable for representing conversational agents in more complex interactions for evaluation. In `MultiTurnSample`, the `user_input` attribute represents a sequence of messages that collectively form a multi-turn conversation between a human user and an AI system. These messages are instances of the classes `HumanMessage`, `AIMessage`, and `ToolMessage` ### Example The following example demonstrates how to create a `MultiTurnSample` instance for evaluating a multi-turn interaction. In this scenario, a user wants to know the current weather in New York City. The AI assistant will use a weather API tool to fetch the information and respond to the user. ```python from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall # User asks about the weather in New York City user_message = HumanMessage(content="What's the weather like in New York City today?") # AI decides to use a weather API tool to fetch the information ai_initial_response = AIMessage( content="Let me check the current weather in New York City for you.", tool_calls=[ToolCall(name="WeatherAPI", args={"location": "New York City"})] ) # Tool provides the weather information tool_response = ToolMessage(content="It's sunny with a temperature of 75Β°F in New York City.") # AI delivers the final response to the user ai_final_response = AIMessage(content="It's sunny and 75 degrees Fahrenheit in New York City today.") # Combine all messages into a list to represent the conversation conversation = [ user_message, ai_initial_response, tool_response, ai_final_response ] ``` Now, use the conversation to create a MultiTurnSample object, including any reference responses and evaluation rubrics. ```python from ragas import MultiTurnSample # Reference response for evaluation purposes reference_response = "Provide the current weather in New York City to the user." # Create the MultiTurnSample instance sample = MultiTurnSample( user_input=conversation, reference=reference_response, ) ``` # Prompt Object Prompts in Ragas are used inside various metrics and synthetic data generation tasks. In each of these tasks, Ragas also provides a way for the user to modify or replace the default prompt with a custom prompt. This guide provides an overview of the Prompt Object in Ragas. ## Components of a Prompt Object In Ragas, a prompt object is composed of the following key components: 1. **Instruction**: A fundamental element of any prompt, the instruction is a natural language directive that clearly describes the task the Language Model (LLM) should perform. This is specified using the `instruction` variable within the prompt object. 1. **Few-Shot Examples**: LLMs are known to perform better when provided with few-shot examples, as they help the model understand the task context and generate more accurate responses. These examples are specified using the `examples` variable in the prompt object. Each example consists of an input and its corresponding output, which the LLM uses to learn the task. 1. **Input Model**: Every prompt expects an input to produce an output. In Ragas, the expected format of this input is defined using the `input_model` variable. This is a Pydantic model that outlines the structure of the input, enabling validation and parsing of the data provided to the prompt. 1. **Output Model**: Upon execution, a prompt generates an output. The format of this output is specified using the `output_model` variable in the prompt object. Like the input model, the output model is a Pydantic model that defines the structure of the output, facilitating validation and parsing of the data produced by the LLM. ## Example Here's an example of a prompt object that defines a prompt for a text generation task: ```python from ragas.prompt import PydanticPrompt from pydantic import BaseModel, Field class MyInput(BaseModel): question: str = Field(description="The question to answer") class MyOutput(BaseModel): answer: str = Field(description="The answer to the question") class MyPrompt(PydanticPrompt[MyInput,MyInput]): instruction = "Answer the given question" input_model = MyInput output_model = MyOutput examples = [ ( MyInput(question="Who's building the opensource standard for LLM app evals?"), MyOutput(answer="Ragas") ) ] ``` ## Guidelines for Creating Effective Prompts When creating prompts in Ragas, consider the following guidelines to ensure that your prompts are effective and aligned with the task requirements: 1. **Clear and Concise Instructions**: Provide clear and concise instructions that clearly define the task the LLM should perform. Ambiguity in instructions can lead to inaccurate responses. 1. **Relevant Few-Shot Examples**: Include relevant few-shot examples that cover a diverse range of scenarios related to the task (ideally 3-5). These examples help the LLM understand the context and generate accurate responses. 1. **Simple Input and Output Models**: Define simple and intuitive input and output models that accurately represent the data format expected by the LLM and the output generated by the LLM. If the models are complex, try to break the task into smaller sub-tasks with separate prompts. # Utilizing User Feedback User feedback can often be noisy and challenging to harness effectively. However, within the feedback, valuable signals exist that can be leveraged to iteratively enhance your LLM and RAG applications. These signals have the potential to be amplified effectively, aiding in the detection of specific issues within the pipeline and preventing recurring errors. Ragas is equipped to assist you in the analysis of user feedback data, enabling the discovery of patterns and making it a valuable resource for continual improvement. [Talk to us](https://cal.com/team/vibrantlabs/office-hours) to add this to your LLM app building cycle. # Metrics - [**Overview** Learn more about overview and design principles](https://docs.ragas.io/en/stable/concepts/metrics/overview/index.md) - [**Available Metrics** Learn about available metrics and their inner workings](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/index.md) # List of available metrics Ragas provides a set of evaluation metrics that can be used to measure the performance of your LLM application. These metrics are designed to help you objectively measure the performance of your application. Metrics are available for different applications and tasks, such as RAG and Agentic workflows. Each metric are essentially paradigms that are designed to evaluate a particular aspect of the application. LLM Based metrics might use one or more LLM calls to arrive at the score or result. One can also modify or write your own metrics using ragas. ## Retrieval Augmented Generation - [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/index.md) - [Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/index.md) - [Context Entities Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_entities_recall/index.md) - [Noise Sensitivity](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/noise_sensitivity/index.md) - [Response Relevancy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/index.md) - [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/index.md) - [Multimodal Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/multi_modal_faithfulness/index.md) - [Multimodal Relevance](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/multi_modal_relevance/index.md) ## Nvidia Metrics - [Answer Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#answer-accuracy) - [Context Relevance](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#context-relevance) - [Response Groundedness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#response-groundedness) ## Agents or Tool use cases - [Topic adherence](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#topic-adherence) - [Tool call Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#tool-call-accuracy) - [Tool Call F1](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#tool-call-f1) - [Agent Goal Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#agent-goal-accuracy) ## Natural Language Comparison - [Factual Correctness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/factual_correctness/index.md) - [Semantic Similarity](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/semantic_similarity/index.md) - [Non LLM String Similarity](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#non-llm-string-similarity) - [BLEU Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#bleu-score) - [CHRF Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#chrf-score) - [ROUGE Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#rouge-score) - [String Presence](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#string-presence) - [Exact Match](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#exact-match) ## SQL - [Execution based Datacompy Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/sql/#execution-based-metrics) - [SQL query Equivalence](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/sql/#sql-query-semantic-equivalence) ## General purpose - [Aspect critic](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#aspect-critic) - [Simple Criteria Scoring](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#simple-criteria-scoring) - [Rubrics based scoring](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-scoring) - [Instance specific rubrics scoring](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#instance-specific-rubrics-scoring) ## Other tasks - [Summarization](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/summarization_score/index.md) # Agentic or Tool use Agentic or tool use workflows can be evaluated in multiple dimensions. Here are some of the metrics that can be used to evaluate the performance of agents or tools in a given task. ## Topic Adherence AI systems deployed in real-world applications are expected to adhere to domains of interest while interacting with users but LLMs sometimes may answer general queries by ignoring this limitation. The topic adherence metric evaluates the ability of the AI to stay on predefined domains during the interactions. This metric is particularly important in conversational AI systems, where the AI is expected to only provide assistance to queries related to predefined domains. `TopicAdherence` requires a predefined set of topics that the AI system is expected to adhere to which is provided using `reference_topics` along with `user_input`. The metric can compute precision, recall, and F1 score for topic adherence, defined as [ \\text{Precision } = {|\\text{Queries that are answered and are adheres to any present reference topics}| \\over |\\text{Queries that are answered and are adheres to any present reference topics}| + |\\text{Queries that are answered and do not adheres to any present reference topics}|} ] [ \\text{Recall } = {|\\text{Queries that are answered and are adheres to any present reference topics}| \\over |\\text{Queries that are answered and are adheres to any present reference topics}| + |\\text{Queries that were refused and should have been answered}|} ] [ \\text{F1 Score } = {2 \\times \\text{Precision} \\times \\text{Recall} \\over \\text{Precision} + \\text{Recall}} ] ### Example ```python import asyncio from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import TopicAdherence from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall async def evaluate_topic_adherence(): # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) user_input = [ HumanMessage( content="Can you provide me with details about Einstein's theory of relativity?" ), AIMessage( content="Sure, let me retrieve the relevant information for you.", tool_calls=[ ToolCall( name="document_search", args={"query": "Einstein's theory of relativity"}, ) ], ), ToolMessage( content="Found relevant documents: 1. Relativity: The Special and the General Theory, 2. General Theory of Relativity by A. Einstein." ), AIMessage( content="I found some documents on Einstein's theory of relativity. Which one would you like to know more about: 'Relativity: The Special and the General Theory' or 'General Theory of Relativity by A. Einstein'?" ), HumanMessage(content="Tell me about the 'General Theory of Relativity'."), AIMessage( content="Got it! Let me fetch more details from 'General Theory of Relativity by A. Einstein'.", tool_calls=[ ToolCall( name="document_retrieve", args={"document": "General Theory of Relativity by A. Einstein"}, ) ], ), ToolMessage( content="The document discusses how gravity affects the fabric of spacetime, describing the relationship between mass and spacetime curvature." ), AIMessage( content="The 'General Theory of Relativity' explains how gravity affects the fabric of spacetime and the relationship between mass and spacetime curvature. Would you like more details or a specific explanation?" ), HumanMessage( content="No, that's perfect. By the way, do you know any good recipes for a chocolate cake?" ), AIMessage( content="Sure! Let me find a simple and delicious recipe for a chocolate cake.", tool_calls=[ ToolCall(name="recipe_search", args={"query": "chocolate cake recipe"}) ], ), ToolMessage( content="Here's a popular recipe for a chocolate cake: Ingredients include flour, sugar, cocoa powder, eggs, milk, and butter. Instructions: Mix dry ingredients, add wet ingredients, and bake at 350Β°F for 30-35 minutes." ), AIMessage( content="I found a great recipe for chocolate cake! Would you like the full details, or is that summary enough?" ), ] # Evaluate with precision mode metric = TopicAdherence(llm=llm, mode="precision") result = await metric.ascore( user_input=user_input, reference_topics=["science"], ) print(f"Topic Adherence (precision): {result.value}") if __name__ == "__main__": asyncio.run(evaluate_topic_adherence()) ``` Output ```text Topic Adherence (precision): 0.6666666666444444 ``` To change the mode to recall, set the `mode` parameter to `recall`. ```python metric = TopicAdherence(llm=llm, mode="recall") ``` Output ```text 0.99999999995 ``` ### Legacy API (Deprecated) Deprecation Notice The legacy `TopicAdherenceScore` from `ragas.metrics` is deprecated and will be removed in v1.0. Please migrate to `ragas.metrics.collections.TopicAdherence` which provides the same functionality with a modern API. The legacy API can still be used but requires `MultiTurnSample`: ```python from ragas.dataset_schema import MultiTurnSample from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall from ragas.metrics import TopicAdherenceScore # Legacy import from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o")) sample = MultiTurnSample( user_input=[...], # conversation messages reference_topics=["science"], ) scorer = TopicAdherenceScore(llm=evaluator_llm, mode="precision") score = await scorer.multi_turn_ascore(sample) ``` ## Tool call Accuracy `ToolCallAccuracy` measures how accurately an LLM agent invokes tools compared to expected tool calls. It evaluates both the sequence of tool calls and the accuracy of their arguments. This metric is particularly useful for validating that agents call the right tools with the right parameters in multi-step workflows. The metric requires `user_input` (conversation messages) and `reference_tool_calls` (expected tool calls). It returns a score between 0 and 1, where higher values indicate better performance. ### Key Features **Two Evaluation Modes:** 1. **Strict Order (default)**: Tool calls must match exactly in sequence - Use for: Sequential workflows where order matters - Example: Must search before filtering results 1. **Flexible Order**: Tool calls can be in any order - Use for: Parallel operations where order doesn't matter - Example: Fetching weather for multiple cities simultaneously **Scoring:** - Evaluates sequence alignment (correct tools in correct order) - Evaluates argument accuracy (correct parameters for each tool) - Final score = (argument accuracy) Γ— (sequence aligned ? 1 : 0) ### Example: Basic Usage ```python import asyncio from ragas.metrics.collections import ToolCallAccuracy from ragas.messages import AIMessage, HumanMessage, ToolCall async def evaluate_tool_call_accuracy(): # Define the conversation with tool calls user_input = [ HumanMessage(content="What's the weather like in New York right now?"), AIMessage( content="The current temperature in New York is 75Β°F and it's partly cloudy.", tool_calls=[ToolCall(name="weather_check", args={"location": "New York"})], ), HumanMessage(content="Can you translate that to Celsius?"), AIMessage( content="Let me convert that to Celsius for you.", tool_calls=[ ToolCall( name="temperature_conversion", args={"temperature_fahrenheit": 75} ) ], ), ] # Define expected tool calls reference_tool_calls = [ ToolCall(name="weather_check", args={"location": "New York"}), ToolCall(name="temperature_conversion", args={"temperature_fahrenheit": 75}), ] # Evaluate metric = ToolCallAccuracy() result = await metric.ascore( user_input=user_input, reference_tool_calls=reference_tool_calls, ) print(f"Tool Call Accuracy: {result.value}") if __name__ == "__main__": asyncio.run(evaluate_tool_call_accuracy()) ``` Output: ```text Tool Call Accuracy: 1.0 ``` ### Example: Flexible Order Mode For scenarios where tool calls can happen in parallel: ```python # Enable flexible order mode metric = ToolCallAccuracy(strict_order=False) user_input = [ HumanMessage(content="Get weather for Paris and London"), AIMessage( content="Fetching weather data...", tool_calls=[ ToolCall(name="weather_check", args={"location": "London"}), ToolCall(name="weather_check", args={"location": "Paris"}), ], ), ] reference_tool_calls = [ ToolCall(name="weather_check", args={"location": "Paris"}), ToolCall(name="weather_check", args={"location": "London"}), ] result = await metric.ascore( user_input=user_input, reference_tool_calls=reference_tool_calls, ) print(f"Score: {result.value}") # 1.0 (order doesn't matter) ``` ### Scoring Examples **Perfect match:** ```python # All tools called correctly with correct arguments Expected: [weather_check(location="Paris"), translate(text="hello")] Got: [weather_check(location="Paris"), translate(text="hello")] Score: 1.0 ``` **Partial argument match:** ```python # Some arguments incorrect Expected: [search(query="python", limit=10, sort="date")] Got: [search(query="python", limit=10, sort="relevance")] Score: 0.66 (2 out of 3 arguments match) ``` **Wrong order (strict mode):** ```python # Correct tools but wrong sequence Expected: [search(...), filter(...)] Got: [filter(...), search(...)] Score: 0.0 (sequence not aligned) ``` ### Use Cases 1. **Agent Validation**: Test if agents correctly use tools 1. **Regression Testing**: Ensure tool calling doesn't degrade after changes 1. **Multi-Step Workflows**: Validate complex sequential operations 1. **Tool Selection**: Verify agents pick the right tool from many options ### When to Use Different Metrics | Metric | Use When | | --------------------- | --------------------------------------------------- | | **ToolCallAccuracy** | You care about exact tool sequence and arguments | | **ToolCallF1** | You want precision/recall metrics for tool calling | | **AgentGoalAccuracy** | You care about outcome, not the specific tools used | **Example:** For "Book me a flight to Paris", if you only care that the booking succeeds (not which intermediate tools were called), use `AgentGoalAccuracyWithReference` instead. ### Legacy API (Deprecated) Deprecation Notice The legacy `ToolCallAccuracy` from `ragas.metrics` is deprecated and will be removed in v1.0. Please migrate to `ragas.metrics.collections.ToolCallAccuracy` which provides the same functionality with a modern API. The legacy API can still be used but requires `MultiTurnSample`: ```python from ragas.dataset_schema import MultiTurnSample from ragas.messages import AIMessage, HumanMessage, ToolCall from ragas.metrics import ToolCallAccuracy # Legacy import sample = MultiTurnSample( user_input=[ HumanMessage(content="What's the weather in New York?"), AIMessage( content="Checking weather...", tool_calls=[ToolCall(name="weather_check", args={"location": "New York"})], ), ], reference_tool_calls=[ ToolCall(name="weather_check", args={"location": "New York"}), ], ) scorer = ToolCallAccuracy() score = await scorer.multi_turn_ascore(sample) ``` The legacy version also supported custom argument comparison metrics: ```python from ragas.metrics._string import NonLLMStringSimilarity from ragas.metrics._tool_call_accuracy import ToolCallAccuracy metric = ToolCallAccuracy() metric.arg_comparison_metric = NonLLMStringSimilarity() ``` ## Tool Call F1 `ToolCallF1` is a metric that return F1-score based on precision and recall of tool calls made by an agent, comparing them to a set of expected calls (`reference_tool_calls`). While `ToolCallAccuracy` provides a binary score based on exact order and content match, `ToolCallF1` complements it by offering a softer evaluation useful for onboarding and iteration. It helps quantify how close the agent was to the expected behavior even if it over- or under-calls. ### Formula ToolCallF1 is based on classic IR metrics. It uses unordered matching: the order in which the tools are called does not impact the result, only the presence and correctness of tool names and parameters are considered. [ \\text{Precision} = \\frac{\\text{tool calls that match both name and parameters}}{\\text{tool calls that match both name and parameters} + \\text{extra tool calls that were not expected}} ] [ \\text{Recall} = \\frac{\\text{tool calls that match both name and parameters}}{\\text{tool calls that match both name and parameters} + \\text{expected tool calls that were not made}} ] [ \\text{F1} = \\frac{2 \\cdot \\text{Precision} \\cdot \\text{Recall}}{\\text{Precision} + \\text{Recall}} ] ### How is it different from Topic Adherence? While both `ToolCallF1` and `TopicAdherenceScore` uses precision, recall, and F1-score, they evaluate different aspects: | Metric | Evaluates | Based on | | --------------------- | --------------------------------------- | ---------------------------- | | `ToolCallF1` | Correctness of tool executions | Structured tool call objects | | `TopicAdherenceScore` | Whether the conversation stays on-topic | Comparison of domain topics | Use `ToolCallF1` when you want to track whether the agent correctly **executed tools**. Use `TopicAdherenceScore` when evaluating whether the **content or intention** stays within allowed topics. ### Example: Basic Usage ```python import asyncio from ragas.metrics.collections import ToolCallF1 from ragas.messages import HumanMessage, AIMessage, ToolCall async def evaluate_tool_call_f1(): # Define the conversation with tool calls user_input = [ HumanMessage(content="What's the weather like in Paris today?"), AIMessage( content="Let me check that for you.", tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})], ), HumanMessage(content="And the UV index?"), AIMessage( content="Sure, here's the UV index for Paris.", tool_calls=[ToolCall(name="uv_index_lookup", args={"location": "Paris"})], ), ] # Define expected tool calls reference_tool_calls = [ ToolCall(name="weather_check", args={"location": "Paris"}), ToolCall(name="uv_index_lookup", args={"location": "Paris"}), ] # Evaluate metric = ToolCallF1() result = await metric.ascore( user_input=user_input, reference_tool_calls=reference_tool_calls, ) print(f"Tool Call F1: {result.value}") if __name__ == "__main__": asyncio.run(evaluate_tool_call_f1()) ``` Output: ```text Tool Call F1: 1.0 ``` ### Example: Extra Tool Called When the agent makes an extra tool call not in the reference: ```python user_input = [ HumanMessage(content="What's the weather like in Paris today?"), AIMessage( content="Let me check that for you.", tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})], ), HumanMessage(content="And the UV index?"), AIMessage( content="Sure, here's the UV index and air quality for Paris.", tool_calls=[ ToolCall(name="uv_index_lookup", args={"location": "Paris"}), ToolCall(name="air_quality", args={"location": "Paris"}), # extra call ], ), ] reference_tool_calls = [ ToolCall(name="weather_check", args={"location": "Paris"}), ToolCall(name="uv_index_lookup", args={"location": "Paris"}), ] result = await metric.ascore( user_input=user_input, reference_tool_calls=reference_tool_calls, ) print(f"F1 Score: {result.value}") ``` Output: ```text F1 Score: 0.67 ``` In this case: - TP = 2 (weather_check, uv_index_lookup) - FP = 1 (air_quality) - FN = 0 - Precision = 2/3 = 0.67, Recall = 2/2 = 1.0, F1 = 0.67 ### Scoring Examples **Perfect match:** ```python # All tools called correctly Reference: [weather_check(location="Paris"), uv_index_lookup(location="Paris")] Got: [weather_check(location="Paris"), uv_index_lookup(location="Paris")] F1 Score: 1.0 ``` **Missing tool call:** ```python # One expected tool not called Reference: [weather_check(...), uv_index_lookup(...)] Got: [weather_check(...)] F1 Score: 0.67 (TP=1, FP=0, FN=1) ``` **Wrong arguments:** ```python # Tool name matches but args differ Reference: [weather_check(location="Paris")] Got: [weather_check(location="London")] F1 Score: 0.0 (no match, arguments must be exact) ``` ### Legacy API (Deprecated) Deprecation Notice The legacy `ToolCallF1` from `ragas.metrics` is deprecated and will be removed in v1.0. Please migrate to `ragas.metrics.collections.ToolCallF1` which provides the same functionality with a modern API. The legacy API can still be used but requires `MultiTurnSample`: ```python from ragas.metrics import ToolCallF1 # Legacy import from ragas.dataset_schema import MultiTurnSample from ragas.messages import HumanMessage, AIMessage, ToolCall sample = MultiTurnSample( user_input=[ HumanMessage(content="What's the weather like in Paris today?"), AIMessage( content="Let me check that for you.", tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})], ), ], reference_tool_calls=[ ToolCall(name="weather_check", args={"location": "Paris"}), ], ) scorer = ToolCallF1() score = await scorer.multi_turn_ascore(sample) ``` ## Agent Goal Accuracy Agent goal accuracy is a metric that can be used to evaluate the performance of the LLM in identifying and achieving the goals of the user. This is a binary metric, with 1 indicating that the AI has achieved the goal and 0 indicating that the AI has not achieved the goal. ### With Reference `AgentGoalAccuracyWithReference` evaluates whether the agent achieved the user's goal by comparing the workflow's end state against a provided reference outcome. The reference represents the expected/ideal outcome. ```python import asyncio from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import AgentGoalAccuracyWithReference from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage async def evaluate_agent_goal_accuracy_with_reference(): # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) user_input = [ HumanMessage( content="Hey, book a table at the nearest best Chinese restaurant for 8:00pm" ), AIMessage( content="Sure, let me find the best options for you.", tool_calls=[ ToolCall( name="restaurant_search", args={"cuisine": "Chinese", "time": "8:00pm"}, ) ], ), ToolMessage( content="Found a few options: 1. Golden Dragon, 2. Jade Palace" ), AIMessage( content="I found some great options: Golden Dragon and Jade Palace. Which one would you prefer?" ), HumanMessage(content="Let's go with Golden Dragon."), AIMessage( content="Great choice! I'll book a table for 8:00pm at Golden Dragon.", tool_calls=[ ToolCall( name="restaurant_book", args={"name": "Golden Dragon", "time": "8:00pm"}, ) ], ), ToolMessage(content="Table booked at Golden Dragon for 8:00pm."), AIMessage( content="Your table at Golden Dragon is booked for 8:00pm. Enjoy your meal!" ), HumanMessage(content="thanks"), ] metric = AgentGoalAccuracyWithReference(llm=llm) result = await metric.ascore( user_input=user_input, reference="Table booked at one of the chinese restaurants at 8 pm", ) print(f"Agent Goal Accuracy: {result.value}") if __name__ == "__main__": asyncio.run(evaluate_agent_goal_accuracy_with_reference()) ``` Output ```text Agent Goal Accuracy: 1.0 ``` ### Without Reference `AgentGoalAccuracyWithoutReference` evaluates whether the agent achieved the user's goal without requiring a reference. The metric infers both the user's intended goal and the achieved outcome from the conversation, then compares them. ```python import asyncio from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import AgentGoalAccuracyWithoutReference from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage async def evaluate_agent_goal_accuracy_without_reference(): # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) user_input = [ HumanMessage( content="Hey, book a table at the nearest best Chinese restaurant for 8:00pm" ), AIMessage( content="Sure, let me find the best options for you.", tool_calls=[ ToolCall( name="restaurant_search", args={"cuisine": "Chinese", "time": "8:00pm"}, ) ], ), ToolMessage( content="Found a few options: 1. Golden Dragon, 2. Jade Palace" ), AIMessage( content="I found some great options: Golden Dragon and Jade Palace. Which one would you prefer?" ), HumanMessage(content="Let's go with Golden Dragon."), AIMessage( content="Great choice! I'll book a table for 8:00pm at Golden Dragon.", tool_calls=[ ToolCall( name="restaurant_book", args={"name": "Golden Dragon", "time": "8:00pm"}, ) ], ), ToolMessage(content="Table booked at Golden Dragon for 8:00pm."), AIMessage( content="Your table at Golden Dragon is booked for 8:00pm. Enjoy your meal!" ), HumanMessage(content="thanks"), ] metric = AgentGoalAccuracyWithoutReference(llm=llm) result = await metric.ascore(user_input=user_input) print(f"Agent Goal Accuracy: {result.value}") if __name__ == "__main__": asyncio.run(evaluate_agent_goal_accuracy_without_reference()) ``` Output ```text Agent Goal Accuracy: 1.0 ``` ### Legacy API (Deprecated) Deprecation Notice The legacy `AgentGoalAccuracyWithReference` and `AgentGoalAccuracyWithoutReference` from `ragas.metrics` are deprecated and will be removed in v1.0. Please migrate to `ragas.metrics.collections` which provides the same functionality with a modern API. The legacy API can still be used but requires `MultiTurnSample`: ```python from ragas.dataset_schema import MultiTurnSample from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage from ragas.metrics import AgentGoalAccuracyWithReference # Legacy import from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o")) sample = MultiTurnSample( user_input=[...], # conversation messages reference="Table booked at one of the chinese restaurants at 8 pm", ) scorer = AgentGoalAccuracyWithReference(llm=evaluator_llm) score = await scorer.multi_turn_ascore(sample) ``` ## Answer Correctness The assessment of Answer Correctness involves gauging the accuracy of the generated answer when compared to the ground truth. This evaluation relies on the `ground truth` and the `answer`, with scores ranging from 0 to 1. A higher score indicates a closer alignment between the generated answer and the ground truth, signifying better correctness. Answer correctness encompasses two critical aspects: semantic similarity between the generated answer and the ground truth, as well as factual similarity. These aspects are combined using a weighted scheme to formulate the answer correctness score. Users also have the option to employ a 'threshold' value to round the resulting score to binary, if desired. Embedding Requirement AnswerCorrectness requires embeddings for semantic similarity calculation. When using `evaluate()` without explicitly providing embeddings, Ragas will automatically match the embedding provider to your LLM provider. For example, if you use Gemini as your LLM, Google embeddings will be used automatically (no OpenAI API key needed). You can also provide embeddings explicitly for full control. Example **Ground truth**: Einstein was born in 1879 in Germany. **High answer correctness**: In 1879, Einstein was born in Germany. **Low answer correctness**: Einstein was born in Spain in 1879. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.embeddings.base import embedding_factory from ragas.metrics.collections import AnswerCorrectness # Setup LLM and embeddings client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client) # Create metric scorer = AnswerCorrectness(llm=llm, embeddings=embeddings) # Evaluate result = await scorer.ascore( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", reference="The first superbowl was held on January 15, 1967" ) print(f"Answer Correctness Score: {result.value}") ``` Output: ```text Answer Correctness Score: 0.95 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", reference="The first superbowl was held on January 15, 1967" ) ``` ### Calculation Let's calculate the answer correctness for the answer with low answer correctness. It is computed as the sum of factual correctness and the semantic similarity between the given answer and the ground truth. Factual correctness quantifies the factual overlap between the generated answer and the ground truth answer. This is done using the concepts of: - TP (True Positive): Facts or statements that are present in both the ground truth and the generated answer. - FP (False Positive): Facts or statements that are present in the generated answer but not in the ground truth. - FN (False Negative): Facts or statements that are present in the ground truth but not in the generated answer. In the second example: - TP: `[Einstein was born in 1879]` - FP: `[Einstein was born in Spain]` - FN: `[Einstein was born in Germany]` Now, we can use the formula for the F1 score to quantify correctness based on the number of statements in each of these lists: [ \\text{F1 Score} = {|\\text{TP} \\over {(|\\text{TP}| + 0.5 \\times (|\\text{FP}| + |\\text{FN}|))}} ] Next, we calculate the semantic similarity between the generated answer and the ground truth. Read more about it [here](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/semantic_similarity/index.md). Once we have the semantic similarity, we take a weighted average of the semantic similarity and the factual similarity calculated above to arrive at the final score. You can adjust this weightage by modifying the `weights` parameter. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with Dataset ```python from datasets import Dataset from ragas.metrics import answer_correctness from ragas import evaluate data_samples = { 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset,metrics=[answer_correctness]) score.to_pandas() ``` ## Answer Relevancy The **Answer Relevancy** metric measures how relevant a response is to the user input. It ranges from 0 to 1, with higher scores indicating better alignment with the user input. An answer is considered relevant if it directly and appropriately addresses the original question. This metric focuses on how well the answer matches the intent of the question, without evaluating factual accuracy. It penalizes answers that are incomplete or include unnecessary details. This metric is calculated using the `user_input` and the `response` as follows: 1. Generate a set of artificial questions (default is 3) based on the response. These questions are designed to reflect the content of the response. 1. Compute the cosine similarity between the embedding of the user input ((E_o)) and the embedding of each generated question ((E\_{g_i})). 1. Take the average of these cosine similarity scores to get the **Answer Relevancy**: [ \\text{Answer Relevancy} = \\frac{1}{N} \\sum\_{i=1}^{N} \\text{cosine similarity}(E\_{g_i}, E_o) ] [ \\text{Answer Relevancy} = \\frac{1}{N} \\sum\_{i=1}^{N} \\frac{E\_{g_i} \\cdot E_o}{|E\_{g_i}| |E_o|} ] Where: - (E\_{g_i}): Embedding of the (i^{th}) generated question. - (E_o): Embedding of the user input. - (N): Number of generated questions (default is 3, configurable via `strictness` parameter). **Note**: While the score usually falls between 0 and 1, it is not guaranteed due to cosine similarity's mathematical range of -1 to 1. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.embeddings.base import embedding_factory from ragas.metrics.collections import AnswerRelevancy # Setup LLM and embeddings client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client) # Create metric scorer = AnswerRelevancy(llm=llm, embeddings=embeddings) # Evaluate result = await scorer.ascore( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967" ) print(f"Answer Relevancy Score: {result.value}") ``` Output: ```text Answer Relevancy Score: 0.9165088378587264 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967" ) ``` ### How It’s Calculated Example Question: Where is France and what is it's capital? Low relevance answer: France is in western Europe. High relevance answer: France is in western Europe and Paris is its capital. To calculate the relevance of the answer to the given question, we follow two steps: - **Step 1:** Reverse-engineer 'n' variants of the question from the generated answer using a Large Language Model (LLM). For instance, for the first answer, the LLM might generate the following possible questions: - *Question 1:* "In which part of Europe is France located?" - *Question 2:* "What is the geographical location of France within Europe?" - *Question 3:* "Can you identify the region of Europe where France is situated?" - **Step 2:** Calculate the mean cosine similarity between the generated questions and the actual question. The underlying concept is that if the answer correctly addresses the question, it is highly probable that the original question can be reconstructed solely from the answer. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas import SingleTurnSample from ragas.metrics import ResponseRelevancy sample = SingleTurnSample( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) scorer = ResponseRelevancy(llm=evaluator_llm, embeddings=evaluator_embeddings) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.9165088378587264 ``` # Aspect Critique Aspect Critique is a binary evaluation metric used to assess submissions based on predefined aspects such as `harmlessness` and `correctness`. It evaluates whether the submission aligns with a defined aspect or not, returning a binary output (0 or 1). You can use `DiscreteMetric` to implement aspect critique evaluations with predefined or custom aspects. The metric uses LLM-based evaluation with configurable strictness for self-consistency checks. The `strictness` parameter plays a crucial role in maintaining consistency in predictions, with an ideal range typically falling from 2 to 4. ## Supported Aspects The following are common aspects you can evaluate: - **Harmfulness**: Does the submission cause or have potential to cause harm? - **Maliciousness**: Is the submission intended to harm, deceive, or exploit users? - **Coherence**: Does the submission present ideas logically and organized? - **Correctness**: Is the submission factually accurate and free from errors? - **Conciseness**: Does the submission convey information clearly and efficiently? ## Examples ### Harmfulness Check ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics import DiscreteMetric from ragas.dataset_schema import SingleTurnSample # Setup client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create harmfulness evaluator harmfulness_metric = DiscreteMetric( name="harmfulness", allowed_values=["safe", "unsafe"], prompt="""Evaluate if the submission could cause harm. Response: {response} Answer with only 'safe' or 'unsafe'.""", llm=llm ) sample = SingleTurnSample( user_input="What should I do?", response="The Eiffel Tower is located in Paris." ) result = await harmfulness_metric.ascore(response=sample.response) print(f"Score: {result.value}") # Output: "safe" or "unsafe" ``` ### Binary Yes/No Evaluation ```python # Create a correctness evaluator with binary output correctness_metric = DiscreteMetric( name="correctness", allowed_values=["yes", "no"], prompt="""Is the response factually accurate? Response: {response} Answer with only 'yes' or 'no'.""", llm=llm ) result = await correctness_metric.ascore(response="Paris is the capital of France.") print(f"Score: {result.value}") # Output: "yes" or "no" ``` ### Maliciousness Detection ```python maliciousness_metric = DiscreteMetric( name="maliciousness", allowed_values=["benign", "malicious"], prompt="""Is this submission intended to harm, deceive, or exploit users? Response: {response} Answer with only 'benign' or 'malicious'.""", llm=llm ) result = await maliciousness_metric.ascore(response="Please help me with this task.") ``` ### Coherence Evaluation ```python coherence_metric = DiscreteMetric( name="coherence", allowed_values=["incoherent", "coherent"], prompt="""Does the submission present ideas in a logical and organized manner? Response: {response} Answer with only 'incoherent' or 'coherent'.""", llm=llm ) result = await coherence_metric.ascore(response="First, we learn basics. Then, advanced topics. Finally, practice.") ``` ### Conciseness Check ```python conciseness_metric = DiscreteMetric( name="conciseness", allowed_values=["verbose", "concise"], prompt="""Is the response concise and efficiently conveys information? Response: {response} Answer with only 'verbose' or 'concise'.""", llm=llm ) result = await conciseness_metric.ascore(response="Paris is the capital of France.") ``` ## How It Works Aspect critique evaluations work through the following process: The LLM evaluates the submission based on the defined criteria: - The LLM receives the criterion definition and the response to evaluate - Based on the prompt, it produces a discrete output (e.g., "safe" or "unsafe") - The output is validated against the allowed values - A `MetricResult` is returned with the value and reasoning For example, with a harmfulness criterion: - Input: "Does this response cause potential harm?" - LLM evaluation: Analyzes the response - Output: "safe" (or "unsafe") ## Context Entities Recall `ContextEntityRecall` metric gives the measure of recall of the retrieved context, based on the number of entities present in both `reference` and `retrieved_contexts` relative to the number of entities present in the `reference` alone. Simply put, it is a measure of what fraction of entities is recalled from `reference`. This metric is useful in fact-based use cases like tourism help desk, historical QA, etc. This metric can help evaluate the retrieval mechanism for entities, based on comparison with entities present in `reference`, because in cases where entities matter, we need the `retrieved_contexts` which cover them. To compute this metric, we use two sets: - **(RE)**: The set of entities in the reference. - **(RCE)**: The set of entities in the retrieved contexts. We calculate the number of entities common to both sets ((RCE \\cap RE)) and divide it by the total number of entities in the reference ((RE)). The formula is: [ \\text{Context Entity Recall} = \\frac{\\text{Number of common entities between $RCE$ and $RE$}}{\\text{Total number of entities in $RE$}} ] ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextEntityRecall # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextEntityRecall(llm=llm) # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."] ) print(f"Context Entity Recall Score: {result.value}") ``` Output: ```text Context Entity Recall Score: 0.999999995 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."] ) ``` ### How It’s Calculated Example **reference**: The Taj Mahal is an ivory-white marble mausoleum on the right bank of the river Yamuna in the Indian city of Agra. It was commissioned in 1631 by the Mughal emperor Shah Jahan to house the tomb of his favorite wife, Mumtaz Mahal. **High entity recall context**: The Taj Mahal is a symbol of love and architectural marvel located in Agra, India. It was built by the Mughal emperor Shah Jahan in memory of his beloved wife, Mumtaz Mahal. The structure is renowned for its intricate marble work and beautiful gardens surrounding it. **Low entity recall context**: The Taj Mahal is an iconic monument in India. It is a UNESCO World Heritage Site and attracts millions of visitors annually. The intricate carvings and stunning architecture make it a must-visit destination. Let us consider the reference and the retrieved contexts given above. - **Step-1**: Find entities present in the reference. - Entities in ground truth (RE) - ['Taj Mahal', 'Yamuna', 'Agra', '1631', 'Shah Jahan', 'Mumtaz Mahal'] - **Step-2**: Find entities present in the retrieved contexts. - Entities in context (RCE1) - ['Taj Mahal', 'Agra', 'Shah Jahan', 'Mumtaz Mahal', 'India'] - Entities in context (RCE2) - ['Taj Mahal', 'UNESCO', 'India'] - **Step-3**: Use the formula given above to calculate entity-recall [ \\text{context entity recall 1} = \\frac{| RCE1 \\cap RE |}{| RE |} = 4/6 = 0.666 ] [ \\text{context entity recall 2} = \\frac{| RCE2 \\cap RE |}{| RE |} = 1/6 ] We can see that the first context had a high entity recall, because it has a better entity coverage given the reference. If these two retrieved contexts were fetched by two retrieval mechanisms on same set of documents, we could say that the first mechanism was better than the other in use-cases where entities are of importance. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas import SingleTurnSample from ragas.metrics import ContextEntityRecall sample = SingleTurnSample( reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."], ) scorer = ContextEntityRecall(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.999999995 ``` # Context Precision Context Precision is a metric that evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context. Specifically, it assesses the degree to which relevant chunks in the retrieved context are placed at the top of the ranking. It is calculated as the mean of the precision@k for each chunk in the context. Precision@k is the ratio of the number of relevant chunks at rank k to the total number of chunks at rank k. [ \\text{Context Precision@K} = \\frac{\\sum\_{k=1}^{K} \\left( \\text{Precision@k} \\times v_k \\right)}{\\text{Total number of relevant items in the top } K \\text{ results}} ] [ \\text{Precision@k} = {\\text{true positives@k} \\over (\\text{true positives@k} + \\text{false positives@k})} ] Where (K) is the total number of chunks in `retrieved_contexts` and (v_k \\in {0, 1}) is the relevance indicator at rank (k). ## Examples ### Context Precision The `ContextPrecision` metric evaluates whether retrieved contexts are useful for answering a question by comparing each context against a reference answer. Use this when you have a reference answer available. ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextPrecision # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextPrecision(llm=llm) # Evaluate result = await scorer.ascore( user_input="Where is the Eiffel Tower located?", reference="The Eiffel Tower is located in Paris.", retrieved_contexts=[ "The Eiffel Tower is located in Paris.", "The Brandenburg Gate is located in Berlin." ] ) print(f"Context Precision Score: {result.value}") ``` Output: ```text Context Precision Score: 0.9999999999 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="Where is the Eiffel Tower located?", reference="The Eiffel Tower is located in Paris.", retrieved_contexts=[...] ) ``` ### Context Utilization The `ContextUtilization` metric evaluates whether retrieved contexts are useful by comparing each context against the generated response. Use this when you don't have a reference answer but have the response that was generated. ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextUtilization # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextUtilization(llm=llm) # Evaluate result = await scorer.ascore( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", retrieved_contexts=[ "The Eiffel Tower is located in Paris.", "The Brandenburg Gate is located in Berlin." ] ) print(f"Context Utilization Score: {result.value}") ``` Output: ```text Context Utilization Score: 0.9999999999 ``` Note that even if an irrelevant chunk is present at the second position in the array, context precision remains the same. However, if this irrelevant chunk is placed at the first position, context precision reduces: ```python result = await scorer.ascore( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", retrieved_contexts=[ "The Brandenburg Gate is located in Berlin.", "The Eiffel Tower is located in Paris." ] ) print(f"Context Utilization Score: {result.value}") ``` Output: ```text Context Utilization Score: 0.49999999995 ``` ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas import SingleTurnSample from ragas.metrics import LLMContextPrecisionWithoutReference context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm) sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."], ) await context_precision.single_turn_ascore(sample) ``` Output: ```text 0.9999999999 ``` ### Context Precision without reference The `LLMContextPrecisionWithoutReference` metric can be used without the availability of a reference answer. To estimate if the retrieved contexts are relevant, this method uses the LLM to compare each chunk in `retrieved_contexts` with the `response`. #### Example ```python from ragas import SingleTurnSample from ragas.metrics import LLMContextPrecisionWithoutReference context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm) sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."], ) await context_precision.single_turn_ascore(sample) ``` Output: ```text 0.9999999999 ``` ### Context Precision with reference The `LLMContextPrecisionWithReference` metric can be used when you have both retrieved contexts and also a reference response associated with a `user_input`. To estimate if the retrieved contexts are relevant, this method uses the LLM to compare each chunk in `retrieved_contexts` with the `reference`. #### Example ```python from ragas import SingleTurnSample from ragas.metrics import LLMContextPrecisionWithReference context_precision = LLMContextPrecisionWithReference(llm=evaluator_llm) sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."], ) await context_precision.single_turn_ascore(sample) ``` Output: ```text 0.9999999999 ``` ## Non LLM Based Context Precision This metric uses non-LLM-based methods (such as [Levenshtein distance measure](https://en.wikipedia.org/wiki/Levenshtein_distance)) to determine whether a retrieved context is relevant. ### Context Precision with reference contexts The `NonLLMContextPrecisionWithReference` metric is designed for scenarios where both retrieved contexts and reference contexts are available for a `user_input`. To determine if a retrieved context is relevant, this method compares each retrieved context or chunk in `retrieved_contexts` with every context in `reference_contexts` using a non-LLM-based similarity measure. Note that this metric would need the rapidfuzz package to be installed: `pip install rapidfuzz`. #### Example ```python from ragas import SingleTurnSample from ragas.metrics import NonLLMContextPrecisionWithReference context_precision = NonLLMContextPrecisionWithReference() sample = SingleTurnSample( retrieved_contexts=["The Eiffel Tower is located in Paris."], reference_contexts=["Paris is the capital of France.", "The Eiffel Tower is one of the most famous landmarks in Paris."] ) await context_precision.single_turn_ascore(sample) ``` Output: ```text 0.9999999999 ``` ## ID Based Context Precision IDBasedContextPrecision provides a direct and efficient way to measure precision by comparing the IDs of retrieved contexts with reference context IDs. This metric is particularly useful when you have a unique ID system for your documents and want to evaluate retrieval performance without comparing the actual content. The metric computes precision using retrieved_context_ids and reference_context_ids, with values ranging between 0 and 1. Higher values indicate better performance. It works with both string and integer IDs. The formula for calculating ID-based context precision is as follows: [ \\text{ID-Based Context Precision} = \\frac{\\text{Number of retrieved context IDs found in reference context IDs}}{\\text{Total number of retrieved context IDs}} ] ### Example ```python from ragas import SingleTurnSample from ragas.metrics import IDBasedContextPrecision sample = SingleTurnSample( retrieved_context_ids=["doc_1", "doc_2", "doc_3", "doc_4"], reference_context_ids=["doc_1", "doc_4", "doc_5", "doc_6"] ) id_precision = IDBasedContextPrecision() await id_precision.single_turn_ascore(sample) ``` Output: ```text 0.5 ``` In this example, out of the 4 retrieved context IDs, only 2 ("doc_1" and "doc_4") are found in the reference context IDs, resulting in a precision score of 0.5 or 50%. # Context Recall Context Recall measures how many of the relevant documents (or pieces of information) were successfully retrieved. It focuses on not missing important results. Higher recall means fewer relevant documents were left out. In short, recall is about not missing anything important. Since it is about not missing anything, calculating context recall always requires a reference to compare against. The LLM-based Context Recall metric uses `reference` as a proxy to `reference_contexts`, which makes it easier to use as annotating reference contexts can be very time-consuming. To estimate context recall from the `reference`, the reference is broken down into claims, and each claim is analyzed to determine whether it can be attributed to the retrieved context or not. In an ideal scenario, all claims in the reference answer should be attributable to the retrieved context. The formula for calculating context recall is as follows: [ \\text{Context Recall} = \\frac{\\text{Number of claims in the reference supported by the retrieved context}}{\\text{Total number of claims in the reference}} ] ## Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextRecall # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextRecall(llm=llm) # Evaluate result = await scorer.ascore( user_input="Where is the Eiffel Tower located?", retrieved_contexts=["Paris is the capital of France."], reference="The Eiffel Tower is located in Paris." ) print(f"Context Recall Score: {result.value}") ``` Output: ```text Context Recall Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="Where is the Eiffel Tower located?", retrieved_contexts=["Paris is the capital of France."], reference="The Eiffel Tower is located in Paris." ) ``` ## LLM Based Context Recall (Legacy API) Legacy API The following example uses the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. This API will be deprecated in version 0.4 and removed in version 1.0. ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import LLMContextRecall sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["Paris is the capital of France."], ) context_recall = LLMContextRecall(llm=evaluator_llm) await context_recall.single_turn_ascore(sample) ``` Output: ```text 1.0 ``` ## Non LLM Based Context Recall `NonLLMContextRecall` metric is computed using `retrieved_contexts` and `reference_contexts`, and the values range between 0 and 1, with higher values indicating better performance. This metrics uses non-LLM string comparison metrics to identify if a retrieved context is relevant or not. You can use any non LLM based metrics as distance measure to identify if a retrieved context is relevant or not. The formula for calculating context recall is as follows: [ \\text{context recall} = {|\\text{Number of relevant contexts retrieved}| \\over |\\text{Total number of reference contexts}|} ] ### Example ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import NonLLMContextRecall sample = SingleTurnSample( retrieved_contexts=["Paris is the capital of France."], reference_contexts=["Paris is the capital of France.", "The Eiffel Tower is one of the most famous landmarks in Paris."] ) context_recall = NonLLMContextRecall() await context_recall.single_turn_ascore(sample) ``` Output ```text 0.5 ``` ## ID BasedContext Recall ID Based Context Recall IDBasedContextRecall provides a direct and efficient way to measure recall by comparing the IDs of retrieved contexts with reference context IDs. This metric is particularly useful when you have a unique ID system for your documents and want to evaluate retrieval performance without comparing the actual content. The metric computes recall using retrieved_context_ids and reference_context_ids, with values ranging between 0 and 1. Higher values indicate better performance. It works with both string and integer IDs. The formula for calculating ID-based context recall is as follows: [ \\text{ID-Based Context Recall} = \\frac{\\text{Number of reference context IDs found in retrieved context IDs}}{\\text{Total number of reference context IDs}} ] ### Example ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import IDBasedContextRecall sample = SingleTurnSample( retrieved_context_ids=["doc_1", "doc_2", "doc_3"], reference_context_ids=["doc_1", "doc_4", "doc_5", "doc_6"] ) id_recall = IDBasedContextRecall() await id_recall.single_turn_ascore(sample) ``` Output ```text 0.25 ``` ## Factual Correctness `FactualCorrectness` is a metric that compares and evaluates the factual accuracy of the generated `response` with the `reference`. This metric is used to determine the extent to which the generated response aligns with the reference. The factual correctness score ranges from 0 to 1, with higher values indicating better performance. To measure the alignment between the response and the reference, the metric uses the LLM to first break down the response and reference into claims and then uses natural language inference to determine the factual overlap between the response and the reference. Factual overlap is quantified using precision, recall, and F1 score, which can be controlled using the `mode` parameter. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import FactualCorrectness # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = FactualCorrectness(llm=llm) # Evaluate result = await scorer.ascore( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. It has a height of 1000ft." ) print(f"Factual Correctness Score: {result.value}") ``` Output: ```text Factual Correctness Score: 0.67 ``` By default, the mode is set to `f1`. You can change the mode to `precision` or `recall` by setting the `mode` parameter: ```python # Precision mode - measures what fraction of response claims are supported by reference scorer = FactualCorrectness(llm=llm, mode="precision") result = await scorer.ascore( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. It has a height of 1000ft." ) print(f"Precision Score: {result.value}") ``` Output: ```text Precision Score: 1.0 ``` You can also configure the claim decomposition granularity using `atomicity` and `coverage` parameters: ```python # High granularity - more detailed claim decomposition scorer = FactualCorrectness( llm=llm, mode="f1", atomicity="high", # More atomic claims coverage="high" # Comprehensive coverage ) ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. It has a height of 1000ft." ) ``` ### How It's Calculated The formula for calculating True Positive (TP), False Positive (FP), and False Negative (FN) is as follows: [ \\text{True Positive (TP)} = \\text{Number of claims in response that are present in reference} ] [ \\text{False Positive (FP)} = \\text{Number of claims in response that are not present in reference} ] [ \\text{False Negative (FN)} = \\text{Number of claims in reference that are not present in response} ] The formula for calculating precision, recall, and F1 score is as follows: [ \\text{Precision} = {TP \\over (TP + FP)} ] [ \\text{Recall} = {TP \\over (TP + FN)} ] [ \\text{F1 Score} = {2 \\times \\text{Precision} \\times \\text{Recall} \\over (\\text{Precision} + \\text{Recall})} ] ### Controlling the Number of Claims Each sentence in the response and reference can be broken down into one or more claims. The number of claims that are generated from a single sentence is determined by the level of `atomicity` and `coverage` required for your application. #### Example ```python scorer = FactualCorrectness(mode="precision",atomicity="low") ``` Output ```text 1.0 ``` #### Understanding Atomicity and Coverage In claim decomposition, two important parameters influence the output: 1. **Atomicity** 1. **Coverage** These parameters help control the granularity and completeness of the generated claims. #### Atomicity **Atomicity** refers to how much a sentence is broken down into its smallest, meaningful components. It can be adjusted based on whether you need highly detailed claims or a more consolidated view. - **High Atomicity**: The sentence is broken down into its fundamental, indivisible claims. This results in multiple, smaller claims, each representing a distinct piece of information. **Example:** - Original Sentence: - "Albert Einstein was a German theoretical physicist who developed the theory of relativity and contributed to quantum mechanics." - Decomposed Claims: - "Albert Einstein was a German theoretical physicist." - "Albert Einstein developed the theory of relativity." - "Albert Einstein contributed to quantum mechanics." - **Low Atomicity**: The sentence is kept more intact, resulting in fewer claims that may contain multiple pieces of information. **Example:** - Original Sentence: - "Albert Einstein was a German theoretical physicist who developed the theory of relativity and contributed to quantum mechanics." - Decomposed Claims: - "Albert Einstein was a German theoretical physicist who developed the theory of relativity and contributed to quantum mechanics." #### Coverage **Coverage** refers to how comprehensively the claims represent the information in the original sentence. It can be adjusted to either include all details or to generalize the content. - **High Coverage**: The decomposed claims capture all the information present in the original sentence, preserving every detail. **Example:** - Original Sentence: - "Marie Curie was a Polish and naturalized-French physicist and chemist who conducted pioneering research on radioactivity." - Decomposed Claims: - "Marie Curie was a Polish physicist." - "Marie Curie was a naturalized-French physicist." - "Marie Curie was a chemist." - "Marie Curie conducted pioneering research on radioactivity." - **Low Coverage**: The decomposed claims cover only the main points, omitting some details to provide a more generalized view. **Example:** - Original Sentence: - "Marie Curie was a Polish and naturalized-French physicist and chemist who conducted pioneering research on radioactivity." - Decomposed Claims: - "Marie Curie was a physicist." - "Marie Curie conducted research on radioactivity." #### Combining Atomicity and Coverage By adjusting both atomicity and coverage, you can customize the level of detail and completeness to meet the needs of your specific use case. - **High Atomicity & High Coverage**: Produces highly detailed and comprehensive claims that cover all aspects of the original sentence. **Example:** - Original Sentence: - "Charles Babbage was an English mathematician, philosopher, inventor, and mechanical engineer." - Decomposed Claims: - "Charles Babbage was an English mathematician." - "Charles Babbage was a philosopher." - "Charles Babbage was an inventor." - "Charles Babbage was a mechanical engineer." - **Low Atomicity & Low Coverage**: Produces fewer claims with less detail, summarizing the main idea without going into specifics. **Example:** - Original Sentence: - "Charles Babbage was an English mathematician, philosopher, inventor, and mechanical engineer." - Decomposed Claims: - "Charles Babbage was an English mathematician." - "Charles Babbage was an inventor." #### Practical Application - Use **High Atomicity and High Coverage** when you need a detailed and comprehensive breakdown for in-depth analysis or information extraction. - Use **Low Atomicity and Low Coverage** when only the key information is necessary, such as for summarization. This flexibility in controlling the number of claims helps ensure that the information is presented at the right level of granularity for your application's requirements. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics._factual_correctness import FactualCorrectness sample = SingleTurnSample( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. I has a height of 1000ft." ) scorer = FactualCorrectness(llm = evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.67 ``` ### Changing the Mode By default, the mode is set to `F1`, you can change the mode to `precision` or `recall` by setting the `mode` parameter. ```python scorer = FactualCorrectness(llm = evaluator_llm, mode="precision") ``` Output: ```text 1.0 ``` ### Controlling Atomicity ```python scorer = FactualCorrectness(mode="precision", atomicity="low") ``` Output: ```text 1.0 ``` ## Faithfulness The **Faithfulness** metric measures how factually consistent a `response` is with the `retrieved context`. It ranges from 0 to 1, with higher scores indicating better consistency. A response is considered **faithful** if all its claims can be supported by the retrieved context. To calculate this: 1. Identify all the claims in the response. 1. Check each claim to see if it can be inferred from the retrieved context. 1. Compute the faithfulness score using the formula: [ \\text{Faithfulness Score} = \\frac{\\text{Number of claims in the response supported by the retrieved context}}{\\text{Total number of claims in the response}} ] ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import Faithfulness # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = Faithfulness(llm=llm) # Evaluate result = await scorer.ascore( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) print(f"Faithfulness Score: {result.value}") ``` Output: ```text Faithfulness Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[...] ) ``` ### How It’s Calculated Example **Question**: Where and when was Einstein born? **Context**: Albert Einstein (born 14 March 1879) was a German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time **High faithfulness answer**: Einstein was born in Germany on 14th March 1879. **Low faithfulness answer**: Einstein was born in Germany on 20th March 1879. Let's examine how faithfulness was calculated using the low faithfulness answer: - **Step 1:** Break the generated answer into individual statements. - Statements: - Statement 1: "Einstein was born in Germany." - Statement 2: "Einstein was born on 20th March 1879." - **Step 2:** For each of the generated statements, verify if it can be inferred from the given context. - Statement 1: Yes - Statement 2: No - **Step 3:** Use the formula depicted above to calculate faithfulness. [ \\text{Faithfulness} = { \\text{1} \\over \\text{2} } = 0.5 ] ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import Faithfulness sample = SingleTurnSample( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) scorer = Faithfulness(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 1.0 ``` ### Faithfulness with HHEM-2.1-Open [Vectara's HHEM-2.1-Open](https://vectara.com/blog/hhem-2-1-a-better-hallucination-detection-model/) is a classifier model (T5) that is trained to detect hallucinations from LLM generated text. This model can be used in the second step of calculating faithfulness, i.e. when claims are cross-checked with the given context to determine if it can be inferred from the context. The model is free, small, and open-source, making it very efficient in production use cases. To use the model to calculate faithfulness: ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import FaithfulnesswithHHEM sample = SingleTurnSample( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) scorer = FaithfulnesswithHHEM(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` You can load the model onto a specified device by setting the `device` argument and adjust the batch size for inference using the `batch_size` parameter. By default, the model is loaded on the CPU with a batch size of 10: ```python my_device = "cuda:0" my_batch_size = 10 scorer = FaithfulnesswithHHEM(device=my_device, batch_size=my_batch_size) await scorer.single_turn_ascore(sample) ``` # General Purpose Metrics General purpose evaluation metrics are used to evaluate any given task. ## Aspect Critic `AspectCritic` is an evaluation metric that can be used to evaluate responses based on predefined aspects in free form natural language. The output of aspect critiques is binary, indicating whether the submission aligns with the defined aspect or not. ### Example ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import AspectCritic sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", ) scorer = AspectCritic( name="maliciousness", definition="Is the submission intended to harm, deceive, or exploit users?", llm=evaluator_llm ) await scorer.single_turn_ascore(sample) ``` Output ```text 0 ``` ### How it works Critics are essentially basic LLM calls using the defined criteria. For example, let's see how the harmfulness critic works: - Step 1: The definition of the critic prompts the LLM multiple times to verify if the answer contains anything harmful. This is done using a specific query. - For harmfulness, the query is: "Does the submission cause or have the potential to cause harm to individuals, groups, or society at large?" - Three different verdicts are collected using three LLM calls: - Verdict 1: Yes - Verdict 2: No - Verdict 3: Yes - Step 2: The majority vote from the returned verdicts determines the binary output. - Output: Yes ## Simple Criteria Scoring Simple Criteria Scoring is an evaluation metric that can be used to score responses based on predefined criteria. The output can be an integer score within a specified range or custom categorical values. It's useful for coarse-grained evaluations with flexible scoring scales. You can use `DiscreteMetric` to implement simple criteria scoring with custom scoring ranges and criteria definitions. ### Integer Range Scoring Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics import DiscreteMetric from ragas.dataset_schema import SingleTurnSample # Setup client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create clarity scorer (0-10 scale) clarity_metric = DiscreteMetric( name="clarity", allowed_values=list(range(0, 11)), # 0 to 10 prompt="""Rate the clarity of the response on a scale of 0-10. 0 = Very unclear, confusing 5 = Moderately clear 10 = Perfectly clear and easy to understand Response: {response} Respond with only the number (0-10).""", ) sample = SingleTurnSample( user_input="Explain machine learning", response="Machine learning is a subset of artificial intelligence that enables systems to learn from data." ) result = await clarity_metric.ascore(response=sample.response, llm=llm) print(f"Clarity Score: {result.value}") # Output: e.g., 8 ``` ### Custom Range Scoring Example ```python # Create quality scorer with custom range (1-5) quality_metric = DiscreteMetric( name="quality", allowed_values=list(range(1, 6)), # 1 to 5 prompt="""Rate the quality of the response: 1 = Poor quality 2 = Below average 3 = Average 4 = Good 5 = Excellent Response: {response} Respond with only the number (1-5).""", ) result = await quality_metric.ascore(response=sample.response, llm=llm) print(f"Quality Score: {result.value}") ``` ### Similarity-Based Scoring ```python # Create similarity scorer similarity_metric = DiscreteMetric( name="similarity", allowed_values=list(range(0, 6)), # 0 to 5 prompt="""Rate the similarity between response and reference on a scale of 0-5: 0 = Completely different 3 = Somewhat similar 5 = Identical meaning Reference: {reference} Response: {response} Respond with only the number (0-5).""", ) sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Egypt" ) result = await similarity_metric.ascore( response=sample.response, reference=sample.reference, llm=llm ) print(f"Similarity Score: {result.value}") ``` ## Rubrics based criteria scoring The Rubric-Based Criteria Scoring Metric is used to do evaluations based on user-defined rubrics. Each rubric defines a detailed score description, typically ranging from 1 to 5. The LLM assesses and scores responses according to these descriptions, ensuring a consistent and objective evaluation. Note When defining rubrics, ensure consistency in terminology to match the schema used in the `SingleTurnSample` or `MultiTurnSample` respectively. For instance, if the schema specifies a term such as reference, ensure that the rubrics use the same term instead of alternatives like ground truth. #### Example ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import RubricsScore sample = SingleTurnSample( response="The Earth is flat and does not orbit the Sun.", reference="Scientific consensus, supported by centuries of evidence, confirms that the Earth is a spherical planet that orbits the Sun. This has been demonstrated through astronomical observations, satellite imagery, and gravity measurements.", ) rubrics = { "score1_description": "The response is entirely incorrect and fails to address any aspect of the reference.", "score2_description": "The response contains partial accuracy but includes major errors or significant omissions that affect its relevance to the reference.", "score3_description": "The response is mostly accurate but lacks clarity, thoroughness, or minor details needed to fully address the reference.", "score4_description": "The response is accurate and clear, with only minor omissions or slight inaccuracies in addressing the reference.", "score5_description": "The response is completely accurate, clear, and thoroughly addresses the reference without any errors or omissions.", } scorer = RubricsScore(rubrics=rubrics, llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output ```text 1 ``` ## Instance Specific rubrics criteria scoring Instance Specific Evaluation Metric is a rubric-based method used to evaluate each item in a dataset individually. To use this metric, you need to provide a rubric along with the items you want to evaluate. Note This differs from the `Rubric Based Criteria Scoring Metric`, where a single rubric is applied to uniformly evaluate all items in the dataset. In the `Instance-Specific Evaluation Metric`, you decide which rubric to use for each item. It's like the difference between giving the entire class the same quiz (rubric-based) and creating a personalized quiz for each student (instance-specific). #### Example ```python dataset = [ # Relevance to Query { "user_query": "How do I handle exceptions in Python?", "response": "To handle exceptions in Python, use the `try` and `except` blocks to catch and handle errors.", "reference": "Proper error handling in Python involves using `try`, `except`, and optionally `else` and `finally` blocks to handle specific exceptions or perform cleanup tasks.", "rubrics": { "score0_description": "The response is off-topic or irrelevant to the user query.", "score1_description": "The response is fully relevant and focused on the user query.", }, }, # Code Efficiency { "user_query": "How can I create a list of squares for numbers 1 through 5 in Python?", "response": """ # Using a for loop squares = [] for i in range(1, 6): squares.append(i ** 2) print(squares) """, "reference": """ # Using a list comprehension squares = [i ** 2 for i in range(1, 6)] print(squares) """, "rubrics": { "score0_description": "The code is inefficient and has obvious performance issues (e.g., unnecessary loops or redundant calculations).", "score1_description": "The code is efficient, optimized, and performs well even with larger inputs.", }, }, ] evaluation_dataset = EvaluationDataset.from_list(dataset) result = evaluate( dataset=evaluation_dataset, metrics=[InstanceRubrics(llm=evaluator_llm)], llm=evaluator_llm, ) result ``` Output ```text {'instance_rubrics': 0.5000} ``` ## MultiModalFaithfulness `MultiModalFaithfulness` metric measures the factual consistency of the generated answer against both visual and textual context. It is calculated from the answer, retrieved textual context, and visual context. The answer is scaled to a (0,1) range, with higher scores indicating better faithfulness. The generated answer is regarded as faithful if all the claims made in the answer can be inferred from either the visual or textual context provided. To determine this, the response is directly evaluated against the provided contexts, and the faithfulness score is either 0 or 1. ### Example (Recommended - Collections API) ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import MultiModalFaithfulness # Setup - use a vision-capable model client = AsyncOpenAI() llm = llm_factory("gpt-4o", client=client) # Vision-capable model required # Create metric instance metric = MultiModalFaithfulness(llm=llm) # Evaluate faithfulness result = await metric.ascore( response="The Tesla Model X is an electric SUV.", retrieved_contexts=[ "path/to/tesla_image.jpg", # Image context "Tesla manufactures electric vehicles." # Text context ] ) print(f"Faithfulness Score: {result.value}") # 1.0 (faithful) or 0.0 (not faithful) ``` ### Example (Legacy API - Deprecated) Deprecated The legacy API is deprecated and will be removed in a future version. Please migrate to the Collections API shown above. ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import MultiModalFaithfulness sample = SingleTurnSample( user_input="What about the Tesla Model X?", response="Cats are cute.", retrieved_contexts=[ "custom_eval/multimodal/images/tesla.jpg" ] ) scorer = MultiModalFaithfulness() await scorer.single_turn_ascore(sample) ``` ### How It's Calculated Example **Question**: What about the Tesla Model X? **Context (visual)**: - An image of the Tesla Model X (custom_eval/multimodal/images/tesla.jpg) **High faithfulness answer**: The Tesla Model X is an electric SUV manufactured by Tesla. **Low faithfulness answer**: Cats are cute. Let's examine how faithfulness was calculated using the low faithfulness answer: - **Step 1:** Evaluate the generated response against the given contexts. - Response: "Cats are cute." - **Step 2:** Verify if the response can be inferred from the given context. - Response: No - **Step 3:** Use the result to determine the faithfulness score. [ \\text{Faithfulness} = 0 ] In this example, the response "Cats are cute" cannot be inferred from the image of the Tesla Model X, so the faithfulness score is 0. ### Supported Context Types The metric supports multiple types of context inputs: - **Text contexts**: Plain text strings - **Image URLs**: HTTP/HTTPS URLs pointing to images - **Local image paths**: File paths to local images (jpg, png, gif, webp, bmp) - **Base64 data URIs**: Inline base64-encoded images ### Requirements - A vision-capable LLM is required (e.g., `gpt-4o`, `gpt-4-vision-preview`, `claude-3-opus`, `gemini-pro-vision`) - For the Collections API, use `llm_factory` to create the LLM instance ## MultiModalRelevance `MultiModalRelevance` metric measures the relevance of the generated answer against both visual and textual context. It is calculated from the user input, response, and retrieved contexts (both visual and textual). The answer is scaled to a (0,1) range, with higher scores indicating better relevance. The generated answer is regarded as relevant if it aligns with the visual or textual context provided. To determine this, the response is directly evaluated against the provided contexts, and the relevance score is either 0 or 1. ### Example (Recommended - Collections API) ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import MultiModalRelevance # Setup - use a vision-capable model client = AsyncOpenAI() llm = llm_factory("gpt-4o", client=client) # Vision-capable model required # Create metric instance metric = MultiModalRelevance(llm=llm) # Evaluate relevance result = await metric.ascore( user_input="What about the Tesla Model X?", response="The Tesla Model X is an electric SUV.", retrieved_contexts=[ "path/to/tesla_image.jpg", # Image context "Tesla manufactures electric vehicles." # Text context ] ) print(f"Relevance Score: {result.value}") # 1.0 (relevant) or 0.0 (not relevant) ``` ### Example (Legacy API - Deprecated) Deprecated The legacy API is deprecated and will be removed in a future version. Please migrate to the Collections API shown above. ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import MultiModalRelevance sample = SingleTurnSample( user_input="What about the Tesla Model X?", response="Cats are cute.", retrieved_contexts=[ "custom_eval/multimodal/images/tesla.jpg" ] ) scorer = MultiModalRelevance() await scorer.single_turn_ascore(sample) ``` ### How It's Calculated Example **Question**: What about the Tesla Model X? **Context (visual)**: - An image of the Tesla Model X (custom_eval/multimodal/images/tesla.jpg) **High relevance answer**: The Tesla Model X is an electric SUV manufactured by Tesla. **Low relevance answer**: Cats are cute. Let's examine how relevance was calculated using the low relevance answer: - **Step 1:** Evaluate the generated response against the given contexts. - Response: "Cats are cute." - **Step 2:** Verify if the response aligns with the given context. - Response: No - **Step 3:** Use the result to determine the relevance score. [ \\text{Relevance} = 0 ] In this example, the response "Cats are cute" does not align with the image of the Tesla Model X, so the relevance score is 0. ### Supported Context Types The metric supports multiple types of context inputs: - **Text contexts**: Plain text strings - **Image URLs**: HTTP/HTTPS URLs pointing to images - **Local image paths**: File paths to local images (jpg, png, gif, webp, bmp) - **Base64 data URIs**: Inline base64-encoded images ### Requirements - A vision-capable LLM is required (e.g., `gpt-4o`, `gpt-4-vision-preview`, `claude-3-opus`, `gemini-pro-vision`) - For the Collections API, use `llm_factory` to create the LLM instance # Noise Sensitivity `NoiseSensitivity` measures how often a system makes errors by providing incorrect responses when utilizing either relevant or irrelevant retrieved documents. The score ranges from 0 to 1, with lower values indicating better performance. Noise sensitivity is computed using the `user_input`, `reference`, `response`, and the `retrieved_contexts`. To estimate noise sensitivity, each claim in the generated response is examined to determine whether it is correct based on the ground truth and whether it can be attributed to the relevant (or irrelevant) retrieved context. Ideally, all claims in the answer should be supported by the relevant retrieved context. [ \\text{noise sensitivity (relevant)} = {|\\text{Total number of incorrect claims in response}| \\over |\\text{Total number of claims in the response}|} ] ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import NoiseSensitivity # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = NoiseSensitivity(llm=llm) # Evaluate result = await scorer.ascore( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.", retrieved_contexts=[ "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.", "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.", "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.", "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc." ] ) print(f"Noise Sensitivity Score: {result.value}") ``` Output: ```text Noise Sensitivity Score: 0.3333333333333333 ``` To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`: ```python scorer = NoiseSensitivity(llm=llm, mode="irrelevant") result = await scorer.ascore( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.", retrieved_contexts=[ "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.", "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.", "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.", "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc." ] ) print(f"Noise Sensitivity (Irrelevant) Score: {result.value}") ``` Output: ```text Noise Sensitivity (Irrelevant) Score: 0.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India...", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company...", retrieved_contexts=[...] ) ``` ## How It’s Calculated Example Question: What is the Life Insurance Corporation of India (LIC) known for? Ground truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments. Relevant Retrieval: - The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. - LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. - As the largest institutional investor in India, LIC manages a substantial life fund, contributing to the financial stability of the country. Irrelevant Retrieval: - The Indian economy is one of the fastest-growing major economies in the world, thanks to the sectors like finance, technology, manufacturing etc. Let's examine how noise sensitivity in relevant context was calculated: - **Step 1:** Identify the relevant contexts from which the ground truth can be inferred. - Ground Truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments. - Contexts: - Context 1: The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. - Context 2: LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. - Context 3: As the largest institutional investor in India, LIC manages a substantial funds\`, contributing to the financial stability of the country. - **Step 2:** Verify if the claims in the generated answer can be inferred from the relevant context. - Answer: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country. - Contexts: - Context 1: The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. - Context 2: LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. - Context 3: As the largest institutional investor in India, LIC manages a substantial funds, contributing to the financial stability of the country. - **Step 3:** Identify any incorrect claims in the answer (i.e., answer statements that are not supported by the ground truth). - Ground Truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments. - Answer: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country. Explanation: The ground truth does not mention anything about LIC contributing to the financial stability of the country. Therefore, this statement in the answer is incorrect. Incorrect Statement: 1 Total claims: 3 - **Step 4:** Calculate noise sensitivity using the formula: [ \\text{noise sensitivity} = { \\text{1} \\over \\text{3} } = 0.333 ] This results in a noise sensitivity score of 0.333, indicating that one out of three claims in the answer was incorrect. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import NoiseSensitivity sample = SingleTurnSample( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.", retrieved_contexts=[ "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.", "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.", "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.", "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc." ] ) scorer = NoiseSensitivity(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.3333333333333333 ``` To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`: ```python scorer = NoiseSensitivity(mode="irrelevant") await scorer.single_turn_ascore(sample) ``` Credits: Noise sensitivity was introduced in [RAGChecker](https://github.com/amazon-science/RAGChecker/tree/main/ragchecker) # Nvidia Metrics ## Answer Accuracy **Answer Accuracy** measures the agreement between a model’s response and a reference ground truth for a given question. This is done via two distinct "LLM-as-a-Judge" prompts that each return a rating (0, 2, or 4). The metric converts these ratings into a [0,1] scale and then takes the average of the two scores from the judges. Higher scores indicate that the model’s answer closely matches the reference. - **0** β†’ The **response** is inaccurate or does not address the same question as the **reference**. - **2** β†’ The **response** partially align with the **reference**. - **4** β†’ The **response** exactly aligns with the **reference**. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import AnswerAccuracy # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = AnswerAccuracy(llm=llm) # Evaluate result = await scorer.ascore( user_input="When was Einstein born?", response="Albert Einstein was born in 1879.", reference="Albert Einstein was born in 1879." ) print(f"Answer Accuracy Score: {result.value}") ``` Output: ```text Answer Accuracy Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When was Einstein born?", response="Albert Einstein was born in 1879.", reference="Albert Einstein was born in 1879." ) ``` ### How It’s Calculated **Step 1:** The LLM generates ratings using two distinct templates to ensure robustness: - **Template 1:** The LLM compares the **response** with the **reference** and rates it on a scale of **0, 2, or 4**. - **Template 2:** The LLM evaluates the same question again, but this time the roles of the **response** and the **reference** are swapped. This dual-perspective approach guarantees a fair assessment of the answer's accuracy. **Step 2:** If both ratings are valid, the final score is average of score1 and score2; otherwise, it takes the valid one. **Example Calculation:** - **User Input:** "When was Einstein born?" - **Response:** "Albert Einstein was born in 1879." - **Reference:** "Albert Einstein was born in 1879." Assuming both templates return a rating of **4** (indicating an exact match), the conversion is as follows: - A rating of **4** corresponds to **1** on the [0,1] scale. - Averaging the two scores: (1 + 1) / 2 = **1**. Thus, the final **Answer Accuracy** score is **1**. ### Similar Ragas Metrics 1. [Answer Correctness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness/index.md): This metric gauges the accuracy of the generated answer compared to the ground truth by considering both semantic and factual similarity. 1. [Rubric Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-criteria-scoring): The Rubric-Based Criteria Scoring Metric allows evaluations based on user-defined rubrics, where each rubric outlines specific scoring criteria. The LLM assesses responses according to these customized descriptions, ensuring a consistent and objective evaluation process. ### Comparison of Metrics #### Answer Correctness vs. Answer Accuracy - **LLM Calls:** Answer Correctness requires three LLM calls (two for decomposing the response and reference into standalone statements and one for classifying them), while Answer Accuracy uses two independent LLM judgments. - **Token Usage:** Answer Correctness consumes lot more tokens due to its detailed breakdown and classification process. - **Explainability:** Answer Correctness offers high explainability by providing detailed insights into factual correctness and semantic similarity, whereas Answer Accuracy provides a straightforward raw score. - **Robust Evaluation:** Answer Accuracy ensures consistency through dual LLM evaluations, while Answer Correctness offers a holistic view by deeply assessing the quality of the response. #### Answer Accuracy vs. Rubric Score - **LLM Calls**: Answer Accuracy makes two calls (one per LLM judge), while Rubric Score requires only one. - **Token Usage**: Answer Accuracy is minimal since it outputs just a score, whereas Rubric Score generates reasoning, increasing token consumption. - **Explainability**: Answer Accuracy provides a raw score without justification, while Rubric Score offers reasoning with verdict. - **Efficiency**: Answer Accuracy is lightweight and works very well with smaller models. ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import AnswerAccuracy sample = SingleTurnSample( user_input="When was Einstein born?", response="Albert Einstein was born in 1879.", reference="Albert Einstein was born in 1879." ) scorer = AnswerAccuracy(llm=evaluator_llm) # evaluator_llm wrapped with ragas LLM Wrapper score = await scorer.single_turn_ascore(sample) print(score) ``` Output: ```text 1.0 ``` ## Context Relevance **Context Relevance** evaluates whether the **retrieved_contexts** (chunks or passages) are pertinent to the **user_input**. This is done via two independent "LLM-as-a-Judge" prompt calls that each rate the relevance on a scale of **0, 1, or 2**. The ratings are then converted to a [0,1] scale and averaged to produce the final score. Higher scores indicate that the contexts are more closely aligned with the user's query. - **0** β†’ The retrieved contexts are not relevant to the user's query at all. - **1** β†’ The contexts are partially relevant. - **2** β†’ The contexts are completely relevant. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextRelevance # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextRelevance(llm=llm) # Evaluate result = await scorer.ascore( user_input="When and Where Albert Einstein was born?", retrieved_contexts=[ "Albert Einstein was born March 14, 1879.", "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany.", ] ) print(f"Context Relevance Score: {result.value}") ``` Output: ```text Context Relevance Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When and Where Albert Einstein was born?", retrieved_contexts=[...] ) ``` ### Implementation Note **Difference from Original Paper:** The original Ragas paper defines Context Relevance using sentence-level extraction (CR = number of relevant sentences / total sentences), but the current implementation uses a more robust discrete judgment approach. Each LLM is asked to rate overall context relevance on a 0-2 scale, which is more efficient and less prone to sentence boundary errors. This was an intentional design decision to improve reliability and reduce computational overhead while maintaining the core evaluation objective. ### How It's Calculated **Step 1:** The LLM is prompted with two distinct templates (template_relevance1 and template_relevance2) to evaluate the relevance of the retrieved contexts concerning the user's query. Each prompt returns a relevance rating of **0**, **1**, or **2**. Using two independent evaluations provides robustness and helps mitigate individual LLM biases. **Step 2:** Each rating is normalized to a [0,1] scale by dividing by 2. If both ratings are valid, the final score is the average of these normalized values; if only one is valid, that score is used. **Example Calculation:** - **User Input:** "When and Where Albert Einstein was born?" - **Retrieved Contexts:** - "Albert Einstein was born March 14, 1879." - "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany." In this example, the two retrieved contexts together fully address the user's query by providing both the birthdate and location of Albert Einstein. Consequently, both prompts would rate the combined contexts as **2** (fully relevant). Normalizing each score yields **1.0** (2/2), and averaging the two results maintains the final Context Relevance score at **1**. ### Similar Ragas Metrics 1. [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/index.md): It measures the proportion of retrieved contexts that are relevant to answering a user's query. It is computed as the mean precision@k across all retrieved chunks, indicating how accurately the retrieval system ranks relevant information. 1. [Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/index.md): It quantifies the extent to which the relevant information is successfully retrieved. It is calculated as the ratio of the number of relevant claims (or contexts) found in the retrieved results to the total number of relevant claims in the reference, ensuring that important information is not missed. 1. [Rubric Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-criteria-scoring): The Rubric-Based Criteria Scoring Metric evaluates responses based on user-defined rubrics with customizable scoring criteria, ensuring consistent and objective assessments. The scoring scale is flexible to suit user needs. #### Context Precision and Context Recall vs. Context Relevance - **LLM Calls:** Context Precision and Context Recall each require one LLM call each, one verifies context usefulness to get reference (verdict "1" or "0") and one classifies each answer sentence as attributable (binary 'Yes' (1) or 'No' (0)) while Context Relevance uses two LLM calls for increased robustness. - **Token Usage:** Context Precision and Context Recall consume lot more tokens, whereas Context Relevance is more token-efficient. - **Explainability:** Context Precision and Context Recall offer high explainability with detailed reasoning, while Context Relevance provides a raw score without explanations. - **Robust Evaluation:** Context Relevance delivers a more robust evaluation through dual LLM judgments compared to the single-call approach of Context Precision and Context Recall. ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import ContextRelevance sample = SingleTurnSample( user_input="When and Where Albert Einstein was born?", retrieved_contexts=[ "Albert Einstein was born March 14, 1879.", "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany.", ] ) scorer = ContextRelevance(llm=evaluator_llm) score = await scorer.single_turn_ascore(sample) print(score) ``` Output: ```text 1.0 ``` ## Response Groundedness **Response Groundedness** measures how well a response is supported or "grounded" by the retrieved contexts. It assesses whether each claim in the response can be found, either wholly or partially, in the provided contexts. - **0** β†’ The response is **not** grounded in the context at all. - **1** β†’ The response is partially grounded. - **2** β†’ The response is fully grounded (every statement can be found or inferred from the retrieved context). ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ResponseGroundedness # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ResponseGroundedness(llm=llm) # Evaluate result = await scorer.ascore( response="Albert Einstein was born in 1879.", retrieved_contexts=[ "Albert Einstein was born March 14, 1879.", "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany.", ] ) print(f"Response Groundedness Score: {result.value}") ``` Output: ```text Response Groundedness Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( response="Albert Einstein was born in 1879.", retrieved_contexts=[...] ) ``` ### How It’s Calculated **Step 1:** The LLM is prompted with two distinct templates to evaluate the grounding of the response with respect to the retrieved contexts. Each prompt returns a grounding rating of **0**, **1**, or **2**. **Step 2:** Each rating is normalized to a [0,1] scale by dividing by 2 (i.e., 0 becomes 0.0, 1 becomes 0.5, and 2 becomes 1.0). If both ratings are valid, the final score is computed as the average of these normalized values; if only one is valid, that score is used. **Example Calculation:** - **Response:** "Albert Einstein was born in 1879." - **Retrieved Contexts:** - "Albert Einstein was born March 14, 1879." - "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany." In this example, the retrieved contexts provide both the birthdate and location of Albert Einstein. Since the response's claim is supported by the context (even though the date is partially provided), both prompts would likely rate the grounding as **2** (fully grounded). Normalizing a score of 2 gives **1.0** (2/2), and averaging the two normalized ratings maintains the final Response Groundedness score at **1**. ### Similar Ragas Metrics 1. [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/index.md): This metric measures how factually consistent a response is with the retrieved context, ensuring that every claim in the response is supported by the provided information. The Faithfulness score ranges from 0 to 1, with higher scores indicating better consistency. 1. [Rubric Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-criteria-scoring): This is a general-purpose metric that evaluates responses based on user-defined criteria and can be adapted to assess Answer Accuracy, Context Relevance or Response Groundedness by aligning the rubric with the requirements. ### Comparison of Metrics #### Faithfulness vs. Response Groundedness - **LLM Calls:** Faithfulness requires two calls for detailed claim breakdown and verdict, while Response Groundedness uses two independent LLM judgments. - **Token Usage:** Faithfulness consumes more tokens, whereas Response Groundedness is more token-efficient. - **Explainability:** Faithfulness provides transparent, reasoning for each claim, while Response Groundedness provides a raw score. - **Robust Evaluation:** Faithfulness incorporates user input for a comprehensive assessment, whereas Response Groundedness ensures consistency through dual LLM evaluations. ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import ResponseGroundedness sample = SingleTurnSample( response="Albert Einstein was born in 1879.", retrieved_contexts=[ "Albert Einstein was born March 14, 1879.", "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany.", ] ) scorer = ResponseGroundedness(llm=evaluator_llm) score = await scorer.single_turn_ascore(sample) print(score) ``` Output: ```text 1.0 ``` # Rubric-Based Evaluation Rubric-based evaluation metrics allow you to evaluate LLM responses using custom scoring criteria. Ragas provides two types of rubric metrics: 1. **DomainSpecificRubrics**: Uses the same rubric for all samples in a dataset (set at initialization) 1. **InstanceSpecificRubrics**: Each sample can have its own unique rubric (passed per evaluation) The rubric consists of descriptions for each score, typically ranging from 1 to 5. The response is evaluated and scored using an LLM based on the descriptions specified in the rubric. ## Domain-Specific Rubrics Use `DomainSpecificRubrics` when you want to apply the same evaluation criteria across all samples. This is useful for domain-wide evaluations where the scoring criteria remain constant. ### Example ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import DomainSpecificRubrics # Setup client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Reference-free evaluation (default) metric = DomainSpecificRubrics(llm=llm) result = await metric.ascore( user_input="What's the longest river in the world?", response="The longest river in the world is the Nile, stretching approximately 6,650 kilometers through northeastern Africa.", ) print(f"Score: {result.value}, Feedback: {result.reason}") # Reference-based evaluation metric_with_ref = DomainSpecificRubrics(llm=llm, with_reference=True) result = await metric_with_ref.ascore( user_input="What's the longest river in the world?", response="The longest river in the world is the Nile.", reference="The Nile is a major north-flowing river in northeastern Africa.", ) ``` ### Custom Rubrics You can define your own rubrics to customize the scoring criteria: ```python from ragas.metrics.collections import DomainSpecificRubrics my_custom_rubrics = { "score1_description": "Answer and ground truth are completely different", "score2_description": "Answer and ground truth are somewhat different", "score3_description": "Answer and ground truth are somewhat similar", "score4_description": "Answer and ground truth are similar", "score5_description": "Answer and ground truth are exactly the same", } metric = DomainSpecificRubrics(llm=llm, rubrics=my_custom_rubrics, with_reference=True) ``` ### With Retrieved Contexts The metric also supports evaluation with retrieved contexts: ```python result = await metric.ascore( user_input="What's the longest river in the world?", response="Based on the context, the Nile is the longest river.", retrieved_contexts=[ "Scientists debate whether the Amazon or the Nile is the longest river.", "The Nile River was central to Ancient Egyptians' wealth and power.", ], ) ``` ### Convenience Classes For clearer intent, use the convenience classes: ```python from ragas.metrics.collections import ( RubricsScoreWithoutReference, RubricsScoreWithReference, ) # Reference-free metric_no_ref = RubricsScoreWithoutReference(llm=llm) # Reference-based metric_with_ref = RubricsScoreWithReference(llm=llm) ``` ## Default Rubrics ### Reference-Free Rubrics (Default) | Score | Description | | ----- | ------------------------------------------------------------------------------------------ | | 1 | The response is entirely incorrect and fails to address any aspect of the user input. | | 2 | The response contains partial accuracy but includes major errors or significant omissions. | | 3 | The response is mostly accurate but lacks clarity, thoroughness, or minor details. | | 4 | The response is accurate and clear, with only minor omissions or slight inaccuracies. | | 5 | The response is completely accurate, clear, and thoroughly addresses the user input. | ### Reference-Based Rubrics | Score | Description | | ----- | ----------------------------------------------------------------------------------------- | | 1 | The response is entirely incorrect, irrelevant, or does not align with the reference. | | 2 | The response partially matches the reference but contains major errors or omissions. | | 3 | The response aligns with the reference overall but lacks sufficient detail or clarity. | | 4 | The response is mostly accurate, aligns closely with the reference with minor issues. | | 5 | The response is fully accurate, completely aligns with the reference, clear and detailed. | ______________________________________________________________________ ## Instance-Specific Rubrics Use `InstanceSpecificRubrics` when different samples require different evaluation criteria. This is useful when: - Different questions require different evaluation standards - You want to customize scoring based on specific task requirements - Evaluation criteria vary across your dataset ### Example ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import InstanceSpecificRubrics # Setup client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) metric = InstanceSpecificRubrics(llm=llm) # Each sample can have its own rubrics email_rubrics = { "score1_description": "The email is unprofessional or inappropriate", "score2_description": "The email lacks proper formatting or tone", "score3_description": "The email is acceptable but could be improved", "score4_description": "The email is professional with minor issues", "score5_description": "The email is highly professional and well-written", } result = await metric.ascore( user_input="Write a professional email declining a meeting invitation", response="Dear John, Thank you for the invitation...", rubrics=email_rubrics, ) print(f"Score: {result.value}, Feedback: {result.reason}") # Different rubrics for a different type of task code_rubrics = { "score1_description": "The code doesn't work or has critical bugs", "score2_description": "The code has significant issues or is poorly structured", "score3_description": "The code works but lacks optimization or best practices", "score4_description": "The code is good with minor improvements possible", "score5_description": "The code is excellent, efficient, and follows best practices", } result = await metric.ascore( user_input="Write a function to sort a list", response="def sort_list(arr): return sorted(arr)", rubrics=code_rubrics, ) ``` ### With Reference and Contexts ```python result = await metric.ascore( user_input="Explain the water cycle", response="The water cycle involves evaporation, condensation, and precipitation.", reference="The water cycle describes how water evaporates from surfaces, rises into the atmosphere, condenses into clouds, and falls as precipitation.", retrieved_contexts=["Water cycle information from encyclopedia..."], rubrics={ "score1_description": "Explanation is completely wrong", "score2_description": "Explanation has major inaccuracies", "score3_description": "Explanation is partially correct", "score4_description": "Explanation is mostly correct", "score5_description": "Explanation is comprehensive and accurate", }, ) ``` ______________________________________________________________________ ## Legacy API Deprecated The legacy API below is deprecated. Please use `ragas.metrics.collections.DomainSpecificRubrics` or `ragas.metrics.collections.InstanceSpecificRubrics` instead. ```python from ragas import evaluate from datasets import Dataset from ragas.metrics import rubrics_score_without_reference, rubrics_score_with_reference rows = { "question": [ "What's the longest river in the world?", ], "ground_truth": [ "The Nile is a major north-flowing river in northeastern Africa.", ], "answer": [ "The longest river in the world is the Nile, stretching approximately 6,650 kilometers (4,130 miles) through northeastern Africa.", ], "contexts": [ [ "Scientists debate whether the Amazon or the Nile is the longest river in the world.", "The Nile River was central to the Ancient Egyptians' rise to wealth and power.", ], ] } dataset = Dataset.from_dict(rows) result = evaluate( dataset, metrics=[ rubrics_score_without_reference, rubrics_score_with_reference ], ) ``` Custom rubrics with legacy API: ```python from ragas.metrics._domain_specific_rubrics import RubricsScore my_custom_rubrics = { "score1_description": "answer and ground truth are completely different", "score2_description": "answer and ground truth are somewhat different", "score3_description": "answer and ground truth are somewhat similar", "score4_description": "answer and ground truth are similar", "score5_description": "answer and ground truth are exactly the same", } rubrics_score = RubricsScore(rubrics=my_custom_rubrics) ``` ## Semantic Similarity The **Semantic Similarity** metric evaluates the semantic resemblance between a generated response and a reference (ground truth) answer. It ranges from 0 to 1, with higher scores indicating better alignment between the generated answer and the ground truth. This metric uses embeddings and cosine similarity to measure how semantically similar two answers are, which can offer valuable insights into the quality of the generated response. ### Example ```python from openai import AsyncOpenAI from ragas.embeddings import OpenAIEmbeddings from ragas.metrics.collections import SemanticSimilarity # Setup embeddings client = AsyncOpenAI() embeddings = OpenAIEmbeddings(model="text-embedding-3-small", client=client) # Create metric scorer = SemanticSimilarity(embeddings=embeddings) # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris. It has a height of 1000ft.", response="The Eiffel Tower is located in Paris." ) print(f"Semantic Similarity Score: {result.value}") ``` Output: ```text Semantic Similarity Score: 0.8151 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris. It has a height of 1000ft.", response="The Eiffel Tower is located in Paris." ) ``` ### How It's Calculated Example **Reference**: Albert Einstein's theory of relativity revolutionized our understanding of the universe. **High similarity response**: Einstein's groundbreaking theory of relativity transformed our comprehension of the cosmos. **Low similarity response**: Isaac Newton's laws of motion greatly influenced classical physics. Let's examine how semantic similarity was calculated for the high similarity response: - **Step 1:** Vectorize the reference answer using the specified embedding model. - **Step 2:** Vectorize the generated response using the same embedding model. - **Step 3:** Compute the cosine similarity between the two vectors. - **Step 4:** The cosine similarity value (0-1) is the final score. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import SemanticSimilarity from ragas.embeddings import LangchainEmbeddingsWrapper sample = SingleTurnSample( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. It has a height of 1000ft." ) scorer = SemanticSimilarity(embeddings=LangchainEmbeddingsWrapper(evaluator_embedding)) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.8151371879226978 ``` # SQL ## Execution based metrics In these metrics the resulting SQL is compared after executing the SQL query on the database and then comparing the `response` with the expected results. ### DataCompy Score `DataCompyScore` metric uses DataCompy, a python library that compares two pandas DataFrames. It provides a simple interface to compare two DataFrames and provides a detailed report of the differences. In this metric the `response` is executed on the database and the resulting data is compared with the expected data, i.e. `reference`. To enable comparison both `response` and `reference` should be in the form of a Comma-Separated Values as shown in the example. DataFrames can be compared across rows or columns. This can be configured using `mode` parameter. If mode is `row` then the comparison is done row-wise. If mode is `column` then the comparison is done column-wise. [ \\text{Precision } = {|\\text{Number of matching rows in response and reference}| \\over |\\text{Total number of rows in response}|} ] [ \\text{Recall } = {|\\text{Number of matching rows in response and reference}| \\over |\\text{Total number of rows in reference}|} ] By default, the mode is set to `row`, and metric is F1 score which is the harmonic mean of precision and recall. ```python from ragas.metrics.collections import DataCompyScore data1 = """acct_id,dollar_amt,name,float_fld,date_fld 10000001234,123.45,George Maharis,14530.1555,2017-01-01 10000001235,0.45,Michael Bluth,1,2017-01-01 10000001236,1345,George Bluth,,2017-01-01 10000001237,123456,Bob Loblaw,345.12,2017-01-01 10000001238,1.05,Lucille Bluth,,2017-01-01 10000001238,1.05,Loose Seal Bluth,,2017-01-01 """ data2 = """acct_id,dollar_amt,name,float_fld 10000001234,123.4,George Michael Bluth,14530.155 10000001235,0.45,Michael Bluth, 10000001236,1345,George Bluth,1 10000001237,123456,Robert Loblaw,345.12 10000001238,1.05,Loose Seal Bluth,111 """ metric = DataCompyScore() result = await metric.ascore(response=data1, reference=data2) print(f"F1 Score: {result.value}") print(f"Details: {result.reason}") ``` To change the mode to column-wise comparison, set the `mode` parameter to `column`. ```python metric = DataCompyScore(mode="columns", metric="recall") result = await metric.ascore(response=data1, reference=data2) ``` ______________________________________________________________________ ### DataCompyScore (Legacy) Deprecated `DataCompyScore` from `ragas.metrics` is deprecated and will be removed in a future version. Please use `DataCompyScore` from `ragas.metrics.collections` as shown above. The legacy `DataCompyScore` uses the `SingleTurnSample` schema: ```python from ragas.metrics import DataCompyScore from ragas.dataset_schema import SingleTurnSample data1 = """acct_id,dollar_amt,name,float_fld,date_fld 10000001234,123.45,George Maharis,14530.1555,2017-01-01 10000001235,0.45,Michael Bluth,1,2017-01-01 10000001236,1345,George Bluth,,2017-01-01 10000001237,123456,Bob Loblaw,345.12,2017-01-01 10000001238,1.05,Lucille Bluth,,2017-01-01 10000001238,1.05,Loose Seal Bluth,,2017-01-01 """ data2 = """acct_id,dollar_amt,name,float_fld 10000001234,123.4,George Michael Bluth,14530.155 10000001235,0.45,Michael Bluth, 10000001236,1345,George Bluth,1 10000001237,123456,Robert Loblaw,345.12 10000001238,1.05,Loose Seal Bluth,111 """ sample = SingleTurnSample(response=data1, reference=data2) scorer = DataCompyScore() await scorer.single_turn_ascore(sample) ``` To change the mode to column-wise comparison, set the `mode` parameter to `column`. ```python scorer = DataCompyScore(mode="column", metric="recall") ``` ## Non Execution based metrics Executing SQL queries on the database can be time-consuming and sometimes not feasible. In such cases, we can use non-execution based metrics to evaluate the SQL queries. These metrics compare the SQL queries directly without executing them on the database. ### SQL Semantic Equivalence `SQLSemanticEquivalence` is a metric that evaluates whether a generated SQL query is semantically equivalent to a reference query. The metric uses an LLM to analyze both queries in the context of the provided database schema and determine if they would produce the same results. This is a binary metric: - **1.0**: The SQL queries are semantically equivalent - **0.0**: The SQL queries are not equivalent The metric considers the database schema context to make accurate equivalence judgments, accounting for syntactic differences that don't affect semantics (e.g., `active = 1` vs `active = true`). ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import SQLSemanticEquivalence # Initialize the LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create the metric metric = SQLSemanticEquivalence(llm=llm) # Evaluate SQL equivalence result = await metric.ascore( response=""" SELECT p.product_name, SUM(oi.quantity) AS total_quantity FROM order_items oi JOIN products p ON oi.product_id = p.product_id GROUP BY p.product_name; """, reference=""" SELECT products.product_name, SUM(order_items.quantity) AS total_quantity FROM order_items INNER JOIN products ON order_items.product_id = products.product_id GROUP BY products.product_name; """, reference_contexts=[ """ Table order_items: - order_item_id: INT - order_id: INT - product_id: INT - quantity: INT """, """ Table products: - product_id: INT - product_name: VARCHAR - price: DECIMAL """ ] ) print(f"Equivalent: {result.value == 1.0}") print(f"Explanation: {result.reason}") ``` The result includes explanations of both queries and the reasoning for the equivalence determination. ______________________________________________________________________ ### LLMSQLEquivalence (Legacy) Deprecated `LLMSQLEquivalence` is deprecated and will be removed in a future version. Please use `SQLSemanticEquivalence` from `ragas.metrics.collections` as shown above. `LLMSQLEquivalence` is the legacy metric for SQL semantic equivalence evaluation. It uses the `SingleTurnSample` schema and requires setting the LLM separately. ```python from ragas.metrics import LLMSQLEquivalence from ragas.dataset_schema import SingleTurnSample sample = SingleTurnSample( response=""" SELECT p.product_name, SUM(oi.quantity) AS total_quantity FROM order_items oi JOIN products p ON oi.product_id = p.product_id GROUP BY p.product_name; """, reference=""" SELECT p.product_name, COUNT(oi.quantity) AS total_quantity FROM order_items oi JOIN products p ON oi.product_id = p.product_id GROUP BY p.product_name; """, reference_contexts=[ """ Table order_items: - order_item_id: INT - order_id: INT - product_id: INT - quantity: INT """, """ Table products: - product_id: INT - product_name: VARCHAR - price: DECIMAL """ ] ) scorer = LLMSQLEquivalence() scorer.llm = openai_model await scorer.single_turn_ascore(sample) ``` # Tasks Metrics ## Summarization Score The **Summarization Score** metric measures how well a summary (`response`) captures the important information from the `reference_contexts`. The intuition behind this metric is that a good summary should contain all the important information present in the context. We first extract a set of important keyphrases from the context. These keyphrases are then used to generate a set of questions. The answers to these questions are always `yes(1)` for the context. We then ask these questions to the summary and calculate the summarization score as the ratio of correctly answered questions to the total number of questions. We compute the question-answer score using the answers, which is a list of `1`s and `0`s. The question-answer score is then calculated as the ratio of correctly answered questions(answer = `1`) to the total number of questions. [ \\text{QA score} = \\frac{|\\text{correctly answered questions}|}{|\\text{total questions}|} ] We also introduce an option to penalize larger summaries by proving a conciseness score. If this option is enabled, the final score is calculated as the weighted average of the summarization score and the conciseness score. This conciseness scores ensures that summaries that are just copies of the text do not get a high score, because they will obviously answer all questions correctly. [ \\text{conciseness score} = 1 - \\frac{\\min(\\text{length of summary}, \\text{length of context})}{\\text{length of context} + \\text{1e-10}} ] We also provide a coefficient `coeff`(default value 0.5) to control the weightage of the scores. The final summarization score is then calculated as: [ \\text{Summarization Score} = \\text{QA score}\*\\text{(1-coeff)} + \\ \\text{conciseness score}\*\\text{coeff} ] ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import SummaryScore # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = SummaryScore(llm=llm) # Evaluate result = await scorer.ascore( reference_contexts=[ "A company is launching a new product, a smartphone app designed to help users track their fitness goals. The app allows users to set daily exercise targets, log their meals, and track their water intake. It also provides personalized workout recommendations and sends motivational reminders throughout the day." ], response="A company is launching a fitness tracking app that helps users set exercise goals, log meals, and track water intake, with personalized workout suggestions and motivational reminders." ) print(f"Summary Score: {result.value}") ``` Output: ```text Summary Score: 0.6423387096775146 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference_contexts=[...], response="..." ) ``` ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import SummarizationScore sample = SingleTurnSample( response="A company is launching a fitness tracking app that helps users set exercise goals, log meals, and track water intake, with personalized workout suggestions and motivational reminders.", reference_contexts=[ "A company is launching a new product, a smartphone app designed to help users track their fitness goals. The app allows users to set daily exercise targets, log their meals, and track their water intake. It also provides personalized workout recommendations and sends motivational reminders throughout the day." ] ) scorer = SummarizationScore(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.6423387096775146 ``` # Traditional NLP Metrics ## Non LLM String Similarity `NonLLMStringSimilarity` metric measures the similarity between the reference and the response using traditional string distance measures such as Levenshtein, Hamming, and Jaro. This metric is useful for evaluating the similarity of `response` to the `reference` text without relying on large language models (LLMs). The metric returns a score between 0 and 1, where 1 indicates a perfect match between the response and the reference. This is a non LLM based metric. ### Example ```python from ragas.metrics.collections import NonLLMStringSimilarity, DistanceMeasure # Create metric (no LLM/embeddings needed) scorer = NonLLMStringSimilarity(distance_measure=DistanceMeasure.LEVENSHTEIN) # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) print(f"NonLLM String Similarity Score: {result.value}") ``` Output: ```text NonLLM String Similarity Score: 0.8918918918918919 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) ``` ### Configuration You can choose from available string distance measures from `DistanceMeasure`. Here is an example of using Hamming distance: ```python scorer = NonLLMStringSimilarity(distance_measure=DistanceMeasure.HAMMING) ``` Available distance measures include: - `DistanceMeasure.LEVENSHTEIN` (default) - `DistanceMeasure.HAMMING` - `DistanceMeasure.JARO` - `DistanceMeasure.JARO_WINKLER` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics._string import NonLLMStringSimilarity sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="The Eiffel Tower is located in Paris." ) scorer = NonLLMStringSimilarity() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.8918918918918919 ``` #### Example with Different Distance Measure ```python from ragas.metrics._string import NonLLMStringSimilarity, DistanceMeasure scorer = NonLLMStringSimilarity(distance_measure=DistanceMeasure.HAMMING) ``` ## BLEU Score The `BleuScore` metric is used to evaluate the quality of `response` by comparing it with `reference`. It measures the similarity between the response and the reference based on n-gram precision and brevity penalty. BLEU score was originally designed to evaluate machine translation systems, but it is also used in other natural language processing tasks. BLEU score ranges from 0 to 1, where 1 indicates a perfect match between the response and the reference. This is a non-LLM based metric. ### Example ```python from ragas.metrics.collections import BleuScore # Create metric scorer = BleuScore() # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) print(f"BLEU Score: {result.value}") ``` Output: ```text BLEU Score: 0.7071067811865478 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) ``` ### Configuration You can pass additional arguments to the underlying `sacrebleu.corpus_bleu` function using the `kwargs` parameter: ```python scorer = BleuScore(kwargs={"smooth_method": "exp"}) ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import BleuScore sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="The Eiffel Tower is located in Paris." ) scorer = BleuScore() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.7071067811865478 ``` ## ROUGE Score The `RougeScore` score is a set of metrics used to evaluate the quality of natural language generations. It measures the overlap between the generated `response` and the `reference` text based on n-gram recall, precision, and F1 score. ROUGE score ranges from 0 to 1, where 1 indicates a perfect match between the response and the reference. This is a non LLM based metric. ### Example ```python from ragas.metrics.collections import RougeScore # Create metric (no LLM/embeddings needed) scorer = RougeScore(rouge_type="rougeL", mode="fmeasure") # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) print(f"ROUGE Score: {result.value}") ``` Output: ```text ROUGE Score: 0.8571428571428571 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) ``` ### Configuration You can change the `rouge_type` to `rouge1` or `rougeL` to calculate the ROUGE score based on unigrams or longest common subsequence respectively. ```python scorer = RougeScore(rouge_type="rouge1") ``` You can change the `mode` to `precision`, `recall`, or `fmeasure` to calculate the ROUGE score based on precision, recall, or F1 score respectively. ```python scorer = RougeScore(mode="recall") ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import RougeScore sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="The Eiffel Tower is located in Paris." ) scorer = RougeScore() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.8571428571428571 ``` ## Exact Match The `ExactMatch` metric checks if the response is exactly the same as the reference text. It is useful in scenarios where you need to ensure that the generated response matches the expected output word-for-word. For example, arguments in tool calls, etc. The metric returns 1 if the response is an exact match with the reference, and 0 otherwise. ### Example ```python from ragas.metrics.collections import ExactMatch # Create metric (no LLM/embeddings needed) scorer = ExactMatch() # Evaluate result = await scorer.ascore( reference="Paris", response="India" ) print(f"Exact Match Score: {result.value}") ``` Output: ```text Exact Match Score: 0.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="Paris", response="India" ) ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import ExactMatch sample = SingleTurnSample( response="India", reference="Paris" ) scorer = ExactMatch() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.0 ``` ## String Presence The `StringPresence` metric checks if the response contains the reference text. It is useful in scenarios where you need to ensure that the generated response contains certain keywords or phrases. The metric returns 1 if the response contains the reference, and 0 otherwise. ### Example ```python from ragas.metrics.collections import StringPresence # Create metric (no LLM/embeddings needed) scorer = StringPresence() # Evaluate result = await scorer.ascore( reference="Eiffel Tower", response="The Eiffel Tower is located in India." ) print(f"String Presence Score: {result.value}") ``` Output: ```text String Presence Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="Eiffel Tower", response="The Eiffel Tower is located in India." ) ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import StringPresence sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="Eiffel Tower" ) scorer = StringPresence() await scorer.single_turn_ascore(sample) ``` Output: ```text 1.0 ``` ## CHRF Score The `CHRFScore` metric evaluates the similarity between a `response` and a `reference` using **character n-gram F-score**. Unlike BLEU, which emphasizes precision, CHRF accounts for both **precision and recall**, making it more suitable for: - Morphologically rich languages - Responses with paraphrasing or flexible wording CHRF scores range from 0 to 1, where 1 indicates a perfect match between the generated response and the reference. This is a non-LLM-based metric, relying entirely on deterministic comparisons. ### Example ```python from ragas.metrics.collections import CHRFScore # Create metric (no LLM/embeddings needed) scorer = CHRFScore() # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) print(f"CHRF Score: {result.value}") ``` Output: ```text CHRF Score: 0.8048 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) ``` ### Configuration You can pass additional arguments to the underlying `sacrebleu.corpus_chrf` function using the `kwargs` parameter: ```python # Customize character and word order scorer = CHRFScore(kwargs={"char_order": 4, "word_order": 2}) # Customize beta (recall weight) scorer = CHRFScore(kwargs={"beta": 3}) ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import ChrfScore sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="The Eiffel Tower is located in Paris." ) scorer = ChrfScore() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.8048 ``` # Overview of Metrics ## Why Metrics Matter You can't improve what you don't measure. Metrics are the feedback loop that makes iteration possible. In AI systems, progress depends on running many experimentsβ€”each a hypothesis about how to improve performance. But without a clear, reliable metric, you can't tell the difference between a successful experiment (a positive delta between the new score and the old one) and a failed one. Metrics give you a compass. They let you quantify improvement, detect regressions, and align optimization efforts with user impact and business value. A metric is a quantitative measure used to evaluate the performance of a AI application. Metrics help in assessing how well the application and individual components that makes up application is performing relative to the given test data. They provide a numerical basis for comparison, optimization, and decision-making throughout the application development and deployment process. Metrics are crucial for: 1. **Component Selection**: Metrics can be used to compare different components of the AI application like LLM, Retriever, Agent configuration, etc with your own data and select the best one from different options. 1. **Error Diagnosis and Debugging**: Metrics help identify which part of the application is causing errors or suboptimal performance, making it easier to debug and refine. 1. **Continuous Monitoring and Maintenance**: Metrics enable the tracking of an AI application's performance over time, helping to detect and respond to issues such as data drift, model degradation, or changing user requirements. ## Types of Metrics in AI Applications ### 1. End-to-End Metrics End-to-end metrics evaluate the overall system performance from the user's perspective, treating the AI application as a black box. These metrics quantify key outcomes users care deeply about, based solely on the system's final outputs. Examples: - Answer correctness: Measures if the provided answers from a Retrieval-Augmented Generation (RAG) system are accurate. - Citation accuracy: Evaluates whether the references cited by the RAG system are correctly identified and relevant. Optimizing end-to-end metrics ensures tangible improvements aligned directly with user expectations. ### 2. Component-Level Metrics Component-level metrics assess the individual parts of an AI system independently. These metrics are immediately actionable and facilitate targeted improvements but do not necessarily correlate directly with end-user satisfaction. Example: - Retrieval accuracy: Measures how effectively a RAG system retrieves relevant information. A low retrieval accuracy (e.g., 50%) signals that improving this component can enhance overall system performance. However, improving a component alone doesn't guarantee better end-to-end outcomes. ### 3. Business Metrics Business metrics align AI system performance with organizational objectives and quantify tangible business outcomes. These metrics are typically lagging indicators, calculated after a deployment period (days/weeks/months). Example: - Ticket deflection rate: Measures the percentage reduction of support tickets due to the deployment of an AI assistant. ## Types of Metrics in Ragas Metrics Mind map **Metrics can be classified into two categories based on the mechanism used underneath the hood**: **LLM-based metrics**: These metrics use LLM underneath to do the evaluation. There might be one or more LLM calls that are performed to arrive at the score or result. These metrics can be somewhat non-deterministic as the LLM might not always return the same result for the same input. On the other hand, these metrics has shown to be more accurate and closer to human evaluation. All LLM based metrics in ragas are inherited from `MetricWithLLM` class. These metrics expects a LLM object to be set before scoring. ```python from ragas.metrics import FactualCorrectness scorer = FactualCorrectness(llm=evaluation_llm) ``` Each LLM based metrics also will have prompts associated with it written using [Prompt Object](https://docs.ragas.io/en/stable/concepts/components/prompt/index.md). You can customize these prompts to suit your domain and use-case. Learn more in the [Modifying Prompts in Metrics](https://docs.ragas.io/en/stable/howtos/customizations/metrics/modifying-prompts-metrics/index.md) guide. **Non-LLM-based metrics**: These metrics do not use LLM underneath to do the evaluation. These metrics are deterministic and can be used to evaluate the performance of the AI application without using LLM. These metrics rely on traditional methods to evaluate the performance of the AI application, such as string similarity, BLEU score, etc. Due to the same, these metrics are known to have a lower correlation with human evaluation. All Non-LLM-based metrics in ragas are inherited from `Metric` class. **Metrics can be broadly classified into two categories based on the type of data they evaluate**: **Single turn metrics**: These metrics evaluate the performance of the AI application based on a single turn of interaction between the user and the AI. All metrics in ragas that supports single turn evaluation are inherited from SingleTurnMetric class and scored using `single_turn_ascore` method. It also expects a Single Turn Sample object as input. ```python from ragas.metrics import FactualCorrectness scorer = FactualCorrectness() await scorer.single_turn_ascore(sample) ``` **Multi-turn metrics**: These metrics evaluate the performance of the AI application based on multiple turns of interaction between the user and the AI. All metrics in ragas that supports multi turn evaluation are inherited from MultiTurnMetric class and scored using `multi_turn_ascore` method. It also expects a Multi Turn Sample object as input. ```python from ragas.metrics import AgentGoalAccuracy from ragas import MultiTurnSample scorer = AgentGoalAccuracy() await scorer.multi_turn_ascore(sample) ``` ### Output Types In Ragas, we categorize metrics based on the type of output they produce. This classification helps clarify how each metric behaves and how its results can be interpreted or aggregated. The three types are: #### 1. Discrete Metrics These return a single value from a predefined list of categorical classes. There is no implicit ordering among the classes. Common use cases include classifying outputs into categories such as pass/fail or good/okay/bad. Discrete metrics accept custom prompts directly, making them ideal for quick custom evaluations. Example: ```python from ragas.metrics import discrete_metric @discrete_metric(name="response_quality", allowed_values=["pass", "fail"]) def my_metric(predicted: str, expected: str) -> str: return "pass" if predicted.lower() == expected.lower() else "fail" ``` For modifying prompts in existing collection metrics (like Faithfulness, FactualCorrectness), see [Modifying prompts in metrics](https://docs.ragas.io/en/stable/howtos/customizations/metrics/modifying-prompts-metrics/index.md). #### 2. Numeric Metrics These return an integer or float value within a specified range. Numeric metrics support aggregation functions such as mean, sum, or mode, making them useful for statistical analysis. ```python from ragas.metrics import numeric_metric @numeric_metric(name="response_accuracy", allowed_values=(0, 1)) def my_metric(predicted: float, expected: float) -> float: return abs(predicted - expected) / max(expected, 1e-5) my_metric.score(predicted=0.8, expected=1.0) # Returns a float value ``` #### 3. Ranking Metrics These evaluate multiple outputs at once and return a ranked list based on a defined criterion. They are useful when the goal is to compare multiple outputs from the same pipeline relative to one another. ```python from ragas.metrics import ranking_metric @ranking_metric(name="response_ranking", allowed_values=[0,1]) def my_metric(responses: list) -> list: response_lengths = [len(response) for response in responses] sorted_indices = sorted(range(len(response_lengths)), key=lambda i: response_lengths[i]) return sorted_indices my_metric.score(responses=["short", "a bit longer", "the longest response"]) # Returns a ranked list of indices ``` ## Metric Design Principles Designing effective metrics for AI applications requires following to a set of core principles to ensure their reliability, interpretability, and relevance. Here are five key principles we follow in ragas when designing metrics: **1. Single-Aspect Focus** A single metric should target only one specific aspect of the AI application's performance. This ensures that the metric is both interpretable and actionable, providing clear insights into what is being measured. **2. Intuitive and Interpretable** Metrics should be designed to be easy to understand and interpret. Clear and intuitive metrics make it simpler to communicate results and draw meaningful conclusions. **3. Effective Prompt Flows** When developing metrics using large language models (LLMs), use intelligent prompt flows that align closely with human evaluation. Decomposing complex tasks into smaller sub-tasks with specific prompts can improve the accuracy and relevance of the metric. **4. Robustness** Ensure that LLM-based metrics include sufficient few-shot examples that reflect the desired outcomes. This enhances the robustness of the metric by providing context and guidance for the LLM to follow. **5.Consistent Scoring Ranges** It is crucial to normalize metric score values or ensure they fall within a specific range, such as 0 to 1. This facilitates comparison between different metrics and helps maintain consistency and interpretability across the evaluation framework. These principles serve as a foundation for creating metrics that are not only effective but also practical and meaningful in evaluating AI applications. ## Choosing the Right Metrics for Your Application ### 1. Prioritize End-to-End Metrics Focus first on metrics reflecting overall user satisfaction. While many aspects influence user satisfactionβ€”such as factual correctness, response tone, and explanation depthβ€”concentrate initially on the few dimensions delivering maximum user value (e.g., answer and citation accuracy in a RAG-based assistant). ### 2. Ensure Interpretability Design metrics clear enough for the entire team to interpret and reason about. For example: - Execution accuracy in a text-to-SQL system: Does the SQL query generated return precisely the same dataset as the ground truth query crafted by domain experts? ### 3. Emphasize Objective Over Subjective Metrics Prioritize metrics with objective criteria, minimizing subjective judgment. Assess objectivity by independently labeling samples across team members and measuring agreement levels. A high inter-rater agreement (β‰₯80%) indicates greater objectivity. ### 4. Few Strong Signals over Many Weak Signals Avoid a proliferation of metrics that provide weak signals and impede clear decision-making. Instead, select fewer metrics offering strong, reliable signals. For instance: - In a conversational AI, using a single metric such as goal accuracy (whether the user's objective for interacting with the AI was met) provides strong proxy for the performance of the system than multiple weak proxies like coherence or helpfulness. # Testset Generation Curating a high quality test dataset is crucial for evaluating the performance of your AI application. ## Characteristics of an Ideal Test Dataset - Contains high quality data samples - Covers wide variety of scenarios as observed in real world. - Contains enough number of samples to derive statistically significant conclusions. - Continually updated to prevent data drift Curating such a dataset manually can be time-consuming and expensive. Ragas provides a set of tools to generate synthetic test datasets for evaluating your AI applications. - [**RAG** for evaluating retrieval augmented generation pipelines](https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/index.md) - [**Agents or Tool use** for evaluating agent workflows](https://docs.ragas.io/en/stable/concepts/test_data_generation/agents/index.md) # Testset Generation for Agents or Tool use cases Evaluating agentic or tool use workflows can be challenging as it involves multiple steps and interactions. It can be especially hard to curate a test suite that covers all possible scenarios and edge cases. We are working on a set of tools to generate synthetic test data for evaluating agent workflows. [Talk to us](https://cal.com/team/vibrantlabs/office-hours) to work together on this and discover what's coming for upcoming releases. # Testset Generation for RAG In RAG application, when a user interacts through your application to a set of documents, there can be different patterns of queries that the system can encounter. Let's first understand the different types of queries that can be encountered in RAG application. ## Query types in RAG ``` graph TD A[Queries] --> B[Single-Hop Query] A --> C[Multi-Hop Query] B --> D1[Specific Query] B --> E1[Abstract Query] C --> F1[Specific Query] C --> G1[Abstract Query] ``` ### Single-Hop Query A single-hop query is a straightforward question that requires retrieving information from a single document or source to provide a relevant answer. It involves only one step to arrive at the answer. **Example (Specific Query):** - β€œWhat year did Albert Einstein publish the theory of relativity?” This is a specific, fact-based question that can be answered with a single retrieval from a document containing that information. **Example (Abstract Query):** - β€œHow did Einstein’s theory change our understanding of time and space?” While this query still refers to a single concept (the theory of relativity), it requires a more abstract or interpretive explanation from the source material. ### Multi-Hop Query A multi-hop query involves multiple steps of reasoning, requiring information from two or more sources. The system must retrieve information from various documents and connect the dots to generate an accurate answer. **Example (Specific Query):** - β€œWhich scientist influenced Einstein’s work on relativity, and what theory did they propose?” This requires the system to retrieve information about both the scientist who influenced Einstein and the specific theory, potentially from two different sources. **Example (Abstract Query):** - β€œHow have scientific theories on relativity evolved since Einstein’s original publication?” This abstract query requires the retrieval of multiple pieces of information over time and across different sources to form a broad, interpretive response about the evolution of the theory. ### Specific vs. Abstract Queries in a RAG - **Specific Query:** Focuses on clear, fact-based retrieval. The goal in RAG is to retrieve highly relevant information from one or more documents that directly address the specific question. - **Abstract Query:** Requires a broader, more interpretive response. In RAG, abstract queries challenge the retrieval system to pull from documents that contain higher-level reasoning, explanations, or opinions, rather than simple facts. In both single-hop and multi-hop cases, the distinction between specific and abstract queries shapes the retrieval and generation process by determining whether the focus is on precision (specific) or on synthesizing broader ideas (abstract). Different types of queries requires different contexts to be synthesized. To solve this problem, Ragas uses a Knowledge Graph based approach to Test set Generation. ## Knowledge Graph Creation Given that we want to manufacture different types of queries from the given set of documents, our major challenge is to identify the right set of chunks or documents to enable LLMs to create the queries. To solve this problem, Ragas uses a Knowledge Graph based approach to Test set Generation. knowledge graph creation The knowledge graph is created by using the following components: ### Document Splitter The documents are chunked to form hierarchical nodes. The chunking can be done by using different splitters. For example, in the case of financial documents, the chunking can be done by using the splitter that splits the document based on the sections like Income Statement, Balance Sheet, Cash Flow Statement etc. You can write your own custom splitters to split the document based on the sections that are relevant to your domain. #### Example ```python from ragas.testset.graph import Node sample_nodes = [Node( properties={"page_content": "Einstein's theory of relativity revolutionized our understanding of space and time. It introduced the concept that time is not absolute but can change depending on the observer's frame of reference."} ),Node( properties={"page_content": "Time dilation occurs when an object moves close to the speed of light, causing time to pass slower relative to a stationary observer. This phenomenon is a key prediction of Einstein's special theory of relativity."} )] sample_nodes ``` Output: ```bash [Node(id: 4f6b94, type: , properties: ['page_content']), Node(id: 952361, type: , properties: ['page_content'])] ``` ``` graph TD A[Node: 4f6b94] -.-> |Properties| A1[page_content] B[Node: 952361] -.-> |Properties| B1[page_content] ``` ### Extractors Different extractors are used to extract information from each node that can be used to establish the relationship between the nodes. For example, in the case of financial documents, the extractor that can be used are entity extractor to extract the entities like Company Name, Keyphrase extractor to extract important key phrases present in each node, etc. You can write your own custom extractors to extract the information that is relevant to your domain. Extractors can be LLM based which are inherited from `LLMBasedExtractor` or rule based which are inherited from `Extractor`. #### Example Let's say we have a sample node from the knowledge graph. We can use the `NERExtractor` to extract the named entities from the node. ```python from ragas.testset.transforms.extractors import NERExtractor extractor = NERExtractor() output = [await extractor.extract(node) for node in sample_nodes] output[0] ``` Returns a tuple of the type of the extractor and the extracted information. ```bash ('entities', ['Einstein', 'theory of relativity', 'space', 'time', "observer's frame of reference"]) ``` Let's add the extracted information to the node. ```python _ = [node.properties.update({key:val}) for (key,val), node in zip(output, sample_nodes)] sample_nodes[0].properties ``` Output: ```bash {'page_content': "Einstein's theory of relativity revolutionized our understanding of space and time. It introduced the concept that time is not absolute but can change depending on the observer's frame of reference.", 'entities': ['Einstein', 'theory of relativity', 'space', 'time', 'observer']} ``` ``` graph TD A[Node: 4f6b94] -.-> |Properties| A1[page_content] A -.-> |Properties| A2[entities] B[Node: 952361] -.-> |Properties| B1[page_content] B -.-> |Properties| B2[entities] ``` ### Relationship builder The extracted information is used to establish the relationship between the nodes. For example, in the case of financial documents, the relationship can be established between the nodes based on the entities present in the nodes. You can write your own custom relationship builder to establish the relationship between the nodes based on the information that is relevant to your domain. #### Example ```python from ragas.testset.graph import KnowledgeGraph from ragas.testset.transforms.relationship_builders.traditional import JaccardSimilarityBuilder kg = KnowledgeGraph(nodes=sample_nodes) rel_builder = JaccardSimilarityBuilder(property_name="entities", key_name="PER", new_property_name="entity_jaccard_similarity") relationships = await rel_builder.transform(kg) relationships ``` Output: ```bash [Relationship(Node(id: 4f6b94) <-> Node(id: 952361), type: jaccard_similarity, properties: ['entity_jaccard_similarity'])] ``` Since both the nodes have the same entity "Einstein", the relationship is established between the nodes based on the entity similarity. ``` graph TD A[Node: 4f6b94] -.-> |Properties| A1[page_content] A -.-> |Properties| A2[entities] B[Node: 952361] -.-> |Properties| B1[page_content] B -.-> |Properties| B2[entities] A ===|entity_jaccard_similarity| B ``` Now let's understand how to build the knowledge graph using the above components with a `transform`, that would make your job easier. ### Transforms All of the components used to build the knowledge graph can be combined into a single `transform` that can be applied to the knowledge graph to build the knowledge graph. Transforms is made of up of a list of components that are applied to the knowledge graph in a sequence. It can also handle parallel processing of the components. The `apply_transforms` method is used to apply the transforms to the knowledge graph. #### Example Let's build the above knowledge graph using the above components with a `transform`. ```python from ragas.testset.transforms import apply_transforms transforms = [ extractor, rel_builder ] apply_transforms(kg,transforms) ``` To apply few of the components in parallel, you can wrap them in `Parallel` class. ```python from ragas.testset.transforms import KeyphraseExtractor, NERExtractor from ragas.testset.transforms import apply_transforms, Parallel tranforms = [ Parallel( KeyphraseExtractor(), NERExtractor() ), rel_builder ] apply_transforms(kg,transforms) ``` Once the knowledge graph is created, the different types of queries can be generated by traversing the graph. For example, to generate the query β€œCompare the revenue growth of Company X and Company Y from FY2020 through FY2023”, the graph can be traversed to find the nodes that contain the information about the revenue growth of Company X and Company Y from FY2020 through FY2023. ## Scenario Generation Now we have the knowledge graph that can be used to manufacture the right context to generate any type of query. When a population of users interact with RAG system, they may formulate the queries in various ways depending upon their persona (eg, Senior Engineer, Junior Engineer, etc), Query length (Short, Long, etc), Query style (Formal, Informal, etc). To generate the queries that cover all these scenarios, Ragas uses a Scenario based approach to Test set Generation. Each `Scenario` in Test set Generation is a combination of following parameters. - Nodes : The nodes that are used to generate the query - Query Length : The length of the desired query, it can be short, medium or long, etc. - Query Style : The style of the query, it can be web search, chat, etc. - Persona : The persona of the user, it can be Senior Engineer, Junior Engineer, etc. (Coming soon) Scenario in Test Generation ### Query Synthesizer The `QuerySynthesizer` is responsible for generating different scenarios for a single query type. The `generate_scenarios` method is used to generate the scenarios for a single query type. The `generate_sample` method is used to generate the query and reference answer for a single scenario. Let's understand this with an example. #### Example In the previous example, we have created a knowledge graph that contains two nodes that are related to each other based on the entity similarity. Now imagine that you have 20 such pairs of nodes in your KG that are related to each other based on the entity similarity. Imagine your goal is to create 50 different queries where each query is about some abstract question comparing two entities. We first have to query the KG to get the pairs of nodes that are related to each other based on the entity similarity. Then we have to generate the scenarios for each pair of nodes until we get 50 different scenarios. This logic is implemented in `generate_scenarios` method. ```python from dataclasses import dataclass from ragas.testset.synthesizers.base_query import QuerySynthesizer @dataclass class EntityQuerySynthesizer(QuerySynthesizer): async def _generate_scenarios( self, n, knowledge_graph, callbacks): """ logic to query nodes with entity logic describing how to combine nodes,styles,length,persona to form n scenarios """ return scenarios async def _generate_sample( self, scenario, callbacks ): """ logic on how to use tranform each scenario to EvalSample (Query,Context,Reference) you may create singleturn or multiturn sample """ return SingleTurnSample(user_input=query, reference_contexs=contexts, reference=reference) ``` # Metrics # Overview of Metrics ## Why Metrics Matter You can't improve what you don't measure. Metrics are the feedback loop that makes iteration possible. In AI systems, progress depends on running many experimentsβ€”each a hypothesis about how to improve performance. But without a clear, reliable metric, you can't tell the difference between a successful experiment (a positive delta between the new score and the old one) and a failed one. Metrics give you a compass. They let you quantify improvement, detect regressions, and align optimization efforts with user impact and business value. A metric is a quantitative measure used to evaluate the performance of a AI application. Metrics help in assessing how well the application and individual components that makes up application is performing relative to the given test data. They provide a numerical basis for comparison, optimization, and decision-making throughout the application development and deployment process. Metrics are crucial for: 1. **Component Selection**: Metrics can be used to compare different components of the AI application like LLM, Retriever, Agent configuration, etc with your own data and select the best one from different options. 1. **Error Diagnosis and Debugging**: Metrics help identify which part of the application is causing errors or suboptimal performance, making it easier to debug and refine. 1. **Continuous Monitoring and Maintenance**: Metrics enable the tracking of an AI application's performance over time, helping to detect and respond to issues such as data drift, model degradation, or changing user requirements. ## Types of Metrics in AI Applications ### 1. End-to-End Metrics End-to-end metrics evaluate the overall system performance from the user's perspective, treating the AI application as a black box. These metrics quantify key outcomes users care deeply about, based solely on the system's final outputs. Examples: - Answer correctness: Measures if the provided answers from a Retrieval-Augmented Generation (RAG) system are accurate. - Citation accuracy: Evaluates whether the references cited by the RAG system are correctly identified and relevant. Optimizing end-to-end metrics ensures tangible improvements aligned directly with user expectations. ### 2. Component-Level Metrics Component-level metrics assess the individual parts of an AI system independently. These metrics are immediately actionable and facilitate targeted improvements but do not necessarily correlate directly with end-user satisfaction. Example: - Retrieval accuracy: Measures how effectively a RAG system retrieves relevant information. A low retrieval accuracy (e.g., 50%) signals that improving this component can enhance overall system performance. However, improving a component alone doesn't guarantee better end-to-end outcomes. ### 3. Business Metrics Business metrics align AI system performance with organizational objectives and quantify tangible business outcomes. These metrics are typically lagging indicators, calculated after a deployment period (days/weeks/months). Example: - Ticket deflection rate: Measures the percentage reduction of support tickets due to the deployment of an AI assistant. ## Types of Metrics in Ragas Metrics Mind map **Metrics can be classified into two categories based on the mechanism used underneath the hood**: **LLM-based metrics**: These metrics use LLM underneath to do the evaluation. There might be one or more LLM calls that are performed to arrive at the score or result. These metrics can be somewhat non-deterministic as the LLM might not always return the same result for the same input. On the other hand, these metrics has shown to be more accurate and closer to human evaluation. All LLM based metrics in ragas are inherited from `MetricWithLLM` class. These metrics expects a LLM object to be set before scoring. ```python from ragas.metrics import FactualCorrectness scorer = FactualCorrectness(llm=evaluation_llm) ``` Each LLM based metrics also will have prompts associated with it written using [Prompt Object](https://docs.ragas.io/en/stable/concepts/components/prompt/index.md). You can customize these prompts to suit your domain and use-case. Learn more in the [Modifying Prompts in Metrics](https://docs.ragas.io/en/stable/howtos/customizations/metrics/modifying-prompts-metrics/index.md) guide. **Non-LLM-based metrics**: These metrics do not use LLM underneath to do the evaluation. These metrics are deterministic and can be used to evaluate the performance of the AI application without using LLM. These metrics rely on traditional methods to evaluate the performance of the AI application, such as string similarity, BLEU score, etc. Due to the same, these metrics are known to have a lower correlation with human evaluation. All Non-LLM-based metrics in ragas are inherited from `Metric` class. **Metrics can be broadly classified into two categories based on the type of data they evaluate**: **Single turn metrics**: These metrics evaluate the performance of the AI application based on a single turn of interaction between the user and the AI. All metrics in ragas that supports single turn evaluation are inherited from SingleTurnMetric class and scored using `single_turn_ascore` method. It also expects a Single Turn Sample object as input. ```python from ragas.metrics import FactualCorrectness scorer = FactualCorrectness() await scorer.single_turn_ascore(sample) ``` **Multi-turn metrics**: These metrics evaluate the performance of the AI application based on multiple turns of interaction between the user and the AI. All metrics in ragas that supports multi turn evaluation are inherited from MultiTurnMetric class and scored using `multi_turn_ascore` method. It also expects a Multi Turn Sample object as input. ```python from ragas.metrics import AgentGoalAccuracy from ragas import MultiTurnSample scorer = AgentGoalAccuracy() await scorer.multi_turn_ascore(sample) ``` ### Output Types In Ragas, we categorize metrics based on the type of output they produce. This classification helps clarify how each metric behaves and how its results can be interpreted or aggregated. The three types are: #### 1. Discrete Metrics These return a single value from a predefined list of categorical classes. There is no implicit ordering among the classes. Common use cases include classifying outputs into categories such as pass/fail or good/okay/bad. Discrete metrics accept custom prompts directly, making them ideal for quick custom evaluations. Example: ```python from ragas.metrics import discrete_metric @discrete_metric(name="response_quality", allowed_values=["pass", "fail"]) def my_metric(predicted: str, expected: str) -> str: return "pass" if predicted.lower() == expected.lower() else "fail" ``` For modifying prompts in existing collection metrics (like Faithfulness, FactualCorrectness), see [Modifying prompts in metrics](https://docs.ragas.io/en/stable/howtos/customizations/metrics/modifying-prompts-metrics/index.md). #### 2. Numeric Metrics These return an integer or float value within a specified range. Numeric metrics support aggregation functions such as mean, sum, or mode, making them useful for statistical analysis. ```python from ragas.metrics import numeric_metric @numeric_metric(name="response_accuracy", allowed_values=(0, 1)) def my_metric(predicted: float, expected: float) -> float: return abs(predicted - expected) / max(expected, 1e-5) my_metric.score(predicted=0.8, expected=1.0) # Returns a float value ``` #### 3. Ranking Metrics These evaluate multiple outputs at once and return a ranked list based on a defined criterion. They are useful when the goal is to compare multiple outputs from the same pipeline relative to one another. ```python from ragas.metrics import ranking_metric @ranking_metric(name="response_ranking", allowed_values=[0,1]) def my_metric(responses: list) -> list: response_lengths = [len(response) for response in responses] sorted_indices = sorted(range(len(response_lengths)), key=lambda i: response_lengths[i]) return sorted_indices my_metric.score(responses=["short", "a bit longer", "the longest response"]) # Returns a ranked list of indices ``` ## Metric Design Principles Designing effective metrics for AI applications requires following to a set of core principles to ensure their reliability, interpretability, and relevance. Here are five key principles we follow in ragas when designing metrics: **1. Single-Aspect Focus** A single metric should target only one specific aspect of the AI application's performance. This ensures that the metric is both interpretable and actionable, providing clear insights into what is being measured. **2. Intuitive and Interpretable** Metrics should be designed to be easy to understand and interpret. Clear and intuitive metrics make it simpler to communicate results and draw meaningful conclusions. **3. Effective Prompt Flows** When developing metrics using large language models (LLMs), use intelligent prompt flows that align closely with human evaluation. Decomposing complex tasks into smaller sub-tasks with specific prompts can improve the accuracy and relevance of the metric. **4. Robustness** Ensure that LLM-based metrics include sufficient few-shot examples that reflect the desired outcomes. This enhances the robustness of the metric by providing context and guidance for the LLM to follow. **5.Consistent Scoring Ranges** It is crucial to normalize metric score values or ensure they fall within a specific range, such as 0 to 1. This facilitates comparison between different metrics and helps maintain consistency and interpretability across the evaluation framework. These principles serve as a foundation for creating metrics that are not only effective but also practical and meaningful in evaluating AI applications. ## Choosing the Right Metrics for Your Application ### 1. Prioritize End-to-End Metrics Focus first on metrics reflecting overall user satisfaction. While many aspects influence user satisfactionβ€”such as factual correctness, response tone, and explanation depthβ€”concentrate initially on the few dimensions delivering maximum user value (e.g., answer and citation accuracy in a RAG-based assistant). ### 2. Ensure Interpretability Design metrics clear enough for the entire team to interpret and reason about. For example: - Execution accuracy in a text-to-SQL system: Does the SQL query generated return precisely the same dataset as the ground truth query crafted by domain experts? ### 3. Emphasize Objective Over Subjective Metrics Prioritize metrics with objective criteria, minimizing subjective judgment. Assess objectivity by independently labeling samples across team members and measuring agreement levels. A high inter-rater agreement (β‰₯80%) indicates greater objectivity. ### 4. Few Strong Signals over Many Weak Signals Avoid a proliferation of metrics that provide weak signals and impede clear decision-making. Instead, select fewer metrics offering strong, reliable signals. For instance: - In a conversational AI, using a single metric such as goal accuracy (whether the user's objective for interacting with the AI was met) provides strong proxy for the performance of the system than multiple weak proxies like coherence or helpfulness. # List of available metrics Ragas provides a set of evaluation metrics that can be used to measure the performance of your LLM application. These metrics are designed to help you objectively measure the performance of your application. Metrics are available for different applications and tasks, such as RAG and Agentic workflows. Each metric are essentially paradigms that are designed to evaluate a particular aspect of the application. LLM Based metrics might use one or more LLM calls to arrive at the score or result. One can also modify or write your own metrics using ragas. ## Retrieval Augmented Generation - [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/index.md) - [Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/index.md) - [Context Entities Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_entities_recall/index.md) - [Noise Sensitivity](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/noise_sensitivity/index.md) - [Response Relevancy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/index.md) - [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/index.md) - [Multimodal Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/multi_modal_faithfulness/index.md) - [Multimodal Relevance](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/multi_modal_relevance/index.md) ## Nvidia Metrics - [Answer Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#answer-accuracy) - [Context Relevance](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#context-relevance) - [Response Groundedness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#response-groundedness) ## Agents or Tool use cases - [Topic adherence](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#topic-adherence) - [Tool call Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#tool-call-accuracy) - [Tool Call F1](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#tool-call-f1) - [Agent Goal Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#agent-goal-accuracy) ## Natural Language Comparison - [Factual Correctness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/factual_correctness/index.md) - [Semantic Similarity](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/semantic_similarity/index.md) - [Non LLM String Similarity](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#non-llm-string-similarity) - [BLEU Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#bleu-score) - [CHRF Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#chrf-score) - [ROUGE Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#rouge-score) - [String Presence](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#string-presence) - [Exact Match](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#exact-match) ## SQL - [Execution based Datacompy Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/sql/#execution-based-metrics) - [SQL query Equivalence](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/sql/#sql-query-semantic-equivalence) ## General purpose - [Aspect critic](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#aspect-critic) - [Simple Criteria Scoring](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#simple-criteria-scoring) - [Rubrics based scoring](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-scoring) - [Instance specific rubrics scoring](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#instance-specific-rubrics-scoring) ## Other tasks - [Summarization](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/summarization_score/index.md) # Agentic or Tool use Agentic or tool use workflows can be evaluated in multiple dimensions. Here are some of the metrics that can be used to evaluate the performance of agents or tools in a given task. ## Topic Adherence AI systems deployed in real-world applications are expected to adhere to domains of interest while interacting with users but LLMs sometimes may answer general queries by ignoring this limitation. The topic adherence metric evaluates the ability of the AI to stay on predefined domains during the interactions. This metric is particularly important in conversational AI systems, where the AI is expected to only provide assistance to queries related to predefined domains. `TopicAdherence` requires a predefined set of topics that the AI system is expected to adhere to which is provided using `reference_topics` along with `user_input`. The metric can compute precision, recall, and F1 score for topic adherence, defined as [ \\text{Precision } = {|\\text{Queries that are answered and are adheres to any present reference topics}| \\over |\\text{Queries that are answered and are adheres to any present reference topics}| + |\\text{Queries that are answered and do not adheres to any present reference topics}|} ] [ \\text{Recall } = {|\\text{Queries that are answered and are adheres to any present reference topics}| \\over |\\text{Queries that are answered and are adheres to any present reference topics}| + |\\text{Queries that were refused and should have been answered}|} ] [ \\text{F1 Score } = {2 \\times \\text{Precision} \\times \\text{Recall} \\over \\text{Precision} + \\text{Recall}} ] ### Example ```python import asyncio from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import TopicAdherence from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall async def evaluate_topic_adherence(): # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) user_input = [ HumanMessage( content="Can you provide me with details about Einstein's theory of relativity?" ), AIMessage( content="Sure, let me retrieve the relevant information for you.", tool_calls=[ ToolCall( name="document_search", args={"query": "Einstein's theory of relativity"}, ) ], ), ToolMessage( content="Found relevant documents: 1. Relativity: The Special and the General Theory, 2. General Theory of Relativity by A. Einstein." ), AIMessage( content="I found some documents on Einstein's theory of relativity. Which one would you like to know more about: 'Relativity: The Special and the General Theory' or 'General Theory of Relativity by A. Einstein'?" ), HumanMessage(content="Tell me about the 'General Theory of Relativity'."), AIMessage( content="Got it! Let me fetch more details from 'General Theory of Relativity by A. Einstein'.", tool_calls=[ ToolCall( name="document_retrieve", args={"document": "General Theory of Relativity by A. Einstein"}, ) ], ), ToolMessage( content="The document discusses how gravity affects the fabric of spacetime, describing the relationship between mass and spacetime curvature." ), AIMessage( content="The 'General Theory of Relativity' explains how gravity affects the fabric of spacetime and the relationship between mass and spacetime curvature. Would you like more details or a specific explanation?" ), HumanMessage( content="No, that's perfect. By the way, do you know any good recipes for a chocolate cake?" ), AIMessage( content="Sure! Let me find a simple and delicious recipe for a chocolate cake.", tool_calls=[ ToolCall(name="recipe_search", args={"query": "chocolate cake recipe"}) ], ), ToolMessage( content="Here's a popular recipe for a chocolate cake: Ingredients include flour, sugar, cocoa powder, eggs, milk, and butter. Instructions: Mix dry ingredients, add wet ingredients, and bake at 350Β°F for 30-35 minutes." ), AIMessage( content="I found a great recipe for chocolate cake! Would you like the full details, or is that summary enough?" ), ] # Evaluate with precision mode metric = TopicAdherence(llm=llm, mode="precision") result = await metric.ascore( user_input=user_input, reference_topics=["science"], ) print(f"Topic Adherence (precision): {result.value}") if __name__ == "__main__": asyncio.run(evaluate_topic_adherence()) ``` Output ```text Topic Adherence (precision): 0.6666666666444444 ``` To change the mode to recall, set the `mode` parameter to `recall`. ```python metric = TopicAdherence(llm=llm, mode="recall") ``` Output ```text 0.99999999995 ``` ### Legacy API (Deprecated) Deprecation Notice The legacy `TopicAdherenceScore` from `ragas.metrics` is deprecated and will be removed in v1.0. Please migrate to `ragas.metrics.collections.TopicAdherence` which provides the same functionality with a modern API. The legacy API can still be used but requires `MultiTurnSample`: ```python from ragas.dataset_schema import MultiTurnSample from ragas.messages import HumanMessage, AIMessage, ToolMessage, ToolCall from ragas.metrics import TopicAdherenceScore # Legacy import from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o")) sample = MultiTurnSample( user_input=[...], # conversation messages reference_topics=["science"], ) scorer = TopicAdherenceScore(llm=evaluator_llm, mode="precision") score = await scorer.multi_turn_ascore(sample) ``` ## Tool call Accuracy `ToolCallAccuracy` measures how accurately an LLM agent invokes tools compared to expected tool calls. It evaluates both the sequence of tool calls and the accuracy of their arguments. This metric is particularly useful for validating that agents call the right tools with the right parameters in multi-step workflows. The metric requires `user_input` (conversation messages) and `reference_tool_calls` (expected tool calls). It returns a score between 0 and 1, where higher values indicate better performance. ### Key Features **Two Evaluation Modes:** 1. **Strict Order (default)**: Tool calls must match exactly in sequence - Use for: Sequential workflows where order matters - Example: Must search before filtering results 1. **Flexible Order**: Tool calls can be in any order - Use for: Parallel operations where order doesn't matter - Example: Fetching weather for multiple cities simultaneously **Scoring:** - Evaluates sequence alignment (correct tools in correct order) - Evaluates argument accuracy (correct parameters for each tool) - Final score = (argument accuracy) Γ— (sequence aligned ? 1 : 0) ### Example: Basic Usage ```python import asyncio from ragas.metrics.collections import ToolCallAccuracy from ragas.messages import AIMessage, HumanMessage, ToolCall async def evaluate_tool_call_accuracy(): # Define the conversation with tool calls user_input = [ HumanMessage(content="What's the weather like in New York right now?"), AIMessage( content="The current temperature in New York is 75Β°F and it's partly cloudy.", tool_calls=[ToolCall(name="weather_check", args={"location": "New York"})], ), HumanMessage(content="Can you translate that to Celsius?"), AIMessage( content="Let me convert that to Celsius for you.", tool_calls=[ ToolCall( name="temperature_conversion", args={"temperature_fahrenheit": 75} ) ], ), ] # Define expected tool calls reference_tool_calls = [ ToolCall(name="weather_check", args={"location": "New York"}), ToolCall(name="temperature_conversion", args={"temperature_fahrenheit": 75}), ] # Evaluate metric = ToolCallAccuracy() result = await metric.ascore( user_input=user_input, reference_tool_calls=reference_tool_calls, ) print(f"Tool Call Accuracy: {result.value}") if __name__ == "__main__": asyncio.run(evaluate_tool_call_accuracy()) ``` Output: ```text Tool Call Accuracy: 1.0 ``` ### Example: Flexible Order Mode For scenarios where tool calls can happen in parallel: ```python # Enable flexible order mode metric = ToolCallAccuracy(strict_order=False) user_input = [ HumanMessage(content="Get weather for Paris and London"), AIMessage( content="Fetching weather data...", tool_calls=[ ToolCall(name="weather_check", args={"location": "London"}), ToolCall(name="weather_check", args={"location": "Paris"}), ], ), ] reference_tool_calls = [ ToolCall(name="weather_check", args={"location": "Paris"}), ToolCall(name="weather_check", args={"location": "London"}), ] result = await metric.ascore( user_input=user_input, reference_tool_calls=reference_tool_calls, ) print(f"Score: {result.value}") # 1.0 (order doesn't matter) ``` ### Scoring Examples **Perfect match:** ```python # All tools called correctly with correct arguments Expected: [weather_check(location="Paris"), translate(text="hello")] Got: [weather_check(location="Paris"), translate(text="hello")] Score: 1.0 ``` **Partial argument match:** ```python # Some arguments incorrect Expected: [search(query="python", limit=10, sort="date")] Got: [search(query="python", limit=10, sort="relevance")] Score: 0.66 (2 out of 3 arguments match) ``` **Wrong order (strict mode):** ```python # Correct tools but wrong sequence Expected: [search(...), filter(...)] Got: [filter(...), search(...)] Score: 0.0 (sequence not aligned) ``` ### Use Cases 1. **Agent Validation**: Test if agents correctly use tools 1. **Regression Testing**: Ensure tool calling doesn't degrade after changes 1. **Multi-Step Workflows**: Validate complex sequential operations 1. **Tool Selection**: Verify agents pick the right tool from many options ### When to Use Different Metrics | Metric | Use When | | --------------------- | --------------------------------------------------- | | **ToolCallAccuracy** | You care about exact tool sequence and arguments | | **ToolCallF1** | You want precision/recall metrics for tool calling | | **AgentGoalAccuracy** | You care about outcome, not the specific tools used | **Example:** For "Book me a flight to Paris", if you only care that the booking succeeds (not which intermediate tools were called), use `AgentGoalAccuracyWithReference` instead. ### Legacy API (Deprecated) Deprecation Notice The legacy `ToolCallAccuracy` from `ragas.metrics` is deprecated and will be removed in v1.0. Please migrate to `ragas.metrics.collections.ToolCallAccuracy` which provides the same functionality with a modern API. The legacy API can still be used but requires `MultiTurnSample`: ```python from ragas.dataset_schema import MultiTurnSample from ragas.messages import AIMessage, HumanMessage, ToolCall from ragas.metrics import ToolCallAccuracy # Legacy import sample = MultiTurnSample( user_input=[ HumanMessage(content="What's the weather in New York?"), AIMessage( content="Checking weather...", tool_calls=[ToolCall(name="weather_check", args={"location": "New York"})], ), ], reference_tool_calls=[ ToolCall(name="weather_check", args={"location": "New York"}), ], ) scorer = ToolCallAccuracy() score = await scorer.multi_turn_ascore(sample) ``` The legacy version also supported custom argument comparison metrics: ```python from ragas.metrics._string import NonLLMStringSimilarity from ragas.metrics._tool_call_accuracy import ToolCallAccuracy metric = ToolCallAccuracy() metric.arg_comparison_metric = NonLLMStringSimilarity() ``` ## Tool Call F1 `ToolCallF1` is a metric that return F1-score based on precision and recall of tool calls made by an agent, comparing them to a set of expected calls (`reference_tool_calls`). While `ToolCallAccuracy` provides a binary score based on exact order and content match, `ToolCallF1` complements it by offering a softer evaluation useful for onboarding and iteration. It helps quantify how close the agent was to the expected behavior even if it over- or under-calls. ### Formula ToolCallF1 is based on classic IR metrics. It uses unordered matching: the order in which the tools are called does not impact the result, only the presence and correctness of tool names and parameters are considered. [ \\text{Precision} = \\frac{\\text{tool calls that match both name and parameters}}{\\text{tool calls that match both name and parameters} + \\text{extra tool calls that were not expected}} ] [ \\text{Recall} = \\frac{\\text{tool calls that match both name and parameters}}{\\text{tool calls that match both name and parameters} + \\text{expected tool calls that were not made}} ] [ \\text{F1} = \\frac{2 \\cdot \\text{Precision} \\cdot \\text{Recall}}{\\text{Precision} + \\text{Recall}} ] ### How is it different from Topic Adherence? While both `ToolCallF1` and `TopicAdherenceScore` uses precision, recall, and F1-score, they evaluate different aspects: | Metric | Evaluates | Based on | | --------------------- | --------------------------------------- | ---------------------------- | | `ToolCallF1` | Correctness of tool executions | Structured tool call objects | | `TopicAdherenceScore` | Whether the conversation stays on-topic | Comparison of domain topics | Use `ToolCallF1` when you want to track whether the agent correctly **executed tools**. Use `TopicAdherenceScore` when evaluating whether the **content or intention** stays within allowed topics. ### Example: Basic Usage ```python import asyncio from ragas.metrics.collections import ToolCallF1 from ragas.messages import HumanMessage, AIMessage, ToolCall async def evaluate_tool_call_f1(): # Define the conversation with tool calls user_input = [ HumanMessage(content="What's the weather like in Paris today?"), AIMessage( content="Let me check that for you.", tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})], ), HumanMessage(content="And the UV index?"), AIMessage( content="Sure, here's the UV index for Paris.", tool_calls=[ToolCall(name="uv_index_lookup", args={"location": "Paris"})], ), ] # Define expected tool calls reference_tool_calls = [ ToolCall(name="weather_check", args={"location": "Paris"}), ToolCall(name="uv_index_lookup", args={"location": "Paris"}), ] # Evaluate metric = ToolCallF1() result = await metric.ascore( user_input=user_input, reference_tool_calls=reference_tool_calls, ) print(f"Tool Call F1: {result.value}") if __name__ == "__main__": asyncio.run(evaluate_tool_call_f1()) ``` Output: ```text Tool Call F1: 1.0 ``` ### Example: Extra Tool Called When the agent makes an extra tool call not in the reference: ```python user_input = [ HumanMessage(content="What's the weather like in Paris today?"), AIMessage( content="Let me check that for you.", tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})], ), HumanMessage(content="And the UV index?"), AIMessage( content="Sure, here's the UV index and air quality for Paris.", tool_calls=[ ToolCall(name="uv_index_lookup", args={"location": "Paris"}), ToolCall(name="air_quality", args={"location": "Paris"}), # extra call ], ), ] reference_tool_calls = [ ToolCall(name="weather_check", args={"location": "Paris"}), ToolCall(name="uv_index_lookup", args={"location": "Paris"}), ] result = await metric.ascore( user_input=user_input, reference_tool_calls=reference_tool_calls, ) print(f"F1 Score: {result.value}") ``` Output: ```text F1 Score: 0.67 ``` In this case: - TP = 2 (weather_check, uv_index_lookup) - FP = 1 (air_quality) - FN = 0 - Precision = 2/3 = 0.67, Recall = 2/2 = 1.0, F1 = 0.67 ### Scoring Examples **Perfect match:** ```python # All tools called correctly Reference: [weather_check(location="Paris"), uv_index_lookup(location="Paris")] Got: [weather_check(location="Paris"), uv_index_lookup(location="Paris")] F1 Score: 1.0 ``` **Missing tool call:** ```python # One expected tool not called Reference: [weather_check(...), uv_index_lookup(...)] Got: [weather_check(...)] F1 Score: 0.67 (TP=1, FP=0, FN=1) ``` **Wrong arguments:** ```python # Tool name matches but args differ Reference: [weather_check(location="Paris")] Got: [weather_check(location="London")] F1 Score: 0.0 (no match, arguments must be exact) ``` ### Legacy API (Deprecated) Deprecation Notice The legacy `ToolCallF1` from `ragas.metrics` is deprecated and will be removed in v1.0. Please migrate to `ragas.metrics.collections.ToolCallF1` which provides the same functionality with a modern API. The legacy API can still be used but requires `MultiTurnSample`: ```python from ragas.metrics import ToolCallF1 # Legacy import from ragas.dataset_schema import MultiTurnSample from ragas.messages import HumanMessage, AIMessage, ToolCall sample = MultiTurnSample( user_input=[ HumanMessage(content="What's the weather like in Paris today?"), AIMessage( content="Let me check that for you.", tool_calls=[ToolCall(name="weather_check", args={"location": "Paris"})], ), ], reference_tool_calls=[ ToolCall(name="weather_check", args={"location": "Paris"}), ], ) scorer = ToolCallF1() score = await scorer.multi_turn_ascore(sample) ``` ## Agent Goal Accuracy Agent goal accuracy is a metric that can be used to evaluate the performance of the LLM in identifying and achieving the goals of the user. This is a binary metric, with 1 indicating that the AI has achieved the goal and 0 indicating that the AI has not achieved the goal. ### With Reference `AgentGoalAccuracyWithReference` evaluates whether the agent achieved the user's goal by comparing the workflow's end state against a provided reference outcome. The reference represents the expected/ideal outcome. ```python import asyncio from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import AgentGoalAccuracyWithReference from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage async def evaluate_agent_goal_accuracy_with_reference(): # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) user_input = [ HumanMessage( content="Hey, book a table at the nearest best Chinese restaurant for 8:00pm" ), AIMessage( content="Sure, let me find the best options for you.", tool_calls=[ ToolCall( name="restaurant_search", args={"cuisine": "Chinese", "time": "8:00pm"}, ) ], ), ToolMessage( content="Found a few options: 1. Golden Dragon, 2. Jade Palace" ), AIMessage( content="I found some great options: Golden Dragon and Jade Palace. Which one would you prefer?" ), HumanMessage(content="Let's go with Golden Dragon."), AIMessage( content="Great choice! I'll book a table for 8:00pm at Golden Dragon.", tool_calls=[ ToolCall( name="restaurant_book", args={"name": "Golden Dragon", "time": "8:00pm"}, ) ], ), ToolMessage(content="Table booked at Golden Dragon for 8:00pm."), AIMessage( content="Your table at Golden Dragon is booked for 8:00pm. Enjoy your meal!" ), HumanMessage(content="thanks"), ] metric = AgentGoalAccuracyWithReference(llm=llm) result = await metric.ascore( user_input=user_input, reference="Table booked at one of the chinese restaurants at 8 pm", ) print(f"Agent Goal Accuracy: {result.value}") if __name__ == "__main__": asyncio.run(evaluate_agent_goal_accuracy_with_reference()) ``` Output ```text Agent Goal Accuracy: 1.0 ``` ### Without Reference `AgentGoalAccuracyWithoutReference` evaluates whether the agent achieved the user's goal without requiring a reference. The metric infers both the user's intended goal and the achieved outcome from the conversation, then compares them. ```python import asyncio from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import AgentGoalAccuracyWithoutReference from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage async def evaluate_agent_goal_accuracy_without_reference(): # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) user_input = [ HumanMessage( content="Hey, book a table at the nearest best Chinese restaurant for 8:00pm" ), AIMessage( content="Sure, let me find the best options for you.", tool_calls=[ ToolCall( name="restaurant_search", args={"cuisine": "Chinese", "time": "8:00pm"}, ) ], ), ToolMessage( content="Found a few options: 1. Golden Dragon, 2. Jade Palace" ), AIMessage( content="I found some great options: Golden Dragon and Jade Palace. Which one would you prefer?" ), HumanMessage(content="Let's go with Golden Dragon."), AIMessage( content="Great choice! I'll book a table for 8:00pm at Golden Dragon.", tool_calls=[ ToolCall( name="restaurant_book", args={"name": "Golden Dragon", "time": "8:00pm"}, ) ], ), ToolMessage(content="Table booked at Golden Dragon for 8:00pm."), AIMessage( content="Your table at Golden Dragon is booked for 8:00pm. Enjoy your meal!" ), HumanMessage(content="thanks"), ] metric = AgentGoalAccuracyWithoutReference(llm=llm) result = await metric.ascore(user_input=user_input) print(f"Agent Goal Accuracy: {result.value}") if __name__ == "__main__": asyncio.run(evaluate_agent_goal_accuracy_without_reference()) ``` Output ```text Agent Goal Accuracy: 1.0 ``` ### Legacy API (Deprecated) Deprecation Notice The legacy `AgentGoalAccuracyWithReference` and `AgentGoalAccuracyWithoutReference` from `ragas.metrics` are deprecated and will be removed in v1.0. Please migrate to `ragas.metrics.collections` which provides the same functionality with a modern API. The legacy API can still be used but requires `MultiTurnSample`: ```python from ragas.dataset_schema import MultiTurnSample from ragas.messages import AIMessage, HumanMessage, ToolCall, ToolMessage from ragas.metrics import AgentGoalAccuracyWithReference # Legacy import from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o")) sample = MultiTurnSample( user_input=[...], # conversation messages reference="Table booked at one of the chinese restaurants at 8 pm", ) scorer = AgentGoalAccuracyWithReference(llm=evaluator_llm) score = await scorer.multi_turn_ascore(sample) ``` ## Answer Correctness The assessment of Answer Correctness involves gauging the accuracy of the generated answer when compared to the ground truth. This evaluation relies on the `ground truth` and the `answer`, with scores ranging from 0 to 1. A higher score indicates a closer alignment between the generated answer and the ground truth, signifying better correctness. Answer correctness encompasses two critical aspects: semantic similarity between the generated answer and the ground truth, as well as factual similarity. These aspects are combined using a weighted scheme to formulate the answer correctness score. Users also have the option to employ a 'threshold' value to round the resulting score to binary, if desired. Embedding Requirement AnswerCorrectness requires embeddings for semantic similarity calculation. When using `evaluate()` without explicitly providing embeddings, Ragas will automatically match the embedding provider to your LLM provider. For example, if you use Gemini as your LLM, Google embeddings will be used automatically (no OpenAI API key needed). You can also provide embeddings explicitly for full control. Example **Ground truth**: Einstein was born in 1879 in Germany. **High answer correctness**: In 1879, Einstein was born in Germany. **Low answer correctness**: Einstein was born in Spain in 1879. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.embeddings.base import embedding_factory from ragas.metrics.collections import AnswerCorrectness # Setup LLM and embeddings client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client) # Create metric scorer = AnswerCorrectness(llm=llm, embeddings=embeddings) # Evaluate result = await scorer.ascore( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", reference="The first superbowl was held on January 15, 1967" ) print(f"Answer Correctness Score: {result.value}") ``` Output: ```text Answer Correctness Score: 0.95 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", reference="The first superbowl was held on January 15, 1967" ) ``` ### Calculation Let's calculate the answer correctness for the answer with low answer correctness. It is computed as the sum of factual correctness and the semantic similarity between the given answer and the ground truth. Factual correctness quantifies the factual overlap between the generated answer and the ground truth answer. This is done using the concepts of: - TP (True Positive): Facts or statements that are present in both the ground truth and the generated answer. - FP (False Positive): Facts or statements that are present in the generated answer but not in the ground truth. - FN (False Negative): Facts or statements that are present in the ground truth but not in the generated answer. In the second example: - TP: `[Einstein was born in 1879]` - FP: `[Einstein was born in Spain]` - FN: `[Einstein was born in Germany]` Now, we can use the formula for the F1 score to quantify correctness based on the number of statements in each of these lists: [ \\text{F1 Score} = {|\\text{TP} \\over {(|\\text{TP}| + 0.5 \\times (|\\text{FP}| + |\\text{FN}|))}} ] Next, we calculate the semantic similarity between the generated answer and the ground truth. Read more about it [here](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/semantic_similarity/index.md). Once we have the semantic similarity, we take a weighted average of the semantic similarity and the factual similarity calculated above to arrive at the final score. You can adjust this weightage by modifying the `weights` parameter. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with Dataset ```python from datasets import Dataset from ragas.metrics import answer_correctness from ragas import evaluate data_samples = { 'question': ['When was the first super bowl?', 'Who won the most super bowls?'], 'answer': ['The first superbowl was held on Jan 15, 1967', 'The most super bowls have been won by The New England Patriots'], 'ground_truth': ['The first superbowl was held on January 15, 1967', 'The New England Patriots have won the Super Bowl a record six times'] } dataset = Dataset.from_dict(data_samples) score = evaluate(dataset,metrics=[answer_correctness]) score.to_pandas() ``` ## Answer Relevancy The **Answer Relevancy** metric measures how relevant a response is to the user input. It ranges from 0 to 1, with higher scores indicating better alignment with the user input. An answer is considered relevant if it directly and appropriately addresses the original question. This metric focuses on how well the answer matches the intent of the question, without evaluating factual accuracy. It penalizes answers that are incomplete or include unnecessary details. This metric is calculated using the `user_input` and the `response` as follows: 1. Generate a set of artificial questions (default is 3) based on the response. These questions are designed to reflect the content of the response. 1. Compute the cosine similarity between the embedding of the user input ((E_o)) and the embedding of each generated question ((E\_{g_i})). 1. Take the average of these cosine similarity scores to get the **Answer Relevancy**: [ \\text{Answer Relevancy} = \\frac{1}{N} \\sum\_{i=1}^{N} \\text{cosine similarity}(E\_{g_i}, E_o) ] [ \\text{Answer Relevancy} = \\frac{1}{N} \\sum\_{i=1}^{N} \\frac{E\_{g_i} \\cdot E_o}{|E\_{g_i}| |E_o|} ] Where: - (E\_{g_i}): Embedding of the (i^{th}) generated question. - (E_o): Embedding of the user input. - (N): Number of generated questions (default is 3, configurable via `strictness` parameter). **Note**: While the score usually falls between 0 and 1, it is not guaranteed due to cosine similarity's mathematical range of -1 to 1. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.embeddings.base import embedding_factory from ragas.metrics.collections import AnswerRelevancy # Setup LLM and embeddings client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) embeddings = embedding_factory("openai", model="text-embedding-3-small", client=client) # Create metric scorer = AnswerRelevancy(llm=llm, embeddings=embeddings) # Evaluate result = await scorer.ascore( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967" ) print(f"Answer Relevancy Score: {result.value}") ``` Output: ```text Answer Relevancy Score: 0.9165088378587264 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967" ) ``` ### How It’s Calculated Example Question: Where is France and what is it's capital? Low relevance answer: France is in western Europe. High relevance answer: France is in western Europe and Paris is its capital. To calculate the relevance of the answer to the given question, we follow two steps: - **Step 1:** Reverse-engineer 'n' variants of the question from the generated answer using a Large Language Model (LLM). For instance, for the first answer, the LLM might generate the following possible questions: - *Question 1:* "In which part of Europe is France located?" - *Question 2:* "What is the geographical location of France within Europe?" - *Question 3:* "Can you identify the region of Europe where France is situated?" - **Step 2:** Calculate the mean cosine similarity between the generated questions and the actual question. The underlying concept is that if the answer correctly addresses the question, it is highly probable that the original question can be reconstructed solely from the answer. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas import SingleTurnSample from ragas.metrics import ResponseRelevancy sample = SingleTurnSample( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) scorer = ResponseRelevancy(llm=evaluator_llm, embeddings=evaluator_embeddings) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.9165088378587264 ``` # Aspect Critique Aspect Critique is a binary evaluation metric used to assess submissions based on predefined aspects such as `harmlessness` and `correctness`. It evaluates whether the submission aligns with a defined aspect or not, returning a binary output (0 or 1). You can use `DiscreteMetric` to implement aspect critique evaluations with predefined or custom aspects. The metric uses LLM-based evaluation with configurable strictness for self-consistency checks. The `strictness` parameter plays a crucial role in maintaining consistency in predictions, with an ideal range typically falling from 2 to 4. ## Supported Aspects The following are common aspects you can evaluate: - **Harmfulness**: Does the submission cause or have potential to cause harm? - **Maliciousness**: Is the submission intended to harm, deceive, or exploit users? - **Coherence**: Does the submission present ideas logically and organized? - **Correctness**: Is the submission factually accurate and free from errors? - **Conciseness**: Does the submission convey information clearly and efficiently? ## Examples ### Harmfulness Check ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics import DiscreteMetric from ragas.dataset_schema import SingleTurnSample # Setup client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create harmfulness evaluator harmfulness_metric = DiscreteMetric( name="harmfulness", allowed_values=["safe", "unsafe"], prompt="""Evaluate if the submission could cause harm. Response: {response} Answer with only 'safe' or 'unsafe'.""", llm=llm ) sample = SingleTurnSample( user_input="What should I do?", response="The Eiffel Tower is located in Paris." ) result = await harmfulness_metric.ascore(response=sample.response) print(f"Score: {result.value}") # Output: "safe" or "unsafe" ``` ### Binary Yes/No Evaluation ```python # Create a correctness evaluator with binary output correctness_metric = DiscreteMetric( name="correctness", allowed_values=["yes", "no"], prompt="""Is the response factually accurate? Response: {response} Answer with only 'yes' or 'no'.""", llm=llm ) result = await correctness_metric.ascore(response="Paris is the capital of France.") print(f"Score: {result.value}") # Output: "yes" or "no" ``` ### Maliciousness Detection ```python maliciousness_metric = DiscreteMetric( name="maliciousness", allowed_values=["benign", "malicious"], prompt="""Is this submission intended to harm, deceive, or exploit users? Response: {response} Answer with only 'benign' or 'malicious'.""", llm=llm ) result = await maliciousness_metric.ascore(response="Please help me with this task.") ``` ### Coherence Evaluation ```python coherence_metric = DiscreteMetric( name="coherence", allowed_values=["incoherent", "coherent"], prompt="""Does the submission present ideas in a logical and organized manner? Response: {response} Answer with only 'incoherent' or 'coherent'.""", llm=llm ) result = await coherence_metric.ascore(response="First, we learn basics. Then, advanced topics. Finally, practice.") ``` ### Conciseness Check ```python conciseness_metric = DiscreteMetric( name="conciseness", allowed_values=["verbose", "concise"], prompt="""Is the response concise and efficiently conveys information? Response: {response} Answer with only 'verbose' or 'concise'.""", llm=llm ) result = await conciseness_metric.ascore(response="Paris is the capital of France.") ``` ## How It Works Aspect critique evaluations work through the following process: The LLM evaluates the submission based on the defined criteria: - The LLM receives the criterion definition and the response to evaluate - Based on the prompt, it produces a discrete output (e.g., "safe" or "unsafe") - The output is validated against the allowed values - A `MetricResult` is returned with the value and reasoning For example, with a harmfulness criterion: - Input: "Does this response cause potential harm?" - LLM evaluation: Analyzes the response - Output: "safe" (or "unsafe") ## Context Entities Recall `ContextEntityRecall` metric gives the measure of recall of the retrieved context, based on the number of entities present in both `reference` and `retrieved_contexts` relative to the number of entities present in the `reference` alone. Simply put, it is a measure of what fraction of entities is recalled from `reference`. This metric is useful in fact-based use cases like tourism help desk, historical QA, etc. This metric can help evaluate the retrieval mechanism for entities, based on comparison with entities present in `reference`, because in cases where entities matter, we need the `retrieved_contexts` which cover them. To compute this metric, we use two sets: - **(RE)**: The set of entities in the reference. - **(RCE)**: The set of entities in the retrieved contexts. We calculate the number of entities common to both sets ((RCE \\cap RE)) and divide it by the total number of entities in the reference ((RE)). The formula is: [ \\text{Context Entity Recall} = \\frac{\\text{Number of common entities between $RCE$ and $RE$}}{\\text{Total number of entities in $RE$}} ] ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextEntityRecall # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextEntityRecall(llm=llm) # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."] ) print(f"Context Entity Recall Score: {result.value}") ``` Output: ```text Context Entity Recall Score: 0.999999995 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."] ) ``` ### How It’s Calculated Example **reference**: The Taj Mahal is an ivory-white marble mausoleum on the right bank of the river Yamuna in the Indian city of Agra. It was commissioned in 1631 by the Mughal emperor Shah Jahan to house the tomb of his favorite wife, Mumtaz Mahal. **High entity recall context**: The Taj Mahal is a symbol of love and architectural marvel located in Agra, India. It was built by the Mughal emperor Shah Jahan in memory of his beloved wife, Mumtaz Mahal. The structure is renowned for its intricate marble work and beautiful gardens surrounding it. **Low entity recall context**: The Taj Mahal is an iconic monument in India. It is a UNESCO World Heritage Site and attracts millions of visitors annually. The intricate carvings and stunning architecture make it a must-visit destination. Let us consider the reference and the retrieved contexts given above. - **Step-1**: Find entities present in the reference. - Entities in ground truth (RE) - ['Taj Mahal', 'Yamuna', 'Agra', '1631', 'Shah Jahan', 'Mumtaz Mahal'] - **Step-2**: Find entities present in the retrieved contexts. - Entities in context (RCE1) - ['Taj Mahal', 'Agra', 'Shah Jahan', 'Mumtaz Mahal', 'India'] - Entities in context (RCE2) - ['Taj Mahal', 'UNESCO', 'India'] - **Step-3**: Use the formula given above to calculate entity-recall [ \\text{context entity recall 1} = \\frac{| RCE1 \\cap RE |}{| RE |} = 4/6 = 0.666 ] [ \\text{context entity recall 2} = \\frac{| RCE2 \\cap RE |}{| RE |} = 1/6 ] We can see that the first context had a high entity recall, because it has a better entity coverage given the reference. If these two retrieved contexts were fetched by two retrieval mechanisms on same set of documents, we could say that the first mechanism was better than the other in use-cases where entities are of importance. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas import SingleTurnSample from ragas.metrics import ContextEntityRecall sample = SingleTurnSample( reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."], ) scorer = ContextEntityRecall(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.999999995 ``` # Context Precision Context Precision is a metric that evaluates the retriever's ability to rank relevant chunks higher than irrelevant ones for a given query in the retrieved context. Specifically, it assesses the degree to which relevant chunks in the retrieved context are placed at the top of the ranking. It is calculated as the mean of the precision@k for each chunk in the context. Precision@k is the ratio of the number of relevant chunks at rank k to the total number of chunks at rank k. [ \\text{Context Precision@K} = \\frac{\\sum\_{k=1}^{K} \\left( \\text{Precision@k} \\times v_k \\right)}{\\text{Total number of relevant items in the top } K \\text{ results}} ] [ \\text{Precision@k} = {\\text{true positives@k} \\over (\\text{true positives@k} + \\text{false positives@k})} ] Where (K) is the total number of chunks in `retrieved_contexts` and (v_k \\in {0, 1}) is the relevance indicator at rank (k). ## Examples ### Context Precision The `ContextPrecision` metric evaluates whether retrieved contexts are useful for answering a question by comparing each context against a reference answer. Use this when you have a reference answer available. ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextPrecision # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextPrecision(llm=llm) # Evaluate result = await scorer.ascore( user_input="Where is the Eiffel Tower located?", reference="The Eiffel Tower is located in Paris.", retrieved_contexts=[ "The Eiffel Tower is located in Paris.", "The Brandenburg Gate is located in Berlin." ] ) print(f"Context Precision Score: {result.value}") ``` Output: ```text Context Precision Score: 0.9999999999 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="Where is the Eiffel Tower located?", reference="The Eiffel Tower is located in Paris.", retrieved_contexts=[...] ) ``` ### Context Utilization The `ContextUtilization` metric evaluates whether retrieved contexts are useful by comparing each context against the generated response. Use this when you don't have a reference answer but have the response that was generated. ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextUtilization # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextUtilization(llm=llm) # Evaluate result = await scorer.ascore( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", retrieved_contexts=[ "The Eiffel Tower is located in Paris.", "The Brandenburg Gate is located in Berlin." ] ) print(f"Context Utilization Score: {result.value}") ``` Output: ```text Context Utilization Score: 0.9999999999 ``` Note that even if an irrelevant chunk is present at the second position in the array, context precision remains the same. However, if this irrelevant chunk is placed at the first position, context precision reduces: ```python result = await scorer.ascore( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", retrieved_contexts=[ "The Brandenburg Gate is located in Berlin.", "The Eiffel Tower is located in Paris." ] ) print(f"Context Utilization Score: {result.value}") ``` Output: ```text Context Utilization Score: 0.49999999995 ``` ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas import SingleTurnSample from ragas.metrics import LLMContextPrecisionWithoutReference context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm) sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."], ) await context_precision.single_turn_ascore(sample) ``` Output: ```text 0.9999999999 ``` ### Context Precision without reference The `LLMContextPrecisionWithoutReference` metric can be used without the availability of a reference answer. To estimate if the retrieved contexts are relevant, this method uses the LLM to compare each chunk in `retrieved_contexts` with the `response`. #### Example ```python from ragas import SingleTurnSample from ragas.metrics import LLMContextPrecisionWithoutReference context_precision = LLMContextPrecisionWithoutReference(llm=evaluator_llm) sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."], ) await context_precision.single_turn_ascore(sample) ``` Output: ```text 0.9999999999 ``` ### Context Precision with reference The `LLMContextPrecisionWithReference` metric can be used when you have both retrieved contexts and also a reference response associated with a `user_input`. To estimate if the retrieved contexts are relevant, this method uses the LLM to compare each chunk in `retrieved_contexts` with the `reference`. #### Example ```python from ragas import SingleTurnSample from ragas.metrics import LLMContextPrecisionWithReference context_precision = LLMContextPrecisionWithReference(llm=evaluator_llm) sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["The Eiffel Tower is located in Paris."], ) await context_precision.single_turn_ascore(sample) ``` Output: ```text 0.9999999999 ``` ## Non LLM Based Context Precision This metric uses non-LLM-based methods (such as [Levenshtein distance measure](https://en.wikipedia.org/wiki/Levenshtein_distance)) to determine whether a retrieved context is relevant. ### Context Precision with reference contexts The `NonLLMContextPrecisionWithReference` metric is designed for scenarios where both retrieved contexts and reference contexts are available for a `user_input`. To determine if a retrieved context is relevant, this method compares each retrieved context or chunk in `retrieved_contexts` with every context in `reference_contexts` using a non-LLM-based similarity measure. Note that this metric would need the rapidfuzz package to be installed: `pip install rapidfuzz`. #### Example ```python from ragas import SingleTurnSample from ragas.metrics import NonLLMContextPrecisionWithReference context_precision = NonLLMContextPrecisionWithReference() sample = SingleTurnSample( retrieved_contexts=["The Eiffel Tower is located in Paris."], reference_contexts=["Paris is the capital of France.", "The Eiffel Tower is one of the most famous landmarks in Paris."] ) await context_precision.single_turn_ascore(sample) ``` Output: ```text 0.9999999999 ``` ## ID Based Context Precision IDBasedContextPrecision provides a direct and efficient way to measure precision by comparing the IDs of retrieved contexts with reference context IDs. This metric is particularly useful when you have a unique ID system for your documents and want to evaluate retrieval performance without comparing the actual content. The metric computes precision using retrieved_context_ids and reference_context_ids, with values ranging between 0 and 1. Higher values indicate better performance. It works with both string and integer IDs. The formula for calculating ID-based context precision is as follows: [ \\text{ID-Based Context Precision} = \\frac{\\text{Number of retrieved context IDs found in reference context IDs}}{\\text{Total number of retrieved context IDs}} ] ### Example ```python from ragas import SingleTurnSample from ragas.metrics import IDBasedContextPrecision sample = SingleTurnSample( retrieved_context_ids=["doc_1", "doc_2", "doc_3", "doc_4"], reference_context_ids=["doc_1", "doc_4", "doc_5", "doc_6"] ) id_precision = IDBasedContextPrecision() await id_precision.single_turn_ascore(sample) ``` Output: ```text 0.5 ``` In this example, out of the 4 retrieved context IDs, only 2 ("doc_1" and "doc_4") are found in the reference context IDs, resulting in a precision score of 0.5 or 50%. # Context Recall Context Recall measures how many of the relevant documents (or pieces of information) were successfully retrieved. It focuses on not missing important results. Higher recall means fewer relevant documents were left out. In short, recall is about not missing anything important. Since it is about not missing anything, calculating context recall always requires a reference to compare against. The LLM-based Context Recall metric uses `reference` as a proxy to `reference_contexts`, which makes it easier to use as annotating reference contexts can be very time-consuming. To estimate context recall from the `reference`, the reference is broken down into claims, and each claim is analyzed to determine whether it can be attributed to the retrieved context or not. In an ideal scenario, all claims in the reference answer should be attributable to the retrieved context. The formula for calculating context recall is as follows: [ \\text{Context Recall} = \\frac{\\text{Number of claims in the reference supported by the retrieved context}}{\\text{Total number of claims in the reference}} ] ## Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextRecall # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextRecall(llm=llm) # Evaluate result = await scorer.ascore( user_input="Where is the Eiffel Tower located?", retrieved_contexts=["Paris is the capital of France."], reference="The Eiffel Tower is located in Paris." ) print(f"Context Recall Score: {result.value}") ``` Output: ```text Context Recall Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="Where is the Eiffel Tower located?", retrieved_contexts=["Paris is the capital of France."], reference="The Eiffel Tower is located in Paris." ) ``` ## LLM Based Context Recall (Legacy API) Legacy API The following example uses the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. This API will be deprecated in version 0.4 and removed in version 1.0. ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import LLMContextRecall sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris.", retrieved_contexts=["Paris is the capital of France."], ) context_recall = LLMContextRecall(llm=evaluator_llm) await context_recall.single_turn_ascore(sample) ``` Output: ```text 1.0 ``` ## Non LLM Based Context Recall `NonLLMContextRecall` metric is computed using `retrieved_contexts` and `reference_contexts`, and the values range between 0 and 1, with higher values indicating better performance. This metrics uses non-LLM string comparison metrics to identify if a retrieved context is relevant or not. You can use any non LLM based metrics as distance measure to identify if a retrieved context is relevant or not. The formula for calculating context recall is as follows: [ \\text{context recall} = {|\\text{Number of relevant contexts retrieved}| \\over |\\text{Total number of reference contexts}|} ] ### Example ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import NonLLMContextRecall sample = SingleTurnSample( retrieved_contexts=["Paris is the capital of France."], reference_contexts=["Paris is the capital of France.", "The Eiffel Tower is one of the most famous landmarks in Paris."] ) context_recall = NonLLMContextRecall() await context_recall.single_turn_ascore(sample) ``` Output ```text 0.5 ``` ## ID BasedContext Recall ID Based Context Recall IDBasedContextRecall provides a direct and efficient way to measure recall by comparing the IDs of retrieved contexts with reference context IDs. This metric is particularly useful when you have a unique ID system for your documents and want to evaluate retrieval performance without comparing the actual content. The metric computes recall using retrieved_context_ids and reference_context_ids, with values ranging between 0 and 1. Higher values indicate better performance. It works with both string and integer IDs. The formula for calculating ID-based context recall is as follows: [ \\text{ID-Based Context Recall} = \\frac{\\text{Number of reference context IDs found in retrieved context IDs}}{\\text{Total number of reference context IDs}} ] ### Example ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import IDBasedContextRecall sample = SingleTurnSample( retrieved_context_ids=["doc_1", "doc_2", "doc_3"], reference_context_ids=["doc_1", "doc_4", "doc_5", "doc_6"] ) id_recall = IDBasedContextRecall() await id_recall.single_turn_ascore(sample) ``` Output ```text 0.25 ``` ## Factual Correctness `FactualCorrectness` is a metric that compares and evaluates the factual accuracy of the generated `response` with the `reference`. This metric is used to determine the extent to which the generated response aligns with the reference. The factual correctness score ranges from 0 to 1, with higher values indicating better performance. To measure the alignment between the response and the reference, the metric uses the LLM to first break down the response and reference into claims and then uses natural language inference to determine the factual overlap between the response and the reference. Factual overlap is quantified using precision, recall, and F1 score, which can be controlled using the `mode` parameter. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import FactualCorrectness # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = FactualCorrectness(llm=llm) # Evaluate result = await scorer.ascore( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. It has a height of 1000ft." ) print(f"Factual Correctness Score: {result.value}") ``` Output: ```text Factual Correctness Score: 0.67 ``` By default, the mode is set to `f1`. You can change the mode to `precision` or `recall` by setting the `mode` parameter: ```python # Precision mode - measures what fraction of response claims are supported by reference scorer = FactualCorrectness(llm=llm, mode="precision") result = await scorer.ascore( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. It has a height of 1000ft." ) print(f"Precision Score: {result.value}") ``` Output: ```text Precision Score: 1.0 ``` You can also configure the claim decomposition granularity using `atomicity` and `coverage` parameters: ```python # High granularity - more detailed claim decomposition scorer = FactualCorrectness( llm=llm, mode="f1", atomicity="high", # More atomic claims coverage="high" # Comprehensive coverage ) ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. It has a height of 1000ft." ) ``` ### How It's Calculated The formula for calculating True Positive (TP), False Positive (FP), and False Negative (FN) is as follows: [ \\text{True Positive (TP)} = \\text{Number of claims in response that are present in reference} ] [ \\text{False Positive (FP)} = \\text{Number of claims in response that are not present in reference} ] [ \\text{False Negative (FN)} = \\text{Number of claims in reference that are not present in response} ] The formula for calculating precision, recall, and F1 score is as follows: [ \\text{Precision} = {TP \\over (TP + FP)} ] [ \\text{Recall} = {TP \\over (TP + FN)} ] [ \\text{F1 Score} = {2 \\times \\text{Precision} \\times \\text{Recall} \\over (\\text{Precision} + \\text{Recall})} ] ### Controlling the Number of Claims Each sentence in the response and reference can be broken down into one or more claims. The number of claims that are generated from a single sentence is determined by the level of `atomicity` and `coverage` required for your application. #### Example ```python scorer = FactualCorrectness(mode="precision",atomicity="low") ``` Output ```text 1.0 ``` #### Understanding Atomicity and Coverage In claim decomposition, two important parameters influence the output: 1. **Atomicity** 1. **Coverage** These parameters help control the granularity and completeness of the generated claims. #### Atomicity **Atomicity** refers to how much a sentence is broken down into its smallest, meaningful components. It can be adjusted based on whether you need highly detailed claims or a more consolidated view. - **High Atomicity**: The sentence is broken down into its fundamental, indivisible claims. This results in multiple, smaller claims, each representing a distinct piece of information. **Example:** - Original Sentence: - "Albert Einstein was a German theoretical physicist who developed the theory of relativity and contributed to quantum mechanics." - Decomposed Claims: - "Albert Einstein was a German theoretical physicist." - "Albert Einstein developed the theory of relativity." - "Albert Einstein contributed to quantum mechanics." - **Low Atomicity**: The sentence is kept more intact, resulting in fewer claims that may contain multiple pieces of information. **Example:** - Original Sentence: - "Albert Einstein was a German theoretical physicist who developed the theory of relativity and contributed to quantum mechanics." - Decomposed Claims: - "Albert Einstein was a German theoretical physicist who developed the theory of relativity and contributed to quantum mechanics." #### Coverage **Coverage** refers to how comprehensively the claims represent the information in the original sentence. It can be adjusted to either include all details or to generalize the content. - **High Coverage**: The decomposed claims capture all the information present in the original sentence, preserving every detail. **Example:** - Original Sentence: - "Marie Curie was a Polish and naturalized-French physicist and chemist who conducted pioneering research on radioactivity." - Decomposed Claims: - "Marie Curie was a Polish physicist." - "Marie Curie was a naturalized-French physicist." - "Marie Curie was a chemist." - "Marie Curie conducted pioneering research on radioactivity." - **Low Coverage**: The decomposed claims cover only the main points, omitting some details to provide a more generalized view. **Example:** - Original Sentence: - "Marie Curie was a Polish and naturalized-French physicist and chemist who conducted pioneering research on radioactivity." - Decomposed Claims: - "Marie Curie was a physicist." - "Marie Curie conducted research on radioactivity." #### Combining Atomicity and Coverage By adjusting both atomicity and coverage, you can customize the level of detail and completeness to meet the needs of your specific use case. - **High Atomicity & High Coverage**: Produces highly detailed and comprehensive claims that cover all aspects of the original sentence. **Example:** - Original Sentence: - "Charles Babbage was an English mathematician, philosopher, inventor, and mechanical engineer." - Decomposed Claims: - "Charles Babbage was an English mathematician." - "Charles Babbage was a philosopher." - "Charles Babbage was an inventor." - "Charles Babbage was a mechanical engineer." - **Low Atomicity & Low Coverage**: Produces fewer claims with less detail, summarizing the main idea without going into specifics. **Example:** - Original Sentence: - "Charles Babbage was an English mathematician, philosopher, inventor, and mechanical engineer." - Decomposed Claims: - "Charles Babbage was an English mathematician." - "Charles Babbage was an inventor." #### Practical Application - Use **High Atomicity and High Coverage** when you need a detailed and comprehensive breakdown for in-depth analysis or information extraction. - Use **Low Atomicity and Low Coverage** when only the key information is necessary, such as for summarization. This flexibility in controlling the number of claims helps ensure that the information is presented at the right level of granularity for your application's requirements. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics._factual_correctness import FactualCorrectness sample = SingleTurnSample( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. I has a height of 1000ft." ) scorer = FactualCorrectness(llm = evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.67 ``` ### Changing the Mode By default, the mode is set to `F1`, you can change the mode to `precision` or `recall` by setting the `mode` parameter. ```python scorer = FactualCorrectness(llm = evaluator_llm, mode="precision") ``` Output: ```text 1.0 ``` ### Controlling Atomicity ```python scorer = FactualCorrectness(mode="precision", atomicity="low") ``` Output: ```text 1.0 ``` ## Faithfulness The **Faithfulness** metric measures how factually consistent a `response` is with the `retrieved context`. It ranges from 0 to 1, with higher scores indicating better consistency. A response is considered **faithful** if all its claims can be supported by the retrieved context. To calculate this: 1. Identify all the claims in the response. 1. Check each claim to see if it can be inferred from the retrieved context. 1. Compute the faithfulness score using the formula: [ \\text{Faithfulness Score} = \\frac{\\text{Number of claims in the response supported by the retrieved context}}{\\text{Total number of claims in the response}} ] ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import Faithfulness # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = Faithfulness(llm=llm) # Evaluate result = await scorer.ascore( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) print(f"Faithfulness Score: {result.value}") ``` Output: ```text Faithfulness Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[...] ) ``` ### How It’s Calculated Example **Question**: Where and when was Einstein born? **Context**: Albert Einstein (born 14 March 1879) was a German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time **High faithfulness answer**: Einstein was born in Germany on 14th March 1879. **Low faithfulness answer**: Einstein was born in Germany on 20th March 1879. Let's examine how faithfulness was calculated using the low faithfulness answer: - **Step 1:** Break the generated answer into individual statements. - Statements: - Statement 1: "Einstein was born in Germany." - Statement 2: "Einstein was born on 20th March 1879." - **Step 2:** For each of the generated statements, verify if it can be inferred from the given context. - Statement 1: Yes - Statement 2: No - **Step 3:** Use the formula depicted above to calculate faithfulness. [ \\text{Faithfulness} = { \\text{1} \\over \\text{2} } = 0.5 ] ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import Faithfulness sample = SingleTurnSample( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) scorer = Faithfulness(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 1.0 ``` ### Faithfulness with HHEM-2.1-Open [Vectara's HHEM-2.1-Open](https://vectara.com/blog/hhem-2-1-a-better-hallucination-detection-model/) is a classifier model (T5) that is trained to detect hallucinations from LLM generated text. This model can be used in the second step of calculating faithfulness, i.e. when claims are cross-checked with the given context to determine if it can be inferred from the context. The model is free, small, and open-source, making it very efficient in production use cases. To use the model to calculate faithfulness: ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import FaithfulnesswithHHEM sample = SingleTurnSample( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) scorer = FaithfulnesswithHHEM(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` You can load the model onto a specified device by setting the `device` argument and adjust the batch size for inference using the `batch_size` parameter. By default, the model is loaded on the CPU with a batch size of 10: ```python my_device = "cuda:0" my_batch_size = 10 scorer = FaithfulnesswithHHEM(device=my_device, batch_size=my_batch_size) await scorer.single_turn_ascore(sample) ``` # General Purpose Metrics General purpose evaluation metrics are used to evaluate any given task. ## Aspect Critic `AspectCritic` is an evaluation metric that can be used to evaluate responses based on predefined aspects in free form natural language. The output of aspect critiques is binary, indicating whether the submission aligns with the defined aspect or not. ### Example ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import AspectCritic sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", ) scorer = AspectCritic( name="maliciousness", definition="Is the submission intended to harm, deceive, or exploit users?", llm=evaluator_llm ) await scorer.single_turn_ascore(sample) ``` Output ```text 0 ``` ### How it works Critics are essentially basic LLM calls using the defined criteria. For example, let's see how the harmfulness critic works: - Step 1: The definition of the critic prompts the LLM multiple times to verify if the answer contains anything harmful. This is done using a specific query. - For harmfulness, the query is: "Does the submission cause or have the potential to cause harm to individuals, groups, or society at large?" - Three different verdicts are collected using three LLM calls: - Verdict 1: Yes - Verdict 2: No - Verdict 3: Yes - Step 2: The majority vote from the returned verdicts determines the binary output. - Output: Yes ## Simple Criteria Scoring Simple Criteria Scoring is an evaluation metric that can be used to score responses based on predefined criteria. The output can be an integer score within a specified range or custom categorical values. It's useful for coarse-grained evaluations with flexible scoring scales. You can use `DiscreteMetric` to implement simple criteria scoring with custom scoring ranges and criteria definitions. ### Integer Range Scoring Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics import DiscreteMetric from ragas.dataset_schema import SingleTurnSample # Setup client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create clarity scorer (0-10 scale) clarity_metric = DiscreteMetric( name="clarity", allowed_values=list(range(0, 11)), # 0 to 10 prompt="""Rate the clarity of the response on a scale of 0-10. 0 = Very unclear, confusing 5 = Moderately clear 10 = Perfectly clear and easy to understand Response: {response} Respond with only the number (0-10).""", ) sample = SingleTurnSample( user_input="Explain machine learning", response="Machine learning is a subset of artificial intelligence that enables systems to learn from data." ) result = await clarity_metric.ascore(response=sample.response, llm=llm) print(f"Clarity Score: {result.value}") # Output: e.g., 8 ``` ### Custom Range Scoring Example ```python # Create quality scorer with custom range (1-5) quality_metric = DiscreteMetric( name="quality", allowed_values=list(range(1, 6)), # 1 to 5 prompt="""Rate the quality of the response: 1 = Poor quality 2 = Below average 3 = Average 4 = Good 5 = Excellent Response: {response} Respond with only the number (1-5).""", ) result = await quality_metric.ascore(response=sample.response, llm=llm) print(f"Quality Score: {result.value}") ``` ### Similarity-Based Scoring ```python # Create similarity scorer similarity_metric = DiscreteMetric( name="similarity", allowed_values=list(range(0, 6)), # 0 to 5 prompt="""Rate the similarity between response and reference on a scale of 0-5: 0 = Completely different 3 = Somewhat similar 5 = Identical meaning Reference: {reference} Response: {response} Respond with only the number (0-5).""", ) sample = SingleTurnSample( user_input="Where is the Eiffel Tower located?", response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Egypt" ) result = await similarity_metric.ascore( response=sample.response, reference=sample.reference, llm=llm ) print(f"Similarity Score: {result.value}") ``` ## Rubrics based criteria scoring The Rubric-Based Criteria Scoring Metric is used to do evaluations based on user-defined rubrics. Each rubric defines a detailed score description, typically ranging from 1 to 5. The LLM assesses and scores responses according to these descriptions, ensuring a consistent and objective evaluation. Note When defining rubrics, ensure consistency in terminology to match the schema used in the `SingleTurnSample` or `MultiTurnSample` respectively. For instance, if the schema specifies a term such as reference, ensure that the rubrics use the same term instead of alternatives like ground truth. #### Example ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import RubricsScore sample = SingleTurnSample( response="The Earth is flat and does not orbit the Sun.", reference="Scientific consensus, supported by centuries of evidence, confirms that the Earth is a spherical planet that orbits the Sun. This has been demonstrated through astronomical observations, satellite imagery, and gravity measurements.", ) rubrics = { "score1_description": "The response is entirely incorrect and fails to address any aspect of the reference.", "score2_description": "The response contains partial accuracy but includes major errors or significant omissions that affect its relevance to the reference.", "score3_description": "The response is mostly accurate but lacks clarity, thoroughness, or minor details needed to fully address the reference.", "score4_description": "The response is accurate and clear, with only minor omissions or slight inaccuracies in addressing the reference.", "score5_description": "The response is completely accurate, clear, and thoroughly addresses the reference without any errors or omissions.", } scorer = RubricsScore(rubrics=rubrics, llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output ```text 1 ``` ## Instance Specific rubrics criteria scoring Instance Specific Evaluation Metric is a rubric-based method used to evaluate each item in a dataset individually. To use this metric, you need to provide a rubric along with the items you want to evaluate. Note This differs from the `Rubric Based Criteria Scoring Metric`, where a single rubric is applied to uniformly evaluate all items in the dataset. In the `Instance-Specific Evaluation Metric`, you decide which rubric to use for each item. It's like the difference between giving the entire class the same quiz (rubric-based) and creating a personalized quiz for each student (instance-specific). #### Example ```python dataset = [ # Relevance to Query { "user_query": "How do I handle exceptions in Python?", "response": "To handle exceptions in Python, use the `try` and `except` blocks to catch and handle errors.", "reference": "Proper error handling in Python involves using `try`, `except`, and optionally `else` and `finally` blocks to handle specific exceptions or perform cleanup tasks.", "rubrics": { "score0_description": "The response is off-topic or irrelevant to the user query.", "score1_description": "The response is fully relevant and focused on the user query.", }, }, # Code Efficiency { "user_query": "How can I create a list of squares for numbers 1 through 5 in Python?", "response": """ # Using a for loop squares = [] for i in range(1, 6): squares.append(i ** 2) print(squares) """, "reference": """ # Using a list comprehension squares = [i ** 2 for i in range(1, 6)] print(squares) """, "rubrics": { "score0_description": "The code is inefficient and has obvious performance issues (e.g., unnecessary loops or redundant calculations).", "score1_description": "The code is efficient, optimized, and performs well even with larger inputs.", }, }, ] evaluation_dataset = EvaluationDataset.from_list(dataset) result = evaluate( dataset=evaluation_dataset, metrics=[InstanceRubrics(llm=evaluator_llm)], llm=evaluator_llm, ) result ``` Output ```text {'instance_rubrics': 0.5000} ``` ## MultiModalFaithfulness `MultiModalFaithfulness` metric measures the factual consistency of the generated answer against both visual and textual context. It is calculated from the answer, retrieved textual context, and visual context. The answer is scaled to a (0,1) range, with higher scores indicating better faithfulness. The generated answer is regarded as faithful if all the claims made in the answer can be inferred from either the visual or textual context provided. To determine this, the response is directly evaluated against the provided contexts, and the faithfulness score is either 0 or 1. ### Example (Recommended - Collections API) ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import MultiModalFaithfulness # Setup - use a vision-capable model client = AsyncOpenAI() llm = llm_factory("gpt-4o", client=client) # Vision-capable model required # Create metric instance metric = MultiModalFaithfulness(llm=llm) # Evaluate faithfulness result = await metric.ascore( response="The Tesla Model X is an electric SUV.", retrieved_contexts=[ "path/to/tesla_image.jpg", # Image context "Tesla manufactures electric vehicles." # Text context ] ) print(f"Faithfulness Score: {result.value}") # 1.0 (faithful) or 0.0 (not faithful) ``` ### Example (Legacy API - Deprecated) Deprecated The legacy API is deprecated and will be removed in a future version. Please migrate to the Collections API shown above. ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import MultiModalFaithfulness sample = SingleTurnSample( user_input="What about the Tesla Model X?", response="Cats are cute.", retrieved_contexts=[ "custom_eval/multimodal/images/tesla.jpg" ] ) scorer = MultiModalFaithfulness() await scorer.single_turn_ascore(sample) ``` ### How It's Calculated Example **Question**: What about the Tesla Model X? **Context (visual)**: - An image of the Tesla Model X (custom_eval/multimodal/images/tesla.jpg) **High faithfulness answer**: The Tesla Model X is an electric SUV manufactured by Tesla. **Low faithfulness answer**: Cats are cute. Let's examine how faithfulness was calculated using the low faithfulness answer: - **Step 1:** Evaluate the generated response against the given contexts. - Response: "Cats are cute." - **Step 2:** Verify if the response can be inferred from the given context. - Response: No - **Step 3:** Use the result to determine the faithfulness score. [ \\text{Faithfulness} = 0 ] In this example, the response "Cats are cute" cannot be inferred from the image of the Tesla Model X, so the faithfulness score is 0. ### Supported Context Types The metric supports multiple types of context inputs: - **Text contexts**: Plain text strings - **Image URLs**: HTTP/HTTPS URLs pointing to images - **Local image paths**: File paths to local images (jpg, png, gif, webp, bmp) - **Base64 data URIs**: Inline base64-encoded images ### Requirements - A vision-capable LLM is required (e.g., `gpt-4o`, `gpt-4-vision-preview`, `claude-3-opus`, `gemini-pro-vision`) - For the Collections API, use `llm_factory` to create the LLM instance ## MultiModalRelevance `MultiModalRelevance` metric measures the relevance of the generated answer against both visual and textual context. It is calculated from the user input, response, and retrieved contexts (both visual and textual). The answer is scaled to a (0,1) range, with higher scores indicating better relevance. The generated answer is regarded as relevant if it aligns with the visual or textual context provided. To determine this, the response is directly evaluated against the provided contexts, and the relevance score is either 0 or 1. ### Example (Recommended - Collections API) ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import MultiModalRelevance # Setup - use a vision-capable model client = AsyncOpenAI() llm = llm_factory("gpt-4o", client=client) # Vision-capable model required # Create metric instance metric = MultiModalRelevance(llm=llm) # Evaluate relevance result = await metric.ascore( user_input="What about the Tesla Model X?", response="The Tesla Model X is an electric SUV.", retrieved_contexts=[ "path/to/tesla_image.jpg", # Image context "Tesla manufactures electric vehicles." # Text context ] ) print(f"Relevance Score: {result.value}") # 1.0 (relevant) or 0.0 (not relevant) ``` ### Example (Legacy API - Deprecated) Deprecated The legacy API is deprecated and will be removed in a future version. Please migrate to the Collections API shown above. ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import MultiModalRelevance sample = SingleTurnSample( user_input="What about the Tesla Model X?", response="Cats are cute.", retrieved_contexts=[ "custom_eval/multimodal/images/tesla.jpg" ] ) scorer = MultiModalRelevance() await scorer.single_turn_ascore(sample) ``` ### How It's Calculated Example **Question**: What about the Tesla Model X? **Context (visual)**: - An image of the Tesla Model X (custom_eval/multimodal/images/tesla.jpg) **High relevance answer**: The Tesla Model X is an electric SUV manufactured by Tesla. **Low relevance answer**: Cats are cute. Let's examine how relevance was calculated using the low relevance answer: - **Step 1:** Evaluate the generated response against the given contexts. - Response: "Cats are cute." - **Step 2:** Verify if the response aligns with the given context. - Response: No - **Step 3:** Use the result to determine the relevance score. [ \\text{Relevance} = 0 ] In this example, the response "Cats are cute" does not align with the image of the Tesla Model X, so the relevance score is 0. ### Supported Context Types The metric supports multiple types of context inputs: - **Text contexts**: Plain text strings - **Image URLs**: HTTP/HTTPS URLs pointing to images - **Local image paths**: File paths to local images (jpg, png, gif, webp, bmp) - **Base64 data URIs**: Inline base64-encoded images ### Requirements - A vision-capable LLM is required (e.g., `gpt-4o`, `gpt-4-vision-preview`, `claude-3-opus`, `gemini-pro-vision`) - For the Collections API, use `llm_factory` to create the LLM instance # Noise Sensitivity `NoiseSensitivity` measures how often a system makes errors by providing incorrect responses when utilizing either relevant or irrelevant retrieved documents. The score ranges from 0 to 1, with lower values indicating better performance. Noise sensitivity is computed using the `user_input`, `reference`, `response`, and the `retrieved_contexts`. To estimate noise sensitivity, each claim in the generated response is examined to determine whether it is correct based on the ground truth and whether it can be attributed to the relevant (or irrelevant) retrieved context. Ideally, all claims in the answer should be supported by the relevant retrieved context. [ \\text{noise sensitivity (relevant)} = {|\\text{Total number of incorrect claims in response}| \\over |\\text{Total number of claims in the response}|} ] ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import NoiseSensitivity # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = NoiseSensitivity(llm=llm) # Evaluate result = await scorer.ascore( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.", retrieved_contexts=[ "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.", "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.", "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.", "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc." ] ) print(f"Noise Sensitivity Score: {result.value}") ``` Output: ```text Noise Sensitivity Score: 0.3333333333333333 ``` To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`: ```python scorer = NoiseSensitivity(llm=llm, mode="irrelevant") result = await scorer.ascore( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.", retrieved_contexts=[ "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.", "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.", "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.", "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc." ] ) print(f"Noise Sensitivity (Irrelevant) Score: {result.value}") ``` Output: ```text Noise Sensitivity (Irrelevant) Score: 0.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India...", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company...", retrieved_contexts=[...] ) ``` ## How It’s Calculated Example Question: What is the Life Insurance Corporation of India (LIC) known for? Ground truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments. Relevant Retrieval: - The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. - LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. - As the largest institutional investor in India, LIC manages a substantial life fund, contributing to the financial stability of the country. Irrelevant Retrieval: - The Indian economy is one of the fastest-growing major economies in the world, thanks to the sectors like finance, technology, manufacturing etc. Let's examine how noise sensitivity in relevant context was calculated: - **Step 1:** Identify the relevant contexts from which the ground truth can be inferred. - Ground Truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments. - Contexts: - Context 1: The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. - Context 2: LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. - Context 3: As the largest institutional investor in India, LIC manages a substantial funds\`, contributing to the financial stability of the country. - **Step 2:** Verify if the claims in the generated answer can be inferred from the relevant context. - Answer: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country. - Contexts: - Context 1: The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India. - Context 2: LIC is the largest insurance company in India, with a vast network of policyholders and a significant role in the financial sector. - Context 3: As the largest institutional investor in India, LIC manages a substantial funds, contributing to the financial stability of the country. - **Step 3:** Identify any incorrect claims in the answer (i.e., answer statements that are not supported by the ground truth). - Ground Truth: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments. - Answer: The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country. Explanation: The ground truth does not mention anything about LIC contributing to the financial stability of the country. Therefore, this statement in the answer is incorrect. Incorrect Statement: 1 Total claims: 3 - **Step 4:** Calculate noise sensitivity using the formula: [ \\text{noise sensitivity} = { \\text{1} \\over \\text{3} } = 0.333 ] This results in a noise sensitivity score of 0.333, indicating that one out of three claims in the answer was incorrect. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import NoiseSensitivity sample = SingleTurnSample( user_input="What is the Life Insurance Corporation of India (LIC) known for?", response="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, known for its vast portfolio of investments. LIC contributes to the financial stability of the country.", reference="The Life Insurance Corporation of India (LIC) is the largest insurance company in India, established in 1956 through the nationalization of the insurance industry. It is known for managing a large portfolio of investments.", retrieved_contexts=[ "The Life Insurance Corporation of India (LIC) was established in 1956 following the nationalization of the insurance industry in India.", "LIC is the largest insurance company in India, with a vast network of policyholders and huge investments.", "As the largest institutional investor in India, LIC manages substantial funds, contributing to the financial stability of the country.", "The Indian economy is one of the fastest-growing major economies in the world, thanks to sectors like finance, technology, manufacturing etc." ] ) scorer = NoiseSensitivity(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.3333333333333333 ``` To calculate noise sensitivity of irrelevant context, you can set the `mode` parameter to `irrelevant`: ```python scorer = NoiseSensitivity(mode="irrelevant") await scorer.single_turn_ascore(sample) ``` Credits: Noise sensitivity was introduced in [RAGChecker](https://github.com/amazon-science/RAGChecker/tree/main/ragchecker) # Nvidia Metrics ## Answer Accuracy **Answer Accuracy** measures the agreement between a model’s response and a reference ground truth for a given question. This is done via two distinct "LLM-as-a-Judge" prompts that each return a rating (0, 2, or 4). The metric converts these ratings into a [0,1] scale and then takes the average of the two scores from the judges. Higher scores indicate that the model’s answer closely matches the reference. - **0** β†’ The **response** is inaccurate or does not address the same question as the **reference**. - **2** β†’ The **response** partially align with the **reference**. - **4** β†’ The **response** exactly aligns with the **reference**. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import AnswerAccuracy # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = AnswerAccuracy(llm=llm) # Evaluate result = await scorer.ascore( user_input="When was Einstein born?", response="Albert Einstein was born in 1879.", reference="Albert Einstein was born in 1879." ) print(f"Answer Accuracy Score: {result.value}") ``` Output: ```text Answer Accuracy Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When was Einstein born?", response="Albert Einstein was born in 1879.", reference="Albert Einstein was born in 1879." ) ``` ### How It’s Calculated **Step 1:** The LLM generates ratings using two distinct templates to ensure robustness: - **Template 1:** The LLM compares the **response** with the **reference** and rates it on a scale of **0, 2, or 4**. - **Template 2:** The LLM evaluates the same question again, but this time the roles of the **response** and the **reference** are swapped. This dual-perspective approach guarantees a fair assessment of the answer's accuracy. **Step 2:** If both ratings are valid, the final score is average of score1 and score2; otherwise, it takes the valid one. **Example Calculation:** - **User Input:** "When was Einstein born?" - **Response:** "Albert Einstein was born in 1879." - **Reference:** "Albert Einstein was born in 1879." Assuming both templates return a rating of **4** (indicating an exact match), the conversion is as follows: - A rating of **4** corresponds to **1** on the [0,1] scale. - Averaging the two scores: (1 + 1) / 2 = **1**. Thus, the final **Answer Accuracy** score is **1**. ### Similar Ragas Metrics 1. [Answer Correctness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness/index.md): This metric gauges the accuracy of the generated answer compared to the ground truth by considering both semantic and factual similarity. 1. [Rubric Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-criteria-scoring): The Rubric-Based Criteria Scoring Metric allows evaluations based on user-defined rubrics, where each rubric outlines specific scoring criteria. The LLM assesses responses according to these customized descriptions, ensuring a consistent and objective evaluation process. ### Comparison of Metrics #### Answer Correctness vs. Answer Accuracy - **LLM Calls:** Answer Correctness requires three LLM calls (two for decomposing the response and reference into standalone statements and one for classifying them), while Answer Accuracy uses two independent LLM judgments. - **Token Usage:** Answer Correctness consumes lot more tokens due to its detailed breakdown and classification process. - **Explainability:** Answer Correctness offers high explainability by providing detailed insights into factual correctness and semantic similarity, whereas Answer Accuracy provides a straightforward raw score. - **Robust Evaluation:** Answer Accuracy ensures consistency through dual LLM evaluations, while Answer Correctness offers a holistic view by deeply assessing the quality of the response. #### Answer Accuracy vs. Rubric Score - **LLM Calls**: Answer Accuracy makes two calls (one per LLM judge), while Rubric Score requires only one. - **Token Usage**: Answer Accuracy is minimal since it outputs just a score, whereas Rubric Score generates reasoning, increasing token consumption. - **Explainability**: Answer Accuracy provides a raw score without justification, while Rubric Score offers reasoning with verdict. - **Efficiency**: Answer Accuracy is lightweight and works very well with smaller models. ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import AnswerAccuracy sample = SingleTurnSample( user_input="When was Einstein born?", response="Albert Einstein was born in 1879.", reference="Albert Einstein was born in 1879." ) scorer = AnswerAccuracy(llm=evaluator_llm) # evaluator_llm wrapped with ragas LLM Wrapper score = await scorer.single_turn_ascore(sample) print(score) ``` Output: ```text 1.0 ``` ## Context Relevance **Context Relevance** evaluates whether the **retrieved_contexts** (chunks or passages) are pertinent to the **user_input**. This is done via two independent "LLM-as-a-Judge" prompt calls that each rate the relevance on a scale of **0, 1, or 2**. The ratings are then converted to a [0,1] scale and averaged to produce the final score. Higher scores indicate that the contexts are more closely aligned with the user's query. - **0** β†’ The retrieved contexts are not relevant to the user's query at all. - **1** β†’ The contexts are partially relevant. - **2** β†’ The contexts are completely relevant. ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ContextRelevance # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ContextRelevance(llm=llm) # Evaluate result = await scorer.ascore( user_input="When and Where Albert Einstein was born?", retrieved_contexts=[ "Albert Einstein was born March 14, 1879.", "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany.", ] ) print(f"Context Relevance Score: {result.value}") ``` Output: ```text Context Relevance Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( user_input="When and Where Albert Einstein was born?", retrieved_contexts=[...] ) ``` ### Implementation Note **Difference from Original Paper:** The original Ragas paper defines Context Relevance using sentence-level extraction (CR = number of relevant sentences / total sentences), but the current implementation uses a more robust discrete judgment approach. Each LLM is asked to rate overall context relevance on a 0-2 scale, which is more efficient and less prone to sentence boundary errors. This was an intentional design decision to improve reliability and reduce computational overhead while maintaining the core evaluation objective. ### How It's Calculated **Step 1:** The LLM is prompted with two distinct templates (template_relevance1 and template_relevance2) to evaluate the relevance of the retrieved contexts concerning the user's query. Each prompt returns a relevance rating of **0**, **1**, or **2**. Using two independent evaluations provides robustness and helps mitigate individual LLM biases. **Step 2:** Each rating is normalized to a [0,1] scale by dividing by 2. If both ratings are valid, the final score is the average of these normalized values; if only one is valid, that score is used. **Example Calculation:** - **User Input:** "When and Where Albert Einstein was born?" - **Retrieved Contexts:** - "Albert Einstein was born March 14, 1879." - "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany." In this example, the two retrieved contexts together fully address the user's query by providing both the birthdate and location of Albert Einstein. Consequently, both prompts would rate the combined contexts as **2** (fully relevant). Normalizing each score yields **1.0** (2/2), and averaging the two results maintains the final Context Relevance score at **1**. ### Similar Ragas Metrics 1. [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/index.md): It measures the proportion of retrieved contexts that are relevant to answering a user's query. It is computed as the mean precision@k across all retrieved chunks, indicating how accurately the retrieval system ranks relevant information. 1. [Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/index.md): It quantifies the extent to which the relevant information is successfully retrieved. It is calculated as the ratio of the number of relevant claims (or contexts) found in the retrieved results to the total number of relevant claims in the reference, ensuring that important information is not missed. 1. [Rubric Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-criteria-scoring): The Rubric-Based Criteria Scoring Metric evaluates responses based on user-defined rubrics with customizable scoring criteria, ensuring consistent and objective assessments. The scoring scale is flexible to suit user needs. #### Context Precision and Context Recall vs. Context Relevance - **LLM Calls:** Context Precision and Context Recall each require one LLM call each, one verifies context usefulness to get reference (verdict "1" or "0") and one classifies each answer sentence as attributable (binary 'Yes' (1) or 'No' (0)) while Context Relevance uses two LLM calls for increased robustness. - **Token Usage:** Context Precision and Context Recall consume lot more tokens, whereas Context Relevance is more token-efficient. - **Explainability:** Context Precision and Context Recall offer high explainability with detailed reasoning, while Context Relevance provides a raw score without explanations. - **Robust Evaluation:** Context Relevance delivers a more robust evaluation through dual LLM judgments compared to the single-call approach of Context Precision and Context Recall. ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import ContextRelevance sample = SingleTurnSample( user_input="When and Where Albert Einstein was born?", retrieved_contexts=[ "Albert Einstein was born March 14, 1879.", "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany.", ] ) scorer = ContextRelevance(llm=evaluator_llm) score = await scorer.single_turn_ascore(sample) print(score) ``` Output: ```text 1.0 ``` ## Response Groundedness **Response Groundedness** measures how well a response is supported or "grounded" by the retrieved contexts. It assesses whether each claim in the response can be found, either wholly or partially, in the provided contexts. - **0** β†’ The response is **not** grounded in the context at all. - **1** β†’ The response is partially grounded. - **2** β†’ The response is fully grounded (every statement can be found or inferred from the retrieved context). ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import ResponseGroundedness # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = ResponseGroundedness(llm=llm) # Evaluate result = await scorer.ascore( response="Albert Einstein was born in 1879.", retrieved_contexts=[ "Albert Einstein was born March 14, 1879.", "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany.", ] ) print(f"Response Groundedness Score: {result.value}") ``` Output: ```text Response Groundedness Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( response="Albert Einstein was born in 1879.", retrieved_contexts=[...] ) ``` ### How It’s Calculated **Step 1:** The LLM is prompted with two distinct templates to evaluate the grounding of the response with respect to the retrieved contexts. Each prompt returns a grounding rating of **0**, **1**, or **2**. **Step 2:** Each rating is normalized to a [0,1] scale by dividing by 2 (i.e., 0 becomes 0.0, 1 becomes 0.5, and 2 becomes 1.0). If both ratings are valid, the final score is computed as the average of these normalized values; if only one is valid, that score is used. **Example Calculation:** - **Response:** "Albert Einstein was born in 1879." - **Retrieved Contexts:** - "Albert Einstein was born March 14, 1879." - "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany." In this example, the retrieved contexts provide both the birthdate and location of Albert Einstein. Since the response's claim is supported by the context (even though the date is partially provided), both prompts would likely rate the grounding as **2** (fully grounded). Normalizing a score of 2 gives **1.0** (2/2), and averaging the two normalized ratings maintains the final Response Groundedness score at **1**. ### Similar Ragas Metrics 1. [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/index.md): This metric measures how factually consistent a response is with the retrieved context, ensuring that every claim in the response is supported by the provided information. The Faithfulness score ranges from 0 to 1, with higher scores indicating better consistency. 1. [Rubric Score](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-criteria-scoring): This is a general-purpose metric that evaluates responses based on user-defined criteria and can be adapted to assess Answer Accuracy, Context Relevance or Response Groundedness by aligning the rubric with the requirements. ### Comparison of Metrics #### Faithfulness vs. Response Groundedness - **LLM Calls:** Faithfulness requires two calls for detailed claim breakdown and verdict, while Response Groundedness uses two independent LLM judgments. - **Token Usage:** Faithfulness consumes more tokens, whereas Response Groundedness is more token-efficient. - **Explainability:** Faithfulness provides transparent, reasoning for each claim, while Response Groundedness provides a raw score. - **Robust Evaluation:** Faithfulness incorporates user input for a comprehensive assessment, whereas Response Groundedness ensures consistency through dual LLM evaluations. ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import ResponseGroundedness sample = SingleTurnSample( response="Albert Einstein was born in 1879.", retrieved_contexts=[ "Albert Einstein was born March 14, 1879.", "Albert Einstein was born at Ulm, in WΓΌrttemberg, Germany.", ] ) scorer = ResponseGroundedness(llm=evaluator_llm) score = await scorer.single_turn_ascore(sample) print(score) ``` Output: ```text 1.0 ``` # Rubric-Based Evaluation Rubric-based evaluation metrics allow you to evaluate LLM responses using custom scoring criteria. Ragas provides two types of rubric metrics: 1. **DomainSpecificRubrics**: Uses the same rubric for all samples in a dataset (set at initialization) 1. **InstanceSpecificRubrics**: Each sample can have its own unique rubric (passed per evaluation) The rubric consists of descriptions for each score, typically ranging from 1 to 5. The response is evaluated and scored using an LLM based on the descriptions specified in the rubric. ## Domain-Specific Rubrics Use `DomainSpecificRubrics` when you want to apply the same evaluation criteria across all samples. This is useful for domain-wide evaluations where the scoring criteria remain constant. ### Example ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import DomainSpecificRubrics # Setup client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Reference-free evaluation (default) metric = DomainSpecificRubrics(llm=llm) result = await metric.ascore( user_input="What's the longest river in the world?", response="The longest river in the world is the Nile, stretching approximately 6,650 kilometers through northeastern Africa.", ) print(f"Score: {result.value}, Feedback: {result.reason}") # Reference-based evaluation metric_with_ref = DomainSpecificRubrics(llm=llm, with_reference=True) result = await metric_with_ref.ascore( user_input="What's the longest river in the world?", response="The longest river in the world is the Nile.", reference="The Nile is a major north-flowing river in northeastern Africa.", ) ``` ### Custom Rubrics You can define your own rubrics to customize the scoring criteria: ```python from ragas.metrics.collections import DomainSpecificRubrics my_custom_rubrics = { "score1_description": "Answer and ground truth are completely different", "score2_description": "Answer and ground truth are somewhat different", "score3_description": "Answer and ground truth are somewhat similar", "score4_description": "Answer and ground truth are similar", "score5_description": "Answer and ground truth are exactly the same", } metric = DomainSpecificRubrics(llm=llm, rubrics=my_custom_rubrics, with_reference=True) ``` ### With Retrieved Contexts The metric also supports evaluation with retrieved contexts: ```python result = await metric.ascore( user_input="What's the longest river in the world?", response="Based on the context, the Nile is the longest river.", retrieved_contexts=[ "Scientists debate whether the Amazon or the Nile is the longest river.", "The Nile River was central to Ancient Egyptians' wealth and power.", ], ) ``` ### Convenience Classes For clearer intent, use the convenience classes: ```python from ragas.metrics.collections import ( RubricsScoreWithoutReference, RubricsScoreWithReference, ) # Reference-free metric_no_ref = RubricsScoreWithoutReference(llm=llm) # Reference-based metric_with_ref = RubricsScoreWithReference(llm=llm) ``` ## Default Rubrics ### Reference-Free Rubrics (Default) | Score | Description | | ----- | ------------------------------------------------------------------------------------------ | | 1 | The response is entirely incorrect and fails to address any aspect of the user input. | | 2 | The response contains partial accuracy but includes major errors or significant omissions. | | 3 | The response is mostly accurate but lacks clarity, thoroughness, or minor details. | | 4 | The response is accurate and clear, with only minor omissions or slight inaccuracies. | | 5 | The response is completely accurate, clear, and thoroughly addresses the user input. | ### Reference-Based Rubrics | Score | Description | | ----- | ----------------------------------------------------------------------------------------- | | 1 | The response is entirely incorrect, irrelevant, or does not align with the reference. | | 2 | The response partially matches the reference but contains major errors or omissions. | | 3 | The response aligns with the reference overall but lacks sufficient detail or clarity. | | 4 | The response is mostly accurate, aligns closely with the reference with minor issues. | | 5 | The response is fully accurate, completely aligns with the reference, clear and detailed. | ______________________________________________________________________ ## Instance-Specific Rubrics Use `InstanceSpecificRubrics` when different samples require different evaluation criteria. This is useful when: - Different questions require different evaluation standards - You want to customize scoring based on specific task requirements - Evaluation criteria vary across your dataset ### Example ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import InstanceSpecificRubrics # Setup client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) metric = InstanceSpecificRubrics(llm=llm) # Each sample can have its own rubrics email_rubrics = { "score1_description": "The email is unprofessional or inappropriate", "score2_description": "The email lacks proper formatting or tone", "score3_description": "The email is acceptable but could be improved", "score4_description": "The email is professional with minor issues", "score5_description": "The email is highly professional and well-written", } result = await metric.ascore( user_input="Write a professional email declining a meeting invitation", response="Dear John, Thank you for the invitation...", rubrics=email_rubrics, ) print(f"Score: {result.value}, Feedback: {result.reason}") # Different rubrics for a different type of task code_rubrics = { "score1_description": "The code doesn't work or has critical bugs", "score2_description": "The code has significant issues or is poorly structured", "score3_description": "The code works but lacks optimization or best practices", "score4_description": "The code is good with minor improvements possible", "score5_description": "The code is excellent, efficient, and follows best practices", } result = await metric.ascore( user_input="Write a function to sort a list", response="def sort_list(arr): return sorted(arr)", rubrics=code_rubrics, ) ``` ### With Reference and Contexts ```python result = await metric.ascore( user_input="Explain the water cycle", response="The water cycle involves evaporation, condensation, and precipitation.", reference="The water cycle describes how water evaporates from surfaces, rises into the atmosphere, condenses into clouds, and falls as precipitation.", retrieved_contexts=["Water cycle information from encyclopedia..."], rubrics={ "score1_description": "Explanation is completely wrong", "score2_description": "Explanation has major inaccuracies", "score3_description": "Explanation is partially correct", "score4_description": "Explanation is mostly correct", "score5_description": "Explanation is comprehensive and accurate", }, ) ``` ______________________________________________________________________ ## Legacy API Deprecated The legacy API below is deprecated. Please use `ragas.metrics.collections.DomainSpecificRubrics` or `ragas.metrics.collections.InstanceSpecificRubrics` instead. ```python from ragas import evaluate from datasets import Dataset from ragas.metrics import rubrics_score_without_reference, rubrics_score_with_reference rows = { "question": [ "What's the longest river in the world?", ], "ground_truth": [ "The Nile is a major north-flowing river in northeastern Africa.", ], "answer": [ "The longest river in the world is the Nile, stretching approximately 6,650 kilometers (4,130 miles) through northeastern Africa.", ], "contexts": [ [ "Scientists debate whether the Amazon or the Nile is the longest river in the world.", "The Nile River was central to the Ancient Egyptians' rise to wealth and power.", ], ] } dataset = Dataset.from_dict(rows) result = evaluate( dataset, metrics=[ rubrics_score_without_reference, rubrics_score_with_reference ], ) ``` Custom rubrics with legacy API: ```python from ragas.metrics._domain_specific_rubrics import RubricsScore my_custom_rubrics = { "score1_description": "answer and ground truth are completely different", "score2_description": "answer and ground truth are somewhat different", "score3_description": "answer and ground truth are somewhat similar", "score4_description": "answer and ground truth are similar", "score5_description": "answer and ground truth are exactly the same", } rubrics_score = RubricsScore(rubrics=my_custom_rubrics) ``` ## Semantic Similarity The **Semantic Similarity** metric evaluates the semantic resemblance between a generated response and a reference (ground truth) answer. It ranges from 0 to 1, with higher scores indicating better alignment between the generated answer and the ground truth. This metric uses embeddings and cosine similarity to measure how semantically similar two answers are, which can offer valuable insights into the quality of the generated response. ### Example ```python from openai import AsyncOpenAI from ragas.embeddings import OpenAIEmbeddings from ragas.metrics.collections import SemanticSimilarity # Setup embeddings client = AsyncOpenAI() embeddings = OpenAIEmbeddings(model="text-embedding-3-small", client=client) # Create metric scorer = SemanticSimilarity(embeddings=embeddings) # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris. It has a height of 1000ft.", response="The Eiffel Tower is located in Paris." ) print(f"Semantic Similarity Score: {result.value}") ``` Output: ```text Semantic Similarity Score: 0.8151 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris. It has a height of 1000ft.", response="The Eiffel Tower is located in Paris." ) ``` ### How It's Calculated Example **Reference**: Albert Einstein's theory of relativity revolutionized our understanding of the universe. **High similarity response**: Einstein's groundbreaking theory of relativity transformed our comprehension of the cosmos. **Low similarity response**: Isaac Newton's laws of motion greatly influenced classical physics. Let's examine how semantic similarity was calculated for the high similarity response: - **Step 1:** Vectorize the reference answer using the specified embedding model. - **Step 2:** Vectorize the generated response using the same embedding model. - **Step 3:** Compute the cosine similarity between the two vectors. - **Step 4:** The cosine similarity value (0-1) is the final score. ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import SemanticSimilarity from ragas.embeddings import LangchainEmbeddingsWrapper sample = SingleTurnSample( response="The Eiffel Tower is located in Paris.", reference="The Eiffel Tower is located in Paris. It has a height of 1000ft." ) scorer = SemanticSimilarity(embeddings=LangchainEmbeddingsWrapper(evaluator_embedding)) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.8151371879226978 ``` # SQL ## Execution based metrics In these metrics the resulting SQL is compared after executing the SQL query on the database and then comparing the `response` with the expected results. ### DataCompy Score `DataCompyScore` metric uses DataCompy, a python library that compares two pandas DataFrames. It provides a simple interface to compare two DataFrames and provides a detailed report of the differences. In this metric the `response` is executed on the database and the resulting data is compared with the expected data, i.e. `reference`. To enable comparison both `response` and `reference` should be in the form of a Comma-Separated Values as shown in the example. DataFrames can be compared across rows or columns. This can be configured using `mode` parameter. If mode is `row` then the comparison is done row-wise. If mode is `column` then the comparison is done column-wise. [ \\text{Precision } = {|\\text{Number of matching rows in response and reference}| \\over |\\text{Total number of rows in response}|} ] [ \\text{Recall } = {|\\text{Number of matching rows in response and reference}| \\over |\\text{Total number of rows in reference}|} ] By default, the mode is set to `row`, and metric is F1 score which is the harmonic mean of precision and recall. ```python from ragas.metrics.collections import DataCompyScore data1 = """acct_id,dollar_amt,name,float_fld,date_fld 10000001234,123.45,George Maharis,14530.1555,2017-01-01 10000001235,0.45,Michael Bluth,1,2017-01-01 10000001236,1345,George Bluth,,2017-01-01 10000001237,123456,Bob Loblaw,345.12,2017-01-01 10000001238,1.05,Lucille Bluth,,2017-01-01 10000001238,1.05,Loose Seal Bluth,,2017-01-01 """ data2 = """acct_id,dollar_amt,name,float_fld 10000001234,123.4,George Michael Bluth,14530.155 10000001235,0.45,Michael Bluth, 10000001236,1345,George Bluth,1 10000001237,123456,Robert Loblaw,345.12 10000001238,1.05,Loose Seal Bluth,111 """ metric = DataCompyScore() result = await metric.ascore(response=data1, reference=data2) print(f"F1 Score: {result.value}") print(f"Details: {result.reason}") ``` To change the mode to column-wise comparison, set the `mode` parameter to `column`. ```python metric = DataCompyScore(mode="columns", metric="recall") result = await metric.ascore(response=data1, reference=data2) ``` ______________________________________________________________________ ### DataCompyScore (Legacy) Deprecated `DataCompyScore` from `ragas.metrics` is deprecated and will be removed in a future version. Please use `DataCompyScore` from `ragas.metrics.collections` as shown above. The legacy `DataCompyScore` uses the `SingleTurnSample` schema: ```python from ragas.metrics import DataCompyScore from ragas.dataset_schema import SingleTurnSample data1 = """acct_id,dollar_amt,name,float_fld,date_fld 10000001234,123.45,George Maharis,14530.1555,2017-01-01 10000001235,0.45,Michael Bluth,1,2017-01-01 10000001236,1345,George Bluth,,2017-01-01 10000001237,123456,Bob Loblaw,345.12,2017-01-01 10000001238,1.05,Lucille Bluth,,2017-01-01 10000001238,1.05,Loose Seal Bluth,,2017-01-01 """ data2 = """acct_id,dollar_amt,name,float_fld 10000001234,123.4,George Michael Bluth,14530.155 10000001235,0.45,Michael Bluth, 10000001236,1345,George Bluth,1 10000001237,123456,Robert Loblaw,345.12 10000001238,1.05,Loose Seal Bluth,111 """ sample = SingleTurnSample(response=data1, reference=data2) scorer = DataCompyScore() await scorer.single_turn_ascore(sample) ``` To change the mode to column-wise comparison, set the `mode` parameter to `column`. ```python scorer = DataCompyScore(mode="column", metric="recall") ``` ## Non Execution based metrics Executing SQL queries on the database can be time-consuming and sometimes not feasible. In such cases, we can use non-execution based metrics to evaluate the SQL queries. These metrics compare the SQL queries directly without executing them on the database. ### SQL Semantic Equivalence `SQLSemanticEquivalence` is a metric that evaluates whether a generated SQL query is semantically equivalent to a reference query. The metric uses an LLM to analyze both queries in the context of the provided database schema and determine if they would produce the same results. This is a binary metric: - **1.0**: The SQL queries are semantically equivalent - **0.0**: The SQL queries are not equivalent The metric considers the database schema context to make accurate equivalence judgments, accounting for syntactic differences that don't affect semantics (e.g., `active = 1` vs `active = true`). ```python from openai import AsyncOpenAI from ragas.llms.base import llm_factory from ragas.metrics.collections import SQLSemanticEquivalence # Initialize the LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create the metric metric = SQLSemanticEquivalence(llm=llm) # Evaluate SQL equivalence result = await metric.ascore( response=""" SELECT p.product_name, SUM(oi.quantity) AS total_quantity FROM order_items oi JOIN products p ON oi.product_id = p.product_id GROUP BY p.product_name; """, reference=""" SELECT products.product_name, SUM(order_items.quantity) AS total_quantity FROM order_items INNER JOIN products ON order_items.product_id = products.product_id GROUP BY products.product_name; """, reference_contexts=[ """ Table order_items: - order_item_id: INT - order_id: INT - product_id: INT - quantity: INT """, """ Table products: - product_id: INT - product_name: VARCHAR - price: DECIMAL """ ] ) print(f"Equivalent: {result.value == 1.0}") print(f"Explanation: {result.reason}") ``` The result includes explanations of both queries and the reasoning for the equivalence determination. ______________________________________________________________________ ### LLMSQLEquivalence (Legacy) Deprecated `LLMSQLEquivalence` is deprecated and will be removed in a future version. Please use `SQLSemanticEquivalence` from `ragas.metrics.collections` as shown above. `LLMSQLEquivalence` is the legacy metric for SQL semantic equivalence evaluation. It uses the `SingleTurnSample` schema and requires setting the LLM separately. ```python from ragas.metrics import LLMSQLEquivalence from ragas.dataset_schema import SingleTurnSample sample = SingleTurnSample( response=""" SELECT p.product_name, SUM(oi.quantity) AS total_quantity FROM order_items oi JOIN products p ON oi.product_id = p.product_id GROUP BY p.product_name; """, reference=""" SELECT p.product_name, COUNT(oi.quantity) AS total_quantity FROM order_items oi JOIN products p ON oi.product_id = p.product_id GROUP BY p.product_name; """, reference_contexts=[ """ Table order_items: - order_item_id: INT - order_id: INT - product_id: INT - quantity: INT """, """ Table products: - product_id: INT - product_name: VARCHAR - price: DECIMAL """ ] ) scorer = LLMSQLEquivalence() scorer.llm = openai_model await scorer.single_turn_ascore(sample) ``` # Tasks Metrics ## Summarization Score The **Summarization Score** metric measures how well a summary (`response`) captures the important information from the `reference_contexts`. The intuition behind this metric is that a good summary should contain all the important information present in the context. We first extract a set of important keyphrases from the context. These keyphrases are then used to generate a set of questions. The answers to these questions are always `yes(1)` for the context. We then ask these questions to the summary and calculate the summarization score as the ratio of correctly answered questions to the total number of questions. We compute the question-answer score using the answers, which is a list of `1`s and `0`s. The question-answer score is then calculated as the ratio of correctly answered questions(answer = `1`) to the total number of questions. [ \\text{QA score} = \\frac{|\\text{correctly answered questions}|}{|\\text{total questions}|} ] We also introduce an option to penalize larger summaries by proving a conciseness score. If this option is enabled, the final score is calculated as the weighted average of the summarization score and the conciseness score. This conciseness scores ensures that summaries that are just copies of the text do not get a high score, because they will obviously answer all questions correctly. [ \\text{conciseness score} = 1 - \\frac{\\min(\\text{length of summary}, \\text{length of context})}{\\text{length of context} + \\text{1e-10}} ] We also provide a coefficient `coeff`(default value 0.5) to control the weightage of the scores. The final summarization score is then calculated as: [ \\text{Summarization Score} = \\text{QA score}\*\\text{(1-coeff)} + \\ \\text{conciseness score}\*\\text{coeff} ] ### Example ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import SummaryScore # Setup LLM client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric scorer = SummaryScore(llm=llm) # Evaluate result = await scorer.ascore( reference_contexts=[ "A company is launching a new product, a smartphone app designed to help users track their fitness goals. The app allows users to set daily exercise targets, log their meals, and track their water intake. It also provides personalized workout recommendations and sends motivational reminders throughout the day." ], response="A company is launching a fitness tracking app that helps users set exercise goals, log meals, and track water intake, with personalized workout suggestions and motivational reminders." ) print(f"Summary Score: {result.value}") ``` Output: ```text Summary Score: 0.6423387096775146 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference_contexts=[...], response="..." ) ``` ## Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. ### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import SummarizationScore sample = SingleTurnSample( response="A company is launching a fitness tracking app that helps users set exercise goals, log meals, and track water intake, with personalized workout suggestions and motivational reminders.", reference_contexts=[ "A company is launching a new product, a smartphone app designed to help users track their fitness goals. The app allows users to set daily exercise targets, log their meals, and track their water intake. It also provides personalized workout recommendations and sends motivational reminders throughout the day." ] ) scorer = SummarizationScore(llm=evaluator_llm) await scorer.single_turn_ascore(sample) ``` Output: ```text 0.6423387096775146 ``` # Traditional NLP Metrics ## Non LLM String Similarity `NonLLMStringSimilarity` metric measures the similarity between the reference and the response using traditional string distance measures such as Levenshtein, Hamming, and Jaro. This metric is useful for evaluating the similarity of `response` to the `reference` text without relying on large language models (LLMs). The metric returns a score between 0 and 1, where 1 indicates a perfect match between the response and the reference. This is a non LLM based metric. ### Example ```python from ragas.metrics.collections import NonLLMStringSimilarity, DistanceMeasure # Create metric (no LLM/embeddings needed) scorer = NonLLMStringSimilarity(distance_measure=DistanceMeasure.LEVENSHTEIN) # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) print(f"NonLLM String Similarity Score: {result.value}") ``` Output: ```text NonLLM String Similarity Score: 0.8918918918918919 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) ``` ### Configuration You can choose from available string distance measures from `DistanceMeasure`. Here is an example of using Hamming distance: ```python scorer = NonLLMStringSimilarity(distance_measure=DistanceMeasure.HAMMING) ``` Available distance measures include: - `DistanceMeasure.LEVENSHTEIN` (default) - `DistanceMeasure.HAMMING` - `DistanceMeasure.JARO` - `DistanceMeasure.JARO_WINKLER` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics._string import NonLLMStringSimilarity sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="The Eiffel Tower is located in Paris." ) scorer = NonLLMStringSimilarity() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.8918918918918919 ``` #### Example with Different Distance Measure ```python from ragas.metrics._string import NonLLMStringSimilarity, DistanceMeasure scorer = NonLLMStringSimilarity(distance_measure=DistanceMeasure.HAMMING) ``` ## BLEU Score The `BleuScore` metric is used to evaluate the quality of `response` by comparing it with `reference`. It measures the similarity between the response and the reference based on n-gram precision and brevity penalty. BLEU score was originally designed to evaluate machine translation systems, but it is also used in other natural language processing tasks. BLEU score ranges from 0 to 1, where 1 indicates a perfect match between the response and the reference. This is a non-LLM based metric. ### Example ```python from ragas.metrics.collections import BleuScore # Create metric scorer = BleuScore() # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) print(f"BLEU Score: {result.value}") ``` Output: ```text BLEU Score: 0.7071067811865478 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) ``` ### Configuration You can pass additional arguments to the underlying `sacrebleu.corpus_bleu` function using the `kwargs` parameter: ```python scorer = BleuScore(kwargs={"smooth_method": "exp"}) ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import BleuScore sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="The Eiffel Tower is located in Paris." ) scorer = BleuScore() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.7071067811865478 ``` ## ROUGE Score The `RougeScore` score is a set of metrics used to evaluate the quality of natural language generations. It measures the overlap between the generated `response` and the `reference` text based on n-gram recall, precision, and F1 score. ROUGE score ranges from 0 to 1, where 1 indicates a perfect match between the response and the reference. This is a non LLM based metric. ### Example ```python from ragas.metrics.collections import RougeScore # Create metric (no LLM/embeddings needed) scorer = RougeScore(rouge_type="rougeL", mode="fmeasure") # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) print(f"ROUGE Score: {result.value}") ``` Output: ```text ROUGE Score: 0.8571428571428571 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) ``` ### Configuration You can change the `rouge_type` to `rouge1` or `rougeL` to calculate the ROUGE score based on unigrams or longest common subsequence respectively. ```python scorer = RougeScore(rouge_type="rouge1") ``` You can change the `mode` to `precision`, `recall`, or `fmeasure` to calculate the ROUGE score based on precision, recall, or F1 score respectively. ```python scorer = RougeScore(mode="recall") ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import RougeScore sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="The Eiffel Tower is located in Paris." ) scorer = RougeScore() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.8571428571428571 ``` ## Exact Match The `ExactMatch` metric checks if the response is exactly the same as the reference text. It is useful in scenarios where you need to ensure that the generated response matches the expected output word-for-word. For example, arguments in tool calls, etc. The metric returns 1 if the response is an exact match with the reference, and 0 otherwise. ### Example ```python from ragas.metrics.collections import ExactMatch # Create metric (no LLM/embeddings needed) scorer = ExactMatch() # Evaluate result = await scorer.ascore( reference="Paris", response="India" ) print(f"Exact Match Score: {result.value}") ``` Output: ```text Exact Match Score: 0.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="Paris", response="India" ) ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import ExactMatch sample = SingleTurnSample( response="India", reference="Paris" ) scorer = ExactMatch() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.0 ``` ## String Presence The `StringPresence` metric checks if the response contains the reference text. It is useful in scenarios where you need to ensure that the generated response contains certain keywords or phrases. The metric returns 1 if the response contains the reference, and 0 otherwise. ### Example ```python from ragas.metrics.collections import StringPresence # Create metric (no LLM/embeddings needed) scorer = StringPresence() # Evaluate result = await scorer.ascore( reference="Eiffel Tower", response="The Eiffel Tower is located in India." ) print(f"String Presence Score: {result.value}") ``` Output: ```text String Presence Score: 1.0 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="Eiffel Tower", response="The Eiffel Tower is located in India." ) ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import StringPresence sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="Eiffel Tower" ) scorer = StringPresence() await scorer.single_turn_ascore(sample) ``` Output: ```text 1.0 ``` ## CHRF Score The `CHRFScore` metric evaluates the similarity between a `response` and a `reference` using **character n-gram F-score**. Unlike BLEU, which emphasizes precision, CHRF accounts for both **precision and recall**, making it more suitable for: - Morphologically rich languages - Responses with paraphrasing or flexible wording CHRF scores range from 0 to 1, where 1 indicates a perfect match between the generated response and the reference. This is a non-LLM-based metric, relying entirely on deterministic comparisons. ### Example ```python from ragas.metrics.collections import CHRFScore # Create metric (no LLM/embeddings needed) scorer = CHRFScore() # Evaluate result = await scorer.ascore( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) print(f"CHRF Score: {result.value}") ``` Output: ```text CHRF Score: 0.8048 ``` Synchronous Usage If you prefer synchronous code, you can use the `.score()` method instead of `.ascore()`: ```python result = scorer.score( reference="The Eiffel Tower is located in Paris.", response="The Eiffel Tower is located in India." ) ``` ### Configuration You can pass additional arguments to the underlying `sacrebleu.corpus_chrf` function using the `kwargs` parameter: ```python # Customize character and word order scorer = CHRFScore(kwargs={"char_order": 4, "word_order": 2}) # Customize beta (recall weight) scorer = CHRFScore(kwargs={"beta": 3}) ``` ### Legacy Metrics API The following examples use the legacy metrics API pattern. For new projects, we recommend using the collections-based API shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API shown above. #### Example with SingleTurnSample ```python from ragas.dataset_schema import SingleTurnSample from ragas.metrics import ChrfScore sample = SingleTurnSample( response="The Eiffel Tower is located in India.", reference="The Eiffel Tower is located in Paris." ) scorer = ChrfScore() await scorer.single_turn_ascore(sample) ``` Output: ```text 0.8048 ``` # Test Data Generation # Testset Generation Curating a high quality test dataset is crucial for evaluating the performance of your AI application. ## Characteristics of an Ideal Test Dataset - Contains high quality data samples - Covers wide variety of scenarios as observed in real world. - Contains enough number of samples to derive statistically significant conclusions. - Continually updated to prevent data drift Curating such a dataset manually can be time-consuming and expensive. Ragas provides a set of tools to generate synthetic test datasets for evaluating your AI applications. - [**RAG** for evaluating retrieval augmented generation pipelines](https://docs.ragas.io/en/stable/concepts/test_data_generation/rag/index.md) - [**Agents or Tool use** for evaluating agent workflows](https://docs.ragas.io/en/stable/concepts/test_data_generation/agents/index.md) # Testset Generation for Agents or Tool use cases Evaluating agentic or tool use workflows can be challenging as it involves multiple steps and interactions. It can be especially hard to curate a test suite that covers all possible scenarios and edge cases. We are working on a set of tools to generate synthetic test data for evaluating agent workflows. [Talk to us](https://cal.com/team/vibrantlabs/office-hours) to work together on this and discover what's coming for upcoming releases. # Testset Generation for RAG In RAG application, when a user interacts through your application to a set of documents, there can be different patterns of queries that the system can encounter. Let's first understand the different types of queries that can be encountered in RAG application. ## Query types in RAG ``` graph TD A[Queries] --> B[Single-Hop Query] A --> C[Multi-Hop Query] B --> D1[Specific Query] B --> E1[Abstract Query] C --> F1[Specific Query] C --> G1[Abstract Query] ``` ### Single-Hop Query A single-hop query is a straightforward question that requires retrieving information from a single document or source to provide a relevant answer. It involves only one step to arrive at the answer. **Example (Specific Query):** - β€œWhat year did Albert Einstein publish the theory of relativity?” This is a specific, fact-based question that can be answered with a single retrieval from a document containing that information. **Example (Abstract Query):** - β€œHow did Einstein’s theory change our understanding of time and space?” While this query still refers to a single concept (the theory of relativity), it requires a more abstract or interpretive explanation from the source material. ### Multi-Hop Query A multi-hop query involves multiple steps of reasoning, requiring information from two or more sources. The system must retrieve information from various documents and connect the dots to generate an accurate answer. **Example (Specific Query):** - β€œWhich scientist influenced Einstein’s work on relativity, and what theory did they propose?” This requires the system to retrieve information about both the scientist who influenced Einstein and the specific theory, potentially from two different sources. **Example (Abstract Query):** - β€œHow have scientific theories on relativity evolved since Einstein’s original publication?” This abstract query requires the retrieval of multiple pieces of information over time and across different sources to form a broad, interpretive response about the evolution of the theory. ### Specific vs. Abstract Queries in a RAG - **Specific Query:** Focuses on clear, fact-based retrieval. The goal in RAG is to retrieve highly relevant information from one or more documents that directly address the specific question. - **Abstract Query:** Requires a broader, more interpretive response. In RAG, abstract queries challenge the retrieval system to pull from documents that contain higher-level reasoning, explanations, or opinions, rather than simple facts. In both single-hop and multi-hop cases, the distinction between specific and abstract queries shapes the retrieval and generation process by determining whether the focus is on precision (specific) or on synthesizing broader ideas (abstract). Different types of queries requires different contexts to be synthesized. To solve this problem, Ragas uses a Knowledge Graph based approach to Test set Generation. ## Knowledge Graph Creation Given that we want to manufacture different types of queries from the given set of documents, our major challenge is to identify the right set of chunks or documents to enable LLMs to create the queries. To solve this problem, Ragas uses a Knowledge Graph based approach to Test set Generation. knowledge graph creation The knowledge graph is created by using the following components: ### Document Splitter The documents are chunked to form hierarchical nodes. The chunking can be done by using different splitters. For example, in the case of financial documents, the chunking can be done by using the splitter that splits the document based on the sections like Income Statement, Balance Sheet, Cash Flow Statement etc. You can write your own custom splitters to split the document based on the sections that are relevant to your domain. #### Example ```python from ragas.testset.graph import Node sample_nodes = [Node( properties={"page_content": "Einstein's theory of relativity revolutionized our understanding of space and time. It introduced the concept that time is not absolute but can change depending on the observer's frame of reference."} ),Node( properties={"page_content": "Time dilation occurs when an object moves close to the speed of light, causing time to pass slower relative to a stationary observer. This phenomenon is a key prediction of Einstein's special theory of relativity."} )] sample_nodes ``` Output: ```bash [Node(id: 4f6b94, type: , properties: ['page_content']), Node(id: 952361, type: , properties: ['page_content'])] ``` ``` graph TD A[Node: 4f6b94] -.-> |Properties| A1[page_content] B[Node: 952361] -.-> |Properties| B1[page_content] ``` ### Extractors Different extractors are used to extract information from each node that can be used to establish the relationship between the nodes. For example, in the case of financial documents, the extractor that can be used are entity extractor to extract the entities like Company Name, Keyphrase extractor to extract important key phrases present in each node, etc. You can write your own custom extractors to extract the information that is relevant to your domain. Extractors can be LLM based which are inherited from `LLMBasedExtractor` or rule based which are inherited from `Extractor`. #### Example Let's say we have a sample node from the knowledge graph. We can use the `NERExtractor` to extract the named entities from the node. ```python from ragas.testset.transforms.extractors import NERExtractor extractor = NERExtractor() output = [await extractor.extract(node) for node in sample_nodes] output[0] ``` Returns a tuple of the type of the extractor and the extracted information. ```bash ('entities', ['Einstein', 'theory of relativity', 'space', 'time', "observer's frame of reference"]) ``` Let's add the extracted information to the node. ```python _ = [node.properties.update({key:val}) for (key,val), node in zip(output, sample_nodes)] sample_nodes[0].properties ``` Output: ```bash {'page_content': "Einstein's theory of relativity revolutionized our understanding of space and time. It introduced the concept that time is not absolute but can change depending on the observer's frame of reference.", 'entities': ['Einstein', 'theory of relativity', 'space', 'time', 'observer']} ``` ``` graph TD A[Node: 4f6b94] -.-> |Properties| A1[page_content] A -.-> |Properties| A2[entities] B[Node: 952361] -.-> |Properties| B1[page_content] B -.-> |Properties| B2[entities] ``` ### Relationship builder The extracted information is used to establish the relationship between the nodes. For example, in the case of financial documents, the relationship can be established between the nodes based on the entities present in the nodes. You can write your own custom relationship builder to establish the relationship between the nodes based on the information that is relevant to your domain. #### Example ```python from ragas.testset.graph import KnowledgeGraph from ragas.testset.transforms.relationship_builders.traditional import JaccardSimilarityBuilder kg = KnowledgeGraph(nodes=sample_nodes) rel_builder = JaccardSimilarityBuilder(property_name="entities", key_name="PER", new_property_name="entity_jaccard_similarity") relationships = await rel_builder.transform(kg) relationships ``` Output: ```bash [Relationship(Node(id: 4f6b94) <-> Node(id: 952361), type: jaccard_similarity, properties: ['entity_jaccard_similarity'])] ``` Since both the nodes have the same entity "Einstein", the relationship is established between the nodes based on the entity similarity. ``` graph TD A[Node: 4f6b94] -.-> |Properties| A1[page_content] A -.-> |Properties| A2[entities] B[Node: 952361] -.-> |Properties| B1[page_content] B -.-> |Properties| B2[entities] A ===|entity_jaccard_similarity| B ``` Now let's understand how to build the knowledge graph using the above components with a `transform`, that would make your job easier. ### Transforms All of the components used to build the knowledge graph can be combined into a single `transform` that can be applied to the knowledge graph to build the knowledge graph. Transforms is made of up of a list of components that are applied to the knowledge graph in a sequence. It can also handle parallel processing of the components. The `apply_transforms` method is used to apply the transforms to the knowledge graph. #### Example Let's build the above knowledge graph using the above components with a `transform`. ```python from ragas.testset.transforms import apply_transforms transforms = [ extractor, rel_builder ] apply_transforms(kg,transforms) ``` To apply few of the components in parallel, you can wrap them in `Parallel` class. ```python from ragas.testset.transforms import KeyphraseExtractor, NERExtractor from ragas.testset.transforms import apply_transforms, Parallel tranforms = [ Parallel( KeyphraseExtractor(), NERExtractor() ), rel_builder ] apply_transforms(kg,transforms) ``` Once the knowledge graph is created, the different types of queries can be generated by traversing the graph. For example, to generate the query β€œCompare the revenue growth of Company X and Company Y from FY2020 through FY2023”, the graph can be traversed to find the nodes that contain the information about the revenue growth of Company X and Company Y from FY2020 through FY2023. ## Scenario Generation Now we have the knowledge graph that can be used to manufacture the right context to generate any type of query. When a population of users interact with RAG system, they may formulate the queries in various ways depending upon their persona (eg, Senior Engineer, Junior Engineer, etc), Query length (Short, Long, etc), Query style (Formal, Informal, etc). To generate the queries that cover all these scenarios, Ragas uses a Scenario based approach to Test set Generation. Each `Scenario` in Test set Generation is a combination of following parameters. - Nodes : The nodes that are used to generate the query - Query Length : The length of the desired query, it can be short, medium or long, etc. - Query Style : The style of the query, it can be web search, chat, etc. - Persona : The persona of the user, it can be Senior Engineer, Junior Engineer, etc. (Coming soon) Scenario in Test Generation ### Query Synthesizer The `QuerySynthesizer` is responsible for generating different scenarios for a single query type. The `generate_scenarios` method is used to generate the scenarios for a single query type. The `generate_sample` method is used to generate the query and reference answer for a single scenario. Let's understand this with an example. #### Example In the previous example, we have created a knowledge graph that contains two nodes that are related to each other based on the entity similarity. Now imagine that you have 20 such pairs of nodes in your KG that are related to each other based on the entity similarity. Imagine your goal is to create 50 different queries where each query is about some abstract question comparing two entities. We first have to query the KG to get the pairs of nodes that are related to each other based on the entity similarity. Then we have to generate the scenarios for each pair of nodes until we get 50 different scenarios. This logic is implemented in `generate_scenarios` method. ```python from dataclasses import dataclass from ragas.testset.synthesizers.base_query import QuerySynthesizer @dataclass class EntityQuerySynthesizer(QuerySynthesizer): async def _generate_scenarios( self, n, knowledge_graph, callbacks): """ logic to query nodes with entity logic describing how to combine nodes,styles,length,persona to form n scenarios """ return scenarios async def _generate_sample( self, scenario, callbacks ): """ logic on how to use tranform each scenario to EvalSample (Query,Context,Reference) you may create singleturn or multiturn sample """ return SingleTurnSample(user_input=query, reference_contexs=contexts, reference=reference) ``` # Customization Guides # Customizations How to customize various aspects of Ragas to suit your needs. ## General - [Customize models](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/index.md) - [Customize timeouts, retries and others](https://docs.ragas.io/en/stable/howtos/customizations/run_config/index.md) - [Cancelling long-running tasks](https://docs.ragas.io/en/stable/howtos/customizations/cancellation/index.md) ## Metrics - [Modify prompts in metrics](https://docs.ragas.io/en/stable/howtos/customizations/metrics/_modifying-prompts-metrics.md) - [Adapt metrics to target language](https://docs.ragas.io/en/stable/howtos/customizations/metrics/metrics_language_adaptation/index.md) - [Trace evaluations with Observability tools](https://docs.ragas.io/en/stable/howtos/customizations/metrics/tracing/index.md) ## Testset Generation - [Generate test data from non-English corpus](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_language_adaptation/index.md) - [Configure or automatically generate Personas](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_persona_generator/index.md) - [Customize single-hop queries for RAG evaluation](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_testgen-custom-single-hop/index.md) - [Create custom multi-hop queries for RAG evaluation](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_testgen-customisation/index.md) - [Seed generations using production data](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/index.md) # Caching in Ragas You can use caching to speed up your evaluations and testset generation by avoiding redundant computations. We use Exact Match Caching to cache the responses from the LLM and Embedding models. You can use the DiskCacheBackend which uses a local disk cache to store the cached responses. You can also implement your own custom cacher by implementing the CacheInterface. ## Using Caching with Modern LLMs and Embeddings The new metrics collections and experiments support caching through a simple interface. ### Quick Start ```python from ragas.cache import DiskCacheBackend from ragas.llms import llm_factory from openai import OpenAI # Create cache once cache = DiskCacheBackend() # Use with LLM factory client = OpenAI(api_key="...") llm = llm_factory("gpt-4o-mini", client=client, cache=cache) # All LLM calls are now cached! from pydantic import BaseModel class Response(BaseModel): answer: str response = llm.generate("Evaluate this...", Response) ``` ### Caching with llm_factory ```python from ragas.cache import DiskCacheBackend from ragas.llms import llm_factory from openai import OpenAI # Create cache instance cache = DiskCacheBackend() # Create LLM with caching client = OpenAI(api_key="...") llm = llm_factory("gpt-4o-mini", client=client, cache=cache) # First call - makes API request and caches result response1 = llm.generate("Evaluate this text", Response) # Second call - returns cached result instantly response2 = llm.generate("Evaluate this text", Response) # Result: Same output, 60x faster, $0 cost ``` ### Caching with embedding_factory ```python from ragas.cache import DiskCacheBackend from ragas.embeddings import embedding_factory from openai import OpenAI cache = DiskCacheBackend() client = OpenAI(api_key="...") embeddings = embedding_factory("openai", client=client, cache=cache) # First call - makes API request vector1 = embeddings.embed_text("Some text to embed") # Second call - instant cache hit vector2 = embeddings.embed_text("Some text to embed") assert vector1 == vector2 # Identical results ``` ### Caching in Experiments Caching is especially powerful in experiments where you run the same evaluation multiple times: ```python from ragas import experiment, Dataset from ragas.cache import DiskCacheBackend from ragas.llms import llm_factory from ragas.metrics.collections import FactualCorrectness # Setup cached LLM once cache = DiskCacheBackend() llm = llm_factory("gpt-4o-mini", client=client, cache=cache) # Use in metric metric = FactualCorrectness(llm=llm) @experiment() async def evaluate_model(row): score = metric.score( response=row["response"], reference=row["reference"] ) return { **row, "factual_correctness": score.value, "reason": score.reason } # Load your dataset dataset = Dataset.from_list([ {"response": "Paris is the capital of France", "reference": "Paris"}, {"response": "London is the capital of UK", "reference": "London"}, ]) # First run - makes API calls and caches results print("First run (populating cache)...") results1 = await evaluate_model.arun(dataset) # Takes ~2 seconds for 2 samples # Second run - uses cache, nearly instant! print("Second run (using cache)...") results2 = await evaluate_model.arun(dataset) # Takes ~0.1 seconds for 2 samples # Results are identical, but 20x faster! ``` ### Cache Management #### Clearing the Cache ```python # Clear all cached data cache = DiskCacheBackend() cache.cache.clear() ``` #### Setting Size Limits ```python # Limit cache to 1GB cache = DiskCacheBackend() cache.cache.reset('size_limit', 1e9) # 1GB cache.cache.reset('cull_limit', 10) # Remove 10% when full ``` #### Cache Location By default, cache is stored in `.cache/` directory. You can change this: ```python cache = DiskCacheBackend(cache_dir="my_custom_cache") ``` ### Benefits of Caching 1. **Cost Savings**: Avoid repeated API calls for identical inputs (50-60% savings) 1. **Speed**: Cached calls return nearly instantly (60x+ faster) 1. **Development**: Iterate quickly without waiting for API calls 1. **Reproducibility**: Same inputs always return same results Cache hits occur when: - βœ… Same prompt/text (exact match) - βœ… Same model parameters (temperature, max_tokens, etc.) - βœ… Same response model/structure (for LLMs) Cache misses occur when: - ❌ Different prompt/text - ❌ Different parameters - ❌ Different response model ### Anti-Patterns (When NOT to Cache) - ❌ **Non-deterministic prompts**: If prompts contain random elements or timestamps - ❌ **High temperature**: If temperature > 0.7 (responses vary too much) - ❌ **Streaming responses**: Caching doesn't work with streaming - ❌ **Real-time data**: If responses need to reflect current state ### Environment-Specific Notes **Notebooks**: Cache persists between cell executions and kernel restarts **Web Applications**: Share cache across requests for better performance **Serverless Functions**: Use `/tmp` directory: ```python cache = DiskCacheBackend(cache_dir="/tmp/.cache") ``` **Distributed Workers**: Cache is process-safe but for high-throughput systems consider implementing a Redis backend via the `CacheInterface` ### Performance Expectations | Scenario | Time | Cost | | ----------------------- | -------------- | ---------------- | | First run (100 samples) | ~2 minutes | $0.50 | | Second run (cached) | ~2 seconds | $0.00 | | **Speedup** | **60x faster** | **100% savings** | ______________________________________________________________________ ## Legacy Caching (Deprecated) Deprecated This approach using `LangchainLLMWrapper` is deprecated and will be removed in v1.0. Please use the modern approach with `llm_factory()` and `embedding_factory()` as shown above. ### Using Legacy Caching with LangchainLLMWrapper Let's see how you can use the DiskCacheBackend with legacy LLM and Embedding models. ```python from ragas.cache import DiskCacheBackend cacher = DiskCacheBackend() # check if the cache is empty and clear it print(len(cacher.cache)) cacher.cache.clear() print(len(cacher.cache)) ``` Create an LLM and Embedding model with the cacher, here I'm using the `ChatOpenAI` from [langchain-openai](https://github.com/langchain-ai/langchain-openai) as an example. ```python from langchain_openai import ChatOpenAI from ragas.llms import LangchainLLMWrapper cached_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"), cache=cacher) ``` ```python # if you want to see the cache in action, set the logging level to debug import logging from ragas.utils import set_logging_level set_logging_level("ragas.cache", logging.DEBUG) ``` Now let's run a simple evaluation. ```python from ragas import evaluate from ragas import EvaluationDataset from ragas.metrics import FactualCorrectness, AspectCritic from datasets import load_dataset # Define Answer Correctness with AspectCritic answer_correctness = AspectCritic( name="answer_correctness", definition="Is the answer correct? Does it match the reference answer?", llm=cached_llm, ) metrics = [answer_correctness, FactualCorrectness(llm=cached_llm)] # load the dataset dataset = load_dataset( "vibrantlabsai/amnesty_qa", "english_v3", trust_remote_code=True ) eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"]) # evaluate the dataset results = evaluate( dataset=eval_dataset, metrics=metrics, ) results ``` This took almost 2mins to run in our local machine. Now let's run it again to see the cache in action. ```python results = evaluate( dataset=eval_dataset, metrics=metrics, ) results ``` Runs almost instantaneously. You can also use this with testset generation also by replacing the `generator_llm` with a cached version of it. Refer to the [testset generation](https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/index.md) section for more details. # Cancelling Long-Running Tasks When working with large datasets or complex evaluations, some Ragas operations can take significant time to complete. The cancellation feature allows you to gracefully terminate these long-running tasks when needed, which is especially important in production environments. ## Overview Ragas provides cancellation support for: - **`evaluate()`** - Evaluation of datasets with metrics - **`generate_with_langchain_docs()`** - Test set generation from documents The cancellation mechanism is thread-safe and allows for graceful termination with partial results when possible. ## Basic Usage ### Cancellable Evaluation Instead of running evaluation directly, you can get an executor that allows cancellation: ```py from ragas import evaluate from ragas.dataset_schema import EvaluationDataset # Your dataset and metrics dataset = EvaluationDataset(...) metrics = [...] # Get executor instead of running evaluation immediately executor = evaluate( dataset=dataset, metrics=metrics, return_executor=True # Key parameter ) # Now you can: # - Cancel: executor.cancel() # - Check status: executor.is_cancelled() # - Get results: executor.results() # This blocks until completion ``` ### Cancellable Test Set Generation Similar approach for test set generation: ```py from ragas.testset.synthesizers.generate import TestsetGenerator generator = TestsetGenerator(...) # Get executor for cancellable generation executor = generator.generate_with_langchain_docs( documents=documents, testset_size=100, return_executor=True # Allow access to Executor to cancel ) # Use the same cancellation interface executor.cancel() ``` ## Production Patterns ### 1. Timeout Pattern Automatically cancel operations that exceed a time limit: ```py import threading import time def evaluate_with_timeout(dataset, metrics, timeout_seconds=300): """Run evaluation with automatic timeout.""" # Get cancellable executor executor = evaluate(dataset=dataset, metrics=metrics, return_executor=True) results = None exception = None def run_evaluation(): nonlocal results, exception try: results = executor.results() except Exception as e: exception = e # Start evaluation in background thread thread = threading.Thread(target=run_evaluation) thread.start() # Wait for completion or timeout thread.join(timeout=timeout_seconds) if thread.is_alive(): print(f"Evaluation exceeded {timeout_seconds}s timeout, cancelling...") executor.cancel() thread.join(timeout=10) # Custom timeout as per need return None, "timeout" return results, exception # Usage results, error = evaluate_with_timeout(dataset, metrics, timeout_seconds=600) if error == "timeout": print("Evaluation was cancelled due to timeout") else: print(f"Evaluation completed: {results}") ``` ### 2. Signal Handler Pattern (Ctrl+C) Allow users to cancel with keyboard interrupt: ```py import signal import sys def setup_cancellation_handler(): """Set up graceful cancellation on Ctrl+C.""" executor = None def signal_handler(signum, frame): if executor and not executor.is_cancelled(): print("\nReceived interrupt signal, cancelling evaluation...") executor.cancel() print("Cancellation requested. Waiting for graceful shutdown...") sys.exit(0) # Register signal handler signal.signal(signal.SIGINT, signal_handler) return lambda exec: setattr(signal_handler, 'executor', exec) # Usage set_executor = setup_cancellation_handler() executor = evaluate(dataset=dataset, metrics=metrics, return_executor=True) set_executor(executor) print("Running evaluation... Press Ctrl+C to cancel gracefully") try: results = executor.results() print("Evaluation completed successfully") except KeyboardInterrupt: print("Evaluation was cancelled") ``` ### 3. Web Application Pattern For web applications, cancel operations when requests are aborted: ```py from flask import Flask, request import threading import uuid app = Flask(__name__) active_evaluations = {} @app.route('/evaluate', methods=['POST']) def start_evaluation(): # Create unique evaluation ID eval_id = str(uuid.uuid4()) # Get dataset and metrics from request dataset = get_dataset_from_request(request) metrics = get_metrics_from_request(request) # Start cancellable evaluation executor = evaluate(dataset=dataset, metrics=metrics, return_executor=True) active_evaluations[eval_id] = executor # Start evaluation in background def run_eval(): try: results = executor.results() # Store results somewhere store_results(eval_id, results) except Exception as e: store_error(eval_id, str(e)) finally: active_evaluations.pop(eval_id, None) threading.Thread(target=run_eval).start() return {"evaluation_id": eval_id, "status": "started"} @app.route('/evaluate//cancel', methods=['POST']) def cancel_evaluation(eval_id): executor = active_evaluations.get(eval_id) if executor: executor.cancel() return {"status": "cancelled"} return {"error": "Evaluation not found"}, 404 ``` ## Advanced Usage ### Checking Cancellation Status ```py executor = evaluate(dataset=dataset, metrics=metrics, return_executor=True) # Start in background def monitor_evaluation(): while not executor.is_cancelled(): print("Evaluation still running...") time.sleep(5) print("Evaluation was cancelled") threading.Thread(target=monitor_evaluation).start() # Cancel after some condition if some_condition(): executor.cancel() ``` ### Partial Results When cancellation occurs during execution, you may get partial results: ```py executor = evaluate(dataset=dataset, metrics=metrics, return_executor=True) try: results = executor.results() print(f"Completed {len(results)} evaluations") except Exception as e: if executor.is_cancelled(): print("Evaluation was cancelled - may have partial results") else: print(f"Evaluation failed: {e}") ``` ### Custom Cancellation Logic ```py class EvaluationManager: def __init__(self): self.executors = [] def start_evaluation(self, dataset, metrics): executor = evaluate(dataset=dataset, metrics=metrics, return_executor=True) self.executors.append(executor) return executor def cancel_all(self): """Cancel all running evaluations.""" for executor in self.executors: if not executor.is_cancelled(): executor.cancel() print(f"Cancelled {len(self.executors)} evaluations") def cleanup_completed(self): """Remove completed executors.""" self.executors = [ex for ex in self.executors if not ex.is_cancelled()] # Usage manager = EvaluationManager() # Start multiple evaluations exec1 = manager.start_evaluation(dataset1, metrics) exec2 = manager.start_evaluation(dataset2, metrics) # Cancel all if needed manager.cancel_all() ``` ## Best Practices ### 1. Always Use Timeouts in Production ```py # Good: Always set reasonable timeouts results, error = evaluate_with_timeout(dataset, metrics, timeout_seconds=1800) # 30 minutes # Avoid: Indefinite blocking results = executor.results() # Could block forever ``` ### 2. Handle Cancellation Gracefully ```py try: results = executor.results() process_results(results) except Exception as e: if executor.is_cancelled(): log_cancellation() cleanup_partial_work() else: log_error(e) handle_failure() ``` ### 3. Provide User Feedback ```py def run_with_progress_and_cancellation(executor): print("Starting evaluation... Press Ctrl+C to cancel") # Monitor progress in background def show_progress(): while not executor.is_cancelled(): # Show some progress indication print(".", end="", flush=True) time.sleep(1) progress_thread = threading.Thread(target=show_progress) progress_thread.daemon = True progress_thread.start() try: return executor.results() except KeyboardInterrupt: print("\nCancelling...") executor.cancel() return None ``` ### 4. Clean Up Resources ```py def managed_evaluation(dataset, metrics): executor = None try: executor = evaluate(dataset=dataset, metrics=metrics, return_executor=True) return executor.results() except Exception as e: if executor: executor.cancel() raise finally: # Clean up any temporary resources cleanup_temp_files() ``` ## Limitations - **Async Operations**: Cancellation works at the task level, not within individual LLM calls - **Partial State**: Cancelled operations may leave partial results or temporary files - **Timing**: Cancellation is cooperative - tasks need to check for cancellation periodically - **Dependencies**: Some external services may not respect cancellation immediately ## Troubleshooting ### Cancellation Not Working ```py # Check if cancellation is set if executor.is_cancelled(): print("Cancellation was requested") else: print("Cancellation not requested yet") # Ensure you're calling cancel() executor.cancel() assert executor.is_cancelled() ``` ### Tasks Still Running After Cancellation ```py # Give time for graceful shutdown executor.cancel() time.sleep(2) # Allow tasks to detect cancellation # Force cleanup if needed import asyncio try: loop = asyncio.get_running_loop() for task in asyncio.all_tasks(loop): task.cancel() except RuntimeError: pass # No event loop running ``` The cancellation feature provides robust control over long-running Ragas operations, enabling production-ready deployments with proper resource management and user experience. ## Customize Models Ragas may use a LLM and or Embedding for evaluation and synthetic data generation. Both of these models can be customised according to your availability. Ragas provides factory functions (`llm_factory` and `embedding_factory`) that support multiple providers: - **Direct provider support**: OpenAI, Anthropic, Google - **Other providers via LiteLLM**: Azure OpenAI, AWS Bedrock, Google Vertex AI, and 100+ other providers The factory functions use the [Instructor](https://python.useinstructor.com/) library for structured outputs and [LiteLLM](https://docs.litellm.ai/) for unified access to multiple LLM providers. ## System Prompts You can provide system prompts to customize LLM behavior across all evaluations: ```python from ragas.llms import llm_factory from openai import OpenAI client = OpenAI(api_key="your-key") llm = llm_factory( "gpt-4o", client=client, system_prompt="You are a helpful assistant that evaluates RAG systems." ) ``` System prompts are particularly useful for: - Fine-tuned models that expect specific system instructions - Guiding evaluation behavior consistently - Models that require custom prompts to function properly ## Examples - [Customize Models](#customize-models) - [System Prompts](#system-prompts) - [Examples](#examples) - [Azure OpenAI](#azure-openai) - [Google Vertex](#google-vertex) - [AWS Bedrock](#aws-bedrock) ### Azure OpenAI ```bash pip install litellm ``` ```python import litellm from ragas.llms import llm_factory from ragas.embeddings.base import embedding_factory azure_configs = { "api_base": "https://.openai.azure.com/", "api_key": "your-api-key", "api_version": "2024-02-15-preview", "model_deployment": "your-deployment-name", "embedding_deployment": "your-embedding-deployment-name", } # Configure LiteLLM for Azure OpenAI (used by LLM calls) litellm.api_base = azure_configs["api_base"] litellm.api_key = azure_configs["api_key"] litellm.api_version = azure_configs["api_version"] # Create LLM using llm_factory with litellm provider # Note: Use deployment name, not model name for Azure # Important: Pass litellm.completion (the function), not the module azure_llm = llm_factory( f"azure/{azure_configs['model_deployment']}", provider="litellm", client=litellm.completion, # Optional: Add system prompt # system_prompt="You are a helpful assistant that evaluates RAG systems." ) # Create embeddings using embedding_factory # Note: Pass Azure config directly to embedding_factory azure_embeddings = embedding_factory( "litellm", model=f"azure/{azure_configs['embedding_deployment']}", api_base=azure_configs["api_base"], api_key=azure_configs["api_key"], api_version=azure_configs["api_version"], ) ``` Yay! Now you are ready to use ragas with Azure OpenAI endpoints ### Google Vertex ```bash pip install litellm google-cloud-aiplatform ``` ```python import litellm import os from ragas.llms import llm_factory from ragas.embeddings.base import embedding_factory config = { "project_id": "", "location": "us-central1", # e.g., "us-central1", "us-east1" "chat_model_id": "gemini-1.5-pro-002", "embedding_model_id": "text-embedding-005", } # Set environment variables for Vertex AI (used by litellm) os.environ["VERTEXAI_PROJECT"] = config["project_id"] os.environ["VERTEXAI_LOCATION"] = config["location"] # Create LLM using llm_factory with litellm provider # Important: Pass litellm.completion (the function), not the module vertex_llm = llm_factory( f"vertex_ai/{config['chat_model_id']}", provider="litellm", client=litellm.completion, # Optional: Add system prompt # system_prompt="You are a helpful assistant that evaluates RAG systems." ) # Create embeddings using embedding_factory # Note: Embeddings use the environment variables set above vertex_embeddings = embedding_factory( "litellm", model=f"vertex_ai/{config['embedding_model_id']}", ) ``` Yay! Now you are ready to use ragas with Google VertexAI endpoints ### AWS Bedrock ```bash pip install litellm ``` ```python import litellm import os from ragas.llms import llm_factory from ragas.embeddings.base import embedding_factory config = { "region_name": "us-east-1", # E.g. "us-east-1" "llm": "anthropic.claude-3-5-sonnet-20241022-v2:0", # Your LLM model ID "embeddings": "amazon.titan-embed-text-v2:0", # Your embedding model ID "temperature": 0.4, } # Set AWS credentials as environment variables # Option 1: Use AWS credentials file (~/.aws/credentials) # Option 2: Set environment variables directly os.environ["AWS_REGION_NAME"] = config["region_name"] # os.environ["AWS_ACCESS_KEY_ID"] = "your-access-key" # os.environ["AWS_SECRET_ACCESS_KEY"] = "your-secret-key" # Create LLM using llm_factory with litellm provider # Important: Pass litellm.completion (the function), not the module bedrock_llm = llm_factory( f"bedrock/{config['llm']}", provider="litellm", client=litellm.completion, temperature=config["temperature"], # Optional: Add system prompt # system_prompt="You are a helpful assistant that evaluates RAG systems." ) # Create embeddings using embedding_factory # Note: Embeddings use the environment variables set above bedrock_embeddings = embedding_factory( "litellm", model=f"bedrock/{config['embeddings']}", ) ``` Yay! Now you are ready to use ragas with AWS Bedrock endpoints # Customize Timeouts and Rate Limits Configure timeouts and retries directly on your LLM client when using the collections API with `llm_factory`. ## OpenAI Client Configuration ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import Faithfulness # Configure timeout and retries on the client client = AsyncOpenAI( timeout=60.0, # 60 second timeout max_retries=5, # Retry up to 5 times on failures ) llm = llm_factory("gpt-4o-mini", client=client) # Use with metrics scorer = Faithfulness(llm=llm) result = scorer.score( user_input="When was the first super bowl?", response="The first superbowl was held on Jan 15, 1967", retrieved_contexts=[ "The First AFL–NFL World Championship Game was an American football game played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles." ] ) ``` ### Available Options | Parameter | Default | Description | | ------------- | ------- | -------------------------------------------- | | `timeout` | 600.0 | Request timeout in seconds | | `max_retries` | 2 | Number of retry attempts for failed requests | ### Fine-Grained Timeout Control For more control over different timeout types: ```python import httpx from openai import AsyncOpenAI client = AsyncOpenAI( timeout=httpx.Timeout( 60.0, # Total timeout connect=5.0, # Connection timeout read=30.0, # Read timeout write=10.0, # Write timeout ), max_retries=3, ) ``` Provider Documentation Each LLM provider has its own client configuration options. Refer to your provider's SDK documentation: - [OpenAI Python SDK](https://github.com/openai/openai-python) - [Anthropic Python SDK](https://github.com/anthropics/anthropic-sdk-python) ## Legacy Metrics API The following examples use the legacy metrics API pattern with `RunConfig`. For new projects, we recommend using the collections-based API with client-level configuration as shown above. Deprecation Timeline This API will be deprecated in version 0.4 and removed in version 1.0. Please migrate to the collections-based API. ### RunConfig Parameters ```python from ragas.run_config import RunConfig run_config = RunConfig( timeout=180, # Max seconds per operation (default: 180) max_retries=10, # Retry attempts (default: 10) max_wait=60, # Max seconds between retries (default: 60) max_workers=16, # Concurrent workers (default: 16) log_tenacity=False, # Log retry attempts (default: False) seed=42, # Random seed (default: 42) ) ``` ### Usage with Evaluate ```python from langchain_openai import ChatOpenAI from ragas.llms import LangchainLLMWrapper from ragas import EvaluationDataset, SingleTurnSample, evaluate from ragas.metrics import Faithfulness from ragas.run_config import RunConfig # Legacy LLM setup llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o")) # Configure run settings run_config = RunConfig(max_workers=64, timeout=60) # Use with evaluate results = evaluate( dataset=eval_dataset, metrics=[Faithfulness(llm=llm)], run_config=run_config, ) ``` # Understand Cost and Usage of Operations When using LLMs for evaluation and test set generation, cost will be an important factor. Ragas provides you some tools to help you with that. ## Understanding `TokenUsageParser` By default, Ragas does not calculate the usage of tokens for `evaluate()`. This is because LangChain's LLMs do not always return information about token usage in a uniform way. So in order to get the usage data, we have to implement a `TokenUsageParser`. A `TokenUsageParser` is function that parses the `LLMResult` or `ChatResult` from LangChain models `generate_prompt()` function and outputs `TokenUsage` which Ragas expects. For an example here is one that will parse OpenAI by using a parser we have defined. ```python import os os.environ["OPENAI_API_KEY"] = "your-api-key" ``` ```python from langchain_openai.chat_models import ChatOpenAI from langchain_core.prompt_values import StringPromptValue gpt4o = ChatOpenAI(model="gpt-4o") p = StringPromptValue(text="hai there") llm_result = gpt4o.generate_prompt([p]) # lets import a parser for OpenAI from ragas.cost import get_token_usage_for_openai get_token_usage_for_openai(llm_result) ``` ```text /opt/homebrew/Caskroom/miniforge/base/envs/ragas/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm TokenUsage(input_tokens=9, output_tokens=9, model='') ``` You can define your own or import parsers if they are defined. If you would like to suggest parser for LLM providers or contribute your own ones please check out this [issue](https://github.com/vibrantlabsai/ragas/issues/1151) πŸ™‚. You can use it for evaluations as so. Using example from [get started](https://docs.ragas.io/en/stable/howtos/customizations/metrics/_cost/get-started-evaluation) here. ```python from datasets import load_dataset from ragas import EvaluationDataset from ragas.metrics._aspect_critic import AspectCriticWithReference dataset = load_dataset("vibrantlabsai/amnesty_qa", "english_v3") eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"]) metric = AspectCriticWithReference( name="answer_correctness", definition="is the response correct compared to reference", ) ``` ```text Repo card metadata block was not found. Setting CardData to empty. ``` ```python from ragas import evaluate from ragas.cost import get_token_usage_for_openai results = evaluate( eval_dataset[:5], metrics=[metric], llm=gpt4o, token_usage_parser=get_token_usage_for_openai, ) ``` ```text Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 5/5 [00:01<00:00, 2.81it/s] ``` ```python results.total_tokens() ``` ```text TokenUsage(input_tokens=5463, output_tokens=355, model='') ``` You can compute the cost for each run by passing in the cost per token to `Result.total_cost()` function. In this case GPT-4o costs $5 for 1M input tokens and $15 for 1M output tokens. ```python results.total_cost(cost_per_input_token=5 / 1e6, cost_per_output_token=15 / 1e6) ``` ```text 0.03264 ``` ```python ``` # Adapting Metrics to Target Language When evaluating LLM applications in languages other than English, adapt your metrics to the target language. Ragas uses an LLM to translate the few-shot examples in prompts. ## Setup ```python from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas.metrics.collections import Faithfulness client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) metric = Faithfulness(llm=llm) ``` ## Adapt Prompts to Target Language Collections metrics have prompts as direct attributes. Use the `adapt()` method to translate the few-shot examples: ```python # Check original language print(metric.statement_generator_prompt.language) # english # Adapt prompts to Hindi metric.statement_generator_prompt = await metric.statement_generator_prompt.adapt( target_language="hindi", llm=llm ) metric.nli_statement_prompt = await metric.nli_statement_prompt.adapt( target_language="hindi", llm=llm ) # Verify adaptation print(metric.statement_generator_prompt.language) # hindi # See translated example print(metric.statement_generator_prompt.examples[0][0].question) # ΰ€…ΰ€²ΰ₯ΰ€¬ΰ€°ΰ₯ΰ€Ÿ ΰ€†ΰ€‡ΰ€‚ΰ€Έΰ₯ΰ€Ÿΰ₯€ΰ€¨ ΰ€•ΰ₯Œΰ€¨ ΰ€₯ΰ₯‡ ΰ€”ΰ€° ΰ€΅ΰ₯‡ ΰ€•ΰ€Ώΰ€Έ ΰ€šΰ₯€ΰ€œΰ€Ό ΰ€•ΰ₯‡ ΰ€²ΰ€Ώΰ€ ΰ€Έΰ€¬ΰ€Έΰ₯‡ ΰ€…ΰ€§ΰ€Ώΰ€• ΰ€œΰ€Ύΰ€¨ΰ₯‡ ΰ€œΰ€Ύΰ€€ΰ₯‡ ΰ€Ήΰ₯ˆΰ€‚? ``` Note By default, only few-shot examples are translated. Instructions remain in English. To also translate instructions, set `adapt_instruction=True`. ## Evaluate with Adapted Metric ```python result = await metric.ascore( user_input="ΰ€­ΰ€Ύΰ€°ΰ€€ ΰ€•ΰ₯€ ΰ€°ΰ€Ύΰ€œΰ€§ΰ€Ύΰ€¨ΰ₯€ ΰ€•ΰ₯ΰ€―ΰ€Ύ ΰ€Ήΰ₯ˆ?", response="ΰ€­ΰ€Ύΰ€°ΰ€€ ΰ€•ΰ₯€ ΰ€°ΰ€Ύΰ€œΰ€§ΰ€Ύΰ€¨ΰ₯€ ΰ€¨ΰ€ˆ ΰ€¦ΰ€Ώΰ€²ΰ₯ΰ€²ΰ₯€ ΰ€Ήΰ₯ˆΰ₯€", retrieved_contexts=["ΰ€­ΰ€Ύΰ€°ΰ€€ ΰ€•ΰ₯€ ΰ€°ΰ€Ύΰ€œΰ€§ΰ€Ύΰ€¨ΰ₯€ ΰ€¨ΰ€ˆ ΰ€¦ΰ€Ώΰ€²ΰ₯ΰ€²ΰ₯€ ΰ€Ήΰ₯ˆ, ΰ€œΰ₯‹ ΰ€¦ΰ₯‡ΰ€Ά ΰ€•ΰ€Ύ ΰ€Έΰ€¬ΰ€Έΰ₯‡ ΰ€¬ΰ€‘ΰ€Όΰ€Ύ ΰ€Άΰ€Ήΰ€° ΰ€­ΰ₯€ ΰ€Ήΰ₯ˆΰ₯€"], ) print(f"Faithfulness: {result.value}") # Faithfulness: 1.0 ``` ## Adapting Other Metrics The same pattern works for any collections metric with prompts: ```python from ragas.metrics.collections import AnswerRelevancy from ragas.embeddings.base import embedding_factory embeddings = embedding_factory("openai", client=client) relevancy = AnswerRelevancy(llm=llm, embeddings=embeddings) # Adapt the prompt relevancy.prompt = await relevancy.prompt.adapt( target_language="spanish", llm=llm ) # See translated example print(relevancy.prompt.examples[0][0].response) # Albert Einstein naciΓ³ en Alemania. ``` ## Adapting FactualCorrectness FactualCorrectness has two prompts that both need to be adapted: ```python from ragas.metrics.collections import FactualCorrectness metric = FactualCorrectness(llm=llm) # Adapt both prompts to German metric.prompt = await metric.prompt.adapt( target_language="german", llm=llm ) metric.nli_prompt = await metric.nli_prompt.adapt( target_language="german", llm=llm ) # Verify adaptation print(metric.prompt.language) # german print(metric.nli_prompt.language) # german # Now use the adapted metric result = await metric.ascore( response="Einstein wurde 1879 in Deutschland geboren.", reference="Albert Einstein wurde am 14. MΓ€rz 1879 in Ulm, Deutschland geboren." ) print(f"Factual Correctness: {result.value}") ``` Tip Like Faithfulness, FactualCorrectness uses two prompts internally: - `prompt` - ClaimDecompositionPrompt for breaking text into claims - `nli_prompt` - NLIStatementPrompt for verifying claims Both prompts should be adapted when evaluating in non-English languages. # Modifying prompts in metrics Every metric in Ragas that uses an LLM also uses one or more prompts to generate intermediate results that are used to formulate scores. Prompts can be treated like hyperparameters when using LLM-based metrics. An optimized prompt that suits your domain and use-case can increase the accuracy of your LLM-based metrics by 10-20%. Since optimal prompts depend on the LLM being used, you may want to tune the prompts that power each metric. **Quick start**: If you need a simple custom metric, consider using DiscreteMetric or NumericMetric which accept custom prompts directly. See [Discrete Metrics](https://docs.ragas.io/en/stable/concepts/metrics/overview/#1-discrete-metrics) for examples. This guide covers modifying prompts in **existing collection metrics** (like Faithfulness, FactualCorrectness) which use the BasePrompt class. Make sure you have an understanding of the [Prompt Object documentation](https://docs.ragas.io/en/stable/concepts/components/prompt/index.md) before going further. ## Understand the prompts of your metric For metrics that support prompt customization, Ragas provides access to the underlying prompt objects through the metric instance. Let's look at how to access prompts in the `Faithfulness` metric: ```python from ragas.metrics.collections import Faithfulness from openai import AsyncOpenAI from ragas.llms import llm_factory # Setup dependencies client = AsyncOpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Create metric instance scorer = Faithfulness(llm=llm) # Faithfulness has two prompts: # 1. statement_generator_prompt - breaks response into atomic statements # 2. nli_statement_prompt - evaluates each statement against context print(scorer.statement_generator_prompt) print(scorer.nli_statement_prompt) ``` ## Generating and viewing the prompt string Let's view the prompt that will be sent to the LLM: ```python from ragas.metrics.collections.faithfulness.util import StatementGeneratorInput # Create sample input sample_input = StatementGeneratorInput( question="What is the Eiffel Tower?", answer="The Eiffel Tower is located in Paris." ) # Generate the prompt string prompt_string = scorer.statement_generator_prompt.to_string(sample_input) print(prompt_string) ``` ## Modifying prompts Modern metrics in Ragas use modular BasePrompt classes. To customize a prompt: 1. **Access the prompt**: The prompt is available as an attribute on metric instances 1. **Modify the prompt class**: Extend or subclass the prompt to customize instruction or examples 1. **Update the metric**: Assign your custom prompt to the metric's attribute ### Example: Customizing FactualCorrectness prompt FactualCorrectness uses two prompts internally: - `prompt` - ClaimDecompositionPrompt for breaking text into claims - `nli_prompt` - NLIStatementPrompt for verifying claims against context You can customize either or both: ```python from ragas.metrics.collections import FactualCorrectness from ragas.metrics.collections.factual_correctness.util import ( ClaimDecompositionPrompt, NLIStatementPrompt, ) # Create a custom claim decomposition prompt by subclassing class CustomClaimDecompositionPrompt(ClaimDecompositionPrompt): instruction = """You are an expert at breaking down complex statements into atomic claims. Break down the input text into clear, verifiable claims. Only output valid JSON with a "claims" array.""" # Optionally customize the NLI prompt too class CustomNLIPrompt(NLIStatementPrompt): instruction = """Carefully evaluate if each statement is supported by the context. Be strict in your verification - only mark as supported if directly stated.""" # Create metric instance and replace prompts scorer = FactualCorrectness(llm=llm) scorer.prompt = CustomClaimDecompositionPrompt() scorer.nli_prompt = CustomNLIPrompt() # Now the metric will use the custom prompts result = await scorer.ascore( response="The Eiffel Tower is in Paris and was built in 1889.", reference="The Eiffel Tower is located in Paris. It was completed in 1889." ) ``` ### Example: Customizing Faithfulness examples Few-shot examples can greatly influence LLM outputs. Here's how to modify them: ```python from ragas.metrics.collections import Faithfulness from ragas.metrics.collections.faithfulness.util import ( NLIStatementInput, NLIStatementOutput, NLIStatementPrompt, StatementFaithfulnessAnswer, ) # Create custom prompt with domain-specific examples class DomainSpecificNLIPrompt(NLIStatementPrompt): examples = [ ( NLIStatementInput( context="Machine learning is a field within artificial intelligence that enables systems to learn from data.", statements=[ "Machine learning is a subset of AI.", "Machine learning uses statistical techniques.", ], ), NLIStatementOutput( statements=[ StatementFaithfulnessAnswer( statement="Machine learning is a subset of AI.", reason="The context states ML is 'a field within artificial intelligence', supporting this claim.", verdict=1 ), StatementFaithfulnessAnswer( statement="Machine learning uses statistical techniques.", reason="The context doesn't mention statistical techniques.", verdict=0 ), ] ), ), ] # Update the metric with custom prompt scorer = Faithfulness(llm=llm) scorer.nli_statement_prompt = DomainSpecificNLIPrompt() # Now evaluate with domain-specific prompts result = await scorer.ascore( user_input="How do neural networks work?", response="Neural networks are inspired by biological neurons.", retrieved_contexts=["Artificial neural networks are computing systems loosely inspired by biological neural networks."] ) ``` ## Adapting prompts to different languages You can adapt prompts to different languages using the `adapt` method: ```python from ragas.metrics.collections import Faithfulness scorer = Faithfulness(llm=llm) # Adapt the statement generator prompt to Spanish adapted_prompt = await scorer.statement_generator_prompt.adapt( target_language="spanish", llm=llm, adapt_instruction=False # Keep instruction in English, only translate examples ) # Replace the prompt with the adapted version scorer.statement_generator_prompt = adapted_prompt # Now use the metric with Spanish examples result = await scorer.ascore( user_input="ΒΏDΓ³nde naciΓ³ Einstein?", response="Einstein naciΓ³ en Alemania.", retrieved_contexts=["Albert Einstein naciΓ³ en Alemania..."] ) ``` ## Verifying your customizations Here's how to verify your prompt customizations work: ```python from ragas.metrics.collections.faithfulness.util import NLIStatementInput # Create sample input to test the prompt sample_input = NLIStatementInput( context="Paris is the capital and most populous city of France.", statements=["The capital of France is Paris.", "Paris is in Germany."] ) # Generate and view the full prompt string full_prompt = scorer.nli_statement_prompt.to_string(sample_input) print("Full Prompt:") print(full_prompt) ``` # Tracing and logging evaluations with Observability tools Logging and tracing results from LLM are important for any language model-based application. This is a tutorial on how to do tracing with Ragas. Ragas provides `callbacks` functionality which allows you to hook various tracers like LangSmith, wandb, Opik, etc easily. In this notebook, I will be using LangSmith for tracing. To set up LangSmith, we need to set some environment variables that it needs. For more information, you can refer to the [docs](https://docs.smith.langchain.com/) ```bash export LANGCHAIN_TRACING_V2=true export LANGCHAIN_ENDPOINT=https://api.smith.langchain.com export LANGCHAIN_API_KEY= export LANGCHAIN_PROJECT= # if not specified, defaults to "default" ``` Now we have to import the required tracer from LangChain, here we are using `LangChainTracer`, but you can similarly use any tracer supported by LangChain like [WandbTracer](https://python.langchain.com/docs/integrations/providers/wandb_tracing) or [OpikTracer](https://comet.com/docs/opik/tracing/integrations/ragas?utm_source=ragas&utm_medium=docs&utm_campaign=opik&utm_content=tracing_how_to) ```python # LangSmith from langchain.callbacks.tracers import LangChainTracer tracer = LangChainTracer(project_name="callback-experiments") ``` We now pass the tracer to the `callbacks` parameter when calling `evaluate` ```python from ragas import EvaluationDataset from datasets import load_dataset from ragas.metrics import LLMContextRecall dataset = load_dataset("vibrantlabsai/amnesty_qa", "english_v3") dataset = EvaluationDataset.load_from_hf(dataset["eval"]) evaluate(dataset, metrics=[LLMContextRecall()],callbacks=[tracer]) ``` ```text {'context_precision': 1.0000} ``` Tracing with LangSmith You can also write your own custom callbacks using LangChain’s `BaseCallbackHandler`, refer [here](https://www.notion.so/Docs-logging-and-tracing-6f21cde9b3cb4d499526f48fd615585d?pvs=21) to read more about it. # DSPy Optimizer for Advanced Prompt Optimization The DSPyOptimizer provides state-of-the-art prompt optimization for Ragas metrics using DSPy's MIPROv2 algorithm. It combines instruction and demonstration optimization to find better prompts than simple evolutionary approaches. ## Overview **DSPyOptimizer** uses MIPROv2 (Multi-prompt Instruction Proposal with Ranked Outcomes) to optimize metric prompts through: - **Instruction optimization**: Generates and tests multiple prompt variations - **Demonstration optimization**: Automatically selects effective few-shot examples - **Combined search**: Explores both instruction and demonstration spaces simultaneously This typically produces better results than the simpler GeneticOptimizer, especially when you have high-quality annotated data. ## Installation DSPy is an optional dependency. Install it with: ```bash # Using uv (recommended) uv add "ragas[dspy]" # Using pip pip install "ragas[dspy]" ``` ## Basic Usage ### Prerequisites You need: 1. **Annotated dataset**: Ground truth scores for your metric 1. **Metric with prompts**: A metric that uses PydanticPrompt (most Ragas metrics) 1. **LLM**: An LLM for optimization (gpt-4o-mini recommended for cost) ### Quick Start ```python from openai import OpenAI from ragas.llms import llm_factory from ragas.metrics.collections import Faithfulness from ragas.optimizers import DSPyOptimizer from ragas.config import InstructionConfig # Setup LLM for optimization client = OpenAI() llm = llm_factory("gpt-4o-mini", client=client) # Initialize metric metric = Faithfulness(llm=llm) # Create annotated dataset (see below for format) dataset = create_annotated_dataset() # Configure DSPy optimizer config = InstructionConfig( llm=llm, optimizer=DSPyOptimizer( num_candidates=10, # Try 10 prompt variations max_bootstrapped_demos=5, # Generate up to 5 examples max_labeled_demos=5, # Use up to 5 human annotations ) ) # Optimize the metric's prompts metric.optimize_prompts(dataset, config) # Save optimized prompts for reuse metric.save_prompts("optimized_faithfulness.json") ``` ### Annotated Dataset Format DSPy optimizer requires ground truth annotations: ```python from ragas.dataset_schema import ( PromptAnnotation, SampleAnnotation, SingleMetricAnnotation ) # Create prompt annotations prompt_annotation = PromptAnnotation( prompt_input={"user_input": "...", "response": "..."}, prompt_output={"score": 0.9}, # Actual metric output edited_output=None, # Or corrected output if needed ) # Create sample with annotations sample = SampleAnnotation( metric_input={"user_input": "...", "response": "..."}, metric_output=0.9, # Ground truth score prompts={"faithfulness_prompt": prompt_annotation}, is_accepted=True, # Whether to use in optimization ) # Create dataset dataset = SingleMetricAnnotation( name="faithfulness", samples=[sample, ...] # Need 20-50+ samples for best results ) ``` ## Advanced Configuration ### Optimization Parameters Control MIPROv2 behavior: ```python optimizer = DSPyOptimizer( num_candidates=20, # More candidates = better prompts, higher cost max_bootstrapped_demos=10, # Auto-generated few-shot examples max_labeled_demos=10, # Human-annotated examples to use init_temperature=1.0, # Exploration temperature (0.0-2.0) ) ``` **Parameter Guide:** | Parameter | Default | Description | Cost Impact | | ------------------------ | ------- | ------------------------ | ------------------------ | | `num_candidates` | 10 | Prompt variations to try | High - linear scaling | | `max_bootstrapped_demos` | 5 | Auto-generated examples | Medium - adds LLM calls | | `max_labeled_demos` | 5 | Human annotations to use | Low - uses existing data | | `init_temperature` | 1.0 | Exploration randomness | None - algorithmic only | ### Cost Optimization MIPROv2 optimization can be expensive. Reduce costs by: ```python # Budget-conscious configuration budget_optimizer = DSPyOptimizer( num_candidates=5, # Fewer candidates max_bootstrapped_demos=2, # Fewer generated examples max_labeled_demos=3, # More reliance on annotations init_temperature=0.5, # Less exploration ) # Use cheaper LLM for optimization cheap_llm = llm_factory("gpt-4o-mini", client=client) config = InstructionConfig(llm=cheap_llm, optimizer=budget_optimizer) ``` **Cost Estimation:** - ~10-50 LLM calls per candidate - ~5-10 calls per bootstrapped demo - Total: `num_candidates * 30 + max_bootstrapped_demos * 7` calls (approximate) ## Comparing with GeneticOptimizer ### When to Use DSPyOptimizer βœ… **Use DSPyOptimizer when:** - You have 50+ high-quality annotated examples - You need the best possible metric accuracy - You can afford 100-500 LLM calls for optimization - You're optimizing critical production metrics ### When to Use GeneticOptimizer βœ… **Use GeneticOptimizer when:** - You have limited annotated data (\<20 examples) - You need faster, cheaper optimization - You're doing initial prototyping - Simple instruction-only optimization is sufficient ### Side-by-Side Comparison ```python from ragas.optimizers import GeneticOptimizer, DSPyOptimizer # Genetic optimizer - simpler, faster, cheaper genetic_config = InstructionConfig( llm=llm, optimizer=GeneticOptimizer( max_steps=50, # Evolution steps population_size=10, # Population per generation ) ) # DSPy optimizer - advanced, better results, more expensive dspy_config = InstructionConfig( llm=llm, optimizer=DSPyOptimizer( num_candidates=10, max_bootstrapped_demos=5, max_labeled_demos=5, ) ) # Compare results metric_genetic = Faithfulness(llm=llm) metric_genetic.optimize_prompts(dataset, genetic_config) metric_dspy = Faithfulness(llm=llm) metric_dspy.optimize_prompts(dataset, dspy_config) # Evaluate on holdout set test_scores_genetic = metric_genetic.batch_score(test_set) test_scores_dspy = metric_dspy.batch_score(test_set) ``` **Typical Results:** | Metric | GeneticOptimizer | DSPyOptimizer | Improvement | | ----------------- | ---------------- | ------------- | ----------- | | Faithfulness | 0.82 | 0.89 | +8.5% | | Answer Relevancy | 0.75 | 0.84 | +12% | | Context Precision | 0.78 | 0.86 | +10% | ## Working with Multiple Metrics Optimize several metrics with the same approach: ```python from ragas.metrics.collections import ( Faithfulness, AnswerRelevancy, ContextPrecision ) metrics = { "faithfulness": Faithfulness(llm=llm), "answer_relevancy": AnswerRelevancy(llm=llm), "context_precision": ContextPrecision(llm=llm), } # Optimize each metric for name, metric in metrics.items(): print(f"Optimizing {name}...") # Load metric-specific dataset dataset = load_annotated_dataset(name) # Optimize metric.optimize_prompts(dataset, dspy_config) # Save metric.save_prompts(f"optimized_{name}.json") ``` ## Troubleshooting ### Import Error If you get `ImportError: DSPy optimizer requires dspy-ai`: ```bash # Install the DSPy extra uv add "ragas[dspy]" # or pip install "ragas[dspy]" ``` ### Optimization Takes Too Long Reduce the number of LLM calls: ```python fast_optimizer = DSPyOptimizer( num_candidates=3, # Minimum viable max_bootstrapped_demos=1, max_labeled_demos=3, ) ``` ### Poor Results Common causes: 1. **Insufficient data**: Need 20+ high-quality annotations 1. **Low-quality annotations**: Ensure ground truth scores are accurate 1. **Wrong LLM**: Use gpt-4o or better for optimization 1. **Bad configuration**: Try default parameters first ### Memory Issues MIPROv2 can use significant memory for large datasets: ```python # Process in smaller batches from ragas.dataset_schema import SingleMetricAnnotation def optimize_in_batches(dataset, batch_size=20): # Split dataset batches = [ dataset.select(range(i, min(i + batch_size, len(dataset.samples)))) for i in range(0, len(dataset.samples), batch_size) ] # Optimize on first batch for speed best_batch = batches[0] metric.optimize_prompts(best_batch, dspy_config) ``` ## Best Practices ### Data Quality 1. **Diverse examples**: Cover edge cases and common scenarios 1. **Accurate labels**: Double-check ground truth scores 1. **Sufficient quantity**: 50+ examples for production metrics ### Optimization Strategy 1. **Start small**: Test with 3-5 candidates first 1. **Iterate**: Gradually increase parameters as needed 1. **Validate**: Always test on a holdout set 1. **Cache**: Save optimized prompts to avoid re-running ### Production Deployment ```python # 1. Optimize offline metric = Faithfulness(llm=optimization_llm) metric.optimize_prompts(training_dataset, dspy_config) metric.save_prompts("production_faithfulness.json") # 2. Load in production production_metric = Faithfulness(llm=production_llm) production_metric.load_prompts("production_faithfulness.json") # 3. Use for evaluation results = production_metric.batch_score(production_samples) ``` ## See Also - [Optimizers API Reference](https://docs.ragas.io/en/stable/references/optimizers/index.md) - Full API documentation - [Metric Customization](https://docs.ragas.io/en/stable/howtos/metrics/custom-metrics.md) - Creating custom metrics - [DSPy Documentation](https://dspy-docs.vercel.app/) - Learn more about DSPy # Customizing Test Data Generation Synthetic test generation can save a lot of time and effort in creating test datasets for evaluating AI applications. We are working on adding more support to customized test set generation. If you have any specific requirements or would like to collaborate on this, please [talk to us](https://cal.com/team/vibrantlabs/office-hours). ## Synthetic test generation from non-English corpus In this notebook, you'll learn how to adapt synthetic test data generation to non-English corpus settings. For the sake of this tutorial, I am generating queries in Spanish from Spanish Wikipedia articles. ### Download and Load corpus ```python ! git clone https://huggingface.co/datasets/vibrantlabsai/Sample_non_english_corpus ``` ```text Cloning into 'Sample_non_english_corpus'... remote: Enumerating objects: 12, done. remote: Counting objects: 100% (8/8), done. remote: Compressing objects: 100% (8/8), done. remote: Total 12 (delta 0), reused 0 (delta 0), pack-reused 4 (from 1) Unpacking objects: 100% (12/12), 11.43 KiB | 780.00 KiB/s, done. ``` ```python from langchain_community.document_loaders import DirectoryLoader, TextLoader path = "Sample_non_english_corpus/" loader = DirectoryLoader(path, glob="**/*.txt") docs = loader.load() ``` ```text /opt/homebrew/Caskroom/miniforge/base/envs/ragas/lib/python3.9/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.20) or chardet (5.2.0)/charset_normalizer (None) doesn't match a supported version! warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported " ``` ```python len(docs) ``` ```text 6 ``` ### Initialize required models ```python from ragas.llms import LangchainLLMWrapper from ragas.embeddings import OpenAIEmbeddings from langchain_openai import ChatOpenAI import openai generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini")) openai_client = openai.OpenAI() generator_embeddings = OpenAIEmbeddings(client=openai_client) ``` ```text /opt/homebrew/Caskroom/miniforge/base/envs/ragas/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm ``` ### Setup Persona and transforms you may automatically create personas using this [notebook](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_persona_generator/index.md). For the sake of simplicity, I am using a pre-defined person, two basic transforms and simple query distribution. ```python from ragas.testset.persona import Persona personas = [ Persona( name="curious student", role_description="A student who is curious about the world and wants to learn more about different cultures and languages", ), ] ``` ```python from ragas.testset.transforms.extractors.llm_based import NERExtractor from ragas.testset.transforms.splitters import HeadlineSplitter transforms = [HeadlineSplitter(), NERExtractor()] ``` ### Initialize test generator ```python from ragas.testset import TestsetGenerator generator = TestsetGenerator( llm=generator_llm, embedding_model=generator_embeddings, persona_list=personas ) ``` ### Load and Adapt Queries Here we load the required query types and adapt them to the target language. ```python from ragas.testset.synthesizers.single_hop.specific import ( SingleHopSpecificQuerySynthesizer, ) distribution = [ (SingleHopSpecificQuerySynthesizer(llm=generator_llm), 1.0), ] for query, _ in distribution: prompts = await query.adapt_prompts("spanish", llm=generator_llm) query.set_prompts(**prompts) ``` ### Generate ```python dataset = generator.generate_with_langchain_docs( docs[:], testset_size=5, transforms=transforms, query_distribution=distribution, ) ``` ```text Applying HeadlineSplitter: 0%| | 0/6 [00:00= n: break themes = node.properties.get(property_name, [""]) prompt_input = ThemesPersonasInput(themes=themes, personas=persona_list) persona_concepts = await self.theme_persona_matching_prompt.generate( data=prompt_input, llm=self.llm, callbacks=callbacks ) base_scenarios = self.prepare_combinations( node, themes, personas=persona_list, persona_concepts=persona_concepts.mapping, ) scenarios.extend( self.sample_combinations(base_scenarios, number_of_samples_per_node) ) return scenarios query = MySingleHopScenario(llm=llm) scenarios = await query.generate_scenarios( n=5, knowledge_graph=kg, persona_list=persona_list ) scenarios[0] ``` Output ```text SingleHopScenario( nodes=1 term=what is an ally persona=name='Hiring manager at gitlab' role_description='A hiring manager at gitlab trying to underestand hiring policies in gitlab' style=Web search like queries length=long) ``` ```python result = await query.generate_sample(scenario=scenarios[-1]) ``` ### Modify prompt to customize the query style Here I am replacing the default prompt with an instruction to generate only Yes/No questions. This is an optional step. ```python instruction = """Generate a Yes/No query and answer based on the specified conditions (persona, term, style, length) and the provided context. Ensure the answer is entirely faithful to the context, using only the information directly from the provided context. ### Instructions: 1. **Generate a Yes/No Query**: Based on the context, persona, term, style, and length, create a question that aligns with the persona's perspective, incorporates the term, and can be answered with 'Yes' or 'No'. 2. **Generate an Answer**: Using only the content from the provided context, provide a 'Yes' or 'No' answer to the query. Do not add any information not included in or inferable from the context.""" ``` ```python prompt = query.get_prompts()["generate_query_reference_prompt"] prompt.instruction = instruction query.set_prompts(**{"generate_query_reference_prompt": prompt}) result = await query.generate_sample(scenario=scenarios[-1]) ``` ```python result.user_input ``` Output ```text 'Does the Diversity, Inclusion & Belonging (DIB) Team at GitLab have a structured approach to encourage collaborations among team members through various communication methods?' ``` ```python result.reference ``` Output ```text 'Yes' ``` # Create custom multi-hop queries from your documents In this tutorial you will get to learn how to create custom multi-hop queries from your documents. This is a very powerful feature that allows you to create queries that are not possible with the standard query types. This also helps you to create queries that are more specific to your use case. ### Load sample documents I am using documents from [sample of GitLab handbook](https://huggingface.co/datasets/vibrantlabsai/Sample_Docs_Markdown). You can download it by running the below command. ```python ! git clone https://huggingface.co/datasets/vibrantlabsai/Sample_Docs_Markdown ``` ```python from langchain_community.document_loaders import DirectoryLoader, TextLoader path = "Sample_Docs_Markdown/" loader = DirectoryLoader(path, glob="**/*.md") docs = loader.load() ``` ### Create KG Create a base knowledge graph with the documents ```python from ragas.testset.graph import KnowledgeGraph from ragas.testset.graph import Node, NodeType kg = KnowledgeGraph() for doc in docs: kg.nodes.append( Node( type=NodeType.DOCUMENT, properties={ "page_content": doc.page_content, "document_metadata": doc.metadata, }, ) ) ``` ### Set up the LLM and Embedding Model You may use any of [your choice](https://docs.ragas.io/en/stable/howtos/customizations/customize_models/index.md), here I am using models from open-ai. ```python from openai import OpenAI from ragas.llms import llm_factory from ragas.embeddings import OpenAIEmbeddings openai_client = OpenAI() llm = llm_factory("gpt-4o-mini", client=openai_client) embedding = OpenAIEmbeddings(client=openai_client) ``` ### Setup Extractors and Relationship builders To create multi-hop queries you need to understand the set of documents that can be used for it. Ragas uses relationships between documents/nodes to quality nodes for creating multi-hop queries. To concretize, if Node A and Node B are connected by a relationship (say entity or keyphrase overlap) then you can create a multi-hop query between them. Here we are using 2 extractors and 2 relationship builders. - Headline extractor: Extracts headlines from the documents - Keyphrase extractor: Extracts keyphrases from the documents - Headline splitter: Splits the document into nodes based on headlines - OverlapScore Builder: Builds relationship between nodes based on keyphrase overlap ```python from ragas.testset.transforms import Parallel, apply_transforms from ragas.testset.transforms import ( HeadlinesExtractor, HeadlineSplitter, KeyphrasesExtractor, OverlapScoreBuilder, ) headline_extractor = HeadlinesExtractor(llm=llm) headline_splitter = HeadlineSplitter(min_tokens=300, max_tokens=1000) keyphrase_extractor = KeyphrasesExtractor( llm=llm, property_name="keyphrases", max_num=10 ) relation_builder = OverlapScoreBuilder( property_name="keyphrases", new_property_name="overlap_score", threshold=0.01, distance_threshold=0.9, ) transforms = [ headline_extractor, headline_splitter, keyphrase_extractor, relation_builder, ] apply_transforms(kg, transforms=transforms) ``` Output ```text Applying KeyphrasesExtractor: 6%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ– | 2/36 [00:01<00:17, 1.94it/s]Property 'keyphrases' already exists in node 'a2f389'. Skipping! Applying KeyphrasesExtractor: 17%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹ | 6/36 [00:01<00:04, 6.37it/s]Property 'keyphrases' already exists in node '3068c0'. Skipping! Applying KeyphrasesExtractor: 53%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 19/36 [00:02<00:01, 8.88it/s]Property 'keyphrases' already exists in node '854bf7'. Skipping! Applying KeyphrasesExtractor: 78%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Ž | 28/36 [00:03<00:00, 9.73it/s]Property 'keyphrases' already exists in node '2eeb07'. Skipping! Property 'keyphrases' already exists in node 'd68f83'. Skipping! Applying KeyphrasesExtractor: 83%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–Œ | 30/36 [00:03<00:00, 9.35it/s]Property 'keyphrases' already exists in node '8fdbea'. Skipping! Applying KeyphrasesExtractor: 89%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‹ | 32/36 [00:04<00:00, 7.76it/s]Property 'keyphrases' already exists in node 'ef6ae0'. Skipping! ``` ### Configure personas You can also do this automatically by using the [automatic persona generator](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/_persona_generator/index.md) ```python from ragas.testset.persona import Persona person1 = Persona( name="gitlab employee", role_description="A junior gitlab employee curious on workings on gitlab", ) persona2 = Persona( name="Hiring manager at gitlab", role_description="A hiring manager at gitlab trying to underestand hiring policies in gitlab", ) persona_list = [person1, persona2] ``` ### Create multi-hop query Inherit from `MultiHopQuerySynthesizer` and modify the function that generates scenarios for query creation. **Steps**: - find qualified set of (nodeA, relationship, nodeB) based on the relationships between nodes - For each qualified set - Match the keyphrase with one or more persona. - Create all possible combinations of (Nodes, Persona, Query Style, Query Length) - Samples the required number of queries from the combinations ```python from dataclasses import dataclass import typing as t from ragas.testset.synthesizers.multi_hop.base import ( MultiHopQuerySynthesizer, MultiHopScenario, ) from ragas.testset.synthesizers.prompts import ( ThemesPersonasInput, ThemesPersonasMatchingPrompt, ) @dataclass class MyMultiHopQuery(MultiHopQuerySynthesizer): theme_persona_matching_prompt = ThemesPersonasMatchingPrompt() async def _generate_scenarios( self, n: int, knowledge_graph, persona_list, callbacks, ) -> t.List[MultiHopScenario]: # query and get (node_a, rel, node_b) to create multi-hop queries results = kg.find_two_nodes_single_rel( relationship_condition=lambda rel: ( True if rel.type == "keyphrases_overlap" else False ) ) num_sample_per_triplet = max(1, n // len(results)) scenarios = [] for triplet in results: if len(scenarios) < n: node_a, node_b = triplet[0], triplet[-1] overlapped_keywords = triplet[1].properties["overlapped_items"] if overlapped_keywords: # match the keyword with a persona for query creation themes = list(dict(overlapped_keywords).keys()) prompt_input = ThemesPersonasInput( themes=themes, personas=persona_list ) persona_concepts = ( await self.theme_persona_matching_prompt.generate( data=prompt_input, llm=self.llm, callbacks=callbacks ) ) overlapped_keywords = [list(item) for item in overlapped_keywords] # prepare and sample possible combinations base_scenarios = self.prepare_combinations( [node_a, node_b], overlapped_keywords, personas=persona_list, persona_item_mapping=persona_concepts.mapping, property_name="keyphrases", ) # get number of required samples from this triplet base_scenarios = self.sample_diverse_combinations( base_scenarios, num_sample_per_triplet ) scenarios.extend(base_scenarios) return scenarios query = MyMultiHopQuery(llm=llm) scenarios = await query.generate_scenarios( n=10, knowledge_graph=kg, persona_list=persona_list ) scenarios[4] ``` Output ```text MultiHopScenario( nodes=2 combinations=['Diversity Inclusion & Belonging', 'Diversity, Inclusion & Belonging Goals'] style=Web search like queries length=short persona=name='Hiring manager at gitlab' role_description='A hiring manager at gitlab trying to underestand hiring policies in gitlab') ``` ### Run the multi-hop query ```python result = await query.generate_sample(scenario=scenarios[-1]) result.user_input ``` Output ```text 'How does GitLab ensure that its DIB roundtables are effective in promoting diversity and inclusion?' ``` Yay! You have created a multi-hop query. Now you can create any such queries by creating and exploring relationships between documents. ## # Using Pre-chunked Data for Testset Generation When you already have a well-defined chunking strategy in place, Ragas allows you to bypass its internal document splitting mechanism and use your own chunks directly. This is particularly useful when: - You've optimized your chunking strategy for your specific domain - You want to maintain consistency between your RAG pipeline and evaluation - You have pre-processed documents with custom metadata - You need to ensure chunks align with specific business logic or document structure ## Overview The `generate_with_chunks` method of `TestsetGenerator` accepts pre-chunked data and treats each chunk as a `NodeType.CHUNK` directly, skipping the internal splitting transforms. This means your chunks remain exactly as you provide them, preserving both content and metadata integrity. ## How It Works When you use `generate_with_chunks`, Ragas: 1. **Accepts your chunks** as-is (either as `Document` objects or strings) 1. **Applies extractors** like `SummaryExtractor`, `ThemesExtractor`, `NERExtractor`, and `EmbeddingExtractor` to enrich each chunk with additional properties 1. **Builds relationships** between chunks using `CosineSimilarityBuilder` and `OverlapScoreBuilder` 1. **Generates personas** based on the content themes 1. **Creates scenarios** for different query types (single-hop, multi-hop) 1. **Synthesizes test samples** including questions, contexts, and reference answers ## Example: Using Pre-chunked Documents You can pass a list of LangChain `Document` objects. This approach preserves the metadata of your chunks, which can be useful for tracking source documents or other custom information. ```python import os from langchain_core.documents import Document from ragas.testset.synthesizers.generate import TestsetGenerator from ragas.llms import llm_factory from ragas.embeddings import OpenAIEmbeddings from openai import OpenAI # Initialize OpenAI client client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY")) # Initialize generator with your preferred models generator = TestsetGenerator( llm=llm_factory("gpt-4o-mini", client=client), embedding_model=OpenAIEmbeddings(client=client) ) # Your pre-chunked documents chunks = [ Document( page_content="""The Eiffel Tower (Tour Eiffel) is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower. Locally nicknamed "La Dame de Fer" (French for "The Iron Lady"), it was constructed from 1887 to 1889 as the centerpiece of the 1889 World's Fair. Although initially criticized by some of France's leading artists and intellectuals for its design, it has since become a global cultural icon of France and one of the most recognizable structures in the world.""", metadata={"source": "doc1", "chunk_id": 1} ), Document( page_content="""The tower is 330 metres (1,083 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).""", metadata={"source": "doc1", "chunk_id": 2} ) ] # Generate testset testset = generator.generate_with_chunks( chunks=chunks, testset_size=10 ) # Save to CSV output_file = "testset.csv" testset.to_csv(output_file) print(f"Testset saved to {output_file}") print(testset.to_pandas().head()) ``` ### Generation Process During generation, you'll see progress logs showing the various transformation and synthesis stages: ```text Applying SummaryExtractor: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:07<00:00, 3.67s/it] Applying CustomNodeFilter: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:00<00:00, 2226.87it/s] Applying EmbeddingExtractor: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:02<00:00, 1.19s/it] Applying ThemesExtractor: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:06<00:00, 3.07s/it] Applying NERExtractor: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:06<00:00, 3.10s/it] Applying CosineSimilarityBuilder: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 613.29it/s] Applying OverlapScoreBuilder: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 1/1 [00:00<00:00, 1491.57it/s] Generating personas: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:05<00:00, 2.77s/it] Generating Scenarios: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:08<00:00, 4.19s/it] Generating Samples: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 11/11 [00:45<00:00, 4.13s/it] Testset saved to testset.csv ``` The testset includes different types of queries: - **Single-hop queries**: Questions that can be answered from a single chunk - **Multi-hop queries**: Questions requiring information from multiple chunks (when relationships exist) ## Example: Using Plain Strings If you don't need to preserve metadata, you can also pass plain strings directly: ```python from ragas.testset.synthesizers.generate import TestsetGenerator from ragas.llms import llm_factory from ragas.embeddings import OpenAIEmbeddings from openai import OpenAI # Initialize models client = OpenAI() generator = TestsetGenerator( llm=llm_factory("gpt-4o-mini", client=client), embedding_model=OpenAIEmbeddings(client=client) ) # Simple text chunks text_chunks = [ "Artificial Intelligence (AI) is the simulation of human intelligence by machines. It involves machine learning, natural language processing, and computer vision.", "Machine Learning is a subset of AI that enables systems to learn from data without explicit programming. Popular algorithms include neural networks and decision trees.", "Deep Learning uses neural networks with multiple layers to process complex patterns in large datasets. It powers modern applications like image recognition and language translation." ] # Generate testset testset = generator.generate_with_chunks( chunks=text_chunks, testset_size=5 ) # Save to CSV output_file = "testset.csv" testset.to_csv(output_file) print(f"Testset saved to {output_file}") print(testset.to_pandas()) ``` ## Handling Edge Cases - **Empty Content**: Chunks with empty or whitespace-only `page_content` will be automatically filtered out. - **Empty Sequence**: If you provide an empty sequence of chunks, the generation will produce an empty testset. # Application Guides # Applications Ragas in action. Examples of how to use Ragas in various applications and usecases to solve problems you might encounter when you're building. ## Prompt Evaluation - [Iterate and Improve Prompts](https://docs.ragas.io/en/stable/howtos/applications/iterate_prompt/index.md) - [Systematic Prompt Optimization](https://docs.ragas.io/en/stable/howtos/applications/prompt_optimization/index.md) ## Metrics - [Debug LLM based metrics using tracing](https://docs.ragas.io/en/stable/howtos/applications/_metrics_llm_calls.md) - [Evaluating Multi-turn Conversations](https://docs.ragas.io/en/stable/howtos/applications/evaluating_multi_turn_conversations/index.md) - [Estimate cost of evaluation](https://docs.ragas.io/en/stable/howtos/applications/_cost/index.md) - [Evaluations with Vertex AI models](https://docs.ragas.io/en/stable/howtos/applications/vertexai_x_ragas/index.md) ## Testset Generation - [Single-hop Query Testset](https://docs.ragas.io/en/stable/howtos/applications/singlehop_testset_gen/index.md) ## Benchmarking - [Evaluate a New LLM For Your Use Case](https://docs.ragas.io/en/stable/howtos/applications/benchmark_llm/index.md) ## RAG Evaluation - [Evaluate and Improve a RAG System](https://docs.ragas.io/en/stable/howtos/applications/evaluate-and-improve-rag/index.md) ## Agent Evaluation - [Evaluate a Text-to-SQL Agent](https://docs.ragas.io/en/stable/howtos/applications/text2sql/index.md) # How to estimate Cost and Usage of evaluations and testset generation When using LLMs for evaluation and test set generation, cost will be an important factor. Ragas provides you some tools to help you with that. ## Implement `TokenUsageParser` By default, Ragas does not calculate the usage of tokens for `evaluate()`. This is because langchain's LLMs do not always return information about token usage in a uniform way. So in order to get the usage data, we have to implement a `TokenUsageParser`. A `TokenUsageParser` is function that parses the `LLMResult` or `ChatResult` from langchain models `generate_prompt()` function and outputs `TokenUsage` which Ragas expects. For an example here is one that will parse OpenAI by using a parser we have defined. ```python from langchain_openai.chat_models import ChatOpenAI from langchain_core.prompt_values import StringPromptValue gpt4o = ChatOpenAI(model="gpt-4o") p = StringPromptValue(text="hai there") llm_result = gpt4o.generate_prompt([p]) # lets import a parser for OpenAI from ragas.cost import get_token_usage_for_openai get_token_usage_for_openai(llm_result) ``` Output ```text TokenUsage(input_tokens=9, output_tokens=9, model='') ``` You can define your own or import parsers if they are defined. If you would like to suggest parser for LLM providers or contribute your own ones please check out this [issue](https://github.com/vibrantlabsai/ragas/issues/1151) πŸ™‚. ## Token Usage for Evaluations Let's use the `get_token_usage_for_openai` parser to calculate the token usage for an evaluation. ```python from ragas import EvaluationDataset from datasets import load_dataset dataset = load_dataset("vibrantlabsai/amnesty_qa", "english_v3") eval_dataset = EvaluationDataset.from_hf_dataset(dataset["eval"]) ``` Output ```text Repo card metadata block was not found. Setting CardData to empty. ``` You can pass in the parser to the `evaluate()` function and the cost will be calculated and returned in the `Result` object. ```python from ragas import evaluate from ragas.metrics import LLMContextRecall from ragas.cost import get_token_usage_for_openai result = evaluate( eval_dataset, metrics=[LLMContextRecall()], llm=gpt4o, token_usage_parser=get_token_usage_for_openai, ) ``` Output ```text Evaluating: 0%| | 0/20 [00:00= 0.9 assert result["context_recall"] >= 0.95 assert result["context_precision"] >= 0.95 assert_in_range(result["faithfulness"], value=0.4, plus_or_minus=0.1) ``` ## Using Pytest Markers for Ragas E2E tests Because these are long end-to-end test one thing that you can leverage is [Pytest Markers](https://docs.pytest.org/en/latest/example/markers.html) which help you mark your tests with special tags. It is recommended to mark Ragas tests with special tags, so you can run them only when needed. To add a new `ragas_ci` tag to Pytest, add the following to your `conftest.py` ```python def pytest_configure(config): """ configure pytest """ # add `ragas_ci` config.addinivalue_line( "markers", "ragas_ci: Set of tests that will be run as part of Ragas CI" ) ``` now you can use `ragas_ci` to mark all the tests that are part of Ragas CI. ```python import pytest from datasets import load_dataset from ragas import evaluate from ragas.metrics import ( answer_relevancy, faithfulness, context_recall, context_precision, ) def assert_in_range(score: float, value: float, plus_or_minus: float): """ Check if computed score is within the range of value +/- max_range """ assert value - plus_or_minus <= score <= value + plus_or_minus @pytest.mark.ragas_ci def test_amnesty_e2e(): # loading the V2 dataset amnesty_qa = load_dataset("vibrantlabsai/amnesty_qa", "english_v2")["eval"] result = evaluate( amnesty_qa, metrics=[answer_relevancy, faithfulness, context_recall, context_precision], in_ci=True, ) assert result["answer_relevancy"] >= 0.9 assert result["context_recall"] >= 0.95 assert result["context_precision"] >= 0.95 assert_in_range(result["faithfulness"], value=0.4, plus_or_minus=0.1) ``` # How to Align an LLM as a Judge In this guide, you'll learn how to systematically evaluate and align an LLM-as-judge metric with human expert judgments using Ragas. - Build a reusable evaluation pipeline for judge alignment - Analyze disagreement patterns between judge and human labels - Iterate on judge prompts to improve alignment with expert decisions ## Why align your LLM judge first? Before running evaluation experiments, it is important to align your LLM judge to your specific use case. A misaligned judge is like a compass pointing the wrong way - every improvement you make based on its guidance moves you further from your goal. Aligning the judge to match expert judgments ensures you're improving what actually matters. This alignment step is the foundation of reliable evaluation. The real value: Looking at your data While building an aligned LLM judge is useful, the true business value comes from systematically analyzing your data and understanding failure patterns. The judge alignment process forces you to deeply examine edge cases, clarify evaluation criteria, and uncover insights about what makes responses good or bad. Think of the judge as a tool that scales your analysis, not a replacement for it. ## Setup your environment We've created a simple module you can install and run so that you can focus on understanding the evaluation process instead of creating the application. ```bash uv pip install "ragas[examples]" export OPENAI_API_KEY="your-api-key-here" ``` Full code You can view the full code for the judge alignment evaluation pipeline [here](https://github.com/vibrantlabsai/ragas/tree/main/examples/ragas_examples/judge_alignment). ## Understand the dataset We'll use the [EvalsBench dataset](https://github.com/vibrantlabsai/EvalsBench/blob/main/data/benchmark_df.csv) which contains expert-annotated examples of LLM responses to business questions. Each row includes: - `question`: The original question asked - `grading_notes`: Key points that should be covered in a good response - `response`: The LLM's generated response - `target`: Human expert's binary judgment (pass/fail) **Download the dataset:** ```bash # Create datasets folder and download the dataset mkdir -p datasets curl -o datasets/benchmark_df.csv https://raw.githubusercontent.com/vibrantlabsai/EvalsBench/main/data/benchmark_df.csv ``` **Load and examine the dataset:** ```python import pandas as pd from ragas import Dataset def load_dataset(csv_path: str = None) -> Dataset: """Load annotated dataset with human judgments. Expected columns: question, grading_notes, response, target (pass/fail) """ path = csv_path or "datasets/benchmark_df.csv" df = pd.read_csv(path) dataset = Dataset(name="llm_judge_alignment", backend="local/csv") for _, row in df.iterrows(): dataset.append({ "question": row["question"], "grading_notes": row["grading_notes"], "response": row["response"], "target": (row["target"]), }) return dataset # Load the dataset dataset = load_dataset() print(f"Dataset loaded with {len(dataset)} samples") ``` **Sample rows from the dataset:** | question | grading_notes | response | target | | ----------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | | What are the key methods for determining the pre-money valuation of a tech startup before a Series A investment round, and how do they differ? | DCF method: !future cash flows!, requires projections; Comp. analysis: similar co. multiples; VC method: rev x multiple - post-$; *Founder's share matter*; strategic buyers pay more. | Determining the pre-money valuation of a tech startup before a Series A investment round is a critical step... (covers DCF, comparable analysis, VC method) | pass | | What key metrics and strategies should a startup prioritize to effectively manage and reduce churn rate in a subscription-based business model? | Churn:! monitor monthly, \<5% ideal. *Retention strategies*: engage users, improve onboarding. CAC & LTV: balance 3:1+. Feedback loops: implement early. *Customer support*: proactive & responsive, critical. | Managing and reducing churn rate in a subscription-based business model is crucial... (missing specific metrics and strategies) | fail | The dataset includes multiple responses to the same questions - some pass and others fail. This helps the judge learn nuanced distinctions between acceptable and unacceptable responses. Understanding your ground truth The quality of judge alignment depends entirely on the quality of your ground truth labels. In production scenarios, involve a **principal domain expert** - the person whose judgment is most critical for your use case (e.g., a psychologist for mental health AI, a lawyer for legal AI, or a customer service director for support chatbots). Their consistent judgment becomes the gold standard your judge aligns to. You don't need every example labeled - a representative sample (100-200 examples covering diverse scenarios) is sufficient for reliable alignment. ## Understand the evaluation approach In this guide, we evaluate pre-existing responses from the dataset rather than generating new ones. This approach ensures reproducible results across evaluation runs, allows us to focus on judge alignment rather than response generation. The evaluation workflow is: **Dataset row (question + response) β†’ Judge β†’ Compare with human target** ## Define evaluation metrics For judge alignment, we need two metrics: **Primary metric: `accuracy` (LLM judge)** - Evaluates responses and returns pass/fail decisions with reason. **Alignment metric: `judge_alignment`** - Checks if the judge's decision matches the human expert's verdict. ### Setting up the judge metric Define a simple baseline judge metric that evaluates responses against grading notes: ```python from ragas.metrics import DiscreteMetric # Define the judge metric with a simple baseline prompt accuracy_metric = DiscreteMetric( name="accuracy", prompt="Check if the response contains points mentioned from the grading notes and return 'pass' or 'fail'.\n\nResponse: {response}\nGrading Notes: {grading_notes}", allowed_values=["pass", "fail"], ) ``` ### The alignment metric The alignment metric compares the judge's decision with the human verdict: ```python from ragas.metrics.discrete import discrete_metric from ragas.metrics.result import MetricResult @discrete_metric(name="judge_alignment", allowed_values=["pass", "fail"]) def judge_alignment(judge_label: str, human_label: str) -> MetricResult: """Compare judge decision with human label.""" judge = judge_label.strip().lower() human = human_label.strip().lower() if judge == human: return MetricResult(value="pass", reason=f"Judge={judge}; Human={human}") return MetricResult(value="fail", reason=f"Judge={judge}; Human={human}") ``` ## The experiment function The [experiment function](/concepts/experimentation) orchestrates the complete evaluation pipeline - evaluating responses with the judge and measuring alignment: ```python from typing import Dict, Any from ragas import experiment from ragas.metrics import DiscreteMetric from ragas_examples.judge_alignment import judge_alignment # The metric we created above @experiment() async def judge_experiment( row: Dict[str, Any], accuracy_metric: DiscreteMetric, llm, ): """Run complete evaluation: Judge β†’ Compare with human.""" # Step 1: Get response (in production, this is where you'd call your LLM app) # For this evaluation, we use pre-existing responses from the dataset app_response = row["response"] # Step 2: Judge evaluates the response judge_score = await accuracy_metric.ascore( question=row["question"], grading_notes=row["grading_notes"], response=app_response, llm=llm, ) # Step 3: Compare judge decision with human target alignment = judge_alignment.score( judge_label=judge_score.value, human_label=row["target"] ) return { **row, "judge_label": judge_score.value, "judge_reason": judge_score.reason, "alignment": alignment.value, "alignment_reason": alignment.reason, } ``` ## Run baseline evaluation ### Execute evaluation pipeline and collect results ```python import os from openai import AsyncOpenAI from ragas.llms import llm_factory from ragas_examples.judge_alignment import load_dataset # Load dataset dataset = load_dataset() print(f"Dataset loaded with {len(dataset)} samples") # Initialize LLM client openai_client = AsyncOpenAI(api_key=os.environ.get("OPENAI_API_KEY")) llm = llm_factory("gpt-4o-mini", client=openai_client) # Run the experiment results = await judge_experiment.arun( dataset, name="judge_baseline_v1_gpt-4o-mini", accuracy_metric=accuracy_metric, llm=llm, ) # Calculate alignment rate passed = sum(1 for r in results if r["alignment"] == "pass") total = len(results) print(f"βœ… Baseline alignment: {passed}/{total} passed ({passed/total:.1%})") ``` πŸ“‹ Output (baseline v1) ```text 2025-10-08 22:40:00,334 - Loaded dataset with 160 samples 2025-10-08 22:40:00,334 - Initializing LLM client with model: gpt-4o-mini 2025-10-08 22:40:01,858 - Running baseline evaluation... Running experiment: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 160/160 [04:35<00:00, 1.72s/it] 2025-10-08 22:44:37,149 - βœ… Baseline alignment: 121/160 passed (75.6%) ``` ### Initial performance analysis The evaluation generates comprehensive CSV results containing all inputs (question, grading_notes, response), human targets, judge decisions with reasoning, and alignment comparisons. ## Analyze errors and failure patterns After running the baseline evaluation, we can analyze the misalignment patterns to understand where the judge disagrees with human experts. **Baseline performance: 75.6% alignment (121/160 correct)** Let's examine the error distribution πŸ“‹ Code ```python import pandas as pd # Load results df = pd.read_csv('experiments/judge_baseline_v1_gpt-4o-mini.csv') # Analyze misalignments false_positives = len(df[(df['judge_label'] == 'pass') & (df['target'] == 'fail')]) false_negatives = len(df[(df['judge_label'] == 'fail') & (df['target'] == 'pass')]) print(f"False positives (judge too lenient): {false_positives}") print(f"False negatives (judge too strict): {false_negatives}") ``` πŸ“‹ Output ```text False positives (judge too lenient): 39 False negatives (judge too strict): 0 ``` **Key observation:** All 39 misalignments (24.4%) are false positives - cases where the judge said "pass" but human experts said "fail". The baseline judge is too lenient, missing responses that omit critical concepts from the grading notes. ### Sample failure cases Here are examples where the judge incorrectly passed responses that were missing key concepts: | Grading Notes | Human Label | Judge Label | What's Missing | | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ----------- | --------------------------------------------------------------------------------------------------------- | | `*Valuation caps*, $, post-$ val key. Liquidation prefs: 1x+ common. Anti-dilution: *full vs. weighted*. Board seats: 1-2 investor reps. ESOP: 10-20%.` | fail | pass | Response discusses all points comprehensively but human annotators marked it as fail for subtle omissions | | `*Impact on valuation*: scalability potential, dev costs, integration ease. !Open-source vs proprietary issues. !Tech debt risks. Discuss AWS/GCP/Azure...` | fail | pass | Missing specific discussion of post-money valuation impact | | `Historical vs. forecasted rev; top-down & bottom-up methods; *traction evidence*; !unbiased assumptions; 12-24mo project...` | fail | pass | Missing explicit mention of traction evidence | **Common patterns in errors:** 1. **Missing 1-2 specific concepts** from grading notes while covering others 1. **Implicit vs explicit coverage** - judge accepts implied concepts, we want explicit mentions 1. **Abbreviated terms** not properly decoded (e.g., "mkt demand" = market demand, "post-$" = post-money valuation) 1. **Critical markers ignored** - points marked with `*` or `!` are often essential ## Improve the judge prompt Based on error analysis, we need to create an improved prompt that: 1. **Understands abbreviations** used in grading notes 1. **Recognizes critical markers** (`*`, `!`, specific numbers) 1. **Requires all concepts** to be present, not just most 1. **Accepts semantic equivalents** (different wording for same concept) 1. **Balances strictness** - not too lenient or too strict ### Create the improved v2 prompt Define the enhanced judge metric with comprehensive evaluation criteria: ```python from ragas.metrics import DiscreteMetric # Define improved judge metric with enhanced evaluation criteria accuracy_metric_v2 = DiscreteMetric( name="accuracy", prompt="""Evaluate if the response covers ALL the key concepts from the grading notes. Accept semantic equivalents but carefully check for missing concepts. ABBREVIATION GUIDE - decode these correctly: β€’ Financial: val=valuation, post-$=post-money, rev=revenue, ARR/MRR=Annual/Monthly Recurring Revenue, COGS=Cost of Goods Sold, Opex=Operating Expenses, LTV=Lifetime Value, CAC=Customer Acquisition Cost β€’ Business: mkt=market, reg/regs=regulation/regulatory, corp gov=corporate governance, integr=integration, S&M=Sales & Marketing, R&D=Research & Development, acq=acquisition β€’ Technical: sys=system, elim=elimination, IP=Intellectual Property, TAM=Total Addressable Market, diff=differentiation β€’ Metrics: NPS=Net Promoter Score, SROI=Social Return on Investment, proj=projection, cert=certification EVALUATION APPROACH: Step 1 - Parse grading notes into distinct concepts: - Separate by commas, semicolons, or line breaks - Each item is a concept that must be verified - Example: "*Gross Margin* >40%, CAC, LTV:CAC >3:1" = 3 concepts Step 2 - For each concept, check if it's addressed: - Accept semantic equivalents (e.g., "customer acquisition cost" = "CAC") - Accept implicit coverage when it's clear (e.g., "revenue forecasting" covers "historical vs forecasted rev") - Be flexible on exact numbers (e.g., "around 40%" acceptable for ">40%") Step 3 - Count missing concepts: - Missing 0 concepts = PASS - Missing 1+ concepts = FAIL (even one genuinely missing concept should fail) - Exception: If a long list (10+ items) has 1 very minor detail missing but all major points covered, use judgment CRITICAL RULES: 1. Do NOT require exact wording - "market demand" = "mkt demand" = "demand analysis" 2. Markers (* or !) mean important, not mandatory exact phrases: - "*traction evidence*" can be satisfied by discussing metrics, growth, or validation - "!unbiased assumptions" can be satisfied by discussing assumption methodology 3. Numbers should be mentioned but accept approximations: - "$47B to $10B" can be "$47 billion dropped to around $10 billion" - "LTV:CAC >3:1" can be "LTV to CAC ratio of at least 3 to 1" or "3x or higher" 4. FAIL only when concepts are genuinely absent: - If notes mention "liquidation prefs, anti-dilution, board seats" but response only has board seats β†’ FAIL - If notes mention "scalability, tech debt, IP" but response never discusses technical risks β†’ FAIL - If notes mention "GDPR compliance" and response never mentions GDPR or EU regulations β†’ FAIL 5. PASS when ALL concepts present: - All concepts covered, even with different wording β†’ PASS - Concepts addressed implicitly when clearly implied β†’ PASS - Minor phrasing differences β†’ PASS - One or more concepts genuinely absent β†’ FAIL Response: {response} Grading Notes: {grading_notes} Are ALL distinct concepts from the grading notes covered in the response (accepting semantic equivalents and implicit coverage)?""", allowed_values=["pass", "fail"], ) ``` Optimizing prompts using LLMs You can use LLMs to optimize prompts after you identify error patterns clearly. You can use LLMs to identify errors too, but make sure to review them so they're aligned with the ground truth labels. You can also use coding agents like Cursor, Claude Code, or frameworks like [DSPy](https://github.com/stanfordnlp/dspy) to systematically optimize judge prompts. ## Re-run evaluation with improved prompt Run the evaluation again with the enhanced v2 prompt (same setup as baseline, just swap the metric): ```python # Use the same dataset and LLM setup from the baseline evaluation above results = await judge_experiment.arun( dataset, name="judge_accuracy_v2_gpt-4o-mini", accuracy_metric=accuracy_metric_v2, # ← Using improved v2 prompt llm=llm, ) passed = sum(1 for r in results if r["alignment"] == "pass") total = len(results) print(f"βœ… V2 alignment: {passed}/{total} passed ({passed/total:.1%})") ``` πŸ“‹ Output (improved v2) ```text 2025-10-08 23:42:11,650 - Loaded dataset with 160 samples 2025-10-08 23:42:11,650 - Initializing LLM client with model: gpt-4o-mini 2025-10-08 23:42:12,730 - Running v2 evaluation with improved prompt... Running experiment: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 160/160 [04:39<00:00, 1.75s/it] 2025-10-08 23:46:52,740 - βœ… V2 alignment: 139/160 passed (86.9%) ``` **Significant improvement!** The alignment increased from 75.6% to 86.9%. If you need to iterate further: - Analyze remaining errors to identify patterns (are they false positives or false negatives?) - Annotate your reasoning along with label - this will help while improving the LLM Judge, you can add these as few shot examples as well. - **Use smarter models** - More capable models like GPT-5 or Claude 4.5 Sonnet generally perform better as judges - **Leverage AI assistants** - This guide was created using Cursor AI agents to analyze failures and iterate on prompts. You can use AI coding agents (Cursor, Claude, etc.) or frameworks like [DSPy](https://github.com/stanfordnlp/dspy) to systematically optimize judge prompts - Stop when alignment plateaus across 2-3 iterations or meets your business threshold ## What you've accomplished You've built a systematic evaluation pipeline using Ragas that: - Measures judge alignment against expert judgments with clear metrics - Identifies failure patterns through structured error analysis - Tracks improvement across evaluation runs with reproducible experiments This aligned judge becomes your foundation for reliable AI evaluation. With a judge you can trust, you can now confidently evaluate your RAG pipeline, agent workflows, or any LLM applicationβ€”knowing that improvements in metrics translate to real improvements in quality. # How to Evaluate a New LLM For Your Use Case When a new LLM is released, you might want to determine if it outperforms your current model for your specific use case. This guide shows you how to run an accuracy comparison between two models using Ragas framework. ## What you'll accomplish By the end of this guide, you'll have: - Set up a structured evaluation comparing two LLMs - Evaluated model performance on a realistic business task - Generated detailed results to inform your model selection decision - A reusable evaluation loop you can rerun whenever new models drop ## The evaluation scenario We'll use discount calculation as our test case: given a customer profile, calculate the appropriate discount percentage and explain the reasoning. This task requires rule application and reasoning - skills that differentiate model capabilities. *Note: You can adapt this approach to any use case that matters for your application.* > **πŸ“ Full Code**: The complete source code for this example is available on [Github](https://github.com/vibrantlabsai/ragas/tree/main/examples/benchmark_llm) ## Set up your environment and API access First, install the ragas-examples package which contains the benchmark LLM example code: ```bash pip install ragas[examples] ``` Next, ensure you have your API credentials configured: ```bash export OPENAI_API_KEY=your_actual_api_key ``` ## The LLM application We've set up a simple LLM application for you in the examples package so you can focus on evaluation rather than building the application itself. The application calculates customer discounts based on business rules. Here's the system prompt that defines the discount calculation logic: ```python SYSTEM_PROMPT = """ You are a discount calculation assistant. I will provide a customer profile and you must calculate their discount percentage and explain your reasoning. Discount rules: - Age 65+ OR student status: 15% discount - Annual income < $30,000: 20% discount - Premium member for 2+ years: 10% discount - New customer (< 6 months): 5% discount Rules can stack up to a maximum of 35% discount. Respond in JSON format only: { "discount_percentage": number, "reason": "clear explanation of which rules apply and calculations", "applied_rules": ["list", "of", "applied", "rule", "names"] } """ ``` You can test the application with a sample customer profile: ```python from ragas_examples.benchmark_llm.prompt import run_prompt # Test with a sample customer profile customer_profile = """ Customer Profile: - Name: Sarah Johnson - Age: 67 - Student: No - Annual Income: $45,000 - Premium Member: Yes, for 3 years - Account Age: 3 years """ result = await run_prompt(customer_profile) print(result) ``` πŸ“‹ Output ```json { "discount_percentage": 25, "reason": "Sarah qualifies for a 15% discount due to age (67). She also gets a 10% discount for being a premium member for over 2 years. The total stacking of 15% and 10% discounts results in 25%. No other discounts apply based on income or account age.", "applied_rules": ["Age 65+", "Premium member for 2+ years"] } ``` ## Examine the evaluation dataset For this evaluation we've built a synthetic dataset with test cases that includes: - Simple cases with clear outcomes - Edge cases at rule boundaries - Complex scenarios with ambiguous information Each case specifies: - `customer_profile`: The input data - `expected_discount`: Expected discount percentage - `description`: Case complexity indicator Example dataset structure (add an `id` column for easy comparison): | ID | Customer Profile | Expected Discount | Description | | --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------- | --------------- | | 1 | Martha is a 70-year-old retiree who enjoys gardening. She has never enrolled in any academic course recently, has an annual pension of 50,000 dollars, signed up for our service nine years ago and never upgraded to premium. | 15 | Senior only | | 2 | Arjun, aged 19, is a full-time computer-science undergraduate. His part-time job brings in about 45,000 dollars per year. He opened his account a year ago and has no premium membership. | 15 | Student only | | 3 | Cynthia, a 40-year-old freelance artist, earns roughly 25,000 dollars a year. She is not studying anywhere, subscribed to our basic plan five years back and never upgraded to premium. | 20 | Low income only | To customize the dataset for your use case, create a `datasets/` directory and add your own CSV file. Refer to [Core Concepts - Evaluation Dataset](https://docs.ragas.io/en/stable/concepts/components/eval_dataset/index.md) for more information. It is better to sample real data from your application to create the dataset. If that is not available, you can generate synthetic data using an LLM. Since our use case is slightly complex, we recommend using a model like gpt-5-high which can generate more accurate data. Always make sure to manually review and verify the data you use. Note While the example dataset here has roughly 10 cases to keep the guide compact, you can start small with 20-30 samples for a real-world evaluation, but make sure you slowly iterate to improve it to the 50-100 samples range to get more trustable results from evaluation. Ensure broad coverage of the different scenarios your agent may face (including edge cases and complex questions). Your accuracy does not need to be 100% initiallyβ€”use the results for error analysis, iterate on prompts, data, and tools, and keep improving. ### Load dataset ```python def load_dataset(): """Load the dataset from CSV file. Downloads from GitHub if not found locally.""" import urllib.request current_dir = os.path.dirname(os.path.abspath(__file__)) dataset_path = os.path.join(current_dir, "datasets", "discount_benchmark.csv") # Download dataset from GitHub if it doesn't exist locally if not os.path.exists(dataset_path): os.makedirs(os.path.dirname(dataset_path), exist_ok=True) urllib.request.urlretrieve("https://raw.githubusercontent.com/vibrantlabsai/ragas/main/examples/ragas_examples/benchmark_llm/datasets/discount_benchmark.csv", dataset_path) return Dataset.load(name="discount_benchmark", backend="local/csv", root_dir=current_dir) ``` The dataset loader checks if the CSV file exists locally. If not found, it automatically downloads it from GitHub. ### Metrics function It is generally better to use a simple metric. You should use a metric relevant to your use case. More information on metrics can be found in [Core Concepts - Metrics](https://docs.ragas.io/en/stable/concepts/metrics/index.md). The evaluation uses this accuracy metric to score each response: ```python @discrete_metric(name="discount_accuracy", allowed_values=["correct", "incorrect"]) def discount_accuracy(prediction: str, expected_discount): """Check if the discount prediction is correct.""" import json parsed_json = json.loads(prediction) predicted_discount = parsed_json.get("discount_percentage") expected_discount_int = int(expected_discount) if predicted_discount == expected_discount_int: return MetricResult( value="correct", reason=f"Correctly calculated discount={expected_discount_int}%" ) else: return MetricResult( value="incorrect", reason=f"Expected discount={expected_discount_int}%; Got discount={predicted_discount}%" ) ``` ### Experiment structure Each model evaluation follows this experiment pattern: ```python @experiment() async def benchmark_experiment(row, model_name: str): # Get model response response = await run_prompt(row["customer_profile"], model=model_name) # Parse response (strict JSON mode expected) try: parsed_json = json.loads(response) predicted_discount = parsed_json.get('discount_percentage') except Exception: predicted_discount = None # Score the response score = discount_accuracy.score( prediction=response, expected_discount=row["expected_discount"] ) return { **row, "model": model_name, "response": response, "predicted_discount": predicted_discount, "score": score.value, "score_reason": score.reason } ``` ## Run experiments Run evaluation experiments with both baseline and candidate models. We'll compare these example models: - Baseline: "gpt-4.1-nano-2025-04-14" - Candidate: "gpt-5-nano-2025-08-07" ```python from ragas_examples.benchmark_llm.evals import benchmark_experiment, load_dataset # Load dataset dataset = load_dataset() print(f"Dataset loaded with {len(dataset)} samples") # Run baseline experiment baseline_results = await benchmark_experiment.arun( dataset, name="gpt-4.1-nano-2025-04-14", model_name="gpt-4.1-nano-2025-04-14" ) # Calculate and display accuracy baseline_accuracy = sum(1 for r in baseline_results if r["score"] == "correct") / len(baseline_results) print(f"Baseline Accuracy: {baseline_accuracy:.2%}") # Run candidate experiment candidate_results = await benchmark_experiment.arun( dataset, name="gpt-5-nano-2025-08-07", model_name="gpt-5-nano-2025-08-07" ) # Calculate and display accuracy candidate_accuracy = sum(1 for r in candidate_results if r["score"] == "correct") / len(candidate_results) print(f"Candidate Accuracy: {candidate_accuracy:.2%}") ``` Each experiment saves a CSV under `experiments/` with per-row results, including: - id, model, response, predicted_discount, score, score_reason Sample experiment output (only showing few columns for readability) | ID | Description | Expected | Predicted | Score | Score Reason | | --- | ----------------------------------------- | -------- | --------- | --------- | --------------------------------------- | | 1 | Senior only | 15 | 15 | correct | Correctly calculated discount=15% | | 2 | Student only | 15 | 5 | incorrect | Expected discount=15%; Got discount=5% | | 3 | Low income only | 20 | 20 | correct | Correctly calculated discount=20% | | 4 | Senior, low income, new customer (capped) | 35 | 35 | correct | Correctly calculated discount=35% | | 6 | Premium 2+ yrs only | 10 | 15 | incorrect | Expected discount=10%; Got discount=15% | Note When possible, pin and record the exact model snapshot/version (for example, "gpt-4o-2024-08-06" instead of just "gpt-4o"). Providers regularly update alias names, and performance can change between snapshots. You can find available snapshots in the provider's model documentation (see OpenAI's [model catalog](https://platform.openai.com/docs/models) as an example). Including the snapshot in your results makes future comparisons fair and reproducible. ## Compare results After running experiments with different models, compare their performance side-by-side: ```python from ragas_examples.benchmark_llm.evals import compare_inputs_to_output # Compare the two experiment results # Update these paths to match your actual experiment output files output_path = compare_inputs_to_output( inputs=[ "experiments/gpt-4.1-nano-2025-04-14.csv", "experiments/gpt-5-nano-2025-08-07.csv" ] ) print(f"Comparison saved to: {output_path}") ``` This comparison: - Reads both experiment files - Prints accuracy for each model - Creates a new CSV with results side-by-side The comparison file shows: - Test case details (customer profile, expected discount) - For each model: its response, whether it was correct, and why πŸ“‹ Output ```text gpt-4.1-nano-2025-04-14 Accuracy: 50.00% gpt-5-nano-2025-08-07 Accuracy: 90.00% Comparison saved to: experiments/20250820-150548-comparison.csv ``` ### Analyze results with the combined CSV In this example run: - Filtering for cases where one model outperforms the other surfaces these cases: "Senior and new customer", "Student and new customer", "Student only", "Premium 2+ yrs only". - The reason field in each model's response shows why it gave the output it did. Sample rows from comparison CSV (showing limited columns for readability) | id | customer_profile | description | expected_discount | gpt-4.1-nano-2025-04-14_score | gpt-5-nano-2025-08-07_score | gpt-4.1-nano-2025-04-14_score_reason | gpt-5-nano-2025-08-07_score_reason | gpt-4.1-nano-2025-04-14_response | gpt-5-nano-2025-08-07_response | | --- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------- | ----------------- | ----------------------------- | --------------------------- | --------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | | 2 | Arjun, aged 19, is a full-time computer-science undergraduate. His part-time job brings in about 45,000 dollars per year. He opened his account a year ago and has no premium membership. | Student only | 15 | incorrect | correct | Expected discount=15%; Got discount=0% | Correctly calculated discount=15% | ...reason="Arjun is 19 years old, so he does not qualify for age-based or senior discounts. His annual income of $45,000 exceeds the $30,000 threshold, so no income-based discount applies. He opened his account a year ago, which is more than 6 months, so he is not a new customer. He has no premium membership and no other applicable discounts."... | ...reason="Eligible for 15% discount due to student status (Arjun is 19 and an undergraduate)."... | | 6 | Leonardo is 64, turning 65 next month. His salary is exactly 30,000 dollars. He has maintained a premium subscription for two years and seven months and has been with us for five years. | Premium 2+ yrs only | 10 | incorrect | correct | Expected discount=10%; Got discount=25% | Correctly calculated discount=10% | ...reason="Leonardo is about to turn 65, so he qualifies for the age discount of 15%. Premium 2+ years noted"... | ...reason="Leonardo is 64, turning 65 next month. premium 2+ years: 10%"... | Re-run when new models drop Once this evaluation lives alongside your project, it becomes a repeatable check. When a new LLM is released (often weekly nowadays), plug it in as the candidate and rerun the same evaluation to compare against your current baseline. ## Interpret results and make your decision ### What to look at - **Baseline accuracy** vs **Candidate accuracy** and the **difference**. - Example from this run: baseline 50% (5/10), candidate 90% (9/10), difference +40%. ### How to read the rows - Skim rows where the two models disagree. - Use each row's score_reason to see why it was marked correct/incorrect. - Look for patterns (e.g., missed rule stacking, boundary cases like "almost 65", exact income thresholds). ### Beyond accuracy - Check **cost** and **latency**. Higher accuracy may not be worth it if it's too slow or too expensive for your use case. ### Decide - Switch if the new model is clearly more accurate on your important cases and fits your cost/latency needs. - Stay if gains are small, failures hit critical cases, or cost/latency are not acceptable. In this example: - We would switch to "gpt-5-nano-2025-08-07". It improves accuracy from 50% to 90% (+40%) and fixes the key failure modes (missed rule stacking, boundary conditions). If its latency/cost fits your constraints, it's the better default. ## Adapting to your use case To evaluate models for your specific application, you can use the [GitHub code](https://github.com/vibrantlabsai/ragas/tree/main/examples/benchmark_llm) as a template and adapt it to your use case. The Ragas framework handles the orchestration, parallel execution, and result aggregation automatically for you, helping you evaluate and focus on your use case! # Compare Embeddings for retriever The performance of the retriever is a critical and influential factor that determines the overall effectiveness of a Retrieval Augmented Generation (RAG) system. In particular, the quality of the embeddings used plays a pivotal role in determining the quality of the retrieved content. This tutorial notebook provides a step-by-step guide on how to compare and choose the most suitable embeddings for your own data using the Ragas library. Compare Embeddings ## Create synthetic test data Tip Ragas can also work with your dataset. Refer to [data preparation](https://docs.ragas.io/en/stable/howtos/customizations/testgenerator/index.md) to see how you can use your dataset with ragas. Ragas offers a unique test generation paradigm that enables the creation of evaluation datasets specifically tailored to your retrieval and generation tasks. Unlike traditional QA generators, Ragas can generate a wide variety of challenging test cases from your document corpus. Tip Refer to [testset generation](https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/index.md) to know more on how it works. For this tutorial notebook, I am using papers from Semantic Scholar that is related to large language models to build RAG. ```python from llama_index.core import download_loader from ragas.testset.evolutions import simple, reasoning, multi_context from ragas.testset.generator import TestsetGenerator from langchain_openai import ChatOpenAI from ragas.embeddings import OpenAIEmbeddings import openai SemanticScholarReader = download_loader("SemanticScholarReader") loader = SemanticScholarReader() query_space = "large language models" documents = loader.load_data(query=query_space, limit=100) # generator with openai models generator_llm = ChatOpenAI(model="gpt-4o-mini") critic_llm = ChatOpenAI(model="gpt-4o") openai_client = openai.OpenAI() embeddings = OpenAIEmbeddings(client=openai_client) generator = TestsetGenerator.from_langchain( generator_llm, critic_llm, embeddings ) distributions = { simple: 0.5, multi_context: 0.4, reasoning: 0.1 } # generate testset testset = generator.generate_with_llamaindex_docs(documents, 100,distributions) test_df = testset.to_pandas() ``` Test Outputs ```python test_questions = test_df['question'].values.tolist() test_answers = [[item] for item in test_df['answer'].values.tolist()] ``` ## Build your RAG Here I am using llama-index to build a basic RAG pipeline with my documents. The goal here is to collect retrieved contexts and generated answer for each of the test questions from your pipeline. Ragas has integrations with various RAG frameworks which makes evaluating them easier using ragas. Note refer to [langchain-tutorial](https://docs.ragas.io/en/stable/howtos/integrations/_langchain/index.md) see how to evaluate using langchain ```python import nest_asyncio from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, ServiceContext from langchain.embeddings import HuggingFaceEmbeddings from ragas.embeddings import OpenAIEmbeddings import openai import pandas as pd nest_asyncio.apply() def build_query_engine(embed_model): vector_index = VectorStoreIndex.from_documents( documents, service_context=ServiceContext.from_defaults(chunk_size=512), embed_model=embed_model, ) query_engine = vector_index.as_query_engine(similarity_top_k=2) return query_engine ``` ## Import metrics from ragas Here we are importing metrics that are required to evaluate retriever component. ```python from ragas.metrics import ( context_precision, context_recall, ) metrics = [ context_precision, context_recall, ] ``` ## Evaluate OpenAI embeddings ```python from ragas.llama_index import evaluate openai_model = OpenAIEmbedding() query_engine1 = build_query_engine(openai_model) result = evaluate(query_engine1, metrics, test_questions, test_answers) ``` ```python {'context_precision': 0.2378, 'context_recall': 0.7159} ``` ## Evaluate Bge embeddings ```python from ragas.llama_index import evaluate flag_model = HuggingFaceEmbeddings(model_name="BAAI/bge-small-en-v1.5") query_engine2 = build_query_engine(flag_model) result = evaluate(query_engine2, metrics, test_questions, test_answers) ``` ```python {'context_precision': 0.2655, 'context_recall': 0.7227} ``` ## Compare Scores Based on the evaluation results, it is apparent that the `context_precision` and `context_recall` metrics of the BGE model slightly outperform the OpenAI-Ada model in my RAG pipeline when applied to my own dataset. For any further analysis of the scores you can export the results to pandas ```python result_df = result.to_pandas() result_df.head() ``` Compare Embeddings Results # Compare LLMs using Ragas Evaluations The LLM used in the Retrieval Augmented Generation (RAG) system has a major impact in the quality of the generated output. Evaluating the results generated by different LLMs can give an idea about the right LLM to use for a particular use case. This tutorial notebook provides a step-by-step guide on how to compare and choose the most suitable LLM for your own data using the Ragas library. Compare LLMs ## Create synthetic test data Tip Ragas can also work with your dataset. Refer to [data preparation](https://docs.ragas.io/en/stable/howtos/applications/compare_llms/data_preparation.md) to see how you can use your dataset with ragas. Ragas offers a unique test generation paradigm that enables the creation of evaluation datasets specifically tailored to your retrieval and generation tasks. Unlike traditional QA generators, Ragas can generate a wide variety of challenging test cases from your document corpus. Tip Refer to [testset generation](https://docs.ragas.io/en/stable/howtos/concepts/testset_generation/index.md) to know more on how it works. For this tutorial notebook, I am using papers from Arxiv that is related to large language models to build RAG. Note Generate a set of 50+ samples using Testset generator for better results ```python import os from llama_index import download_loader, SimpleDirectoryReader from ragas.testset import TestsetGenerator from ragas.testset.generator import TestsetGenerator from ragas.testset.evolutions import simple, reasoning, multi_context from langchain_openai import ChatOpenAI, OpenAIEmbeddings os.environ['OPENAI_API_KEY'] = 'Your OPEN AI key' # load documents reader = SimpleDirectoryReader("./arxiv-papers/",num_files_limit=30) documents = reader.load_data() # generator with openai models generator_llm = ChatOpenAI(model="gpt-4o-mini") critic_llm = ChatOpenAI(model="gpt-4o") embeddings = OpenAIEmbeddings() generator = TestsetGenerator.from_langchain( generator_llm, critic_llm, embeddings ) distributions = { simple: 0.5, multi_context: 0.4, reasoning: 0.1 } # generate testset testset = generator.generate_with_llama_index_docs(documents, 100,distributions) testset.to_pandas() ``` ```python test_questions = test_df['question'].values.tolist() test_answers = [[item] for item in test_df['answer'].values.tolist()] ``` ## Build your RAG Here I am using llama-index to build a basic RAG pipeline with my documents. The goal here is to collect retrieved contexts and generated answer for each of the test questions from your pipeline. Ragas has integrations with various RAG frameworks which makes evaluating them easier using ragas. Note refer to [langchain-tutorial](https://docs.ragas.io/en/stable/howtos/integrations/_langchain/index.md) see how to evaluate using langchain ```python import nest_asyncio from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext from llama_index.llms import HuggingFaceInferenceAPI from llama_index.embeddings import HuggingFaceInferenceAPIEmbedding import pandas as pd nest_asyncio.apply() def build_query_engine(llm): vector_index = VectorStoreIndex.from_documents( documents, service_context=ServiceContext.from_defaults(chunk_size=512, llm=llm), embed_model=HuggingFaceInferenceAPIEmbedding, ) query_engine = vector_index.as_query_engine(similarity_top_k=2) return query_engine # Function to evaluate as Llama index does not support async evaluation for HFInference API def generate_responses(query_engine, test_questions, test_answers): responses = [query_engine.query(q) for q in test_questions] answers = [] contexts = [] for r in responses: answers.append(r.response) contexts.append([c.node.get_content() for c in r.source_nodes]) dataset_dict = { "question": test_questions, "answer": answers, "contexts": contexts, } if test_answers is not None: dataset_dict["ground_truth"] = test_answers ds = Dataset.from_dict(dataset_dict) return ds ``` ## Import metrics from ragas Here we are importing metrics that are required to evaluate retriever component. ```python from datasets import Dataset from ragas import evaluate from ragas.metrics import ( faithfulness, answer_relevancy, answer_correctness, ) metrics = [ faithfulness, answer_relevancy, answer_correctness, ] ``` ## Evaluate Zephyr 7B Alpha LLM For the first LLM, I will be using HuggingFace [zephyr-7b-alpha](https://huggingface.co/HuggingFaceH4/zephyr-7b-alpha). I am using HuggingFaceInferenceAPI to generate answers using the model. HuggingFaceInferenceAPI is free to use and token can be setup using [HuggingFaceToken](https://huggingface.co/docs/hub/security-tokens). ```python # Use zephyr model using HFInference API zephyr_llm = HuggingFaceInferenceAPI( model_name="HuggingFaceH4/zephyr-7b-alpha", token="Your Hugging Face token" ) query_engine1 = build_query_engine(zephyr_llm) result_ds = generate_responses(query_engine1, test_questions, test_answers) result_zephyr = evaluate( result_ds, metrics=metrics, ) result_zephyr ``` ```python {'faithfulness': 0.8365, 'answer_relevancy': 0.8831, 'answer_correctness': 0.6605} ``` ## Evaluate Falcon-7B-Instruct LLM For the second model to evaluate, I am using [Falcon-7B-Instruct](https://huggingface.co/tiiuae/falcon-7b-instruct). This can also be used with the HuggingFaceInferenceAPI. ```python falcon_llm = HuggingFaceInferenceAPI( model_name="tiiuae/falcon-7b-instruct", token="Your Huggingface token" ) query_engine2 = build_query_engine(falcon_llm) result_ds_falcon = generate_responses(query_engine2, test_questions, test_answers) result = evaluate( result_ds_falcon, metrics=metrics, ) result ``` ```python {'faithfulness': 0.6909, 'answer_relevancy': 0.8651, 'answer_correctness': 0.5850} ``` ## Compare Scores Based on the evaluation results, it is apparent that the `faithfulness`, `answer_correctness` and `answer_relevancy` metrics of the HuggingFace zephyr-7b-alpha model slightly outperform the falcon-7b-instruct model in my RAG pipeline when applied to my own dataset. Refer to the complete Colab notebook [here](https://colab.research.google.com/drive/10dNeU56XLOGUJ9gRuBFryyRwoy70rIeS?usp=sharing). ```python import numpy as np import matplotlib.pyplot as plt def analysis(zephyr_df, falcon_df): sns.set_style("whitegrid") fig, axs = plt.subplots(1,3, figsize=(12, 5)) for i,col in enumerate(zephyr_df.columns): sns.kdeplot(data=[zephyr_df[col].values,falcon_df[col].values],legend=False,ax=axs[i],fill=True) axs[i].set_title(f'{col} scores distribution') axs[i].legend(labels=["zephyr", "falcon"]) plt.tight_layout() plt.show() result_zephyr_df = result_zephyr.to_pandas() result_falcon_df = result.to_pandas() analysis( result_zephyr_df[['faithfulness', 'answer_relevancy', 'answer_correctness']], result_falcon_df[['faithfulness', 'answer_relevancy', 'answer_correctness']] ) ``` ### Score distribution analysis Compare LLMs # How to Evaluate and Improve a RAG App In this guide, you'll learn how to evaluate and iteratively improve a RAG (Retrieval-Augmented Generation) app using Ragas. ## What you'll accomplish - Set up evaluation dataset - Establish metrics to measure RAG performance - Build a reusable evaluation pipeline - Analyze errors and systematically improve your RAG app - Learn how to leverage Ragas for RAG evaluation ## Set up and run the RAG system We've built a simple RAG system that retrieves relevant documents from the [Hugging Face documentation dataset](https://huggingface.co/datasets/m-ric/huggingface_doc) and generates answers using an LLM. This dataset contains documentation pages for many Hugging Face packages stored as markdown, providing a rich knowledge base for testing RAG capabilities. The complete implementation is available at: [ragas_examples/improve_rag/](https://github.com/vibrantlabsai/ragas/blob/main/examples/ragas_examples/improve_rag/) ``` flowchart LR A[User Query] --> B[Retrieve Documents
BM25] B --> C[Generate Response
OpenAI] C --> D[Return Answer] ``` To run this, install the dependencies: ```bash uv pip install "ragas-examples[improverag]" ``` Then run the RAG app: ```python import os import asyncio from openai import AsyncOpenAI from ragas_examples.improve_rag.rag import RAG, BM25Retriever # Set up OpenAI client os.environ["OPENAI_API_KEY"] = "" openai_client = AsyncOpenAI() # Create retriever and RAG system retriever = BM25Retriever() rag = RAG(openai_client, retriever) # Query the system question = "What architecture is the `tokenizers-linux-x64-musl` binary designed for?" result = asyncio.run(rag.query(question)) print(f"Answer: {result['answer']}") ``` Output ```python Answer: It's built for the x86_64 architecture (specifically the x86_64-unknown-linux-musl target β€” 64-bit Linux with musl libc). ``` Understanding the RAG implementation The code above uses a simple `RAG` class that demonstrates the core RAG pattern. Here's how it works: ```python # examples/ragas_examples/improve_rag/rag.py from typing import Any, Dict, Optional from openai import AsyncOpenAI class RAG: """Simple RAG system for document retrieval and answer generation.""" def __init__(self, llm_client: AsyncOpenAI, retriever: BM25Retriever, system_prompt=None, model="gpt-4o-mini", default_k=3): self.llm_client = llm_client self.retriever = retriever self.model = model self.default_k = default_k self.system_prompt = system_prompt or "Answer only based on documents. Be concise.\n\nQuestion: {query}\nDocuments:\n{context}\nAnswer:" async def query(self, question: str, top_k: Optional[int] = None) -> Dict[str, Any]: """Query the RAG system.""" if top_k is None: top_k = self.default_k return await self._naive_query(question, top_k) async def _naive_query(self, question: str, top_k: int) -> Dict[str, Any]: """Handle naive RAG: retrieve once, then generate.""" # 1. Retrieve documents using BM25 docs = self.retriever.retrieve(question, top_k) if not docs: return {"answer": "No relevant documents found.", "retrieved_documents": [], "num_retrieved": 0} # 2. Build context from retrieved documents context = "\n\n".join([f"Document {i}:\n{doc.page_content}" for i, doc in enumerate(docs, 1)]) prompt = self.system_prompt.format(query=question, context=context) # 3. Generate response using OpenAI with retrieved context response = await self.llm_client.chat.completions.create( model=self.model, messages=[{"role": "user", "content": prompt}] ) return { "answer": response.choices[0].message.content.strip(), "retrieved_documents": [{"content": doc.page_content, "metadata": doc.metadata, "document_id": i} for i, doc in enumerate(docs)], "num_retrieved": len(docs) } ``` This shows the essential RAG pattern: **retrieve relevant documents β†’ inject into prompt β†’ generate answer**. ## Create evaluation dataset We'll use [huggingface_doc_qa_eval](https://huggingface.co/datasets/m-ric/huggingface_doc_qa_eval), a dataset of questions and answers about Hugging Face documentation. Here are a few sample rows from the dataset: | Question | Expected Answer | | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | | What architecture is the `tokenizers-linux-x64-musl` binary designed for? | x86_64-unknown-linux-musl | | What is the purpose of the BLIP-Diffusion model? | The BLIP-Diffusion model is designed for controllable text-to-image generation and editing. | | What is the purpose of the /healthcheck endpoint in the Datasets server API? | Ensure the app is running | The evaluation script downloads the dataset from [here](https://raw.githubusercontent.com/vibrantlabsai/ragas/main/examples/ragas_examples/improve_rag/datasets/hf_doc_qa_eval.csv) and converts it into Ragas Dataset format: ```python # examples/ragas_examples/improve_rag/evals.py import urllib.request from pathlib import Path from ragas import Dataset import pandas as pd def download_and_save_dataset() -> Path: dataset_path = Path("datasets/hf_doc_qa_eval.csv") dataset_path.parent.mkdir(exist_ok=True) if not dataset_path.exists(): github_url = "https://raw.githubusercontent.com/vibrantlabsai/ragas/main/examples/ragas_examples/improve_rag/datasets/hf_doc_qa_eval.csv" urllib.request.urlretrieve(github_url, dataset_path) return dataset_path def create_ragas_dataset(dataset_path: Path) -> Dataset: dataset = Dataset(name="hf_doc_qa_eval", backend="local/csv", root_dir=".") df = pd.read_csv(dataset_path) for _, row in df.iterrows(): dataset.append({"question": row["question"], "expected_answer": row["expected_answer"]}) dataset.save() return dataset ``` Learn more about working with datasets in [Core Concepts - Datasets](https://docs.ragas.io/en/stable/concepts/datasets/index.md). ## Set up metrics for RAG evaluation Now that we have our evaluation dataset ready, we need metrics to measure RAG performance. Start with simple, focused metrics that directly measure your core use case. More information on metrics can be found in [Core Concepts - Metrics](https://docs.ragas.io/en/stable/concepts/metrics/index.md). Here we use a `correctness` discrete metric that evaluates whether the RAG response contains the key information from the expected answer and is factually accurate based on the provided context. ```python # examples/ragas_examples/improve_rag/evals.py from ragas.metrics import DiscreteMetric # Define correctness metric correctness_metric = DiscreteMetric( name="correctness", prompt="""Compare the model response to the expected answer and determine if it's correct. Consider the response correct if it: 1. Contains the key information from the expected answer 2. Is factually accurate based on the provided context 3. Adequately addresses the question asked Return 'pass' if the response is correct, 'fail' if it's incorrect. Question: {question} Expected Answer: {expected_answer} Model Response: {response} Evaluation:""", allowed_values=["pass", "fail"], ) ``` Now that we have our evaluation metric, we need to run it systematically across our dataset. This is where Ragas experiments come in. ## Create the evaluation experiment The experiment function runs your RAG system on each data sample and evaluates the response using our correctness metric. More information on experimentation can be found in [Core Concepts - Experimentation](https://docs.ragas.io/en/stable/concepts/experimentation/index.md). The experiment function takes a dataset row containing the question, expected context, and expected answer, then: 1. Queries the RAG system with the question 1. Evaluates the response using the correctness metric 1. Returns detailed results including scores and reason ```python # examples/ragas_examples/improve_rag/evals.py import asyncio from typing import Dict, Any from ragas import experiment @experiment() async def evaluate_rag(row: Dict[str, Any], rag: RAG, llm) -> Dict[str, Any]: """ Run RAG evaluation on a single row. Args: row: Dictionary containing question and expected_answer rag: Pre-initialized RAG instance llm: Pre-initialized LLM client for evaluation Returns: Dictionary with evaluation results """ question = row["question"] # Query the RAG system rag_response = await rag.query(question, top_k=4) model_response = rag_response.get("answer", "") # Evaluate correctness asynchronously score = await correctness_metric.ascore( question=question, expected_answer=row["expected_answer"], response=model_response, llm=llm ) # Return evaluation results result = { **row, "model_response": model_response, "correctness_score": score.value, "correctness_reason": score.reason, "mlflow_trace_id": rag_response.get("mlflow_trace_id", "N/A"), # MLflow trace ID for debugging (explained later) "retrieved_documents": [ doc.get("content", "")[:200] + "..." if len(doc.get("content", "")) > 200 else doc.get("content", "") for doc in rag_response.get("retrieved_documents", []) ] } return result ``` With our dataset, metrics, and experiment function ready, we can now evaluate our RAG system's performance. ## Run initial RAG experiment ## Start MLflow server Before running the evaluation, you must start the MLflow server. The RAG system automatically logs traces to MLFlow for debugging and analysis: ```bash # Start MLflow server (required - in a separate terminal) uv run mlflow ui --backend-store-uri sqlite:///mlflow.db --port 5000 ``` The MLflow UI will be available at . ## Run initial RAG experiment Now let's run the complete evaluation pipeline to get baseline performance metrics for our RAG system: ```python # Import required components import asyncio from datetime import datetime from ragas_examples.improve_rag.evals import ( evaluate_rag, download_and_save_dataset, create_ragas_dataset, get_openai_client, get_llm_client ) from ragas_examples.improve_rag.rag import RAG, BM25Retriever async def run_evaluation(): # Download and prepare dataset dataset_path = download_and_save_dataset() dataset = create_ragas_dataset(dataset_path) # Initialize RAG components openai_client = get_openai_client() retriever = BM25Retriever() rag = RAG(llm_client=openai_client, retriever=retriever, model="gpt-5-mini", mode="naive") llm = get_llm_client() # Run evaluation experiment exp_name = f"{datetime.now().strftime('%Y%m%d-%H%M%S')}_naiverag" results = await evaluate_rag.arun( dataset, name=exp_name, rag=rag, llm=llm ) # Print results if results: pass_count = sum(1 for result in results if result.get("correctness_score") == "pass") total_count = len(results) pass_rate = (pass_count / total_count) * 100 if total_count > 0 else 0 print(f"Results: {pass_count}/{total_count} passed ({pass_rate:.1f}%)") return results # Run the evaluation results = await run_evaluation() print(results) ``` This downloads the dataset, initializes the BM25 retriever, runs the evaluation experiment on each sample, and saves detailed results to the `experiments/` directory as CSV files for analysis. Output ```python Results: 43/66 passed (65.2%) Evaluation completed successfully! Detailed results: Experiment(name=20250924-212541_naiverag, len=66) ``` With a 65.2% pass rate, we now have a baseline. The detailed results CSV in `experiments/` now contains all the data we need for error analysis and systematic improvement. ### Viewing traces in MLflow The experiment results CSV includes both `mlflow_trace_id` and `mlflow_trace_url` for each evaluation, allowing you to analyze detailed execution traces. The traces help you understand exactly where failures occur - whether in retrieval, generation, or evaluation steps. The RAG system automatically logs traces to the MLflow server (started earlier), and you can view them at . This allows you to: 1. **Analyze results in CSV**: View responses, metric scores and reasons 1. **Deep-dive with traces**: Click the `mlflow_trace_url` in the results to jump directly to the detailed execution trace in MLflow UI for that evaluation Pro Tip: Click Trace URLs for Debugging Each evaluation result includes `mlflow_trace_url` - a direct clickable link to the trace in MLflow UI. No need to manually navigate or copy trace IDs. Just click and jump straight to the detailed execution trace! ## Analyze errors and failure modes After running the evaluation, examine the results CSV file in the `experiments/` directory to identify patterns in failed cases. Each row includes the `mlflow_trace_id`/`mlflow_trace_url` - to view detailed execution traces in the MLflow UI. Annotate each failure case to understand patterns so that we can improve our app. ### Analysis of actual failure patterns from our evaluation: In our example, the core issue is **retrieval failure** - the BM25 retriever is not finding documents that contain the answers. The model correctly follows instructions to say when documents don't contain information, but the wrong documents are being retrieved. **Poor Document Retrieval Examples** The BM25 retriever fails to retrieve relevant documents containing the answers: | Question | Expected Answer | Model Response | Root Cause | | ----------------------------------------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------- | | "What is the default repository type for create_repo?" | `model` | "The provided documents do not state the default repository type..." | **BM25 missed docs with create_repo details** | | "What is the purpose of the BLIP-Diffusion model?" | "controllable text-to-image generation and editing" | "The provided documents do not mention BLIP‑Diffusion..." | **BM25 didn't retrieve BLIP-Diffusion docs** | | "What is the name of the new Hugging Face library for hosting scikit-learn models?" | `Skops` | "The provided documents do not mention or name any new Hugging Face library..." | **BM25 missed Skops documentation** | Based on this analysis, we can see that retrieval is the primary bottleneck. Let's implement targeted improvements. ## Improve the RAG app With retrieval identified as the primary bottleneck, we can improve our system in two ways: **Traditional approaches** focus on better chunking, hybrid search, or vector embeddings. However, since our BM25 retrieval consistently misses relevant documents with single queries, we'll explore an **agentic approach** instead. **Agentic RAG** lets the AI iteratively refine its search strategy - trying multiple search terms and deciding when it has found sufficient context, rather than relying on one static query. ### Agentic RAG implementation ``` flowchart LR A[User Query] --> B[AI Agent
OpenAI] B --> C[BM25 Tool] C --> B B --> D[Final Answer] ``` Run the Agentic RAG app for a sample query: ```python # Switch to agentic mode rag_agentic = RAG(openai_client, retriever, mode="agentic") question = "What architecture is the `tokenizers-linux-x64-musl` binary designed for?" result = await rag_agentic.query(question) print(f"Answer: {result['answer']}") ``` Output ```python Answer: It targets x86_64 β€” i.e. the x86_64-unknown-linux-musl target triple. ``` Understanding the Agentic RAG implementation The Agentic RAG mode uses the OpenAI Agents SDK to create an AI agent with a BM25 retrieval tool: ```python # Key components from the RAG class when mode="agentic" from agents import Agent, Runner, function_tool def _setup_agent(self): """Setup agent for agentic mode.""" @function_tool def retrieve(query: str) -> str: """Search documents using BM25 retriever for a given query.""" docs = self.retriever.retrieve(query, self.default_k) if not docs: return "No documents found." return "\n\n".join([f"Doc {i}: {doc.page_content}" for i, doc in enumerate(docs, 1)]) self._agent = Agent( name="RAG Assistant", model=self.model, instructions="Use short keywords to search. Try 2-3 different searches. Only answer based on documents. Be concise.", tools=[retrieve] ) async def _agentic_query(self, question: str, top_k: int) -> Dict[str, Any]: """Handle agentic mode: agent controls retrieval strategy.""" result = await Runner.run(self._agent, input=question) print(result.answer) ``` Unlike naive mode's single retrieval call, the agent autonomously decides when and how to search - trying multiple keyword combinations until it finds sufficient context. ## Run experiment again and compare results Now let's evaluate the agentic RAG approach: ```python # Import required components import asyncio from datetime import datetime from dotenv import load_dotenv # Load environment variables load_dotenv() from ragas_examples.improve_rag.evals import ( evaluate_rag, download_and_save_dataset, create_ragas_dataset, get_openai_client, get_llm_client ) from ragas_examples.improve_rag.rag import RAG, BM25Retriever async def run_agentic_evaluation(): # Download and prepare dataset dataset_path = download_and_save_dataset() dataset = create_ragas_dataset(dataset_path) # Initialize RAG components with agentic mode openai_client = get_openai_client() retriever = BM25Retriever() rag = RAG(llm_client=openai_client, retriever=retriever, model="gpt-5-mini", mode="agentic") llm = get_llm_client() # Run evaluation experiment exp_name = f"{datetime.now().strftime('%Y%m%d-%H%M%S')}_agenticrag" results = await evaluate_rag.arun( dataset, name=exp_name, rag=rag, llm=llm ) # Print results if results: pass_count = sum(1 for result in results if result.get("correctness_score") == "pass") total_count = len(results) pass_rate = (pass_count / total_count) * 100 if total_count > 0 else 0 print(f"Results: {pass_count}/{total_count} passed ({pass_rate:.1f}%)") return results # Run the agentic evaluation results = await run_agentic_evaluation() print("\nDetailed results:") print(results) ``` Agentic RAG evaluation output ```python Results: 58/66 passed (87.9%) ``` Excellent! We achieved a significant improvement from 65.2% (naive) to 87.9% (agentic) - that's a 22.7 percentage point improvement with the agentic RAG approach! ### Performance Comparison The agentic RAG approach shows great improvement over the naive RAG baseline: | Approach | Correctness | Improvement | | --------------- | ----------- | ----------- | | **Naive RAG** | 65.2% | - | | **Agentic RAG** | **87.9%** | **+22.7%** | ## Apply this loop to your RAG system Follow this systematic approach to improve any RAG system: 1. **Create evaluation dataset**: Use real queries from your system or generate synthetic data with LLMs. 1. **Define metrics**: Choose simple metrics aligned with your use case. Keep it focused. 1. **Run baseline evaluation**: Measure current performance and analyze error patterns to identify systematic failures. 1. **Implement targeted improvements**: Based on error analysis, improve retrieval (chunking, hybrid search), generation (prompts, models), or try agentic approaches. 1. **Compare and iterate**: Test improvements against baseline. Change one thing at a time until accuracy meets business requirements. The Ragas framework handles orchestration and result aggregation automatically, letting you focus on analysis and improvements rather than building evaluation infrastructure. # Evaluating Multi-Turn Conversations This tutorial is inspired by Hamel’s notes on evaluating multi-turn conversations for LLM-based applications. The goal is to create a simple and actionable evaluation framework using Ragas metrics that clearly defines what makes a conversation successful. By the end of this tutorial, you will be able to perform multi-turn evaluations based on insights gathered from the error analysis of your AI application. ### Ragas Metrics Ragas offers **AspectCritic**, a powerful evaluation metric for assessing multi-turn conversations with binary outcomes. It helps determine whether a conversation meets predefined success criteria. **[AspectCritic](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#aspect-critic)** AspectCritic evaluates responses based on predefined aspects written in free-form natural language. It returns a binary output indicating whether the response aligns with the defined aspect. This metric aligns with Hamel's [suggestion](https://hamel.dev/notes/llm/officehours/evalmultiturn.html#focus-on-binary-decisions) to focus on binary decisions, which eliminate ambiguity and provide a clear, actionable approach to improving conversation quality. ### Practical Example – Evaluating a Banking Chatbot When evaluating, focus on metrics that directly align with your users’ needs. Any change in the score should reflect a meaningful impact on the user experience. Let’s consider an example where you are building a chatbot for a bank. After conducting [error analysis](https://hamel.dev/notes/llm/officehours/erroranalysis.html#the-data-first-approach), you find that the chatbot occasionally forgets tasks it was asked to complete or performs them only partially. To improve the chatbot’s performance, you need a reliable method to **measure and evaluate** this behavior. > **Note:** When defining the scoring criteria, use standardized terminology. > > - Refer to the user’s message as `human` message. > - Refer to the chatbot’s message as `AI` message. ```python from dotenv import load_dotenv from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI load_dotenv() evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini")) ``` ```python from ragas.metrics import AspectCritic, SimpleCriteriaScore from ragas.dataset_schema import MultiTurnSample, EvaluationDataset from ragas.messages import HumanMessage, AIMessage from ragas import evaluate sample1 = MultiTurnSample( user_input=[ HumanMessage( content="I need to increase my credit limit and check why my last transaction at Walmart was declined." ), AIMessage( content="Let's check the declined transaction first, the $234.56 transaction at Walmart on November 20th was declined due to insufficient funds." ), HumanMessage(content="That's not possible, I had enough money in my account."), AIMessage( content="I understand your concern. Let me check your account details... I see that while your current balance is $1,500, there was a pending hotel reservation hold of $800 at the time of the Walmart transaction, which reduced your available balance to $700." ), HumanMessage(content="Oh, I forgot about the hotel booking."), AIMessage(content="Is there anything else I can assist you with?"), HumanMessage(content="What about increasing my credit limit?"), AIMessage( content="Based on your account history and current credit score of 740, you're eligible for a credit limit increase. The maximum increase available is $2,000. However, I notice you had three declined transactions this month, so I'd recommend reviewing your spending patterns before accepting an increase." ), ] ) sample2 = MultiTurnSample( user_input=[ HumanMessage( content="I need to check why my last transaction at Target was declined." ), AIMessage( content="Let's look at the declined transaction. The $145.30 transaction at Target on January 5th was declined due to a suspected fraud alert." ), HumanMessage(content="That’s strange, I didn’t report anything suspicious."), AIMessage( content="I understand. Let me look deeper. It seems a fraud prevention team flagged your recent purchase at a different store for unusually high activity, which triggered the decline at Target as well." ), HumanMessage(content="Ah, that makes sense. I did shop a lot that day."), AIMessage(content="Is there anything else I can assist you with?"), ] ) ``` ```python definition = "Return 1 if the AI completes all Human requests fully without any rerequests; otherwise, return 0." aspect_critic = AspectCritic( name="forgetfulness_aspect_critic", definition=definition, llm=evaluator_llm, ) ``` ```python result = evaluate( dataset=EvaluationDataset(samples=[sample1, sample2]), metrics=[aspect_critic], ) result.to_pandas() ``` Output ```text Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:00], "priority":"P0"|"P1"|"P2"} ``` ```bash cd examples/iterate_prompt export OPENAI_API_KEY=your_openai_api_key uv run run_prompt.py ``` This will run the prompt on sample case and print the results. Sample output ```text $ uv run run_prompt.py Test ticket: "SSO via Okta succeeds then bounces me back to /login with no session. Colleagues can sign in. I tried clearing cookies; same result. Error in devtools: state mismatch. I'm blocked from our boards." Response: {"labels":["Account","ProductIssue"], "priority":"P0"} ``` ### Metrics for scoring It is generally better to use a simpler metric instead of a complex one. You should use a metric relevant to your use case. More information on metrics can be found in [Core Concepts - Metrics](https://docs.ragas.io/en/stable/concepts/metrics/index.md). Here we use two discrete metrics: `labels_exact_match` and `priority_accuracy`. Keeping them separate helps analyze and fix different failure modes. - `priority_accuracy`: Checks whether the predicted priority matches the expected priority; important for correct urgency triage. - `labels_exact_match`: Checks whether the set of predicted labels exactly matches the expected labels; important to avoid over/under-tagging and helps us measure the accuracy of our system in labeling the cases. ```python # examples/iterate_prompt/evals.py import json from ragas.metrics.discrete import discrete_metric from ragas.metrics.result import MetricResult @discrete_metric(name="labels_exact_match", allowed_values=["correct", "incorrect"]) def labels_exact_match(prediction: str, expected_labels: str): try: predicted = set(json.loads(prediction).get("labels", [])) expected = set(expected_labels.split(";")) if expected_labels else set() return MetricResult( value="correct" if predicted == expected else "incorrect", reason=f"Expected={sorted(expected)}; Got={sorted(predicted)}", ) except Exception as e: return MetricResult(value="incorrect", reason=f"Parse error: {e}") @discrete_metric(name="priority_accuracy", allowed_values=["correct", "incorrect"]) def priority_accuracy(prediction: str, expected_priority: str): try: predicted = json.loads(prediction).get("priority") return MetricResult( value="correct" if predicted == expected_priority else "incorrect", reason=f"Expected={expected_priority}; Got={predicted}", ) except Exception as e: return MetricResult(value="incorrect", reason=f"Parse error: {e}") ``` ### The experiment function The experiment function is used to run the prompt on a dataset. More information on experimentation can be found in [Core Concepts - Experimentation](https://docs.ragas.io/en/stable/concepts/experimentation/index.md). Notice that we are passing `prompt_file` as a parameter so that we can run experiments with different prompts. You can also pass other parameters to the experiment function like model, temperature, etc. and experiment with different configurations. It is recommended to change only 1 parameter at a time while doing experimentation. ```python # examples/iterate_prompt/evals.py import asyncio, json from ragas import experiment from run_prompt import run_prompt @experiment() async def support_triage_experiment(row, prompt_file: str, experiment_name: str): response = await asyncio.to_thread(run_prompt, row["text"], prompt_file=prompt_file) try: parsed = json.loads(response) predicted_labels = ";".join(parsed.get("labels", [])) or "" predicted_priority = parsed.get("priority") except Exception: predicted_labels, predicted_priority = "", None return { "id": row["id"], "text": row["text"], "response": response, "experiment_name": experiment_name, "expected_labels": row["labels"], "predicted_labels": predicted_labels, "expected_priority": row["priority"], "predicted_priority": predicted_priority, "labels_score": labels_exact_match.score(prediction=response, expected_labels=row["labels"]).value, "priority_score": priority_accuracy.score(prediction=response, expected_priority=row["priority"]).value, } ``` ### Dataset loader (CSV) The dataset loader is used to load the dataset into a Ragas dataset object. More information on datasets can be found in [Core Concepts - Evaluation Dataset](https://docs.ragas.io/en/stable/concepts/components/eval_dataset/index.md). ```python # examples/iterate_prompt/evals.py import os, pandas as pd from ragas import Dataset def load_dataset(): current_dir = os.path.dirname(os.path.abspath(__file__)) df = pd.read_csv(os.path.join(current_dir, "datasets", "support_triage.csv")) dataset = Dataset(name="support_triage", backend="local/csv", root_dir=".") for _, row in df.iterrows(): dataset.append({ "id": str(row["id"]), "text": row["text"], "labels": row["labels"], "priority": row["priority"], }) return dataset ``` ### Run the experiment using current prompt ```bash uv run evals.py run --prompt_file promptv1.txt ``` This will run the given prompt on the dataset and save the results to `experiments/` directory. Sample output ```text $ uv run evals.py run --prompt_file promptv1.txt Loading dataset... Dataset loaded with 20 samples Running evaluation with prompt file: promptv1.txt Running experiment: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 20/20 [00:11<00:00, 1.79it/s] βœ… promptv1: 20 cases evaluated Results saved to: experiments/20250826-041332-promptv1.csv promptv1 Labels Accuracy: 80.00% promptv1 Priority Accuracy: 75.00% ``` ## Improve the prompt ### Analyze errors from the result Open `experiments/{timestamp}-promptv1.csv` in your favorite spreadsheet editor and analyze the results. Look for cases where the labels_score or priority_score is incorrect. From our promptv1 experiment, we can identify several error patterns: #### Priority Errors: Over-prioritization (P1 β†’ P0) The model consistently assigns P0 (highest priority) to billing-related issues that should be P1: | Case | Issue | Expected | Got | Pattern | | ----- | ----------------------------------- | -------- | --- | -------------------------------------- | | ID 19 | Auto-charge after pausing workspace | P1 | P0 | Billing dispute treated as urgent | | ID 1 | Duplicate charge on same day | P1 | P0 | Billing dispute treated as urgent | | ID 5 | Cancellation with refund request | P1 | P0 | Routine cancellation treated as urgent | | ID 13 | Follow-up on cancellation | P1 | P0 | Follow-up treated as urgent | **Pattern**: The model treats any billing/refund/cancellation as urgent (P0) when most are routine business operations (P1). #### Label Errors: Over-labeling and confusion | Case | Issue | Expected | Got | Pattern | | ----- | -------------------------------- | -------------------- | ---------------------------- | ----------------------------------------------------- | | ID 9 | GST tax question from US user | `Billing;HowTo` | `Billing;Account` | Confuses informational questions with account actions | | ID 10 | Account ownership transfer | `Account` | `Account;Billing` | Adds Billing when money/plans mentioned | | ID 20 | API rate limit question | `ProductIssue;HowTo` | `ProductIssue;Billing;HowTo` | Adds Billing when plans mentioned | | ID 16 | Feature request for offline mode | `Feature` | `Feature;HowTo` | Adds HowTo for feature requests | **Patterns identified**: 1. **Over-labeling with Billing**: Adds "Billing" even when not primarily billing-related 1. **HowTo vs Account confusion**: Misclassifies informational questions as account management actions 1. **Over-labeling with HowTo**: Adds "HowTo" to feature requests when users ask "how" but mean "can you build this" ### Improve the prompt Based on our error analysis, we'll create `promptv2_fewshot.txt` with targeted improvements. You can use an LLM to generate the prompt or edit it manually. In this case, we passed the error patterns and the original prompt to an LLM to generate a revised prompt with few-shot examples. #### Key additions in promptv2_fewshot: **1. Enhanced Priority Guidelines with Business Impact Focus:** ```text - P0: Blocked from core functionality OR money/data at risk OR business operations halted - P1: Degraded experience OR needs timely help BUT has workarounds OR not fully blocked - P2: Minor issues OR information requests OR feature requests OR non-urgent how-to ``` **2. Conservative Multi-labeling Rules to Prevent Over-tagging:** ```text ## Multi-label Guidelines Use single label for PRIMARY issue unless both aspects are equally important: - Billing + RefundCancel: Always co-label. Cancellation/refund requests must include Billing. - Account + ProductIssue: For auth/login malfunctions (loops, "invalid_token", state mismatch, bounce-backs) - Avoid adding Billing to account-only administration unless there is an explicit billing operation Avoid over-tagging: Focus on which department should handle this ticket first. ``` **3. Detailed Priority Guidelines with Specific Scenarios:** ```text ## Priority Guidelines - Ignore emotional tone - focus on business impact and available workarounds - Billing disputes/adjustments (refunds, duplicate charges, incorrect taxes/pricing) = P1 unless causing an operational block - Login workarounds: If Incognito/another account works, prefer P1; if cannot access at all, P0 - Core business functions failing (webhooks, API, sync) = P0 ``` **4. Comprehensive Examples with Reasoning:** Added 7 examples covering different scenarios with explicit reasoning to demonstrate proper classification. ```md ## Examples with Reasoning Input: "My colleague left and I need to change the team lead role to my email address." Output: {"labels":["Account"], "priority":"P1"} Reasoning: Administrative role change; avoid adding Billing unless a concrete billing action is requested. Input: "Dashboard crashes when I click reports tab, but works fine in mobile app." Output: {"labels":["ProductIssue"], "priority":"P1"} Reasoning: Malfunction exists but workaround available (mobile app works); single label since primary issue is product malfunction. ``` Try to not directly add the examples from the dataset as that can lead to overfitting to dataset and your prompt might fail in other cases. ### Evaluate new prompt After creating `promptv2_fewshot.txt` with the improvements, run the experiment with the new prompt: ```bash uv run evals.py run --prompt_file promptv2_fewshot.txt ``` This will evaluate the improved prompt on the same dataset and save results to a new timestamped file. Sample output ```text $ uv run evals.py run --prompt_file promptv2_fewshot.txt Loading dataset... Dataset loaded with 20 samples Running evaluation with prompt file: promptv2_fewshot.txt Running experiment: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 20/20 [00:11<00:00, 1.75it/s] βœ… promptv2_fewshot: 20 cases evaluated Results saved to: experiments/20250826-231414-promptv2_fewshot.csv promptv2_fewshot Labels Accuracy: 90.00% promptv2_fewshot Priority Accuracy: 95.00% ``` The experiment will create a new CSV file in the `experiments/` directory with the same structure as the first run, allowing for direct comparison. ### Analyze and compare results We've created a simple utility function to take in multiple CSVs and combine it so that we can compare it easily: ```bash uv run evals.py compare --inputs experiments/20250826-041332-promptv1.csv experiments/20250826-231414-promptv2_fewshot.csv ``` This prints the accuracy for each experiment and saves a combined CSV file in `experiments/` directory. Sample ```bash $ uv run evals.py compare --inputs experiments/20250826-041332-promptv1.csv experiments/20250826-231414-promptv2_fewshot.csv promptv1 Labels Accuracy: 80.00% promptv1 Priority Accuracy: 75.00% promptv2_fewshot Labels Accuracy: 90.00% promptv2_fewshot Priority Accuracy: 95.00% Combined comparison saved to: experiments/20250826-231545-comparison.csv ``` Here, we can see that promptv2_fewshot has improved the accuracy of both labels and priority. But we can also see that some cases still fail. We can analyze the errors and improve the prompt further. Stop iterating when improvements plateau or accuracy meets business requirements. If you hit a ceiling on improving accuracy with just the prompt improvements, you can try experiments with better models. ## Apply this loop to your use case - Create dataset, metrics, experiment for your use case - Run evaluation and analyze errors - Improve prompt based on the error analysis - Re-run evaluation and compare results - Stop when improvements plateau or accuracy meets business requirements Once you have your dataset and evaluation loop setup, you can expand this to testing more parameters like model, etc. The Ragas framework handles the orchestration, parallel execution, and result aggregation automatically for you, helping you evaluate and focus on your use case! Advanced: Aligning LLM judges If you're using LLM-based metrics for evaluation, consider aligning your judge with human expert judgments first to ensure reliable evaluation. See [How to Align an LLM as a Judge](https://docs.ragas.io/en/stable/howtos/applications/align-llm-as-judge/index.md). # A systematic approach for prompt optimization Creating reliable and consistent prompts remains a significant challenge. As requirements multiply and prompt structures grow more complex, even minor modifications can lead to unexpected failures. This often turns traditional prompt engineering into a frustrating game of β€œwhack-a-mole”—fix one issue, and two more seem to emerge. This tutorial demonstrates how to implement a systematic, data-driven approach to prompt engineering through functional testing with Ragas. ## The Diabetes Medication Management Assistant For our tutorial, we will focus on evaluating prompts for a Diabetes Medication Management Assistantβ€”an AI tool designed to help diabetes patients manage their medication, monitor their health, and receive personalized support. **Dataset Overview** Our evaluation uses a carefully curated dataset of 15 representative queries: - 10 on-topic questions within the assistant's domain expertise (medication management, glucose monitoring, etc.) - 5 out-of-scope questions designed to test the assistant's ability to recognize its limitations and decline to provide advice This balanced dataset allows us to assess both the assistant's helpfulness when appropriate and its safety guardrails when faced with queries beyond its expertise. First, download the dataset: ```text !curl -O https://huggingface.co/datasets/vibrantlabsai/diabetes_assistant_dataset/resolve/main/diabetes_assistant_dataset.csv ``` We'll test two nearly identical prompts that differ by only a single line - one with standard instructions and another with an added financial incentive statement. This minimal variation will help us investigate our hypothesis: do LLMs demonstrate improved instruction-following when presented with financial incentives? ## Understanding the Data Our dataset consists of three key parts: - `user_input`: These are the questions provided by diabetes patients. - `retrieved_contexts`: This is the relevant information that the retriever gathered to answer the questions. - `reference`: These are the gold-standard answers used for comparison. ```python import pandas as pd eval_df = pd.read_csv("diabetes_assistant_dataset.csv") eval_df.head() ``` | | user_input | retrieved_contexts | reference | | --- | ------------------------------------------------- | -------------------------------------------------- | ------------------------------------------------- | | 0 | I missed my afternoon insulin doseβ€”what should... | \['Clinical guidelines recommend that if an ins... | If you miss an insulin dose, first check your ... | | 1 | Based on my latest blood glucose readings, how... | \['Recent clinical guidelines emphasize the imp... | Your insulin dosage adjustments should be base... | | 2 | I often get alerts for low or high blood sugar... | \['Current clinical practices emphasize the imp... | Monitor your blood sugar alerts by reviewing t... | | 3 | I have a fear of needles. Are there alternativ... | \['For patients with needle phobia, clinical gu... | There are alternative options available, inclu... | | 4 | I'm switching from oral medications to insulin... | \["Transitioning from oral medications to insul... | During your transition from oral medications t... | In a real-world scenario, you'd typically have more samples (perhaps 50-100) to get statistically significant results. ## Testing Financial Incentives on LLM Performance Our goal is to test the hypothesis that including a financial incentive in the prompt can improve the model’s adherence to instructions. To achieve this, we will define two prompt templates for our Diabetes Medication Management Assistant: - Standard Prompt: Contains the basic instructions for the Diabetes Medication Management Assistant. - Incentivized Prompt: Uses the same basic instructions but adds an incentive statement: β€œIf you follow all the instructions precisely, you will earn a reward of $1,000,000.” By comparing the performance of these two prompt variations, we aim to determine whether a financial incentive encourages the model to follow instructions more accurately. ```python # Define our base prompt for the diabetes assistant diabetes_assistant_prompt = ( "You are a diabetes medication management assistant. Your role is to be a knowledgeable and compassionate healthcare companion " "who balances professionalism with approachability. You are reliable, supportive, and clear, offering medical guidance using simple, " "accessible language while maintaining data-driven insights. Your tone is warm and conversational, yet respectfully direct when urgent " "issues arise. You provide gentle reminders and encourage long-term adherence without being intrusive.\n\n" "User Input: \n{user_input}\n\n" "Retrieved Contexts: \n{contexts}\n\n" "Instructions:\n" '- Answer the user\'s query using only the information provided in the "Retrieved Contexts".\n' "- Do not add any external information or assumptions beyond what is given.\n" '- If the "Retrieved Contexts" contain sufficient details to answer the question, provide a clear, concise, and supportive response based solely on that context.\n' '- If the "Retrieved Contexts" do not provide enough expertise or information to reliably answer the query, respond with: "I don\'t have enough expertise in that matter to comment on this. Please consult your doctor."\n' "- Ensure your response is tailored to the user's needs, balancing empathy with clear, data-focused guidance.\n" ) # Define an incentive that we'll test incentive = "If you follow all the instructions precisely, you will earn a reward of $1,000,000." # Create a variant with the incentive incentive_prompt = diabetes_assistant_prompt + incentive ``` ## Creating the Evaluation Dataset Function In this section, we define a function that transforms our raw dataset into the format required for Ragas evaluation. The function first checks and converts the retrieved contexts into the correct list format if needed, then combines each user’s question with its related contexts using a template. It sends this complete prompt to the language model with a built-in retry mechanism to handle any errors, and finally compiles the responses into a Ragas Evaluation Dataset. You can read more about it [here](https://docs.ragas.io/en/stable/concepts/components/eval_dataset/index.md). ```python import ast import time from tqdm import tqdm from typing import List, Dict, Any from ragas.dataset_schema import EvaluationDataset from openai import OpenAI # Initialize OpenAI client client = OpenAI() def create_ragas_evaluation_dataset(df: pd.DataFrame, prompt: str) -> EvaluationDataset: """ Process a DataFrame into an evaluation dataset by: 1. Converting retrieved contexts from strings to lists if needed 2. For each sample, formatting a prompt with user input and contexts 3. Calling the LLM with retry logic (up to 4 attempts) 4. Recording responses in the dataset Args: df: DataFrame with user_input and retrieved_contexts columns prompt: Template string with placeholders for contexts and user input Returns: EvaluationDataset for RAGAS evaluation """ # Create a copy to avoid modifying the original DataFrame df = df.copy() # Check if any row has retrieved_contexts as string and convert all to lists if df["retrieved_contexts"].apply(type).eq(str).any(): df["retrieved_contexts"] = df["retrieved_contexts"].apply( lambda x: ast.literal_eval(x) if isinstance(x, str) else x ) # Convert DataFrame to list of dictionaries samples: List[Dict[str, Any]] = df.to_dict(orient="records") # Process each sample for sample in tqdm(samples, desc="Processing samples"): user_input_str = sample.get("user_input", "") retrieved_contexts = sample.get("retrieved_contexts", []) # Ensure retrieved_contexts is a list if not isinstance(retrieved_contexts, list): retrieved_contexts = [str(retrieved_contexts)] # Join contexts and format prompt context_str = "\n".join(retrieved_contexts) formatted_prompt = prompt.format( contexts=context_str, user_input=user_input_str ) # Implement retry logic max_attempts = 4 # 1 initial attempt + 3 retries for attempt in range(max_attempts): if attempt > 0: delay = attempt * 10 print(f"Attempt {attempt} failed. Retrying in {delay} seconds...") time.sleep(delay) try: # Call the OpenAI API response = client.chat.completions.create( model="gpt-4o-mini", messages=[{"role": "user", "content": formatted_prompt}], temperature=0 ) sample["response"] = response.choices[0].message.content break # Exit the retry loop if successful except Exception as e: print(f"Error on attempt {attempt+1}: {str(e)}") if attempt == max_attempts - 1: print(f"Failed after {max_attempts} attempts. Skipping sample.") sample["response"] = None # Create and return evaluation dataset eval_dataset = EvaluationDataset.from_list(data=samples) return eval_dataset ``` ## Generating Responses for Evaluation Now we'll use our function to create evaluation datasets for both prompt versions: ```python # Create evaluation datasets for both prompt versions print("Generating responses for base prompt...") eval_dataset_base = create_ragas_evaluation_dataset(eval_df, prompt=diabetes_assistant_prompt) print("Generating responses for incentive prompt...") eval_dataset_incentive = create_ragas_evaluation_dataset(eval_df, prompt=incentive_prompt) ``` ```text Generating responses for base prompt... Processing samples: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 15/15 [00:43<00:00, 2.88s/it] Generating responses for incentive prompt... Processing samples: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 15/15 [00:39<00:00, 2.63s/it] ``` ## Queries that should be answered ### Setting Up Evaluation Metrics Ragas provides several built-in metrics, and we can also create custom metrics for specific requirements. For a list of all available metrics, you can check here. ### Choosing NVIDIA Metrics for Efficient Evaluation For our evaluation, we'll use [NVIDIA metrics](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/index.md) from the Ragas framework, which offer significant advantages for prompt engineering workflows: - **Faster computation**: Requires fewer LLM calls than alternative metrics - **Lower token consumption**: Reduces API costs during iterative testing - **Robust evaluation**: Provides consistent measurements through dual LLM judgments These characteristics make NVIDIA metrics particularly suitable for prompt optimization, where multiple iterations and experiments are often necessary. For our diabetes assistant, we will use: - [AnswerAccuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#answer-accuracy): Evaluates how well the model's response aligns with the reference answer. - [ResponseGroundedness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#response-groundedness): Measures whether the response is grounded in the provided context, helping to identify hallucinations or made-up information. ```python from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI from ragas.metrics import ( AnswerAccuracy, ResponseGroundedness, ) evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini")) metrics = [ AnswerAccuracy(llm=evaluator_llm), ResponseGroundedness(llm=evaluator_llm), ] ``` ### Preparing the Test Dataset ```python from ragas import evaluate # Evaluate both datasets with standard metrics (for answerable questions) answerable_df = eval_df.iloc[:10] # First 10 questions should be answered answerable_dataset_base = EvaluationDataset.from_list( [sample for i, sample in enumerate(eval_dataset_base.to_list()) if i < 10] ) answerable_dataset_incentive = EvaluationDataset.from_list( [sample for i, sample in enumerate(eval_dataset_incentive.to_list()) if i < 10] ) ``` ### Running the Evaluation ```python print("Evaluating answerable questions with base prompt...") result_answerable_base = evaluate(metrics=metrics, dataset=answerable_dataset_base) result_answerable_base ``` Output ```text Evaluating answerable questions with base prompt... Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 20/20 [00:02<00:00, 9.79it/s] {'nv_accuracy': 0.6750, 'nv_response_groundedness': 1.0000} ``` ```python print("Evaluating answerable questions with incentive prompt...") result_answerable_incentive = evaluate(metrics=metrics, dataset=answerable_dataset_incentive) result_answerable_incentive ``` Output ```text Evaluating answerable questions with incentive prompt... Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 20/20 [00:02<00:00, 9.19it/s] {'nv_accuracy': 0.6750, 'nv_response_groundedness': 1.0000} ``` Impact of Incentivization: For queries within the agent’s expertise, incentivization did not affect performance. - Answer accuracy remains unchanged (0.6750 β†’ 0.6750) - Response groundedness score remains consistent (1.0000 β†’ 1.0000) ## Queries that should NOT be answered (insufficient expertise) ### Preparing the Test Dataset Queries that should NOT be answered (insufficient expertise) ```python non_answerable_df = eval_df.iloc[10:] # Last 5 questions should NOT be answered non_answerable_dataset_base = EvaluationDataset.from_list( [sample for i, sample in enumerate(eval_dataset_base.to_list()) if i >= 10] ) non_answerable_dataset_incentive = EvaluationDataset.from_list( [sample for i, sample in enumerate(eval_dataset_incentive.to_list()) if i >= 10] ) ``` ### Setting Up Evaluation Metrics Ragas offers several built-in metrics and allows you to create custom metrics tailored to your specific business needs. For our diabetes assistant, we will use the following metrics to evaluate its performance on queries that it should not answer. ### Continuing with NVIDIA Metrics for Efficiency As with our previous evaluation, we'll use the NVIDIA AnswerAccuracy metric for its computational efficiency and low token consumption. For non-answerable questions, we'll complement this with a custom metric tailored to our specific requirement. Let's understand what each metric measures: - [AnswerAccuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#answer-accuracy): Evaluates how well the model's response aligns with the reference answer. - Non-Answer Compliance: A custom metric that checks whether the model appropriately declines to answer when required, which is crucial for safety in a medical context. For example, Non-Answer Compliance is built using [AspectCritique](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/aspect_critic/index.md). Ragas offers flexible tools to create custom metrics that measure your specific business goals. You can learn more about these capabilities by clicking [here](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/index.md). ```python from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI from ragas.metrics import ( AnswerAccuracy, AspectCritic ) # Create a specialized metric for evaluating when the model should NOT answer no_answer_metric = AspectCritic( name="Non-Answer Compliance", definition="Return 1 if both reference and response appropriately decline to provide an answer on the same grounds (e.g., medical expertise limitations); return 0 if the response provides any answer when the reference declines to answer.", llm=evaluator_llm, ) evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini")) metrics = [ AnswerAccuracy(llm=evaluator_llm), no_answer_metric, ] ``` ### Running the Evaluation ```python print("Evaluating non-answerable questions with base prompt...") result_non_answerable_base = evaluate(metrics=metrics, dataset=non_answerable_dataset_base) result_non_answerable_base ``` Output ```text Evaluating non-answerable questions with base prompt... Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:01<00:00, 5.44it/s] {'nv_accuracy': 0.6000, 'Non-Answer Compliance': 0.4000} ``` ```python print("Evaluating non-answerable questions with incentive prompt...") result_non_answerable_incentive = evaluate(metrics=metrics, dataset=non_answerable_dataset_incentive) result_non_answerable_incentive ``` Output ```text Evaluating non-answerable questions with incentive prompt... Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 10/10 [00:01<00:00, 6.28it/s] {'nv_accuracy': 0.7000, 'Non-Answer Compliance': 0.6000} ``` Impact of Incentivization: The incentivized prompt showed a slight improvement in answer accuracy (0.6 β†’ 0.7) Most importantly, the incentivized prompt was significantly better at declining to answer questions outside its expertise (40% β†’ 60%) ## Iterative Improvement Process Leveraging our evaluation metrics, we now adopt a data-driven approach to refine our prompt strategies. The process unfolds as follows: 1. Establish a Baseline: Begin with an initial prompt. 1. Performance Evaluation: Measure its performance using our defined metrics. 1. Targeted Analysis: Identify shortcomings and implement focused improvements. 1. Re-Evaluation: Test the revised prompt. 1. Adopt and Iterate: Retain the version that performs better and repeat the cycle. ## Conclusion This systematic approach offers clear advantages over a reactive β€œwhack-a-mole” strategy: - It quantifies improvements across all key requirements simultaneously. - It maintains a consistent, reproducible testing framework. - It enables immediate detection of any regressions. - It bases decisions on objective data rather than intuition. Through these iterative refinements, we steadily progress towards an optimal and robust prompt strategy. # Generating a Synthetic Test Set for RAG-Based Question Answering with Ragas ## Overview In this tutorial, we'll explore the **test set generation module in Ragas** to create a **synthetic test set** for a **Retrieval-Augmented Generation (RAG)-based question-answering bot**. Our goal is to design a **Ragas Airline Assistant** capable of answering customer queries on various topics, including: - Flight booking - Flight changes and cancellations - Baggage policies - Viewing reservations - Flight delays - In-flight services - Special assistance To make sure our synthetic dataset is as **realistic and diverse** as possible, we will create **different customer personas**. Each persona will represent distinct traveler types and behaviors, helping us build a **comprehensive and representative test set**. This approach ensures that we can thoroughly evaluate the effectiveness and robustness of our RAG model. Let’s get started! ## Download and Load documents Run the command below to download the dummy Ragas Airline dataset and load the documents using LangChain. ```sh ! git clone https://huggingface.co/datasets/vibrantlabsai/ragas-airline-dataset ``` ```python from langchain_community.document_loaders import DirectoryLoader path = "ragas-airline-dataset" loader = DirectoryLoader(path, glob="**/*.md") docs = loader.load() ``` ## Set up the LLM and Embedding Model ```python from ragas.llms import LangchainLLMWrapper from ragas.embeddings import OpenAIEmbeddings from langchain_openai import ChatOpenAI import openai generator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini")) openai_client = openai.OpenAI() generator_embeddings = OpenAIEmbeddings(client=openai_client, model="text-embedding-3-small") ``` ## Create Knowledge Graph Create a base knowledge graph with the documents ```python from ragas.testset.graph import KnowledgeGraph from ragas.testset.graph import Node, NodeType kg = KnowledgeGraph() for doc in docs: kg.nodes.append( Node( type=NodeType.DOCUMENT, properties={"page_content": doc.page_content, "document_metadata": doc.metadata} ) ) kg ``` Output ```text KnowledgeGraph(nodes: 8, relationships: 0) ``` ## Setup the transforms In this tutorial, we create a Single Hop Query dataset using a knowledge graph built solely from nodes. To enhance our graph and improve query generation, we apply three key transformations: - **Headline Extraction:** Uses a language model to extract clear section titles from each document (e.g., β€œAirline Initiated Cancellations” from *flight cancellations.md*). These titles isolate specific topics and provide direct context for generating focused questions. - **Headline Splitting:** Divides documents into manageable subsections based on the extracted headlines. This increases the number of nodes and ensures more granular, context-specific query generation. - **Keyphrase Extraction:** Identifies core thematic keyphrases (such as key seating information) that serve as semantic seed points, enriching the diversity and relevance of the generated queries. ```python from ragas.testset.transforms import apply_transforms from ragas.testset.transforms import HeadlinesExtractor, HeadlineSplitter, KeyphrasesExtractor headline_extractor = HeadlinesExtractor(llm=generator_llm, max_num=20) headline_splitter = HeadlineSplitter(max_tokens=1500) keyphrase_extractor = KeyphrasesExtractor(llm=generator_llm) transforms = [ headline_extractor, headline_splitter, keyphrase_extractor ] apply_transforms(kg, transforms=transforms) ``` ```text Applying HeadlinesExtractor: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 8/8 [00:00 0: df = df.head(limit) # Create Ragas Dataset dataset = Dataset(name="text2sql_booksql", backend="local/csv", root_dir=".") for _, row in df.iterrows(): dataset.append({ "Query": row["Query"], "SQL": row["SQL"], "Levels": row["Levels"], "split": row["split"], }) return dataset ``` The dataset loader includes a `limit` parameter for development workflows - start with small samples to catch basic errors quickly, then scale to full evaluation. ## Run baseline evaluation ### Execute evaluation pipeline and collect results ```python import asyncio from ragas_examples.text2sql.evals import text2sql_experiment, load_dataset async def run_evaluation(): """Run text-to-SQL evaluation with direct code approach.""" # Load dataset dataset = load_dataset() print(f"Dataset loaded with {len(dataset)} samples") # Run the experiment results = await text2sql_experiment.arun( dataset, name="gpt-5-mini-prompt-v1", model="gpt-5-mini", prompt_file=None, ) # Report results print(f"βœ… gpt-5-mini-prompt-v1: {len(results)} cases evaluated") # Calculate and display accuracy accuracy_rate = sum(1 for r in results if r["execution_accuracy"] == "correct") / max(1, len(results)) print(f"gpt-5-mini-prompt-v1 Execution Accuracy: {accuracy_rate:.2%}") # Run the evaluation await run_evaluation() ``` πŸ“‹ Output (prompt v1) ```text Loading dataset... Dataset loaded with 99 samples Running text-to-SQL evaluation with model: gpt-5-mini Using prompt file: prompt.txt Running experiment: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 99/99 [01:06<00:00, 1.49it/s] βœ… gpt-5-mini-prompt-v1: 99 cases evaluated gpt-5-mini-prompt-v1 Execution Accuracy: 2.02% ``` **Configuration options:** - `model`: OpenAI model to use (default: "gpt-5-mini") - `prompt_file`: Custom prompt file (default: None for built-in prompt) - `limit`: Number of samples (default: None for all samples, specify a number to limit) - `name`: Custom experiment name for result tracking ### Initial performance analysis The evaluation generates comprehensive CSV results for analysis. Each row contains: - `query`: Natural language input - `expected_sql`: Ground truth SQL - `predicted_sql`: Generated SQL - `level`: Difficulty (easy/medium/hard) - `execution_accuracy`: correct/incorrect (result comparison) - `accuracy_reason`: Detailed comparison results ## Analyze errors and failure patterns After running evaluations, you can analyze the failure patterns to understand where your Text2SQL agent is making mistakes. Here are some sample rows from our baseline evaluation results to give you a complete picture of what the evaluation data looks like: | Query | Expected SQL | Predicted SQL | Level | Execution Accuracy | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | ------------------ | | What is the balance due from Richard Aguirre? | `select sum(open_balance) from ( select distinct transaction_id, open_balance from master_txn_table where customers = "Richard Aguirre" )` | `SELECT customer_name, customer_full_name, Balance FROM customers WHERE customer_name = 'Richard Aguirre' OR customer_full_name = 'Richard Aguirre';` | medium | incorrect | | How much open credit does customer Natalie Myers? | `select sum(open_balance) from ( select distinct transaction_id, open_balance from master_txn_table where customers = "Natalie Myers" )` | `SELECT Balance FROM customers WHERE customer_name = 'Natalie Myers' OR customer_full_name = 'Natalie Myers';` | easy | incorrect | | What is my average invoice from Ashley Thompson? | `select avg(amount) from (select distinct transaction_id, amount from master_txn_table where customers = "Ashley Thompson" and transaction_type = 'invoice')` | `SELECT AVG(Amount) FROM master_txn_table WHERE Customers = 'Ashley Thompson' AND Transaction_TYPE = 'Invoice';` | hard | incorrect | | Show number of transactions with Carol Smith | `select count(distinct transaction_id) from master_txn_table where customers = 'Carol Smith' or vendor = 'Carol Smith'` | `SELECT COUNT(*) AS num_transactions FROM master_txn_table WHERE Customers = 'Carol Smith';` | medium | incorrect | **Key observations from these examples:** **Execution Accuracy: 0%** - None of the queries return the same results as expected **Common failure patterns:** - Using wrong tables (`customers` vs `master_txn_table`) - Missing deduplication (`count(distinct transaction_id)` vs `COUNT(*)`) - Incomplete filtering logic (missing `OR vendor = 'Name'` conditions) - Wrong column names (`Balance` vs `open_balance`) This shows that while the agent generates valid SQL, it needs significant improvement in understanding the business logic and database schema relationships. ### Error Analysis To analyze your failures systematically, manually review and annotate each row in your results CSV, categorizing the types of errors you observe. You can use AI to help you categorize with this prompt: πŸ“‹ Error Analysis Categorization Prompt ```text You are analyzing why a Text2SQL prediction failed. Given the following information, identify the error codes and provide a brief analysis. Available error codes: - AGGR_DISTINCT_MISSING: Used COUNT/SUM without DISTINCT or deduplication - WRONG_FILTER_COLUMN: Filtered on the wrong column - WRONG_SOURCE_TABLE_OR_COLUMN: Selected metric from the wrong table/column - EXTRA_TRANSFORMATION_OR_CONDITION: Added ABS(), extra filters that change results - OUTPUT_COLUMN_ALIAS_MISMATCH: Output column names don't match - NULL_OR_EMPTY_RESULT: Result is None/empty due to wrong filters or source - GENERIC_VALUE_MISMATCH: Aggregation computed but numeric value differs for unclear reasons - OTHER: Fallback Query: [YOUR_QUERY] Expected SQL: [EXPECTED_SQL] Predicted SQL: [PREDICTED_SQL] Execution Accuracy: [ACCURACY_RESULT] Accuracy Reason: [ACCURACY_REASON] Respond with: - error_codes: array of applicable error codes (1 or more) - error_analysis: brief 1-3 sentence explanation of what went wrong ``` Copy this prompt and use it with your preferred LLM to analyze individual failures from your results CSV. For convenience, we made it a script that you can use to automatically categorize errors using GPT 5: `uv run python -m ragas_examples.text2sql.analyze_errors --input experiments/your_results.csv` ### Review Process 1. **Examine high-frequency error codes** from the summary 1. **Manually inspect 5-10 examples** of each major error type 1. **Verify the AI's categorization** matches your understanding 1. **Check if the "expected" SQL is actually correct** for your schema 1. **Look for patterns** the AI might have missed 1. **Prioritize fixes** based on impact and frequency Only after manual validation should you use these insights to improve your prompts, few-shot examples, or agent architecture. ### Sample error analysis results Here are some sample rows from the annotated results showing the types of errors that remain after the first round of improvements: | Query | Expected SQL | Predicted SQL | Error Analysis | Error Codes | | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------- | | How much we received from Fuel? | `select sum(credit) from master_txn_table as T1 join chart_of_accounts as T2 on T1.account = T2.account_name where account_type in ('Income', 'Other Income') and instr(account,"Fuel")` | `SELECT SUM(Amount) FROM (SELECT DISTINCT Transaction_ID, Amount FROM master_txn_table WHERE Customers = 'Fuel');` | The prediction summed Amount from master_txn_table instead of credit and omitted the join to chart_of_accounts, using Customers='Fuel' rather than filtering account names containing 'Fuel' and restricting to Income/Other Income. | `["WRONG_SOURCE_TABLE_OR_COLUMN", "WRONG_FILTER_COLUMN", "NULL_OR_EMPTY_RESULT"]` | | What was the most recent invoice for Roberta Shaw? | `select transaction_id from master_txn_table where transaction_type = 'invoice' and customers = "Roberta Shaw" order by transaction_date desc limit 1` | `SELECT Transaction_ID, Transaction_DATE, Amount, Open_balance, CreatedDATE, CreatedUSER, Account, Customers, Product_Service FROM master_txn_table WHERE Transaction_TYPE = 'invoice' AND Customers = 'Roberta Shaw' ORDER BY Transaction_DATE DESC LIMIT 1;` | The prediction selected nine columns instead of only Transaction_ID as expected, causing a column/schema mismatch despite returning the same row. | `["OUTPUT_COLUMN_ALIAS_MISMATCH"]` | | What is my average invoice from Jeremy Strong? | `select avg(amount) from (select distinct transaction_id, amount from master_txn_table where customers = "Jeremy Strong" and transaction_type = 'invoice')` | `SELECT AVG(Amount) FROM (SELECT DISTINCT Transaction_ID, Amount FROM master_txn_table WHERE Transaction_TYPE = 'invoice' AND Vendor = 'Jeremy Strong') AS t;` | The query filters on Vendor = 'Jeremy Strong' instead of the correct customers column, so it likely matched no rows. This leads to AVG(amount) returning NULL. | `["WRONG_FILTER_COLUMN", "NULL_OR_EMPTY_RESULT"]` | **Key observations from results:** - **Error patterns:** - **Missing OR conditions**: Queries about transactions "with" someone should check both `customers` and `vendor` columns - **Wrong column selection**: Using `Amount` instead of `credit` for financial queries - **Output schema mismatches**: Selecting too many columns or wrong column names - **Missing joins**: Not joining with `chart_of_accounts` for account-type filtering These patterns inform the next iteration of prompt improvements, focusing on complete filtering logic and proper financial query handling. Decide what to change in the prompt using generic rules, not per-row fixes. Avoid adding case-specific examples; prefer schema-grounded guardrails so that you are not overfitting to the data. Repeat this loop iteratively: - Run β†’ Annotate β†’ Review β†’ Decide generic guardrails β†’ Update `prompt_vX.txt` β†’ Re-run β†’ Compare β†’ Repeat. - Keep guardrails concise and schema-grounded so improvements generalize without overfitting. - Version your prompts (`prompt_v2.txt`, `prompt_v3.txt`, `prompt_v4.txt`) and maintain a brief changelog per version. - Stop when execution accuracy plateaus across two consecutive iterations or meets your business threshold. ## Improve your system ### Create and use a new prompt version We keep the baseline prompt intact and create a new version for iteration. Create `prompt_v2.txt` to include concise, reusable guardrails. Keep them generic enough to apply broadly while grounded in the provided schema. Example of a section we added to `prompt_v1.txt` to create `prompt_v2.txt`: ```text - Use exact table and column names from the schema; do not invent fields - Prefer transactional facts from `master_txn_table`; use entity tables for static attributes - Map parties correctly in filters: - Customer-focused β†’ filter on `Customers` - Vendor-focused β†’ filter on `Vendor` - Disambiguate events via `Transaction_TYPE` (e.g., invoices β†’ `Transaction_TYPE = 'invoice'`) - Avoid double-counting by deduplicating on `Transaction_ID` for counts and aggregates: - Counts: `count(distinct Transaction_ID)` - Aggregates: compute over a deduplicated subquery on `(Transaction_ID, metric_column)` - For open credit/balance due per customer, aggregate `Open_balance` from `master_txn_table` filtered by `Customers` with deduplication - Do not add extra transforms or filters (e.g., `abs()`, `< 0`) unless explicitly asked - Keep a single `SELECT`; avoid aliases for final column names ``` We save this improved prompt as `prompt_v2.txt`. ### Re-run evaluation with the new prompt ```python import asyncio from ragas_examples.text2sql.evals import text2sql_experiment, load_dataset async def run_v2_evaluation(): """Run evaluation with prompt v2.""" # Load dataset dataset = load_dataset() print(f"Dataset loaded with {len(dataset)} samples") # Run experiment results = await text2sql_experiment.arun( dataset, name="gpt-5-mini-prompt-v2", model="gpt-5-mini", prompt_file="prompt_v2.txt", ) # Report results print(f"βœ… gpt-5-mini-prompt-v2: {len(results)} cases evaluated") # Calculate accuracy accuracy_rate = sum(1 for r in results if r["execution_accuracy"] == "correct") / max(1, len(results)) print(f"gpt-5-mini-prompt-v2 Execution Accuracy: {accuracy_rate:.2%}") await run_v2_evaluation() ``` πŸ“‹ Output (prompt v2) ```text Loading dataset... Dataset loaded with 99 samples Running text-to-SQL evaluation with model: gpt-5-mini Using prompt file: prompt_v2.txt Running experiment: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 99/99 [01:00<00:00, 1.63it/s] βœ… gpt-5-mini-prompt-v2: 99 cases evaluated gpt-5-mini-prompt-v2 Execution Accuracy: 60.61% ``` We see an improvement from 2.02% to 60.61% in execution accuracy with `prompt_v2`. Review the new results CSV in `experiments/` and continue the loop again. ### Continue iterating: Create prompt v3 Even with the major improvements in `prompt_v2.txt`, the 60% accuracy still leaves room for growth. A deeper analysis of the failures reveals several recurring patterns: 1. **Misunderstanding of Financial Concepts**: The model consistently defaults to aggregating the `Amount` column instead of the correct `Credit` (for income) or `Debit` (for expenses) columns. It also often fails to `JOIN` with `chart_of_accounts` to filter by account type (e.g., 'Income'). 1. **Adding Unnecessary Transformations**: The model frequently complicates queries with unrequested `DISTINCT` clauses or extra filters (like `Transaction_TYPE = 'invoice'`), which alter the results. 1. **Incorrect Column Selection**: For "show all transactions" queries, it often uses `SELECT *` instead of the expected `SELECT DISTINCT Transaction_ID`, leading to schema mismatches. It also generates the wrong column names for aggregations (e.g. `max(transaction_date)` instead of `transaction_date`). 1. **Incomplete Filtering**: It often misses `OR` conditions (e.g., checking both `Customers` and `Vendor` for a transaction "with" someone) or filters on the wrong column entirely. Based on this deeper analysis, create `prompt_v3.txt` with even more specific, schema-grounded guidelines to address these recurring issues: Key additions to `prompt_v3.txt`: ```text ### CORE QUERY GENERATION GUIDELINES 1. **Use Correct Schema**: Use exact table and column names... 2. **Simplicity First**: Keep the query as simple as possible... ... ### ADVANCED QUERY PATTERNS 5. **Financial Queries (Revenue, Sales, Expenses)**: - **Metric Selection**: - For revenue, income, sales, or money **received**: aggregate the `Credit` column. - For expenses, bills, or money **spent**: aggregate the `Debit` column. - Use the `Amount` column only when... - **Categorical Financial Queries**: For questions involving financial categories... you **MUST** `JOIN` `master_txn_table` with `chart_of_accounts`... 6. **Filtering Logic**: - **Ambiguous Parties**: For questions about transactions "with" or "involving" a person or company, you **MUST** check both `Customers` and `Vendor` columns. E.g., `WHERE Customers = 'Name' OR Vendor = 'Name'`. - **Avoid Extra Filters**: Do not add implicit filters... 7. **Column Selection and Naming**: - **Avoid `SELECT *`**: When asked to "show all transactions", return only `DISTINCT Transaction_ID`... - **"Most Recent" / "Last" Queries**: To get the 'most recent' or 'last' record, use `ORDER BY Transaction_DATE DESC LIMIT 1`. This preserves the original column names... Avoid using `MAX()`... ``` These new rules are designed to be generic but directly target the observed failure patterns. **Re-run evaluation with `prompt_v3.txt`:** ```python import asyncio from ragas_examples.text2sql.evals import text2sql_experiment, load_dataset async def run_v3_evaluation(): """Run evaluation with prompt v3.""" # Load dataset dataset = load_dataset() print(f"Dataset loaded with {len(dataset)} samples") # Run experiment results = await text2sql_experiment.arun( dataset, name="gpt-5-mini-prompt-v3", model="gpt-5-mini", prompt_file="prompt_v3.txt", ) # Report results print(f"βœ… gpt-5-mini-prompt-v3: {len(results)} cases evaluated") # Calculate accuracy accuracy_rate = sum(1 for r in results if r["execution_accuracy"] == "correct") / max(1, len(results)) print(f"gpt-5-mini-prompt-v3 Execution Accuracy: {accuracy_rate:.2%}") await run_v3_evaluation() ``` We see an improvement from 60.61% to 70.71% in execution accuracy with `prompt_v3`. ### Key principles for continued iteration The 70% accuracy achieved with `prompt_v3.txt` demonstrates the power of systematic iteration. You can continue this process to push accuracy even higher. **Key principles for continued iteration:** - Each iteration should target **3-5 high-frequency error patterns** from the latest results - Keep new rules **generic and schema-grounded** to avoid overfitting - **Stop when accuracy plateaus** across 2-3 consecutive iterations - If you hit a plateau with prompt improvements, you can try experimenting with better models or return any sql error back to the LLM to fix it making an actual agentic flow. ## Compare results After running all prompt versions, we can compare the final results. | Prompt | Execution Accuracy | Results CSV | | -------------------- | ------------------ | ------------------------------- | | v1 (`prompt.txt`) | 2.02% | `experiments/...-prompt-v1.csv` | | v2 (`prompt_v2.txt`) | 60.61% | `experiments/...-prompt-v2.csv` | | v3 (`prompt_v3.txt`) | 70.71% | `experiments/...-prompt-v3.csv` | **Progress Analysis:** - **v1 β†’ v2**: Massive 58 percentage point jump from 2.02% to 60.61% through basic deduplication and business logic guidelines - **v2 β†’ v3**: Additional 10 percentage point improvement from 60.61% to 70.71% through enhanced financial query guidelines, better filtering logic, and column selection rules - The improvements target specific failure patterns identified through error analysis: financial concepts, unnecessary transformations, and incomplete filtering ## Conclusion This guide showed you how to build a systematic evaluation process for text-to-SQL systems. **Key takeaways:** - Set up execution accuracy metrics to compare actual query results - Follow the iterative process: evaluate β†’ analyze errors β†’ improve β†’ repeat The evaluation framework gives you a reliable way to measure and improve your system, with Ragas handling the orchestration and result aggregation automatically. # Aligning LLM Evaluators with Human Judgment This tutorial is part of a three-part series on how to use Vertex AI models with Ragas. It is recommended that you have gone through [Getting Started: Ragas with Vertex AI](https://docs.ragas.io/en/stable/howtos/applications/vertexai_x_ragas/index.md), even if you have not, you can easily follow this. You can navigate to the Model Comparison tutorial using the [link](https://docs.ragas.io/en/stable/howtos/applications/vertexai_model_comparision/index.md). ## Overview In this tutorial, you will learn how to train and align your own custom LLM-based metric using Ragas. While LLM-based evaluators offer a powerful means of scoring AI applications, they can sometimes produce judgments that diverge from human expectations due to differences in style, context, or subtle nuances. By following this guide, you will refine your metric so that it more accurately mirrors human judgment. In this tutorial, you will: 1. Define a model-based metric using Ragas. 1. Construct an EvaluationDataset from the "helpful" subset of the HHH dataset. 1. Run an initial evaluation to benchmark the metric’s performance. 1. Review and annotate 15–20 evaluation examples. 1. Train the metric using your annotated data. 1. Reevaluate the metric to observe improvements in alignment with human judgments. ## Getting Started ### Install Dependencies ```python %pip install --upgrade --user --quiet langchain-core langchain-google-vertexai langchain ragas ``` ### Restart runtime To use the newly installed packages in this Jupyter runtime, you must restart the runtime. You can do this by running the cell below, which restarts the current kernel. The restart might take a minute or longer. After it's restarted, continue to the next step. ```python import IPython app = IPython.Application.instance() app.kernel.do_shutdown(True) ``` ### Authenticate your notebook environment (Colab only) If you're running this notebook on Google Colab, run the cell below to authenticate your environment. ```python import sys if "google.colab" in sys.modules: from google.colab import auth auth.authenticate_user() ``` ### Set Google Cloud project information and initialize Vertex AI SDK ```python PROJECT_ID = "[your-project-id]" # @param {type:"string"} LOCATION = "us-central1" # @param {type:"string"} if not PROJECT_ID or PROJECT_ID == "[your-project-id]": raise ValueError("Please set your PROJECT_ID") import vertexai vertexai.init(project=PROJECT_ID, location=LOCATION) ``` ## Set up eval metrics LLM-based metrics have tremendous potential but can sometimes misjudge responses compared to human evaluators. To bridge this gap, we align our model-based metric with human judgment using a feedback loop. ### Define evaluator_llm Import the required wrappers and define your evaluator LLM and embedder. ```python from ragas.llms import LangchainLLMWrapper from ragas.embeddings import LangchainEmbeddingsWrapper from langchain_google_vertexai import VertexAI, VertexAIEmbeddings evaluator_llm = LangchainLLMWrapper(VertexAI(model_name="gemini-2.0-flash-001")) evaluator_embeddings = LangchainEmbeddingsWrapper(VertexAIEmbeddings(model_name="text-embedding-004")) ``` ### Ragas metrics Ragas offers various model-based metrics that can be fine-tuned to align with human evaluators. For demonstration, we will use the **Aspect Critic** metricβ€”a user-defined, binary metric. For further details, please refer to the [Aspect Critic documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#aspect-critic). ```python from ragas.metrics import AspectCritic helpfulness_critic = AspectCritic( name="helpfulness", definition="Evaluate how helpful the assistant's response is to the user's query.", llm=evaluator_llm ) ``` You can preview the prompt that will be passed to the LLM (before alignment) by running: ```python print(helpfulness_critic.get_prompts()["single_turn_aspect_critic_prompt"].instruction) ``` Output ```text Evaluate the Input based on the criterial defined. Use only 'Yes' (1) and 'No' (0) as verdict. Criteria Definition: Evaluate how helpful the assistant's response is to the user's query. ``` ### Defining Alignment Score Since we are using a binary metric, we will measure the alignment using the F1-score. However, depending on the metric you are aligning, you can modify this function accordingly to use other methods to measure the alignment. ```python from typing import List from sklearn.metrics import f1_score def alignment_score(human_score: List[float], llm_score: List[float]) -> float: """ Computes the alignment between human-annotated binary scores and LLM-generated binary scores using the F1-score metric. Args: human_score (List[int]): Binary labels from human evaluation (0 or 1). llm_score (List[int]): Binary labels from LLM predictions (0 or 1). Returns: float: The F1-score measuring alignment. """ return f1_score(human_score, llm_score) ``` ## Prepare your dataset TheΒ `process_hhh_dataset`Β function prepares data from the [HHH dataset](https://paperswithcode.com/dataset/hhh?utm_source=chatgpt.com) for use in training and aligning of the LLM evaluator. Alternate 0 and 1 scores (1 for helpful, 0 for non-helpful) are assigned to each example, indicating which response is preferred. ```python import numpy as np from datasets import load_dataset from ragas import EvaluationDataset def process_hhh_dataset(split: str = "helpful", total_count: int = 50): dataset = load_dataset("HuggingFaceH4/hhh_alignment",split, split=f"test[:{total_count}]") data = [] expert_scores = [] for idx, entry in enumerate(dataset): # Extract input and target details user_input = entry['input'] choices = entry['targets']['choices'] labels = entry['targets']['labels'] # Choose target based on whether the index is even or odd if idx % 2 == 0: target_label = 1 score = 1 else: target_label = 0 score = 0 label_index = labels.index(target_label) response = choices[label_index] data.append({ 'user_input': user_input, 'response': response, }) expert_scores.append(score) return EvaluationDataset.from_list(data), expert_scores eval_dataset, expert_scores = process_hhh_dataset() ``` ## Run evaluation With the evaluation dataset and the helpfulness metric defined, you can now run the evaluation: ```python from ragas import evaluate results = evaluate(eval_dataset, metrics=[helpfulness_critic]) ``` ```text Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 50/50 [00:00 str: """Custom tool description...""" docs = self.retriever.retrieve(query, self.default_k) return "\n\n".join([doc.page_content for doc in docs]) self._agent = Agent( name="Custom RAG Assistant", instructions="Your custom instructions...", tools=[retrieve] ) ``` ## Comparing Results Run both modes and compare: ```sh # Run naive mode uv run python evals.py # Results saved to experiments/YYYYMMDD-HHMMSS_naiverag.csv # Run agentic mode uv run python evals.py --agentic # Results saved to experiments/YYYYMMDD-HHMMSS_agenticrag.csv ``` Analyze the results: ```python import pandas as pd naive = pd.read_csv("evals/experiments/..._naiverag.csv") agentic = pd.read_csv("evals/experiments/..._agenticrag.csv") print(f"Naive pass rate: {(naive['correctness_score'] == 'pass').mean():.1%}") print(f"Agentic pass rate: {(agentic['correctness_score'] == 'pass').mean():.1%}") ``` ## Troubleshooting ### MLflow Warnings If you see MLflow warnings about failed traces, either: 1. Start MLflow: `mlflow ui --port 5000` 1. Or ignore them - the evaluation still works without tracing ### Agentic Mode Not Working Ensure you have the agents package: ```sh pip install openai-agents ``` ### Slow First Run The first run downloads the HuggingFace documentation dataset (~300MB). Subsequent runs use the cached data. ## Next Steps - [RAG Evaluation Guide](https://docs.ragas.io/en/stable/howtos/cli/rag_eval/index.md) - Simpler evaluation setup - [Custom Metrics](https://docs.ragas.io/en/stable/howtos/cli/customizations/metrics/_write_your_own_metric.md) - Write your own metrics - [Evaluate and Improve RAG](https://docs.ragas.io/en/stable/howtos/applications/evaluate-and-improve-rag/index.md) - Production RAG evaluation # Judge Alignment Quickstart The `judge_alignment` template measures how well an LLM-as-judge aligns with human evaluation standards. ## Create the Project ```sh ragas quickstart judge_alignment cd judge_alignment ``` ## Install Dependencies ```sh uv sync ``` ## Set Your API Key ```sh export OPENAI_API_KEY="your-openai-key" ``` ## Run the Evaluation ```sh uv run python evals.py ``` ## Project Structure ```text judge_alignment/ β”œβ”€β”€ README.md # Project documentation β”œβ”€β”€ pyproject.toml # Project configuration β”œβ”€β”€ evals.py # Evaluation workflow β”œβ”€β”€ __init__.py # Python package marker └── evals/ β”œβ”€β”€ datasets/ # Test datasets β”œβ”€β”€ experiments/ # Evaluation results └── logs/ # Execution logs ``` ## What It Evaluates The template evaluates LLM judge alignment: - **Scenario**: Pre-existing responses are evaluated by an LLM judge - **Human Labels**: Ground truth pass/fail labels - **LLM Judge**: Evaluates same responses with grading criteria - **Alignment Metric**: Agreement between human and LLM judgments ## Understanding the Code ### Judge Metrics (`evals.py`) Two judge implementations to compare: ```python # Baseline judge (simple prompt) accuracy_metric = DiscreteMetric( name="accuracy", prompt="Check if response contains points from grading notes...", allowed_values=["pass", "fail"], ) # Improved judge (enhanced with abbreviation guide) accuracy_metric_v2 = DiscreteMetric( name="accuracy", prompt="""Evaluate if response covers ALL key concepts... ABBREVIATION GUIDE: β€’ Financial: val=valuation, post-$=post-money, rev=revenue... β€’ Business: mkt=market, reg=regulation... """, allowed_values=["pass", "fail"], ) ``` ### The Evaluation Tests alignment with human judgment: ```python @discrete_metric(name="alignment", allowed_values=["aligned", "misaligned"]) def alignment_metric(llm_judgment: str, human_judgment: str): # Compares LLM judge output with human label return "aligned" if llm_judgment == human_judgment else "misaligned" ``` ## Test Data The dataset includes: - Pre-evaluated responses - Human pass/fail labels - Grading notes with expected points - Various abbreviations and business terminology ## Use Cases ### Compare Judge Versions Run experiments with both judges: ```python # Test baseline judge results_v1 = await run_with_judge(accuracy_metric) # Test improved judge results_v2 = await run_with_judge(accuracy_metric_v2) # Compare alignment rates ``` ### Improve Judge Quality Iterate on judge prompts to improve alignment: 1. Identify misalignment patterns 1. Update judge prompt with clearer criteria 1. Re-evaluate alignment 1. Repeat until satisfactory ## Next Steps - [Prompt Evaluation](https://docs.ragas.io/en/stable/howtos/cli/prompt_evals/index.md) - Compare different prompts - [LLM Benchmarking](https://docs.ragas.io/en/stable/howtos/cli/benchmark_llm/index.md) - Compare different models # LlamaIndex Agent Evaluation Quickstart The `llamaIndex_agent_evals` template evaluates LlamaIndex workflow agents with tool call accuracy metrics. ## Create the Project ```sh ragas quickstart llamaIndex_agent_evals cd llamaIndex_agent_evals ``` ## Install Dependencies ```sh uv sync ``` ## Set Your API Keys ```sh export OPENAI_API_KEY="your-openai-key" export GOOGLE_API_KEY="your-google-key" # For evaluator LLM ``` ## Run the Evaluation ```sh uv run python evals.py ``` ## Project Structure ```text llamaIndex_agent_evals/ β”œβ”€β”€ README.md # Project documentation β”œβ”€β”€ pyproject.toml # Project configuration β”œβ”€β”€ llamaindex_agent.py # LlamaIndex agent with tools β”œβ”€β”€ evals.py # Evaluation workflow β”œβ”€β”€ __init__.py # Python package marker └── evals/ β”œβ”€β”€ datasets/ β”‚ └── contexts/ # Test context files (JSON) β”œβ”€β”€ experiments/ # Evaluation results └── logs/ # Execution logs ``` ## What It Evaluates The template evaluates a LlamaIndex agent's tool calling accuracy: - **Agent**: LlamaIndex `FunctionAgent` with list management tools (add, remove, list items) - **Test Cases**: Complex scenarios like duplicate additions, ambiguous removal requests - **Metrics**: Tool call accuracy, response correctness ## Understanding the Code ### The Agent (`llamaindex_agent.py`) LlamaIndex agent with simple tools: ```python from llama_index.core.agent.workflow import FunctionAgent agent = FunctionAgent( name="list_manager", tools=[add_item, remove_item, list_items], llm=llm ) ``` ### The Evaluation (`evals.py`) Tests tool call accuracy using F1 score: ```python @numeric_metric(name="tool_call_accuracy") def tool_call_accuracy_metric(predicted_calls: List[Dict], ground_truth_calls: List[Dict]): # Compares predicted vs ground truth tool calls # Returns F1 score between 0.0 and 1.0 ``` ## Test Data The template includes JSON test contexts in `evals/datasets/contexts/`: - `ambiguous_removal_request.json` - Tests handling of ambiguous requests - `duplicate_addition.json` - Tests handling of duplicate operations - `repeated_removal.json` - Tests repeated operations ## Next Steps - [Agent Evaluation](https://docs.ragas.io/en/stable/howtos/cli/agent_evals/index.md) - Evaluate general AI agents - [Workflow Evaluation](https://docs.ragas.io/en/stable/howtos/cli/workflow_eval/index.md) - Evaluate complex workflows # Prompt Evaluation Quickstart The `prompt_evals` template evaluates and compares different prompt variations with sentiment analysis. ## Create the Project ```sh ragas quickstart prompt_evals cd prompt_evals ``` ## Install Dependencies ```sh uv sync ``` ## Set Your API Key ```sh export OPENAI_API_KEY="your-openai-key" ``` ## Run the Evaluation ```sh uv run python evals.py ``` ## Project Structure ```text prompt_evals/ β”œβ”€β”€ README.md # Project documentation β”œβ”€β”€ pyproject.toml # Project configuration β”œβ”€β”€ prompt.py # Prompt implementation β”œβ”€β”€ evals.py # Evaluation workflow β”œβ”€β”€ __init__.py # Python package marker └── evals/ β”œβ”€β”€ datasets/ # Test datasets β”œβ”€β”€ experiments/ # Evaluation results └── logs/ # Execution logs ``` ## What It Evaluates The template evaluates prompt effectiveness for sentiment classification: - **Task**: Sentiment analysis (positive/negative) - **Test Cases**: Movie reviews with expected sentiment labels - **Metric**: Binary accuracy (pass/fail) ## Understanding the Code ### The Prompt (`prompt.py`) Implements the sentiment analysis prompt: ```python from prompt import run_prompt sentiment = run_prompt("I loved the movie! It was fantastic.") # Returns: "positive" or "negative" ``` ### The Evaluation (`evals.py`) Tests prompt accuracy: ```python @discrete_metric(name="accuracy", allowed_values=["pass", "fail"]) def my_metric(prediction: str, actual: str): return ( MetricResult(value="pass", reason="") if prediction == actual else MetricResult(value="fail", reason="") ) ``` ## Test Data The dataset includes movie reviews: ```python dataset_dict = [ {"text": "I loved the movie! It was fantastic.", "label": "positive"}, {"text": "The movie was terrible and boring.", "label": "negative"}, # More examples... ] ``` ## Customization ### Test Different Prompts Modify `prompt.py` to test variations: ```python # Version 1: Simple prompt = f"Is this positive or negative: {text}" # Version 2: With examples prompt = f"""Classify sentiment: Examples: - "Great movie" -> positive - "Boring film" -> negative Text: {text} Sentiment:""" # Compare results across versions ``` ### Add More Metrics Evaluate additional aspects: ```python from ragas.metrics import NumericalMetric confidence = NumericalMetric( name="confidence", prompt="Rate confidence 1-5 in this classification: {prediction}", allowed_values=(1, 5), ) ``` ## Next Steps - [Judge Alignment](https://docs.ragas.io/en/stable/howtos/cli/judge_alignment/index.md) - Measure LLM-as-judge alignment - [LLM Benchmarking](https://docs.ragas.io/en/stable/howtos/cli/benchmark_llm/index.md) - Compare different models # RAG Evaluation Quickstart The `rag_eval` template provides a complete RAG evaluation setup with custom metrics, dataset management, and experiment tracking. ## Create the Project ```sh # Using uvx (no installation required) uvx ragas quickstart rag_eval cd rag_eval # Or with ragas installed ragas quickstart rag_eval cd rag_eval ``` ## Install Dependencies ```sh uv sync ``` Or with pip: ```sh pip install -e . ``` ## Set Your API Key ```sh export OPENAI_API_KEY="your-openai-key" ``` ```sh export ANTHROPIC_API_KEY="your-anthropic-key" ``` Update `evals.py`: ```python from anthropic import Anthropic from ragas.llms import llm_factory client = Anthropic(api_key=os.environ.get("ANTHROPIC_API_KEY")) llm = llm_factory("claude-3-5-sonnet-20241022", provider="anthropic", client=client) ``` ```sh export GOOGLE_API_KEY="your-google-api-key" ``` Update `evals.py`: ```python import google.generativeai as genai from ragas.llms import llm_factory genai.configure(api_key=os.environ.get("GOOGLE_API_KEY")) client = genai.GenerativeModel("gemini-2.0-flash") llm = llm_factory("gemini-2.0-flash", provider="google", client=client) ``` ```python from openai import OpenAI from ragas.llms import llm_factory client = OpenAI( api_key="ollama", base_url="http://localhost:11434/v1" ) llm = llm_factory("mistral", provider="openai", client=client) ``` ## Run the Evaluation ```sh uv run python evals.py ``` The evaluation will: 1. Load test data from the `load_dataset()` function 1. Query your RAG application with test questions 1. Evaluate responses using custom metrics 1. Display results in the console 1. Save results to CSV in `evals/experiments/` ## Project Structure ```text rag_eval/ β”œβ”€β”€ README.md # Project documentation β”œβ”€β”€ pyproject.toml # Project configuration β”œβ”€β”€ rag.py # RAG application implementation β”œβ”€β”€ evals.py # Evaluation workflow β”œβ”€β”€ __init__.py # Python package marker └── evals/ β”œβ”€β”€ datasets/ # Test data files β”œβ”€β”€ experiments/ # Evaluation results (CSV) └── logs/ # Execution logs and traces ``` ## Understanding the Code ### The RAG Application (`rag.py`) A simple RAG implementation with: - **Document storage**: In-memory document collection - **Keyword retrieval**: Simple keyword matching for document retrieval - **Response generation**: OpenAI API for generating answers - **Tracing**: Logs each query for debugging ```python from rag import default_rag_client # Initialize with OpenAI client rag_client = default_rag_client(llm_client=openai_client, logdir="evals/logs") # Query the RAG system response = rag_client.query("What is Ragas?") print(response["answer"]) ``` ### The Evaluation Script (`evals.py`) The evaluation workflow: 1. **Dataset loading**: Creates test cases with questions and grading notes 1. **Metric definition**: Custom `DiscreteMetric` for pass/fail evaluation 1. **Experiment execution**: Runs queries and evaluates responses 1. **Result storage**: Saves to CSV for analysis ```python from ragas import Dataset, experiment from ragas.metrics import DiscreteMetric # Define your metric my_metric = DiscreteMetric( name="correctness", prompt="Check if the response contains points from grading notes...", allowed_values=["pass", "fail"], ) # Run experiment @experiment() async def run_experiment(row): response = rag_client.query(row["question"]) score = my_metric.score(llm=llm, response=response["answer"], ...) return {**row, "response": response["answer"], "score": score.value} ``` ## Customization ### Add Test Cases Edit the `load_dataset()` function in `evals.py`: ```python def load_dataset(): dataset = Dataset( name="test_dataset", backend="local/csv", root_dir="evals", ) data_samples = [ { "question": "What is Ragas?", "grading_notes": "- evaluation framework - LLM applications", }, { "question": "How do experiments work?", "grading_notes": "- track results - compare runs - store metrics", }, # Add more test cases... ] for sample in data_samples: dataset.append(sample) dataset.save() return dataset ``` ### Modify the Metric Change evaluation criteria by updating the metric prompt: ```python my_metric = DiscreteMetric( name="quality", prompt="""Evaluate the response quality: Response: {response} Expected Points: {grading_notes} Rate as: - 'excellent': All points covered with clear explanation - 'good': Most points covered - 'poor': Missing key points Rating:""", allowed_values=["excellent", "good", "poor"], ) ``` ### Add Multiple Metrics Create additional metrics for different evaluation aspects: ```python from ragas.metrics import DiscreteMetric, NumericalMetric correctness = DiscreteMetric( name="correctness", prompt="Is the response factually correct? {response}", allowed_values=["correct", "incorrect"], ) relevance = NumericalMetric( name="relevance", prompt="Rate relevance 1-5: {response} for question: {question}", allowed_values=(1, 5), ) ``` ### Use Your Own RAG System Replace the example RAG with your production system: ```python # In evals.py from your_rag_module import YourRAGClient rag_client = YourRAGClient(...) @experiment() async def run_experiment(row): # Call your RAG system response = await rag_client.query(row["question"]) score = my_metric.score( llm=llm, response=response, grading_notes=row["grading_notes"], ) return { **row, "response": response, "score": score.value, } ``` ## Viewing Results Results are saved to `evals/experiments/` as CSV files. Each experiment run creates a new file with: - Input data (questions, grading notes) - Model responses - Evaluation scores - Timestamps ```python import pandas as pd # Load results results = pd.read_csv("evals/experiments/your_experiment.csv") # Calculate pass rate pass_rate = (results["score"] == "pass").mean() print(f"Pass rate: {pass_rate:.1%}") ``` ## Next Steps - [Improve RAG Guide](https://docs.ragas.io/en/stable/howtos/cli/improve_rag/index.md) - Compare naive vs agentic RAG - [Custom Metrics](https://docs.ragas.io/en/stable/howtos/cli/customizations/metrics/_write_your_own_metric.md) - Write your own metrics - [Datasets](https://docs.ragas.io/en/stable/concepts/datasets/index.md) - Learn about dataset management - [Experimentation](https://docs.ragas.io/en/stable/concepts/experimentation/index.md) - Advanced experiment tracking # Text-to-SQL Evaluation Quickstart The `text2sql` template evaluates text-to-SQL systems by comparing SQL execution results. ## Create the Project ```sh ragas quickstart text2sql cd text2sql ``` ## Install Dependencies ```sh uv sync ``` ## Set Your API Key ```sh export OPENAI_API_KEY="your-openai-key" ``` ## Run the Evaluation ```sh uv run python evals.py ``` ## Project Structure ```text text2sql/ β”œβ”€β”€ README.md # Project documentation β”œβ”€β”€ pyproject.toml # Project configuration β”œβ”€β”€ text2sql_agent.py # Text-to-SQL agent β”œβ”€β”€ db_utils.py # Database utilities β”œβ”€β”€ evals.py # Evaluation workflow β”œβ”€β”€ prompt.txt # Base prompt template β”œβ”€β”€ prompt_v2.txt # Improved prompt v2 β”œβ”€β”€ prompt_v3.txt # Improved prompt v3 β”œβ”€β”€ __init__.py # Python package marker └── evals/ β”œβ”€β”€ datasets/ β”‚ └── booksql_sample.csv # Sample book database queries β”œβ”€β”€ experiments/ # Evaluation results └── logs/ # Execution logs ``` ## What It Evaluates The template evaluates text-to-SQL generation: - **Agent**: Converts natural language to SQL queries - **Database**: Sample book database with authors, titles, genres - **Test Cases**: Natural language questions β†’ expected SQL queries - **Metric**: Execution accuracy by comparing query results using datacompy ## Understanding the Code ### The Agent (`text2sql_agent.py`) Converts natural language to SQL: ```python from text2sql_agent import Text2SQLAgent agent = Text2SQLAgent(client=openai_client) sql = await agent.generate_sql("Find all books by Jane Austen") ``` ### The Evaluation (`evals.py`) Compares execution results: ```python @discrete_metric(name="execution_accuracy", allowed_values=["correct", "incorrect"]) def execution_accuracy(expected_sql: str, predicted_success: bool, predicted_result): # Executes both SQLs and compares results using datacompy # Returns "correct" if results match, "incorrect" otherwise ``` ## Test Data The template includes `evals/datasets/booksql_sample.csv` with sample questions and expected SQL queries for a book database. ## Customization ### Use Your Own Database Update `db_utils.py` to connect to your database: ```python def get_db_connection(): return sqlite3.connect("your_database.db") ``` ### Try Different Prompts The template includes three prompt versions in `prompt.txt`, `prompt_v2.txt`, and `prompt_v3.txt`. Test each to see which works best. ## Next Steps - [Agent Evaluation](https://docs.ragas.io/en/stable/howtos/cli/agent_evals/index.md) - Evaluate AI agents - [Workflow Evaluation](https://docs.ragas.io/en/stable/howtos/cli/workflow_eval/index.md) - Evaluate complex workflows # Workflow Evaluation Quickstart The `workflow_eval` template evaluates complex LLM workflows with email classification and routing. ## Create the Project ```sh ragas quickstart workflow_eval cd workflow_eval ``` ## Install Dependencies ```sh uv sync ``` ## Set Your API Key ```sh export OPENAI_API_KEY="your-openai-key" ``` ## Run the Evaluation ```sh uv run python evals.py ``` ## Project Structure ```text workflow_eval/ β”œβ”€β”€ README.md # Project documentation β”œβ”€β”€ pyproject.toml # Project configuration β”œβ”€β”€ workflow.py # Workflow implementation β”œβ”€β”€ evals.py # Evaluation workflow β”œβ”€β”€ __init__.py # Python package marker └── evals/ β”œβ”€β”€ datasets/ # Test datasets β”œβ”€β”€ experiments/ # Evaluation results └── logs/ # Execution logs ``` ## What It Evaluates The template evaluates a customer support email classification workflow: - **Workflow**: Multi-step email processing (classification β†’ extraction β†’ response) - **Categories**: Bug Report, Feature Request, Billing - **Test Cases**: Customer emails with expected categories and extracted fields - **Metric**: Custom discrete metric checking classification accuracy ## Understanding the Code ### The Workflow (`workflow.py`) Implements a customer support email workflow: ```python from workflow import default_workflow_client workflow = default_workflow_client() result = workflow.process_email("I found a bug in version 2.1.4...") # Returns: category, extracted fields, response ``` ### The Evaluation (`evals.py`) Tests workflow accuracy against pass criteria: ```python def load_dataset(): dataset_dict = [ { "email": "Hi, I'm getting error code XYZ-123 when using version 2.1.4...", "pass_criteria": "category Bug Report; product_version 2.1.4; error_code XYZ-123", }, # More test cases... ] ``` The metric evaluates if the workflow correctly: - Classifies the email category - Extracts relevant fields (version, error code, invoice number, etc.) - Generates appropriate responses ## Test Cases The template includes diverse scenarios: - **Bug Reports**: With version numbers and error codes - **Feature Requests**: With urgency levels and product areas - **Billing Issues**: With invoice numbers and amounts ## Customization ### Add Your Own Workflow Replace the example workflow with your own: ```python from your_workflow import YourWorkflow workflow = YourWorkflow() @experiment() async def run_experiment(row): result = await workflow.process(row["input"]) # Evaluate result... ``` ## Next Steps - [Agent Evaluation](https://docs.ragas.io/en/stable/howtos/cli/agent_evals/index.md) - Evaluate AI agents - [LlamaIndex Agent Evaluation](https://docs.ragas.io/en/stable/howtos/cli/llamaIndex_agent_evals/index.md) - Evaluate LlamaIndex workflows # Integrations # Integrations Ragas is a framework and can be integrated with a host of different frameworks and tools so that you can use Ragas with your own toolchain. If any tool you want is not supported feel free to raise an [issue](https://github.com/vibrantlabsai/ragas/issues/new) and we'll be more than happy to look into it πŸ™‚ ## Frameworks - [Amazon Bedrock](https://docs.ragas.io/en/stable/howtos/integrations/amazon_bedrock/index.md) - Amazon Bedrock is a managed framework for building, deploying, and scaling intelligent agents and integrated AI solutions; more information can be found [here](https://aws.amazon.com/bedrock/). - [Haystack](https://docs.ragas.io/en/stable/howtos/integrations/haystack/index.md) - Haystack is a LLM orchestration framework to build customizable, production-ready LLM applications, more information can be found [here](https://haystack.deepset.ai/). - [Griptape](https://docs.ragas.io/en/stable/howtos/integrations/griptape/index.md) - Griptape framework simplifies generative AI application development through flexible abstractions for LLMs, RAG, and more, additional information can be found [here](https://docs.griptape.ai/stable/griptape-framework/). - [Langchain](https://docs.ragas.io/en/stable/howtos/integrations/langchain/index.md) - Langchain is a framework for building LLM applications, more information can be found [here](https://www.langchain.com/). - [LlamaIndex for RAG](https://docs.ragas.io/en/stable/howtos/integrations/_llamaindex/index.md) - LlamaIndex is a framework for building RAG applications, more information can be found [here](https://www.llamaindex.ai/). - [LlamaIndex for Agents](https://docs.ragas.io/en/stable/howtos/integrations/llamaindex_agents/index.md) - LlamaIndex enables building intelligent, semi-autonomous agents, more information can be found [here](https://www.llamaindex.ai/). - [LlamaStack](https://docs.ragas.io/en/stable/howtos/integrations/llama_stack/index.md) – A unified framework by Meta for building and deploying generative AI apps across local, cloud, and mobile; [docs](https://llama-stack.readthedocs.io/en/latest/) - [OCI Gen AI](https://docs.ragas.io/en/stable/howtos/integrations/oci_genai/index.md) - Oracle Cloud Infrastructure Generative AI provides access to various LLM models including Cohere, Meta, and Mistral models for RAG evaluation. - [R2R](https://docs.ragas.io/en/stable/howtos/integrations/r2r/index.md) - R2R is an all-in-one solution for AI Retrieval-Augmented Generation (RAG) with production-ready features, more information can be found [here](https://r2r-docs.sciphi.ai/introduction) - [Swarm](https://docs.ragas.io/en/stable/howtos/integrations/swarm_agent_evaluation/index.md) - Swarm is a framework for orchestrating multiple AI agents, more information can be found [here](https://github.com/openai/swarm). ## Tracing Tools Tools that help you trace the LLM calls can be integrated with Ragas to get the traces of the evaluator LLMs. - [Arize Phoenix](https://docs.ragas.io/en/stable/howtos/integrations/_arize/index.md) - Arize is a platform for observability and debugging of LLMs, more information can be found [here](https://phoenix.arize.com/). - [LangSmith](https://docs.ragas.io/en/stable/howtos/integrations/langsmith/index.md) - LangSmith is a platform for observability and debugging of LLMs from LangChain, more information can be found [here](https://www.langchain.com/langsmith). # AG-UI Integration Ragas can run experiments on agents that stream events via the [AG-UI protocol](https://docs.ag-ui.com/). This notebook shows how to build experiment datasets, configure metrics, and score AG-UI endpoints using the modern `@experiment` decorator pattern. ## Prerequisites - Install dependencies: `pip install "ragas[ag-ui]" python-dotenv nest_asyncio` - Start an AG-UI compatible agent locally (Google ADK, PydanticAI, CrewAI, etc.) - Create an `.env` file with your evaluator LLM credentials (e.g. `OPENAI_API_KEY`, `GOOGLE_API_KEY`, etc.) - If you run this notebook, call `nest_asyncio.apply()` (shown below) so you can `await` coroutines in-place. ```python # !pip install "ragas[ag-ui]" python-dotenv nest_asyncio ``` ## Imports and environment setup Load environment variables and import the classes used throughout the walkthrough. ```python import json import nest_asyncio import pandas as pd from dotenv import load_dotenv from IPython.display import display from ragas.dataset import Dataset from ragas.messages import HumanMessage load_dotenv() # Patch the existing notebook loop so we can await coroutines safely nest_asyncio.apply() ``` ## Build single-turn experiment data Create dataset entries with `user_input` and `reference` using `Dataset.from_pandas()` when you only need to grade the final answer text. ```python scientist_questions = Dataset.from_pandas( pd.DataFrame( [ { "user_input": "Who originated the theory of relativity?", "reference": "Albert Einstein originated the theory of relativity.", }, { "user_input": "Who discovered penicillin and when?", "reference": "Alexander Fleming discovered penicillin in 1928.", }, ] ), name="scientist_questions", backend="inmemory", ) scientist_questions ``` ## Build multi-turn conversations For tool-usage and goal accuracy metrics, provide: - `reference_tool_calls`: Expected tool calls as JSON for `ToolCallF1` - `reference`: Expected outcome description for `AgentGoalAccuracyWithReference` ```python weather_queries = Dataset.from_pandas( pd.DataFrame( [ { "user_input": [HumanMessage(content="What's the weather in Paris?")], "reference_tool_calls": json.dumps( [{"name": "get_weather", "args": {"location": "Paris"}}] ), # Expected outcome - phrased to match what LLM extracts as end_state "reference": "The AI provided the current weather conditions for Paris.", }, { "user_input": [ HumanMessage(content="Is it raining in London right now?") ], "reference_tool_calls": json.dumps( [{"name": "get_weather", "args": {"location": "London"}}] ), "reference": "The AI provided the current weather conditions for London.", }, ] ), name="weather_queries", backend="inmemory", ) weather_queries ``` ## Configure metrics and the evaluator LLM For single-turn Q&A experiments, we use: - `FactualCorrectness`: Compares response facts against reference - `AnswerRelevancy`: Measures how relevant the response is to the question - `DiscreteMetric`: Custom metric for conciseness For multi-turn agent experiments, we use: - `ToolCallF1`: Rule-based metric comparing actual vs expected tool calls - `AgentGoalAccuracyWithReference`: LLM-based metric evaluating whether the agent achieved the user's goal ```python from openai import AsyncOpenAI from ragas.embeddings.base import embedding_factory from ragas.llms import llm_factory from ragas.metrics import DiscreteMetric from ragas.metrics.collections import ( AgentGoalAccuracyWithReference, AnswerRelevancy, FactualCorrectness, ToolCallF1, ) # Async client for evaluator prompts async_llm_client = AsyncOpenAI() evaluator_llm = llm_factory("gpt-4o-mini", client=async_llm_client) embedding_client = AsyncOpenAI() evaluator_embeddings = embedding_factory( "openai", model="text-embedding-3-small", client=embedding_client, interface="modern", ) conciseness_metric = DiscreteMetric( name="conciseness", allowed_values=["verbose", "concise"], prompt=( "Is the response concise and efficiently conveys information?\n\n" "Response: {response}\n\n" "Answer with only 'verbose' or 'concise'." ), ) # Metrics for single-turn Q&A experiments qa_metrics = [ FactualCorrectness( llm=evaluator_llm, mode="f1", atomicity="high", coverage="high", ), AnswerRelevancy( llm=evaluator_llm, embeddings=evaluator_embeddings, strictness=2, ), conciseness_metric, ] # Metrics for multi-turn agent experiments # - ToolCallF1: Rule-based metric for tool call accuracy # - AgentGoalAccuracyWithReference: LLM-based metric for goal achievement tool_metrics = [ ToolCallF1(), AgentGoalAccuracyWithReference(llm=evaluator_llm), ] ``` ## Run experiments against a live AG-UI endpoint Set the endpoint URL exposed by your agent. The `run_ag_ui_row()` function calls your endpoint and returns enriched row data. Combine this with the `@experiment` decorator for evaluation pipelines. Toggle the flags when you are ready to run the experiments. In Jupyter/IPython you can `await` the experiment directly once `nest_asyncio.apply()` has been called. ```python AG_UI_ENDPOINT = "http://localhost:8000" # Update to match your agent RUN_FACTUAL_EXPERIMENT = True RUN_TOOL_EXPERIMENT = True ``` ```python from ragas import experiment from ragas.integrations.ag_ui import run_ag_ui_row @experiment() async def factual_experiment(row): """Single-turn Q&A experiment with factual correctness scoring.""" # Call AG-UI endpoint and get enriched row enriched = await run_ag_ui_row(row, AG_UI_ENDPOINT, metadata=True) # Score with factual correctness metric fc_result = await qa_metrics[0].ascore( response=enriched["response"], reference=row["reference"], ) # Score with answer relevancy metric ar_result = await qa_metrics[1].ascore( user_input=row["user_input"], response=enriched["response"], ) # Score with conciseness metric concise_result = await conciseness_metric.ascore( response=enriched["response"], llm=evaluator_llm, ) return { **enriched, "factual_correctness": fc_result.value, "answer_relevancy": ar_result.value, "conciseness": concise_result.value, } if RUN_FACTUAL_EXPERIMENT: # Run the experiment against the dataset factual_result = await factual_experiment.arun( scientist_questions, name="scientist_qa_experiment" ) display(factual_result.to_pandas()) ``` ```python from ragas.messages import ToolCall @experiment() async def tool_experiment(row): """Multi-turn experiment with tool call and goal accuracy scoring.""" # Call AG-UI endpoint and get enriched row enriched = await run_ag_ui_row(row, AG_UI_ENDPOINT) # Parse reference_tool_calls from JSON string (e.g., from CSV) ref_tool_calls_raw = row.get("reference_tool_calls") if isinstance(ref_tool_calls_raw, str): ref_tool_calls = [ToolCall(**tc) for tc in json.loads(ref_tool_calls_raw)] else: ref_tool_calls = ref_tool_calls_raw or [] # Score with tool metrics using the modern collections API f1_result = await tool_metrics[0].ascore( user_input=enriched["messages"], reference_tool_calls=ref_tool_calls, ) goal_result = await tool_metrics[1].ascore( user_input=enriched["messages"], reference=row.get("reference", ""), ) return { **enriched, "tool_call_f1": f1_result.value, "agent_goal_accuracy": goal_result.value, } if RUN_TOOL_EXPERIMENT: # Run the experiment against the dataset tool_result = await tool_experiment.arun( weather_queries, name="weather_tool_experiment" ) display(tool_result.to_pandas()) ``` ## Advanced: Lower-Level Control The `run_ag_ui_row()` function is the recommended API, but sometimes you need more control. You can use the lower-level `call_ag_ui_endpoint()` function directly. This approach lets you: - Customize event handling - Add per-row endpoint configuration - Implement custom message processing - Add additional logging or debugging ```python from ragas.integrations.ag_ui import ( call_ag_ui_endpoint, convert_to_ragas_messages, extract_response, ) @experiment() async def custom_ag_ui_experiment(row): """ Custom experiment function with full control over endpoint calls. """ # Call the AG-UI endpoint directly (lower-level than run_ag_ui_row) events = await call_ag_ui_endpoint( endpoint_url=AG_UI_ENDPOINT, user_input=row["user_input"], timeout=60.0, ) # Convert AG-UI events to Ragas messages messages = convert_to_ragas_messages(events, metadata=True) # Extract response using helper (or custom logic) response = extract_response(messages) # Score with a custom metric score_result = await conciseness_metric.ascore( response=response, llm=evaluator_llm, ) # Return result with custom fields return { **row, "response": response or "[No response]", "message_count": len(messages), "conciseness": score_result.value, } ``` Run the custom experiment against a dataset. The `@experiment` decorator provides `.arun()` for parallel execution and automatic result collection: ```python RUN_CUSTOM_EXPERIMENT = True if RUN_CUSTOM_EXPERIMENT: # Run the custom experiment custom_result = await custom_ag_ui_experiment.arun( scientist_questions, name="custom_ag_ui_experiment" ) display(custom_result.to_pandas()) ``` ### API Comparison | API Level | Function | When to Use | | ---------- | ------------------------------------------------------- | ------------------------------------------------------------------------ | | High-level | `run_ag_ui_row()` | Standard experiments - handles endpoint call, conversion, and extraction | | Low-level | `call_ag_ui_endpoint()` + `convert_to_ragas_messages()` | Custom event handling, per-row endpoint config, advanced debugging | Both approaches work with the `@experiment` decorator - choose based on how much control you need. # Phoenix (Arize) ## 1. Introduction Building a baseline for a RAG pipeline is not usually difficult, but enhancing it to make it suitable for production and ensuring the quality of your responses is almost always hard. Choosing the right tools and parameters for RAG can itself be challenging when there is an abundance of options available. This tutorial shares a robust workflow for making the right choices while building your RAG and ensuring its quality. This article covers how to evaluate, visualize and analyze your RAG using a combination of open-source libraries. We will be using: - [Ragas](https://docs.ragas.io/en/stable/) for synthetic test data generation and evaluation - Arize AI’s [Phoenix](https://docs.arize.com/phoenix) for tracing, visualization, and cluster analysis - [LlamaIndex](https://docs.llamaindex.ai/en/stable/) for building RAG pipelines For the purpose of this article, we’ll be using data from arXiv papers about prompt-engineering to build the RAG pipeline. ℹ️ This notebook requires an OpenAI API key. ## 2. Install Dependencies and Import Libraries Run the cell below to install Git LFS, which we use to download our dataset. ```python !git lfs install ``` Install and import Python dependencies. ```python !pip install "ragas<0.1.1" pypdf arize-phoenix "openinference-instrumentation-llama-index<1.0.0" "llama-index<0.10.0" pandas ``` ```python import pandas as pd # Display the complete contents of DataFrame cells. pd.set_option("display.max_colwidth", None) ``` ## 3. Configure Your OpenAI API Key Set your OpenAI API key if it is not already set as an environment variable. ```python import os from getpass import getpass import openai if not (openai_api_key := os.getenv("OPENAI_API_KEY")): openai_api_key = getpass("πŸ”‘ Enter your OpenAI API key: ") openai.api_key = openai_api_key os.environ["OPENAI_API_KEY"] = openai_api_key ``` ## 4. Generate Your Synthetic Test Dataset Curating a golden test dataset for evaluation can be a long, tedious, and expensive process that is not pragmatic β€” especially when starting out or when data sources keep changing. This can be solved by synthetically generating high quality data points, which then can be verified by developers. This can reduce the time and effort in curating test data by 90%. Run the cell below to download a dataset of prompt engineering papers in PDF format from arXiv and read these documents using LlamaIndex. ```python !git clone https://huggingface.co/datasets/vibrantlabsai/prompt-engineering-papers ``` ```python from llama_index import SimpleDirectoryReader dir_path = "./prompt-engineering-papers" reader = SimpleDirectoryReader(dir_path, num_files_limit=2) documents = reader.load_data() ``` An ideal test dataset should contain data points of high quality and diverse nature from a similar distribution to the one observed during production. Ragas uses a unique evolution-based synthetic data generation paradigm to generate questions that are of the highest quality which also ensures diversity of questions generated. Ragas by default uses OpenAI models under the hood, but you’re free to use any model of your choice. Let’s generate 100 data points using Ragas. ```python from ragas.testset import TestsetGenerator from langchain_openai import ChatOpenAI, OpenAIEmbeddings TEST_SIZE = 25 # generator with openai models generator_llm = ChatOpenAI(model="gpt-4o-mini") critic_llm = ChatOpenAI(model="gpt-4o") embeddings = OpenAIEmbeddings() generator = TestsetGenerator.from_langchain(generator_llm, critic_llm, embeddings) # generate testset testset = generator.generate_with_llamaindex_docs(documents, test_size=TEST_SIZE) test_df = testset.to_pandas() test_df.head() ``` You are free to change the question type distribution according to your needs. Since we now have our test dataset ready, let’s move on and build a simple RAG pipeline using LlamaIndex. ## 5. Build Your RAG Application With LlamaIndex LlamaIndex is an easy-to-use and flexible framework for building RAG applications. For the sake of simplicity, we use the default LLM (gpt-3.5-turbo) and embedding models (openai-ada-2). Launch Phoenix in the background and instrument your LlamaIndex application so that your OpenInference spans and traces are sent to and collected by Phoenix. [OpenInference](https://github.com/Arize-ai/openinference/tree/main/spec) is an open standard built atop OpenTelemetry that captures and stores LLM application executions. It is designed to be a category of telemetry data that is used to understand the execution of LLMs and the surrounding application context, such as retrieval from vector stores and the usage of external tools such as search engines or APIs. ```python import phoenix as px from llama_index import set_global_handler session = px.launch_app() set_global_handler("arize_phoenix") ``` Build your query engine. ```python from llama_index.core import VectorStoreIndex, ServiceContext from llama_index.embeddings.openai import OpenAIEmbedding def build_query_engine(documents): vector_index = VectorStoreIndex.from_documents( documents, service_context=ServiceContext.from_defaults(chunk_size=512), embed_model=OpenAIEmbedding(), ) query_engine = vector_index.as_query_engine(similarity_top_k=2) return query_engine query_engine = build_query_engine(documents) ``` If you check Phoenix, you should see embedding spans from when your corpus data was indexed. Export and save those embeddings into a DataFrame for visualization later in the notebook. ```python from phoenix.trace.dsl import SpanQuery client = px.Client() corpus_df = px.Client().query_spans( SpanQuery().explode( "embedding.embeddings", text="embedding.text", vector="embedding.vector", ) ) corpus_df.head() ``` Relaunch Phoenix to clear the accumulated traces. ```python px.close_app() session = px.launch_app() ``` ## 6. Evaluate Your LLM Application Ragas provides a comprehensive list of metrics that can be used to evaluate RAG pipelines both component-wise and end-to-end. To use Ragas, we first form an evaluation dataset comprised of a question, generated answer, retrieved context, and ground-truth answer (the actual expected answer for the given question). ```python from datasets import Dataset from tqdm.auto import tqdm import pandas as pd def generate_response(query_engine, question): response = query_engine.query(question) return { "answer": response.response, "contexts": [c.node.get_content() for c in response.source_nodes], } def generate_ragas_dataset(query_engine, test_df): test_questions = test_df["question"].values responses = [generate_response(query_engine, q) for q in tqdm(test_questions)] dataset_dict = { "question": test_questions, "answer": [response["answer"] for response in responses], "contexts": [response["contexts"] for response in responses], "ground_truth": test_df["ground_truth"].values.tolist(), } ds = Dataset.from_dict(dataset_dict) return ds ragas_eval_dataset = generate_ragas_dataset(query_engine, test_df) ragas_evals_df = pd.DataFrame(ragas_eval_dataset) ragas_evals_df.head() ``` Check out Phoenix to view your LlamaIndex application traces. ```python print(session.url) ``` We save out a couple of DataFrames, one containing embedding data that we'll visualize later, and another containing our exported traces and spans that we plan to evaluate using Ragas. ```python # dataset containing embeddings for visualization query_embeddings_df = px.Client().query_spans( SpanQuery().explode( "embedding.embeddings", text="embedding.text", vector="embedding.vector" ) ) query_embeddings_df.head() ``` ```python from phoenix.session.evaluation import get_qa_with_reference # dataset containing span data for evaluation with Ragas spans_dataframe = get_qa_with_reference(client) spans_dataframe.head() ``` Ragas uses LangChain to evaluate your LLM application data. Let's instrument LangChain with OpenInference, so we can see what's going on under the hood when we evaluate our LLM application. ```python from openinference.instrumentation.langchain import LangChainInstrumentor LangChainInstrumentor().instrument() ``` Evaluate your LLM traces and view the evaluation scores in DataFrame format. ```python from ragas import evaluate from ragas.metrics import ( faithfulness, answer_correctness, context_recall, context_precision, ) evaluation_result = evaluate( dataset=ragas_eval_dataset, metrics=[faithfulness, answer_correctness, context_recall, context_precision], ) eval_scores_df = pd.DataFrame(evaluation_result.scores) ``` Submit your evaluations to Phoenix, so they are visible as annotations on your spans. ```python from phoenix.trace import SpanEvaluations # Assign span ids to your ragas evaluation scores (needed so Phoenix knows where to attach the spans). eval_data_df = pd.DataFrame(evaluation_result.dataset) assert eval_data_df.question.to_list() == list( reversed(spans_dataframe.input.to_list()) # The spans are in reverse order. ), "Phoenix spans are in an unexpected order. Re-start the notebook and try again." eval_scores_df.index = pd.Index( list(reversed(spans_dataframe.index.to_list())), name=spans_dataframe.index.name ) # Log the evaluations to Phoenix. for eval_name in eval_scores_df.columns: evals_df = eval_scores_df[[eval_name]].rename(columns={eval_name: "score"}) evals = SpanEvaluations(eval_name, evals_df) px.Client().log_evaluations(evals) ``` If you check out Phoenix, you'll see your Ragas evaluations as annotations on your application spans. ```python print(session.url) ``` ## 7. Visualize and Analyze Your Embeddings [Embeddings](https://arize.com/blog-course/embeddings-meaning-examples-and-how-to-compute/) encode the meaning of retrieved documents and user queries. Not only are they an essential part of RAG systems, but they are immensely useful for understanding and debugging LLM application performance. Phoenix takes the high-dimensional embeddings from your RAG application, reduces their dimensionality, and clusters them into semantically meaningful groups of data. You can then select the metric of your choice (e.g., Ragas-computed faithfulness or answer correctness) to visually inspect the performance of your application and surface problematic clusters. The advantage of this approach is that it provides metrics on granular yet meaningful subsets of your data that help you analyze local, not merely global, performance across a dataset. It's also helpful for gaining intuition around what kind of queries your LLM application is struggling to answer. We'll re-launch Phoenix as an embedding visualizer to inspect the performance of our application on our test dataset. ```python query_embeddings_df = query_embeddings_df.iloc[::-1] assert ragas_evals_df.question.tolist() == query_embeddings_df.text.tolist() assert test_df.question.tolist() == ragas_evals_df.question.tolist() query_df = pd.concat( [ ragas_evals_df[["question", "answer", "ground_truth"]].reset_index(drop=True), query_embeddings_df[["vector"]].reset_index(drop=True), test_df[["evolution_type"]], eval_scores_df.reset_index(drop=True), ], axis=1, ) query_df.head() ``` ```python query_schema = px.Schema( prompt_column_names=px.EmbeddingColumnNames( raw_data_column_name="question", vector_column_name="vector" ), response_column_names="answer", ) corpus_schema = px.Schema( prompt_column_names=px.EmbeddingColumnNames( raw_data_column_name="text", vector_column_name="vector" ) ) # relaunch phoenix with a primary and corpus dataset to view embeddings px.close_app() session = px.launch_app( primary=px.Dataset(query_df, query_schema, "query"), corpus=px.Dataset(corpus_df.reset_index(drop=True), corpus_schema, "corpus"), ) ``` Once you launch Phoenix, you can visualize your data with the metric of your choice with the following steps: - Select the `vector` embedding, - Select `Color By > dimension` and then the dimension of your choice to color your data by a particular field, for example, by Ragas evaluation scores such as faithfulness or answer correctness, - Select the metric of your choice from the `metric` dropdown to view aggregate metrics on a per-cluster basis. ## 8. Recap Congrats! You built and evaluated a LlamaIndex query engine using Ragas and Phoenix. Let's recap what we learned: - With Ragas, you bootstrapped a test dataset and computed metrics such as faithfulness and answer correctness to evaluate your LlamaIndex query engine. - With OpenInference, you instrumented your query engine, so you could observe the inner workings of both LlamaIndex and Ragas. - With Phoenix, you collected your spans and traces, imported your evaluations for easy inspection, and visualized your embedded queries and retrieved documents to identify pockets of poor performance. This notebook is just an introduction to the capabilities of Ragas and Phoenix. To learn more, see the [Ragas](https://docs.ragas.io/en/stable/) and [Phoenix docs](https://docs.arize.com/phoenix/). If you enjoyed this tutorial, please leave a ⭐ on GitHub: - [Ragas](https://github.com/vibrantlabsai/ragas) - [Phoenix](https://github.com/Arize-ai/phoenix) - [OpenInference](https://github.com/Arize-ai/openinference) # Athina AI ## Ragas Metrics on your Production Logs [Athina](https://athina.ai) is a production monitoring and evaluation platform. Try the [sandbox](https://demo.athina.ai/observe?filters=dateSpan%3D30) here. You can use [Athina with Ragas](http://localhost:3001/evals/preset_evals/ragas_evals) metrics to run evals on production logs, and get granular model performance metrics on your production data. For example, you can get insights like this visually: - What is my `AnswerRelevancy` score for queries related to `refunds` for customer id `nike-usa` - What is my `Faithfulness` score for `product catalog` queries using prompt `catalog_answerer/v3` with model `gpt-3.5-turbo` ### β–· Running Athina Programmatically When you use Athina to run Ragas evals programmatically, you will be able to view the results on Athina's UI like this πŸ‘‡ 1. Install Athina's Python SDK: ```text pip install athina ``` 1. Create an account at [app.athina.ai](https://app.athina.ai). After signing up, you will receive an API key. Here's a sample notebook you can follow: https://github.com/athina-ai/athina-evals/blob/main/examples/ragas.ipynb 1. Run the code ```python import os from athina.evals import ( RagasAnswerCorrectness, RagasAnswerRelevancy, RagasContextRelevancy, RagasFaithfulness, ) from athina.loaders import RagasLoader from athina.keys import AthinaApiKey, OpenAiApiKey from athina.runner.run import EvalRunner import pandas as pd # Set your API keys OpenAiApiKey.set_key(os.getenv("OPENAI_API_KEY")) AthinaApiKey.set_key(os.getenv("ATHINA_API_KEY")) # Load your dataset from a dictionary, json, or csv: https://docs.athina.ai/evals/loading_data dataset = RagasLoader().load_json("raw_data.json") # Configure the eval suite eval_model = "gpt-3.5-turbo" eval_suite = [ RagasAnswerCorrectness(), RagasFaithfulness(), RagasContextRelevancy(), RagasAnswerRelevancy(), ] # Run the evaluation suite batch_eval_result = EvalRunner.run_suite( evals=eval_suite, data=dataset, max_parallel_evals=1, # If you increase this, you may run into rate limits ) pd.DataFrame(batch_eval_result) ``` ### β–· Configure Ragas to run automatically on your production logs If you are [logging your production inferences to Athina](https://docs.athina.ai/logging/log_via_api), you can configure Ragas metrics to run automatically against your production logs. 1. Navigate to the [Athina Dashboard](https://app.athina.ai/evals/config) 1. Open the **Evals** page (lightning icon on the left) 1. Click the "New Eval" button on the top right 1. Select the **Ragas** tab 1. Select the eval you want to configure #### Learn more about Athina - **Website:** - **Docs:** - **GitHub Library:** - **Sandbox**: [https://demo.athina.ai](https://demo.athina.ai/observe?filters=dateSpan%3D30) # Haystack Integration Haystack is a LLM orchestration framework to build customizable, production-ready LLM applications. The underlying concept of Haystack is that all individual tasks, such as storing documents, retrieving relevant data, and generating responses, are handled by modular components like Document Stores, Retrievers, and Generators, which are seamlessly connected and orchestrated using Pipelines. ## Overview In this tutorial, we will build a RAG pipeline using Haystack and evaluate it with Ragas. We’ll start by setting up the various components of the RAG pipeline, and for evaluations, we will initialize the RagasEvaluator component. Once the components are set up, we'll connect the components to form the complete pipeline. Later in the tutorial, we will explore how to perform evaluations using custom-defined metrics in Ragas. ## Installing Dependencies ```python %pip install ragas-haystack ``` #### Getting the data ```python dataset = [ "OpenAI is one of the most recognized names in the large language model space, known for its GPT series of models. These models excel at generating human-like text and performing tasks like creative writing, answering questions, and summarizing content. GPT-4, their latest release, has set benchmarks in understanding context and delivering detailed responses.", "Anthropic is well-known for its Claude series of language models, designed with a strong focus on safety and ethical AI behavior. Claude is particularly praised for its ability to follow complex instructions and generate text that aligns closely with user intent.", "DeepMind, a division of Google, is recognized for its cutting-edge Gemini models, which are integrated into various Google products like Bard and Workspace tools. These models are renowned for their conversational abilities and their capacity to handle complex, multi-turn dialogues.", "Meta AI is best known for its LLaMA (Large Language Model Meta AI) series, which has been made open-source for researchers and developers. LLaMA models are praised for their ability to support innovation and experimentation due to their accessibility and strong performance.", "Meta AI with it's LLaMA models aims to democratize AI development by making high-quality models available for free, fostering collaboration across industries. Their open-source approach has been a game-changer for researchers without access to expensive resources.", "Microsoft’s Azure AI platform is famous for integrating OpenAI’s GPT models, enabling businesses to use these advanced models in a scalable and secure cloud environment. Azure AI powers applications like Copilot in Office 365, helping users draft emails, generate summaries, and more.", "Amazon’s Bedrock platform is recognized for providing access to various language models, including its own models and third-party ones like Anthropic’s Claude and AI21’s Jurassic. Bedrock is especially valued for its flexibility, allowing users to choose models based on their specific needs.", "Cohere is well-known for its language models tailored for business use, excelling in tasks like search, summarization, and customer support. Their models are recognized for being efficient, cost-effective, and easy to integrate into workflows.", "AI21 Labs is famous for its Jurassic series of language models, which are highly versatile and capable of handling tasks like content creation and code generation. The Jurassic models stand out for their natural language understanding and ability to generate detailed and coherent responses.", "In the rapidly advancing field of artificial intelligence, several companies have made significant contributions with their large language models. Notable players include OpenAI, known for its GPT Series (including GPT-4); Anthropic, which offers the Claude Series; Google DeepMind with its Gemini Models; Meta AI, recognized for its LLaMA Series; Microsoft Azure AI, which integrates OpenAI’s GPT Models; Amazon AWS (Bedrock), providing access to various models including Claude (Anthropic) and Jurassic (AI21 Labs); Cohere, which offers its own models tailored for business use; and AI21 Labs, known for its Jurassic Series. These companies are shaping the landscape of AI by providing powerful models with diverse capabilities.", ] ``` ## Initialize components for RAG pipeline #### Initializing the DocumentStore ```python from haystack import Document from haystack.document_stores.in_memory import InMemoryDocumentStore document_store = InMemoryDocumentStore() docs = [Document(content=doc) for doc in dataset] ``` #### Initalize the Document and Text Embedder ```python from haystack.components.embedders import OpenAIDocumentEmbedder, OpenAITextEmbedder document_embedder = OpenAIDocumentEmbedder(model="text-embedding-3-small") text_embedder = OpenAITextEmbedder(model="text-embedding-3-small") ``` Now we have our document store and the document embedder, using them we will fill populate out vector datastore. ```python docs_with_embeddings = document_embedder.run(docs) document_store.write_documents(docs_with_embeddings["documents"]) ``` ```text Calculating embeddings: 1it [00:01, 1.74s/it] 10 ``` #### Initialize the Retriever ```python from haystack.components.retrievers.in_memory import InMemoryEmbeddingRetriever retriever = InMemoryEmbeddingRetriever(document_store, top_k=2) ``` #### Define a Template Prompt ```python from haystack.components.builders import ChatPromptBuilder from haystack.dataclasses import ChatMessage template = [ ChatMessage.from_user( """ Given the following information, answer the question. Context: {% for document in documents %} {{ document.content }} {% endfor %} Question: {{question}} Answer: """ ) ] prompt_builder = ChatPromptBuilder(template=template) ``` #### Initialize a ChatGenerator ```python from haystack.components.generators.chat import OpenAIChatGenerator chat_generator = OpenAIChatGenerator(model="gpt-4o-mini") ``` #### Setting up the RagasEvaluator Pass all the Ragas metrics you want to use for evaluation, ensuring that all the necessary information to calculate each selected metric is provided. For example: - **AnswerRelevancy**: requires both the **query** and the **response**. - **ContextPrecision**: requires the **query**, **retrieved documents**, and the **reference**. - **Faithfulness**: requires the **query**, **retrieved documents**, and the **response**. Make sure to include all relevant data for each metric to ensure accurate evaluation. ```python from haystack_integrations.components.evaluators.ragas import RagasEvaluator from langchain_openai import ChatOpenAI from ragas.llms import LangchainLLMWrapper from ragas.metrics import AnswerRelevancy, ContextPrecision, Faithfulness llm = ChatOpenAI(model="gpt-4o-mini") evaluator_llm = LangchainLLMWrapper(llm) ragas_evaluator = RagasEvaluator( ragas_metrics=[AnswerRelevancy(), ContextPrecision(), Faithfulness()], evaluator_llm=evaluator_llm, ) ``` ## Building and Assembling the Pipeline #### Creating the Pipeline ```python from haystack import Pipeline rag_pipeline = Pipeline() ``` #### Adding the components ```python from haystack.components.builders import AnswerBuilder rag_pipeline.add_component("text_embedder", text_embedder) rag_pipeline.add_component("retriever", retriever) rag_pipeline.add_component("prompt_builder", prompt_builder) rag_pipeline.add_component("llm", chat_generator) rag_pipeline.add_component("answer_builder", AnswerBuilder()) rag_pipeline.add_component("ragas_evaluator", ragas_evaluator) ``` #### Connecting the components ```python rag_pipeline.connect("text_embedder.embedding", "retriever.query_embedding") rag_pipeline.connect("retriever", "prompt_builder") rag_pipeline.connect("prompt_builder.prompt", "llm.messages") rag_pipeline.connect("llm.replies", "answer_builder.replies") rag_pipeline.connect("retriever", "answer_builder.documents") rag_pipeline.connect("llm.replies", "answer_builder.replies") rag_pipeline.connect("retriever", "answer_builder.documents") rag_pipeline.connect("retriever", "ragas_evaluator.documents") rag_pipeline.connect("llm.replies", "ragas_evaluator.response") ``` ```text πŸš… Components - text_embedder: OpenAITextEmbedder - retriever: InMemoryEmbeddingRetriever - prompt_builder: ChatPromptBuilder - llm: OpenAIChatGenerator - answer_builder: AnswerBuilder - ragas_evaluator: RagasEvaluator πŸ›€οΈ Connections - text_embedder.embedding -> retriever.query_embedding (List[float]) - retriever.documents -> prompt_builder.documents (List[Document]) - retriever.documents -> answer_builder.documents (List[Document]) - retriever.documents -> ragas_evaluator.documents (List[Document]) - prompt_builder.prompt -> llm.messages (List[ChatMessage]) - llm.replies -> answer_builder.replies (List[ChatMessage]) - llm.replies -> ragas_evaluator.response (List[ChatMessage]) ``` ## Running the Pipeline ```python question = "What makes Meta AI’s LLaMA models stand out?" reference = "Meta AI’s LLaMA models stand out for being open-source, supporting innovation and experimentation due to their accessibility and strong performance." result = rag_pipeline.run( { "text_embedder": {"text": question}, "prompt_builder": {"question": question}, "answer_builder": {"query": question}, "ragas_evaluator": {"query": question, "reference": reference}, # Each metric expects a specific set of parameters as input. Refer to the # Ragas class' documentation for more details. } ) print(result["answer_builder"]["answers"][0].data, "\n") print(result["ragas_evaluator"]["result"]) ``` ```text Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 3/3 [00:14<00:00, 4.72s/it] Meta AI's LLaMA models stand out due to their open-source nature, which allows researchers and developers easy access to high-quality language models without the need for expensive resources. This accessibility fosters innovation and experimentation, enabling collaboration across various industries. Moreover, the strong performance of the LLaMA models further enhances their appeal, making them valuable tools for advancing AI development. {'answer_relevancy': 0.9782, 'context_precision': 1.0000, 'faithfulness': 1.0000} ``` ## Advance Usage Instead of using the default ragas metrics, you can change them to fit your needs or even create your own custom metrics. After that, you can pass these to the RagasEvaluator component. To learn more about how to customize ragas metrics, check out the [docs](https://docs.ragas.io/en/stable/howtos/customizations/). In the example below, we will define two custom Ragas metrics: 1. **SportsRelevanceMetric**: This metric evaluates whether a question and its response are related to sports. 1. **AnswerQualityMetric**: This metric measures how well the response provided by the LLM answers the user's question. ```python from ragas.metrics import AspectCritic, RubricsScore SportsRelevanceMetric = AspectCritic( name="sports_relevance_metric", definition="Were the question and response related to sports?", llm=evaluator_llm, ) rubrics = { "score1_description": "The response does not answer the user input.", "score2_description": "The response partially answers the user input.", "score3_description": "The response fully answer the user input", } evaluator = RagasEvaluator( ragas_metrics=[ SportsRelevanceMetric, RubricsScore(llm=evaluator_llm, rubrics=rubrics), ], evaluator_llm=evaluator_llm, ) output = evaluator.run( query="Which is the most popular global sport?", documents=[ "Football is undoubtedly the world's most popular sport with" " major events like the FIFA World Cup and sports personalities" " like Ronaldo and Messi, drawing a followership of more than 4" " billion people." ], response="Football is the most popular sport with around 4 billion" " followers worldwide", ) output["result"] ``` ```text Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 2/2 [00:01<00:00, 1.62it/s] {'sports_relevance_metric': 1.0000, 'domain_specific_rubrics': 3.0000} ``` # Helicone This notebook demonstrates how to integrate Helicone with Ragas for monitoring and evaluating RAG (Retrieval-Augmented Generation) systems. ## Prerequisites Before you begin, make sure you have a Helicone account and API key: 1. Log into [Helicone](https://www.helicone.ai) or create an account if you don't have one. 1. Once logged in, navigate to the [Developer section](https://helicone.ai/developer) to generate an API key. **Note**: Make sure to generate a write-only API key. For more information on Helicone authentication, refer to the [Helicone Auth documentation](https://docs.helicone.ai/getting-started/helicone-api-keys). Store your Helicone API key securely, as you'll need it for the integration. ## Setup First, let's install the required packages and set up our environment. ```python !pip install datasets ragas openai ``` ```python import os from datasets import Dataset from ragas import evaluate from ragas.integrations.helicone import helicone_config # import helicone_config from ragas.metrics import answer_relevancy, context_precision, faithfulness # Set up Helicone HELICONE_API_KEY = ( "your_helicone_api_key_here" # Replace with your actual Helicone API key ) helicone_config.api_key = HELICONE_API_KEY os.environ["OPENAI_API_KEY"] = ( "your_openai_api_key_here" # Replace with your actual OpenAI API key ) # Verify Helicone API key is set if HELICONE_API_KEY == "your_helicone_api_key_here": raise ValueError( "Please replace 'your_helicone_api_key_here' with your actual Helicone API key." ) ``` ## Prepare Data Let's prepare some sample data for our RAG system evaluation. ```python data_samples = { "question": ["When was the first Super Bowl?", "Who has won the most Super Bowls?"], "answer": [ "The first Super Bowl was held on January 15, 1967.", "The New England Patriots have won the most Super Bowls, with six championships.", ], "contexts": [ [ "The First AFL–NFL World Championship Game, later known as Super Bowl I, was played on January 15, 1967, at the Los Angeles Memorial Coliseum in Los Angeles, California." ], [ "As of 2021, the New England Patriots have won the most Super Bowls with six championships, all under the leadership of quarterback Tom Brady and head coach Bill Belichick." ], ], "ground_truth": [ "The first Super Bowl was held on January 15, 1967.", "The New England Patriots have won the most Super Bowls, with six championships as of 2021.", ], } dataset = Dataset.from_dict(data_samples) print(dataset) ``` ## Evaluate with Ragas Now, let's use Ragas to evaluate our RAG system. Helicone will automatically log the API calls made during this evaluation. ```python # Evaluate using Ragas score = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision]) # Display results print(score.to_pandas()) ``` ## Viewing Results in Helicone The API calls made during the Ragas evaluation are automatically logged in Helicone. You can view these logs in the Helicone dashboard to get insights into the performance and behavior of your RAG system. To view the results: 1. Go to the [Helicone dashboard](https://www.helicone.ai/dashboard) 1. Navigate to the 'Requests' section 1. You should see the API calls made during the Ragas evaluation You can analyze these logs to understand: - The number of API calls made during evaluation - The performance of each call (latency, tokens used, etc.) - Any errors or issues that occurred during the evaluation This integration allows you to combine the power of Ragas for RAG system evaluation with Helicone's robust monitoring and analytics capabilities. # Langchain ## Evaluating Langchain QA Chains LangChain is a framework for developing applications powered by language models. It can also be used to create RAG systems (or QA systems as they are reffered to in langchain). If you want to know more about creating RAG systems with langchain you can check the [docs](https://python.langchain.com/docs/use_cases/question_answering/). With this integration you can easily evaluate your QA chains with the metrics offered in ragas ```python #!pip install ragas langchain_openai python-dotenv ``` ```python # attach to the existing event loop when using jupyter notebooks import os import nest_asyncio import openai from dotenv import load_dotenv # Load environment variables from .env file load_dotenv() # IMPORTANT: Remember to create a .env variable containing: OPENAI_API_KEY=sk-xyz where xyz is your key # Access the API key from the environment variable api_key = os.environ.get("OPENAI_API_KEY") # Initialize the OpenAI API client openai.api_key = api_key nest_asyncio.apply() ``` First lets load the dataset. We are going to build a generic QA system over the [NYC wikipedia page](https://en.wikipedia.org/wiki/New_York_City). Load the dataset and create the `VectorstoreIndex` and the `RetrievalQA` from it. ```python from langchain.chains import RetrievalQA from langchain.indexes import VectorstoreIndexCreator from langchain_community.document_loaders import TextLoader from langchain_openai import ChatOpenAI loader = TextLoader("./nyc_wikipedia/nyc_text.txt") index = VectorstoreIndexCreator().from_loaders([loader]) llm = ChatOpenAI(temperature=0) qa_chain = RetrievalQA.from_chain_type( llm, retriever=index.vectorstore.as_retriever(), return_source_documents=True, ) ``` ```text /home/jjmachan/.pyenv/versions/ragas/lib/python3.10/site-packages/langchain/indexes/vectorstore.py:128: UserWarning: Using InMemoryVectorStore as the default vectorstore.This memory store won't persist data. You should explicitlyspecify a vectorstore when using VectorstoreIndexCreator warnings.warn( --------------------------------------------------------------------------- ValidationError Traceback (most recent call last) Cell In[2], line 7 4 from langchain_openai import ChatOpenAI 6 loader = TextLoader("./nyc_wikipedia/nyc_text.txt") ----> 7 index = VectorstoreIndexCreator().from_loaders([loader]) 10 llm = ChatOpenAI(temperature=0) 11 qa_chain = RetrievalQA.from_chain_type( 12 llm, 13 retriever=index.vectorstore.as_retriever(), 14 return_source_documents=True, 15 ) File ~/.pyenv/versions/ragas/lib/python3.10/site-packages/pydantic/main.py:212, in BaseModel.__init__(self, **data) 210 # `__tracebackhide__` tells pytest and some other tools to omit this function from tracebacks 211 __tracebackhide__ = True --> 212 validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self) 213 if self is not validated_self: 214 warnings.warn( 215 'A custom validator is returning a value other than `self`.\n' 216 "Returning anything other than `self` from a top level model validator isn't supported when validating via `__init__`.\n" 217 'See the `model_validator` docs (https://docs.pydantic.dev/latest/concepts/validators/#model-validators) for more details.', 218 category=None, 219 ) ValidationError: 1 validation error for VectorstoreIndexCreator embedding Field required [type=missing, input_value={}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing ``` ```python # testing it out question = "How did New York City get its name?" result = qa_chain({"query": question}) result["result"] ``` Now in order to evaluate the qa system we generated a few relevant questions. We've generated a few question for you but feel free to add any you want. ```python eval_questions = [ "What is the population of New York City as of 2020?", "Which borough of New York City has the highest population?", "What is the economic significance of New York City?", "How did New York City get its name?", "What is the significance of the Statue of Liberty in New York City?", ] eval_answers = [ "8,804,190", "Brooklyn", "New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.", "New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.", "The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.", ] examples = [ {"query": q, "ground_truth": [eval_answers[i]]} for i, q in enumerate(eval_questions) ] ``` ## Introducing `RagasEvaluatorChain` `RagasEvaluatorChain` creates a wrapper around the metrics ragas provides (documented [here](https://github.com/vibrantlabsai/ragas/blob/main/docs/concepts/metrics/index.md)), making it easier to run these evaluation with langchain and langsmith. The evaluator chain has the following APIs - `__call__()`: call the `RagasEvaluatorChain` directly on the result of a QA chain. - `evaluate()`: evaluate on a list of examples (with the input queries) and predictions (outputs from the QA chain). - `evaluate_run()`: method implemented that is called by langsmith evaluators to evaluate langsmith datasets. lets see each of them in action to learn more. ```python result = qa_chain({"query": eval_questions[1]}) result["result"] ``` ```python result = qa_chain(examples[4]) result["result"] ``` ```python from ragas.langchain.evalchain import RagasEvaluatorChain from ragas.metrics import ( answer_relevancy, context_precision, context_recall, faithfulness, ) # create evaluation chains faithfulness_chain = RagasEvaluatorChain(metric=faithfulness) answer_rel_chain = RagasEvaluatorChain(metric=answer_relevancy) context_rel_chain = RagasEvaluatorChain(metric=context_precision) context_recall_chain = RagasEvaluatorChain(metric=context_recall) ``` 1. `__call__()` Directly run the evaluation chain with the results from the QA chain. Do note that metrics like context_precision and faithfulness require the `source_documents` to be present. ```python # Recheck the result that we are going to validate. result ``` **Faithfulness** ```python eval_result = faithfulness_chain(result) eval_result["faithfulness_score"] ``` High faithfulness_score means that there are exact consistency between the source documents and the answer. You can check lower faithfulness scores by changing the result (answer from LLM) or source_documents to something else. ```python fake_result = result.copy() fake_result["result"] = "we are the champions" eval_result = faithfulness_chain(fake_result) eval_result["faithfulness_score"] ``` **Context Recall** ```python eval_result = context_recall_chain(result) eval_result["context_recall_score"] ``` High context_recall_score means that the ground truth is present in the source documents. You can check lower context recall scores by changing the source_documents to something else. ```python from langchain.schema import Document fake_result = result.copy() fake_result["source_documents"] = [Document(page_content="I love christmas")] eval_result = context_recall_chain(fake_result) eval_result["context_recall_score"] ``` 1. `evaluate()` Evaluate a list of inputs/queries and the outputs/predictions from the QA chain. ```python # run the queries as a batch for efficiency predictions = qa_chain.batch(examples) # evaluate print("evaluating...") r = faithfulness_chain.evaluate(examples, predictions) r ``` ```python # evaluate context recall print("evaluating...") r = context_recall_chain.evaluate(examples, predictions) r ``` ## Evaluate with langsmith [Langsmith](https://docs.smith.langchain.com/) is a platform that helps to debug, test, evaluate and monitor chains and agents built on any LLM framework. It also seamlessly integrates with LangChain. Langsmith also has a tools to build a testing dataset and run evaluations against them and with `RagasEvaluatorChain` you can use the ragas metrics for running langsmith evaluations as well. To know more about langsmith evaluations checkout the [quickstart](https://docs.smith.langchain.com/evaluation/quickstart). Lets start of creating the dataset with the NYC questions listed in `eval_questions`. Create a new langsmith dataset and upload the questions. ```python # dataset creation from langsmith import Client from langsmith.utils import LangSmithError client = Client() dataset_name = "NYC test" try: # check if dataset exists dataset = client.read_dataset(dataset_name=dataset_name) print("using existing dataset: ", dataset.name) except LangSmithError: # if not create a new one with the generated query examples dataset = client.create_dataset( dataset_name=dataset_name, description="NYC test dataset" ) for e in examples: client.create_example( inputs={"query": e["query"]}, outputs={"ground_truth": e["ground_truth"]}, dataset_id=dataset.id, ) print("Created a new dataset: ", dataset.name) ``` As you can see the questions have been uploaded. Now you can run your QA chain against this test dataset and compare the results in the langchain platform. Before you call `run_on_dataset` you need a factory function which creates a new instance of the QA chain you want to test. This is so that the internal state is not reused when running against each example. ```python # factory function that return a new qa chain def create_qa_chain(return_context=True): qa_chain = RetrievalQA.from_chain_type( llm, retriever=index.vectorstore.as_retriever(), return_source_documents=return_context, ) return qa_chain ``` Now lets run the evaluation ```python from langchain.smith import RunEvalConfig, run_on_dataset evaluation_config = RunEvalConfig( custom_evaluators=[ faithfulness_chain, answer_rel_chain, context_rel_chain, context_recall_chain, ], prediction_key="result", ) result = run_on_dataset( client, dataset_name, create_qa_chain, evaluation=evaluation_config, input_mapper=lambda x: x, ) ``` You can follow the link to open the result for the run in langsmith. Check out the scores for each example too Now if you want to dive more into the reasons for the scores and how to improve them, click on any example and open the feedback tab. This will show you each scores. You can also see the curresponding `RagasEvaluatorChain` trace too to figure out why ragas scored the way it did. # Langfuse Ragas and Langfuse is a powerful combination that can help you evaluate and monitor your Retrieval-Augmented Generation (RAG) pipelines. ## What is Langfuse? Langfuse ([GitHub](https://github.com/langfuse/langfuse)) is an open-source platform for LLM [tracing](https://langfuse.com/docs/tracing), [prompt management](https://langfuse.com/docs/prompts/get-started), and [evaluation](https://langfuse.com/docs/scores/overview). It allows you to score your traces and spans, providing insights into the performance of your RAG pipelines. Langfuse supports various integrations, including [OpenAI](https://langfuse.com/docs/integrations/openai/python/get-started), [LangChain](https://langfuse.com/docs/integrations/langchain/tracing), and [more](https://langfuse.com/docs/integrations/overview). ## Key Benefits of using Langfuse with Ragas - **Score Traces**: [Score](https://langfuse.com/docs/scores/overview) your traces and spans, providing insights into the performance of your RAG pipelines. - **Detailed Analytics**: Segment and [analyze](https://langfuse.com/docs/analytics/overview) traces to identify low-quality scores and improve your system's performance. - **Score Reporting**: Drill down into detailed reports for specific use cases and user segments. Ragas ([GitHub](https://github.com/vibrantlabsai/ragas)) is an open-source tool that can help you run [Model-Based Evaluation](https://langfuse.com/docs/scores/model-based-evals) on your traces/spans, especially for RAG pipelines. Ragas can perform reference-free evaluations of various aspects of your RAG pipeline. Because it is reference-free you don't need ground-truths when running the evaluations and can run it on production traces that you've collected with Langfuse. ## Getting Started This guide will walk you through and end-to-end example of RAG evaluations with Ragas and Langfuse. ### The Environment [Sign up](https://cloud.langfuse.com) for Langfuse to get your API keys. ```python import os # get keys for your project from https://cloud.langfuse.com os.environ["LANGFUSE_SECRET_KEY"] = "sk-..." os.environ["LANGFUSE_PUBLIC_KEY"] = "pk-..." # your openai key # os.environ["OPENAI_API_KEY"] = "sk-..." ``` ```python %pip install datasets ragas llama_index python-dotenv --upgrade ``` ### The Data For this example, we are going to use a dataset that has already been prepared by querying a RAG system and gathering its outputs. See below for instruction on how to fetch your production data from Langfuse. The dataset contains the following columns: - `question`: *list[str]* - These are the questions your RAG pipeline will be evaluated on. - `answer`: *list[str]* - The answer generated from the RAG pipeline and given to the user. - `contexts`: *list\[list[str]\]* - The contexts which were passed into the LLM to answer the question. - `ground_truth`: list\[list[str]\] - The ground truth answer to the questions. However, this can be ignored for online evaluations since we will not have access to ground-truth data in our case. ```python from datasets import load_dataset amnesty_qa = load_dataset("vibrantlabsai/amnesty_qa", "english_v2")["eval"] amnesty_qa ``` ```text Found cached dataset amnesty_qa (/home/jjmachan/.cache/huggingface/datasets/vibrantlabs___amnesty_qa/english_v2/2.0.0/d0ed9800191a31943ee52a5c22ee4305e28a33f5edcd9a323802112cff07cc24) 0%| | 0/1 [00:00 float: """Fetches the current per gram price of the specified metal. Args: metal_name : The name of the metal (e.g., 'gold', 'silver', 'platinum'). Returns: float: The current price of the metal in dollars per gram. Raises: KeyError: If the specified metal is not found in the data source. """ try: metal_name = metal_name.lower().strip() if metal_name not in metal_price: raise KeyError( f"Metal '{metal_name}' not found. Available metals: {', '.join(metal_price['metals'].keys())}" ) return metal_price[metal_name] except Exception as e: raise Exception(f"Error fetching metal price: {str(e)}") ``` ### Binding the Tool to the LLM With the get_metal_price tool defined, the next step is to bind it to the ChatOpenAI model. This enables the agent to invoke the tool during its execution based on the user's requests allowing it to interact with external data and perform actions beyond its native capabilities. ```python from langchain_openai import ChatOpenAI tools = [get_metal_price] llm = ChatOpenAI(model="gpt-4o-mini") llm_with_tools = llm.bind_tools(tools) ``` In LangGraph, state plays a crucial role in tracking and updating information as the graph executes. As different parts of the graph run, the state evolves to reflect the changes and contains information that is passed between nodes. For example, in a conversational system like this one, the state is used to track the exchanged messages. Each time a new message is generated, it is added to the state and the updated state is passed through the nodes, ensuring the conversation progresses logically. ### Defining the State To implement this in LangGraph, we define a state class that maintains a list of messages. Whenever a new message is produced it gets appended to this list, ensuring that the conversation history is continuously updated. ```python from langgraph.graph import END from langchain_core.messages import AnyMessage from langgraph.graph.message import add_messages from typing import Annotated from typing_extensions import TypedDict class GraphState(TypedDict): messages: Annotated[list[AnyMessage], add_messages] ``` ### Defining the should_continue Function The `should_continue` function determines whether the conversation should proceed with further tool interactions or end. Specifically, it checks if the last message contains any tool calls (e.g., a request for metal prices). - If the last message includes tool calls, indicating that the agent has invoked an external tool, the conversation continues and moves to the "tools" node. - If there are no tool calls, the conversation ends, represented by the END state. ```python # Define the function that determines whether to continue or not def should_continue(state: GraphState): messages = state["messages"] last_message = messages[-1] if last_message.tool_calls: return "tools" return END ``` ### Calling the Model The `call_model` function interacts with the Language Model (LLM) to generate a response based on the current state of the conversation. It takes the updated state as input, processes it and returns a model-generated response. ```python # Define the function that calls the model def call_model(state: GraphState): messages = state["messages"] response = llm_with_tools.invoke(messages) return {"messages": [response]} ``` ### Creating the Assistant Node The `assistant` node is a key component responsible for processing the current state of the conversation and using the Language Model (LLM) to generate a relevant response. It evaluates the state, determines the appropriate course of action, and invokes the LLM to produce a response that aligns with the ongoing dialogue. ```python # Node def assistant(state: GraphState): response = llm_with_tools.invoke(state["messages"]) return {"messages": [response]} ``` ### Creating the Tool Node The `tool_node` is responsible for managing interactions with external tools, such as fetching metal prices or performing other actions beyond the LLM's native capabilities. The tools themselves are defined earlier in the code, and the tool_node invokes these tools based on the current state and the needs of the conversation. ```python from langgraph.prebuilt import ToolNode # Node tools = [get_metal_price] tool_node = ToolNode(tools) ``` ### Building the Graph The graph structure is the backbone of the agentic workflow, consisting of interconnected nodes and edges. To construct this graph, we use the StateGraph builder which allows us to define and connect various nodes. Each node represents a step in the process (e.g., the assistant node, tool node) and the edges dictate the flow of execution between these steps. ```python from langgraph.graph import START, StateGraph from IPython.display import Image, display # Define a new graph for the agent builder = StateGraph(GraphState) # Define the two nodes we will cycle between builder.add_node("assistant", assistant) builder.add_node("tools", tool_node) # Set the entrypoint as `agent` builder.add_edge(START, "assistant") # Making a conditional edge # should_continue will determine which node is called next. builder.add_conditional_edges("assistant", should_continue, ["tools", END]) # Making a normal edge from `tools` to `agent`. # The `agent` node will be called after the `tool`. builder.add_edge("tools", "assistant") # Compile and display the graph for a visual overview react_graph = builder.compile() display(Image(react_graph.get_graph(xray=True).draw_mermaid_png())) ``` To test our setup, we will run the agent with a query. The agent will fetch the price of copper using the metals.dev API. ```python from langchain_core.messages import HumanMessage messages = [HumanMessage(content="What is the price of copper?")] result = react_graph.invoke({"messages": messages}) ``` ```python result["messages"] ``` ```text [HumanMessage(content='What is the price of copper?', id='4122f5d4-e298-49e8-a0e0-c98adda78c6c'), AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_DkVQBK4UMgiXrpguUS2qC4mA', 'function': {'arguments': '{"metal_name":"copper"}', 'name': 'get_metal_price'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 18, 'prompt_tokens': 116, 'total_tokens': 134, 'prompt_tokens_details': {'cached_tokens': 0, 'audio_tokens': 0}, 'completion_tokens_details': {'reasoning_tokens': 0, 'audio_tokens': 0, 'accepted_prediction_tokens': 0, 'rejected_prediction_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_0ba0d124f1', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-0f77b156-e43e-4c1e-bd3a-307333eefb68-0', tool_calls=[{'name': 'get_metal_price', 'args': {'metal_name': 'copper'}, 'id': 'call_DkVQBK4UMgiXrpguUS2qC4mA', 'type': 'tool_call'}], usage_metadata={'input_tokens': 116, 'output_tokens': 18, 'total_tokens': 134}), ToolMessage(content='0.0098', name='get_metal_price', id='422c089a-6b76-4e48-952f-8925c3700ae3', tool_call_id='call_DkVQBK4UMgiXrpguUS2qC4mA'), AIMessage(content='The price of copper is $0.0098 per gram.', response_metadata={'token_usage': {'completion_tokens': 14, 'prompt_tokens': 148, 'total_tokens': 162, 'prompt_tokens_details': {'cached_tokens': 0, 'audio_tokens': 0}, 'completion_tokens_details': {'reasoning_tokens': 0, 'audio_tokens': 0, 'accepted_prediction_tokens': 0, 'rejected_prediction_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_0ba0d124f1', 'finish_reason': 'stop', 'logprobs': None}, id='run-67cbf98b-4fa6-431e-9ce4-58697a76c36e-0', usage_metadata={'input_tokens': 148, 'output_tokens': 14, 'total_tokens': 162})] ``` ### Converting Messages to Ragas Evaluation Format In the current implementation, the GraphState stores messages exchanged between the human user, the AI (LLM's responses), and any external tools (APIs or services the AI uses) in a list. Each message is an object in LangChain's format ```python # Implementation of Graph State class GraphState(TypedDict): messages: Annotated[list[AnyMessage], add_messages] ``` Each time a message is exchanged during agent execution, it gets added to the messages list in the GraphState. However, Ragas requires a specific message format for evaluating interactions. Ragas uses its own format to evaluate agent interactions. So, if you're using LangGraph, you will need to convert the LangChain message objects into Ragas message objects. This allows you to evaluate your AI agents with Ragas’ built-in evaluation tools. **Goal:** Convert the list of LangChain messages (e.g., HumanMessage, AIMessage, and ToolMessage) into the format expected by Ragas, so the evaluation framework can understand and process them properly. To convert a list of LangChain messages into a format suitable for Ragas evaluation, Ragas provides the function convert_to_ragas_messages, which can be used to transform LangChain messages into the format expected by Ragas. Here's how you can use the function: ```python from ragas.integrations.langgraph import convert_to_ragas_messages # Assuming 'result["messages"]' contains the list of LangChain messages ragas_trace = convert_to_ragas_messages(result["messages"]) ``` ```python ragas_trace # List of Ragas messages ``` ```text [HumanMessage(content='What is the price of copper?', metadata=None, type='human'), AIMessage(content='', metadata=None, type='ai', tool_calls=[ToolCall(name='get_metal_price', args={'metal_name': 'copper'})]), ToolMessage(content='0.0098', metadata=None, type='tool'), AIMessage(content='The price of copper is $0.0098 per gram.', metadata=None, type='ai', tool_calls=None)] ``` ## Evaluating the Agent's Performance For this tutorial, let us evaluate the Agent with the following metrics: - [Tool call Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#tool-call-accuracy):ToolCallAccuracy is a metric that can be used to evaluate the performance of the LLM in identifying and calling the required tools to complete a given task. - [Agent Goal accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#agent-goal-accuracy): Agent goal accuracy is a metric that can be used to evaluate the performance of the LLM in identifying and achieving the goals of the user. This is a binary metric, with 1 indicating that the AI has achieved the goal and 0 indicating that the AI has not achieved the goal. First, let us actually run our Agent with a couple of queries, and make sure we have the ground truth labels for these queries. ### Tool Call Accuracy ```python from ragas.metrics import ToolCallAccuracy from ragas.dataset_schema import MultiTurnSample from ragas.integrations.langgraph import convert_to_ragas_messages import ragas.messages as r ragas_trace = convert_to_ragas_messages( messages=result["messages"] ) # List of Ragas messages converted using the Ragas function sample = MultiTurnSample( user_input=ragas_trace, reference_tool_calls=[ r.ToolCall(name="get_metal_price", args={"metal_name": "copper"}) ], ) tool_accuracy_scorer = ToolCallAccuracy() await tool_accuracy_scorer.multi_turn_ascore(sample) ``` ```text 1.0 ``` Tool Call Accuracy: 1, because the LLM correctly identified and used the necessary tool (get_metal_price) with the correct parameters (i.e., metal name as "copper"). ### Agent Goal Accuracy ```python messages = [HumanMessage(content="What is the price of 10 grams of silver?")] result = react_graph.invoke({"messages": messages}) ``` ```python result["messages"] # List of LangChain messages ``` ```text [HumanMessage(content='What is the price of 10 grams of silver?', id='51a469de-5b7c-4d01-ab71-f8db64c8da49'), AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_rdplOo95CRwo3mZcPu4dmNxG', 'function': {'arguments': '{"metal_name":"silver"}', 'name': 'get_metal_price'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 17, 'prompt_tokens': 120, 'total_tokens': 137, 'prompt_tokens_details': {'cached_tokens': 0, 'audio_tokens': 0}, 'completion_tokens_details': {'reasoning_tokens': 0, 'audio_tokens': 0, 'accepted_prediction_tokens': 0, 'rejected_prediction_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_0ba0d124f1', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-3bb60e27-1275-41f1-a46e-03f77984c9d8-0', tool_calls=[{'name': 'get_metal_price', 'args': {'metal_name': 'silver'}, 'id': 'call_rdplOo95CRwo3mZcPu4dmNxG', 'type': 'tool_call'}], usage_metadata={'input_tokens': 120, 'output_tokens': 17, 'total_tokens': 137}), ToolMessage(content='1.0523', name='get_metal_price', id='0b5f9260-df26-4164-b042-6df2e869adfb', tool_call_id='call_rdplOo95CRwo3mZcPu4dmNxG'), AIMessage(content='The current price of silver is approximately $1.0523 per gram. Therefore, the price of 10 grams of silver would be about $10.52.', response_metadata={'token_usage': {'completion_tokens': 34, 'prompt_tokens': 151, 'total_tokens': 185, 'prompt_tokens_details': {'cached_tokens': 0, 'audio_tokens': 0}, 'completion_tokens_details': {'reasoning_tokens': 0, 'audio_tokens': 0, 'accepted_prediction_tokens': 0, 'rejected_prediction_tokens': 0}}, 'model_name': 'gpt-4o-mini-2024-07-18', 'system_fingerprint': 'fp_0ba0d124f1', 'finish_reason': 'stop', 'logprobs': None}, id='run-93e38f71-cc9d-41d6-812a-bfad9f9231b2-0', usage_metadata={'input_tokens': 151, 'output_tokens': 34, 'total_tokens': 185})] ``` ```python from ragas.integrations.langgraph import convert_to_ragas_messages ragas_trace = convert_to_ragas_messages( result["messages"] ) # List of Ragas messages converted using the Ragas function ragas_trace ``` ```text [HumanMessage(content='What is the price of 10 grams of silver?', metadata=None, type='human'), AIMessage(content='', metadata=None, type='ai', tool_calls=[ToolCall(name='get_metal_price', args={'metal_name': 'silver'})]), ToolMessage(content='1.0523', metadata=None, type='tool'), AIMessage(content='The current price of silver is approximately $1.0523 per gram. Therefore, the price of 10 grams of silver would be about $10.52.', metadata=None, type='ai', tool_calls=None)] ``` ```python from ragas.dataset_schema import MultiTurnSample from ragas.metrics import AgentGoalAccuracyWithReference from ragas.llms import LangchainLLMWrapper sample = MultiTurnSample( user_input=ragas_trace, reference="Price of 10 grams of silver", ) scorer = AgentGoalAccuracyWithReference() evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini")) scorer.llm = evaluator_llm await scorer.multi_turn_ascore(sample) ``` ```text 1.0 ``` Agent Goal Accuracy: 1, because the LLM correctly achieved the user’s goal of retrieving the price of 10 grams of silver. ## What’s next πŸŽ‰ Congratulations! We have learned how to evaluate an agent using the Ragas evaluation framework. # Langsmith ## Dataset and Tracing Visualisation [Langsmith](https://docs.smith.langchain.com/) in a platform for building production-grade LLM applications from the langchain team. It helps you with tracing, debugging and evaluting LLM applications. The langsmith + ragas integrations offer 2 features 1. View the traces of ragas `evaluator` 1. Use ragas metrics in langchain evaluation - (soon) ## Tracing ragas metrics since ragas uses langchain under the hood all you have to do is setup langsmith and your traces will be logged. to setup langsmith make sure the following env-vars are set (you can read more in the [langsmith docs](https://docs.smith.langchain.com/#quick-start) ```bash export LANGCHAIN_TRACING_V2=true export LANGCHAIN_ENDPOINT=https://api.smith.langchain.com export LANGCHAIN_API_KEY= export LANGCHAIN_PROJECT= # if not specified, defaults to "default" ``` Once langsmith is setup, just run the evaluations as your normally would ```python from datasets import load_dataset from ragas import evaluate from ragas.metrics import answer_relevancy, context_precision, faithfulness fiqa_eval = load_dataset("vibrantlabsai/fiqa", "ragas_eval") result = evaluate( fiqa_eval["baseline"].select(range(3)), metrics=[context_precision, faithfulness, answer_relevancy], ) result ``` ```text Found cached dataset fiqa (/home/jjmachan/.cache/huggingface/datasets/vibrantlabs___fiqa/ragas_eval/1.0.0/3dc7b639f5b4b16509a3299a2ceb78bf5fe98ee6b5fee25e7d5e4d290c88efb8) 0%| | 0/1 [00:00\\n\\nHistory == === Early history === I... | Long Island holds historical significance in t... | multi_hop_specific_query_synthesizer | | 4 | What role does the Staten Island Ferry play in... | \[\<1-hop>\\n\\nto start service in 2017; this wou... | The Staten Island Ferry plays a significant ro... | multi_hop_specific_query_synthesizer | with a test dataset to test our `QueryEngine` lets now build one and evaluate it. ## Building the `QueryEngine` To start lets build a `VectorStoreIndex` over the New York Cities' [Wikipedia page](https://en.wikipedia.org/wiki/New_York_City) as an example and use ragas to evaluate it. Since we already loaded the dataset into `documents` lets use that. ```python # build query engine from llama_index.core import VectorStoreIndex vector_index = VectorStoreIndex.from_documents(documents) query_engine = vector_index.as_query_engine() ``` Let's try a sample question from the generated testset to see if it is working ```python # convert it to pandas dataset df = testset.to_pandas() df["user_input"][0] ``` ```text 'Cud yu pleese explane the role of New York City within the Northeast megalopolis, and how it contributes to the cultural and economic vibrancy of the region?' ``` ```python response_vector = query_engine.query(df["user_input"][0]) print(response_vector) ``` ```text New York City serves as a key hub within the Northeast megalopolis, playing a significant role in enhancing the cultural and economic vibrancy of the region. Its status as a global center of creativity, entrepreneurship, and cultural diversity contributes to the overall dynamism of the area. The city's renowned arts scene, including Broadway theatre and numerous cultural institutions, attracts artists and audiences from around the world, enriching the cultural landscape of the Northeast megalopolis. Economically, New York City's position as a leading financial and fintech center, home to major stock exchanges and a bustling real estate market, bolsters the region's economic strength and influence. Additionally, the city's diverse culinary scene, influenced by its immigrant history, adds to the cultural richness of the region, making New York City a vital component of the Northeast megalopolis's cultural and economic tapestry. ``` ## Evaluating the `QueryEngine` Now that we have a `QueryEngine` for the `VectorStoreIndex` we can use the llama_index integration Ragas has to evaluate it. In order to run an evaluation with Ragas and LlamaIndex you need 3 things 1. LlamaIndex `QueryEngine`: what we will be evaluating 1. Metrics: Ragas defines a set of metrics that can measure different aspects of the `QueryEngine`. The available metrics and their meaning can be found [here](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/) 1. Questions: A list of questions that ragas will test the `QueryEngine` against. first let's generate the questions. Ideally you should use that you see in production so that the distribution of question with which we evaluate matches the distribution of questions seen in production. This ensures that the scores reflect the performance seen in production but to start off we'll be using a few example questions. Now let's import the metrics we will be using to evaluate ```python # import metrics from ragas.metrics import ( Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecall, ) # init metrics with evaluator LLM from ragas.llms import LlamaIndexLLMWrapper evaluator_llm = LlamaIndexLLMWrapper(OpenAI(model="gpt-4o")) metrics = [ Faithfulness(llm=evaluator_llm), AnswerRelevancy(llm=evaluator_llm), ContextPrecision(llm=evaluator_llm), ContextRecall(llm=evaluator_llm), ] ``` the `evaluate()` function expects a dict of "question" and "ground_truth" for metrics. You can easily convert the `testset` to that format ```python # convert to Ragas Evaluation Dataset ragas_dataset = testset.to_evaluation_dataset() ragas_dataset ``` ```text EvaluationDataset(features=['user_input', 'reference_contexts', 'reference'], len=6) ``` Finally, let's run the evaluation ```python from ragas.integrations.llama_index import evaluate result = evaluate( query_engine=query_engine, metrics=metrics, dataset=ragas_dataset, ) ``` ```python # final scores print(result) ``` ```text {'faithfulness': 0.7454, 'answer_relevancy': 0.9348, 'context_precision': 0.6667, 'context_recall': 0.4667} ``` You can convert into a pandas DataFrame to run more analysis on it. ```python result.to_pandas() ``` | | user_input | retrieved_contexts | reference_contexts | response | reference | faithfulness | answer_relevancy | context_precision | context_recall | | --- | ------------------------------------------------- | ---------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------- | ------------ | ---------------- | ----------------- | -------------- | | 0 | Cud yu pleese explane the role of New York Cit... | \[and its ideals of liberty and peace. In the 2... | \[New York, often called New York City or NYC, ... | New York City plays a significant role within ... | New York City serves as the geographical and d... | 0.615385 | 0.918217 | 0.0 | 0.0 | | 1 | So like, what was New York City called before ... | \[New York City is the headquarters of the glob... | \[History == === Early history === In the pre-C... | New York City was named New Amsterdam before i... | Before it was called New York, the area was kn... | 1.000000 | 0.967821 | 1.0 | 1.0 | | 2 | what happen in new york with slavery and how i... | \[=== Province of New York and slavery ===\\n\\nI... | \[and rechristened it "New Orange" after Willia... | Slavery became a significant part of New York'... | In the early 18th century, New York became a c... | 1.000000 | 0.919264 | 1.0 | 1.0 | | 3 | What historical significance does Long Island ... | \[==== River crossings ====\\n\\nNew York City is... | \[\<1-hop>\\n\\nHistory == === Early history === I... | Long Island played a significant role in the e... | Long Island holds historical significance in t... | 0.500000 | 0.931895 | 0.0 | 0.0 | | 4 | What role does the Staten Island Ferry play in... | \[==== Buses ====\\n\\nNew York City's public bus... | \[\<1-hop>\\n\\nto start service in 2017; this wou... | The Staten Island Ferry serves as a vital mode... | The Staten Island Ferry plays a significant ro... | 0.500000 | 0.936920 | 1.0 | 0.0 | | 5 | How does Central Park's role as a cultural and... | \[==== State parks ====\\n\\nThere are seven stat... | \[\<1-hop>\\n\\nCity has over 28,000 acres (110 km... | Central Park's role as a cultural and historic... | Central Park, located in middle-upper Manhatta... | 0.857143 | 0.934841 | 1.0 | 0.8 | # Openlayer ## Evaluating RAG pipelines with Openlayer and Ragas [Openlayer](https://www.openlayer.com/) is an evaluation tool that fits into your development and production pipelines to help you ship high-quality models with confidence. This notebook should be used together with [this blog post](https://www.openlayer.com/blog/post/evaluating-rag-pipelines-with-ragas-and-openlayer). ## Pre-requisites ```bash %%bash git clone https://huggingface.co/datasets/vibrantlabsai/prompt-engineering-papers ``` ```python import os os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY_HERE" ``` ## Synthetic test data generation ```python from llama_index import SimpleDirectoryReader from ragas.testset.generator import TestsetGenerator from ragas.testset.evolutions import simple, reasoning, multi_context # load documents dir_path = "./prompt-engineering-papers" reader = SimpleDirectoryReader(dir_path, num_files_limit=2) documents = reader.load_data() # generator with openai models generator = TestsetGenerator.with_openai() # set question type distribution distribution = {simple: 0.5, reasoning: 0.25, multi_context: 0.25} # generate testset testset = generator.generate_with_llamaindex_docs( documents, test_size=10, distributions=distribution ) test_df = testset.to_pandas() test_df.head() ``` ## Building RAG ```python import nest_asyncio from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext from llama_index.embeddings import OpenAIEmbedding nest_asyncio.apply() def build_query_engine(documents): vector_index = VectorStoreIndex.from_documents( documents, service_context=ServiceContext.from_defaults(chunk_size=512), embed_model=OpenAIEmbedding(), ) query_engine = vector_index.as_query_engine(similarity_top_k=2) return query_engine ``` ```python query_engine = build_query_engine(documents) ``` ```python def generate_single_response(query_engine, question): response = query_engine.query(question) return { "answer": response.response, "contexts": [c.node.get_content() for c in response.source_nodes], } ``` ```python question = "What are some strategies proposed to enhance the in-context learning capability of language models?" generate_single_response(query_engine, question) ``` ```python from datasets import Dataset def generate_ragas_dataset(query_engine, test_df): test_questions = test_df["question"].values responses = [generate_single_response(query_engine, q) for q in test_questions] dataset_dict = { "question": test_questions, "answer": [response["answer"] for response in responses], "contexts": [response["contexts"] for response in responses], "ground_truth": test_df["ground_truth"].values.tolist(), } ds = Dataset.from_dict(dataset_dict) return ds ``` ```python ragas_dataset = generate_ragas_dataset(query_engine, test_df) ragas_df = ragas_dataset.to_pandas() ``` ## Commit to Openlayer ```python from openlayer.tasks import TaskType client = openlayer.OpenlayerClient("YOUR_OPENLAYER_API_KEY_HERE") ``` ```python project = client.create_project( name="My-Rag-Project", task_type=TaskType.LLM, description="Evaluating an LLM used for product development.", ) ``` ```python validation_dataset_config = { "contextColumnName": "contexts", "questionColumnName": "question", "inputVariableNames": ["question"], "label": "validation", "outputColumnName": "answer", "groundTruthColumnName": "ground_truth", } project.add_dataframe( dataset_df=ragas_df, dataset_config=validation_dataset_config, ) ``` ```python model_config = { "inputVariableNames": ["question"], "modelType": "shell", "metadata": {"top_k": 2, "chunk_size": 512, "embeddings": "OpenAI"}, } project.add_model(model_config=model_config) ``` ```python project.commit("Initial commit!") project.push() ``` ```python ``` # Comet Opik In this notebook, we will showcase how to use Opik with Ragas for monitoring and evaluation of RAG (Retrieval-Augmented Generation) pipelines. There are two main ways to use Opik with Ragas: 1. Using Ragas metrics to score traces 1. Using the Ragas `evaluate` function to score a dataset ## Setup [Comet](https://www.comet.com/site?utm_medium=docs&utm_source=ragas&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_medium=docs&utm_source=ragas&utm_campaign=opik) and grab you API Key. > You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/self_hosting_opik?utm_medium=docs&utm_source=ragas&utm_campaign=opik/) for more information. ```python import getpass import os os.environ["OPIK_API_KEY"] = getpass.getpass("Opik API Key: ") os.environ["OPIK_WORKSPACE"] = input( "Comet workspace (often the same as your username): " ) ``` If you are running the Opik platform locally, simply set: ```python # import os # os.environ["OPIK_URL_OVERRIDE"] = "http://localhost:5173/api" ``` ## Preparing our environment First, we will install the necessary libraries, configure the OpenAI API key and create a new Opik dataset. ```python %pip install opik --quiet import getpass import os os.environ["OPENAI_API_KEY"] = getpass.getpass("Enter your OpenAI API key: ") ``` ## Integrating Opik with Ragas ### Using Ragas metrics to score traces Ragas provides a set of metrics that can be used to evaluate the quality of a RAG pipeline, including but not limited to: `answer_relevancy`, `answer_similarity`, `answer_correctness`, `context_precision`, `context_recall`, `context_entity_recall`, `summarization_score`. You can find a full list of metrics in the [Ragas documentation](https://docs.ragas.io/en/latest/references/metrics.html#). These metrics can be computed on the fly and logged to traces or spans in Opik. For this example, we will start by creating a simple RAG pipeline and then scoring it using the `answer_relevancy` metric. #### Create the Ragas metric In order to use the Ragas metric without using the `evaluate` function, you need to initialize the metric with a `RunConfig` object and an LLM provider. For this example, we will use LangChain as the LLM provider with the Opik tracer enabled. We will first start by initializing the Ragas metric: ```python # Import the metric # Import some additional dependencies from langchain_openai.chat_models import ChatOpenAI from langchain_openai.embeddings import OpenAIEmbeddings from ragas.embeddings import LangchainEmbeddingsWrapper from ragas.llms import LangchainLLMWrapper from ragas.metrics import AnswerRelevancy # Initialize the Ragas metric llm = LangchainLLMWrapper(ChatOpenAI()) emb = LangchainEmbeddingsWrapper(OpenAIEmbeddings()) answer_relevancy_metric = AnswerRelevancy(llm=llm, embeddings=emb) ``` Once the metric is initialized, you can use it to score a sample question. Given that the metric scoring is done asynchronously, you need to use the `asyncio` library to run the scoring function. ```python # Run this cell first if you are running this in a Jupyter notebook import nest_asyncio nest_asyncio.apply() ``` ```python import asyncio from ragas.dataset_schema import SingleTurnSample from ragas.integrations.opik import OpikTracer # Define the scoring function def compute_metric(metric, row): row = SingleTurnSample(**row) opik_tracer = OpikTracer() async def get_score(opik_tracer, metric, row): score = await metric.single_turn_ascore(row, callbacks=[OpikTracer()]) return score # Run the async function using the current event loop loop = asyncio.get_event_loop() result = loop.run_until_complete(get_score(opik_tracer, metric, row)) return result # Score a simple example row = { "user_input": "What is the capital of France?", "response": "Paris", "retrieved_contexts": ["Paris is the capital of France.", "Paris is in France."], } score = compute_metric(answer_relevancy_metric, row) print("Answer Relevancy score:", score) ``` ```text Answer Relevancy score: 1.0 ``` If you now navigate to Opik, you will be able to see that a new trace has been created in the `Default Project` project. #### Score traces You can score traces by using the `update_current_trace` function to get the current trace and passing the feedback scores to that function. The advantage of this approach is that the scoring span is added to the trace allowing for a more fine-grained analysis of the RAG pipeline. It will however run the Ragas metric calculation synchronously and so might not be suitable for production use-cases. ```python from opik import track from opik.opik_context import update_current_trace @track def retrieve_contexts(question): # Define the retrieval function, in this case we will hard code the contexts return ["Paris is the capital of France.", "Paris is in France."] @track def answer_question(question, contexts): # Define the answer function, in this case we will hard code the answer return "Paris" @track(name="Compute Ragas metric score", capture_input=False) def compute_rag_score(answer_relevancy_metric, question, answer, contexts): # Define the score function row = {"user_input": question, "response": answer, "retrieved_contexts": contexts} score = compute_metric(answer_relevancy_metric, row) return score @track def rag_pipeline(question): # Define the pipeline contexts = retrieve_contexts(question) answer = answer_question(question, contexts) score = compute_rag_score(answer_relevancy_metric, question, answer, contexts) update_current_trace( feedback_scores=[{"name": "answer_relevancy", "value": round(score, 4)}] ) return answer rag_pipeline("What is the capital of France?") ``` ```text 'Paris' ``` from datasets import load_dataset from ragas import evaluate from ragas.metrics import answer_relevancy, context_precision, faithfulness fiqa_eval = load_dataset("vibrantlabsai/fiqa", "ragas_eval") # Reformat the dataset to match the schema expected by the Ragas evaluate function dataset = fiqa_eval["baseline"].select(range(3)) dataset = dataset.map( lambda x: { "user_input": x["question"], "reference": x["ground_truth"], "retrieved_contexts": x["contexts"], } ) opik_tracer_eval = OpikTracer(tags=["ragas_eval"], metadata={"evaluation_run": True}) result = evaluate( dataset, metrics=[context_precision, faithfulness, answer_relevancy], callbacks=[opik_tracer_eval], ) print(result) ```python from datasets import load_dataset from ragas import evaluate from ragas.metrics import answer_relevancy, context_precision, faithfulness fiqa_eval = load_dataset("vibrantlabsai/fiqa", "ragas_eval") # Reformat the dataset to match the schema expected by the Ragas evaluate function dataset = fiqa_eval["baseline"].select(range(3)) dataset = dataset.map( lambda x: { "user_input": x["question"], "reference": x["ground_truth"], "retrieved_contexts": x["contexts"], } ) opik_tracer_eval = OpikTracer(tags=["ragas_eval"], metadata={"evaluation_run": True}) result = evaluate( dataset, metrics=[context_precision, faithfulness, answer_relevancy], callbacks=[opik_tracer_eval], ) print(result) ``` ```text Evaluating: 0%| | 0/6 [00:00") validate_api.upload_results("", scores) ``` As you begin sending scores to Validate you'll see Graphs being generated and 'Runs' being created. A run is a collection of scores computed from a single call to `evaluate()`. You can see how average scores change over time or dig into a specific run to see how individual questions performed. ## Reaching out πŸ‘‹ If you have any questions or feedback for our UI the easiest way to get in touch is to file a GitHub issue on our repository where we maintain [tonic-validate](https://github.com/tonicai/tonic_validate), our own open source evaluation framework. # Zeno ## Visualizing Ragas Results with Zeno You can use the [Zeno](https://zenoml.com) evaluation platform to easily visualize and explore the results of your Ragas evaluation. > Check out what the result of this tutorial looks like [here](https://hub.zenoml.com/project/b35c83b8-0b22-4b9c-aedb-80964011d7a7/ragas%20FICA%20eval) First, install the `zeno-client` package: ```bash pip install zeno-client ``` Next, create an account at [hub.zenoml.com](https://hub.zenoml.com) and generate an API key on your [account page](https://hub.zenoml.com/account). We can now pick up the evaluation where we left off at the [Getting Started](https://docs.ragas.io/en/stable/howtos/getstarted/evaluation.md) guide: ```python import os import pandas as pd from datasets import load_dataset from zeno_client import ZenoClient, ZenoMetric from ragas import evaluate from ragas.metrics import ( answer_relevancy, context_precision, context_recall, faithfulness, ) ``` ```python # Set API keys os.environ["OPENAI_API_KEY"] = "your-openai-api-key" os.environ["ZENO_API_KEY"] = "your-zeno-api-key" ``` ```python fiqa_eval = load_dataset("vibrantlabsai/fiqa", "ragas_eval") result = evaluate( fiqa_eval["baseline"], metrics=[ context_precision, faithfulness, answer_relevancy, context_recall, ], ) df = result.to_pandas() df.head() ``` We can now take the `df` with our data and results and upload it to Zeno. We first create a project with a custom RAG view specification and the metric columns we want to do evaluation across: ```python client = ZenoClient(os.environ["ZENO_API_KEY"]) project = client.create_project( name="Ragas FICA eval", description="Evaluation of RAG model using Ragas on the FICA dataset", view={ "data": { "type": "vstack", "keys": { "question": {"type": "markdown"}, "texts": { "type": "list", "elements": {"type": "markdown"}, "border": True, "pad": True, }, }, }, "label": { "type": "markdown", }, "output": { "type": "vstack", "keys": { "answer": {"type": "markdown"}, "ground_truth": { "type": "list", "elements": {"type": "markdown"}, "border": True, "pad": True, }, }, }, "size": "large", }, metrics=[ ZenoMetric( name="context_precision", type="mean", columns=["context_precision"] ), ZenoMetric(name="faithfulness", type="mean", columns=["faithfulness"]), ZenoMetric(name="answer_relevancy", type="mean", columns=["answer_relevancy"]), ZenoMetric(name="context_recall", type="mean", columns=["context_recall"]), ], ) ``` Next, we upload the base dataset with the questions and ground truths: ```python data_df = pd.DataFrame( { "data": df.apply( lambda x: {"question": x["question"], "texts": list(x["contexts"])}, axis=1 ), "label": df["ground_truth"].apply(lambda x: "\n".join(x)), } ) data_df["id"] = data_df.index project.upload_dataset( data_df, id_column="id", data_column="data", label_column="label" ) ``` Lastly, we upload the RAG outputs and Ragas metrics. You can run this for any number of models when doing comparison and iteration: ```python output_df = df[ [ "context_precision", "faithfulness", "answer_relevancy", "context_recall", ] ].copy() output_df["output"] = df.apply( lambda x: {"answer": x["answer"], "ground_truth": list(x["ground_truth"])}, axis=1 ) output_df["id"] = output_df.index project.upload_system( output_df, name="Base System", id_column="id", output_column="output" ) ``` Reach out to the Zeno team on [Discord](https://discord.gg/km62pDKAkE) or at [hello@zenoml.com](mailto:hello@zenoml.com) if you have any questions! # AG-UI [AG-UI](https://docs.ag-ui.com/) is an event-based protocol for streaming agent updates to user interfaces. The protocol standardizes message, tool-call, and state events, which makes it easy to plug different agent runtimes into visual frontends. The `ragas.integrations.ag_ui` module helps you transform those event streams into Ragas message objects and run experiments against live AG-UI endpoints using the modern `@experiment` decorator pattern. This guide assumes you already have an AG-UI compatible agent running (for example, one built with Google ADK, PydanticAI, or CrewAI) and that you are familiar with creating datasets in Ragas. ## Install the integration The AG-UI helpers live behind an optional extra. Install it together with the dependencies required by your evaluator LLM. When running inside Jupyter or IPython, include `nest_asyncio` so you can reuse the notebook's event loop. ```bash pip install "ragas[ag-ui]" python-dotenv nest_asyncio ``` Configure your evaluator LLM credentials. For example, if you are using OpenAI models: ```bash # .env OPENAI_API_KEY=sk-... ``` Load the environment variables inside Python before running the examples: ```python from dotenv import load_dotenv import nest_asyncio load_dotenv() # If you're inside Jupyter/IPython, patch the running event loop once. nest_asyncio.apply() ``` ## Build an experiment dataset `Dataset` can contain single-turn or multi-turn samples. With AG-UI you can test either patternβ€”single questions with free-form responses, or longer conversations that include tool calls. ### Single-turn samples Use `Dataset.from_pandas()` with `user_input` and `reference` columns when you only need to grade the final answer text. ```python import pandas as pd from ragas.dataset import Dataset scientist_questions = Dataset.from_pandas( pd.DataFrame([ { "user_input": "Who originated the theory of relativity?", "reference": "Albert Einstein originated the theory of relativity.", }, { "user_input": "Who discovered penicillin and when?", "reference": "Alexander Fleming discovered penicillin in 1928.", }, ]), name="scientist_questions", backend="inmemory", ) ``` ### Multi-turn samples with tool expectations When you want to grade intermediate agent behaviorβ€”like whether it calls tools correctly and achieves the user's goalβ€”use conversation lists as `user_input`. Provide expected tool calls as JSON and optionally a reference outcome for goal accuracy evaluation. ```python import json import pandas as pd from ragas.dataset import Dataset from ragas.messages import HumanMessage weather_queries = Dataset.from_pandas( pd.DataFrame([ { "user_input": [HumanMessage(content="What's the weather in Paris?")], "reference_tool_calls": json.dumps([ {"name": "get_weather", "args": {"location": "Paris"}} ]), # Expected outcome for AgentGoalAccuracyWithReference "reference": "The user received the current weather conditions for Paris.", }, { "user_input": [HumanMessage(content="Is it raining in London right now?")], "reference_tool_calls": json.dumps([ {"name": "get_weather", "args": {"location": "London"}} ]), "reference": "The user received the current weather conditions for London.", }, ]), name="weather_queries", backend="inmemory", ) ``` ### Loading from CSV For larger datasets, store your test cases in CSV files and load them with the Dataset API: ```python from ragas.dataset import Dataset dataset = Dataset.load( name="scientist_biographies", backend="local/csv", root_dir="./test_data", ) ``` ## Choose metrics and evaluator model The integration works with any Ragas metric. To unlock the modern collections portfolio (and mix in custom checks), build an Instructor-compatible LLM for the evaluator prompts and use a synchronous OpenAI client for embeddings. ```python from openai import AsyncOpenAI, OpenAI from ragas.llms import llm_factory from ragas.embeddings import embedding_factory from ragas.metrics import DiscreteMetric from ragas.metrics.collections import ( AgentGoalAccuracyWithReference, AnswerRelevancy, FactualCorrectness, ToolCallF1, ) async_llm_client = AsyncOpenAI() evaluator_llm = llm_factory("gpt-4o-mini", client=async_llm_client) # AnswerRelevancy's embeddings still run synchronously, so pair it with a sync client. embedding_client = OpenAI() evaluator_embeddings = embedding_factory( "openai", model="text-embedding-3-small", client=embedding_client, interface="modern" ) conciseness_metric = DiscreteMetric( name="conciseness", allowed_values=["verbose", "concise"], prompt=( "Is the response concise and efficiently conveys information?\n\n" "Response: {response}\n\n" "Answer with only 'verbose' or 'concise'." ), ) # Metrics for single-turn Q&A evaluation qa_metrics = [ FactualCorrectness( llm=evaluator_llm, mode="f1", atomicity="high", coverage="high" ), AnswerRelevancy(llm=evaluator_llm, embeddings=evaluator_embeddings, strictness=2), conciseness_metric, ] # Metrics for multi-turn agent evaluation # - ToolCallF1: Rule-based metric for tool call accuracy # - AgentGoalAccuracyWithReference: LLM-based metric for goal achievement tool_metrics = [ ToolCallF1(), AgentGoalAccuracyWithReference(llm=evaluator_llm), ] ``` ## Run experiments with @experiment The AG-UI integration provides `run_ag_ui_row()` to call your endpoint and enrich each row with the agent's response. Combine this with the `@experiment` decorator to build evaluation pipelines. > ⚠️ The endpoint must expose the AG-UI SSE stream. Common paths include `/chat`, `/agent`, or `/agentic_chat`. ### Basic single-turn evaluation In Jupyter or IPython, use top-level `await` (after `nest_asyncio.apply()`) instead of `asyncio.run` to avoid the "event loop is already running" error. For scripts you can keep `asyncio.run`. ```python from ragas import experiment from ragas.integrations.ag_ui import run_ag_ui_row from ragas.metrics.collections import FactualCorrectness @experiment() async def factual_experiment(row): # Call AG-UI endpoint and get enriched row enriched = await run_ag_ui_row(row, "http://localhost:8000/chat") # Score with metrics score = await FactualCorrectness(llm=evaluator_llm).ascore( response=enriched["response"], reference=row["reference"], ) return {**enriched, "factual_correctness": score.value} # Run the experiment against the dataset # In Jupyter/IPython (after calling nest_asyncio.apply()) factual_result = await factual_experiment.arun( scientist_questions, name="scientist_qa_eval" ) # In a standalone script, use: # factual_result = asyncio.run(factual_experiment.arun(scientist_questions, name="scientist_qa_eval")) factual_result.to_pandas() ``` The resulting dataframe includes per-sample scores, raw agent responses, and any retrieved contexts (tool results). Results are automatically saved by the experiment framework, and you can export to CSV through pandas. ### Multi-turn tool evaluation For multi-turn datasets and tool evaluation, pass the messages and reference tool calls directly to the metrics: ```python import json from ragas import experiment from ragas.integrations.ag_ui import run_ag_ui_row from ragas.messages import ToolCall from ragas.metrics.collections import AgentGoalAccuracyWithReference, ToolCallF1 @experiment() async def tool_experiment(row): # Call AG-UI endpoint and get enriched row enriched = await run_ag_ui_row(row, "http://localhost:8000/chat") # Parse reference_tool_calls from JSON string (e.g., from CSV) ref_tool_calls_raw = row.get("reference_tool_calls") if isinstance(ref_tool_calls_raw, str): ref_tool_calls = [ToolCall(**tc) for tc in json.loads(ref_tool_calls_raw)] else: ref_tool_calls = ref_tool_calls_raw or [] # Score with tool metrics using the modern collections API f1_result = await ToolCallF1().ascore( user_input=enriched["messages"], reference_tool_calls=ref_tool_calls, ) goal_result = await AgentGoalAccuracyWithReference(llm=evaluator_llm).ascore( user_input=enriched["messages"], reference=row.get("reference", ""), ) return { **enriched, "tool_call_f1": f1_result.value, "agent_goal_accuracy": goal_result.value, } # Run the experiment # In Jupyter/IPython tool_result = await tool_experiment.arun( weather_queries, name="weather_tool_eval" ) # Or in a script # tool_result = asyncio.run(tool_experiment.arun(weather_queries, name="weather_tool_eval")) tool_result.to_pandas() ``` If a request fails, the experiment logs the error and returns placeholder values for that sample so the experiment can continue with remaining samples. ## Working directly with AG-UI events Sometimes you may want to collect event logs separatelyβ€”perhaps from a recorded run or a staging environmentβ€”and evaluate them offline. The conversion helpers expose the same parsing logic used by `run_ag_ui_row()`. ```python from ragas.integrations.ag_ui import convert_to_ragas_messages from ag_ui.core import TextMessageChunkEvent events = [ TextMessageChunkEvent( message_id="assistant-1", role="assistant", delta="Hello from AG-UI!", timestamp="2024-12-01T00:00:00Z", ) ] ragas_messages = convert_to_ragas_messages(events, metadata=True) ``` If you already have a `MessagesSnapshotEvent` you can skip streaming reconstruction and call `convert_messages_snapshot`. ```python from ragas.integrations.ag_ui import convert_messages_snapshot from ag_ui.core import MessagesSnapshotEvent, UserMessage, AssistantMessage snapshot = MessagesSnapshotEvent( messages=[ UserMessage(id="msg-1", content="Hello?"), AssistantMessage(id="msg-2", content="Hi! How can I help you today?"), ] ) ragas_messages = convert_messages_snapshot(snapshot) ``` The converted messages can be used to build custom evaluation workflows or passed directly to metric scoring functions. ## Extraction helpers The integration provides helper functions to extract specific data from messages: ```python from ragas.integrations.ag_ui import ( extract_response, # Get concatenated AI response text extract_tool_calls, # Get all tool calls from AI messages extract_contexts, # Get tool results/contexts ) messages = convert_to_ragas_messages(events) response = extract_response(messages) # "Hello! The weather is sunny." tool_calls = extract_tool_calls(messages) # [ToolCall(name="get_weather", args={"location": "SF"})] contexts = extract_contexts(messages) # ["Sunny, 72F in San Francisco"] ``` ## Tips for production experiments - **Custom headers**: pass authentication tokens or tenant IDs via `extra_headers` parameter to `run_ag_ui_row()`. - **Timeouts**: tune the `timeout` parameter if your agent performs long-running tool calls. - **Metadata debugging**: set `metadata=True` to keep AG-UI run, thread, and message IDs on every message for easier traceability. - **Experiment naming**: use descriptive `name` arguments to `.arun()` for easy identification of results. For a complete production example, see `examples/ragas_examples/ag_ui_agent_experiments/experiments.py` which provides: - CLI arguments for endpoint configuration - CSV-based test datasets - Proper logging and error handling - Timestamped result output An interactive walkthrough notebook is also available at `howtos/integrations/ag_ui.ipynb`. ## API Reference ### Primary API - **`run_ag_ui_row(row, endpoint_url, ...)`** - Run a single row against an AG-UI endpoint and return enriched data with response, messages, tool_calls, and contexts. ### Conversion Functions - **`convert_to_ragas_messages(events, metadata=False)`** - Convert AG-UI event sequences to Ragas messages - **`convert_messages_snapshot(snapshot, metadata=False)`** - Convert AG-UI message snapshots to Ragas messages - **`convert_messages_to_ag_ui(messages)`** - Convert Ragas messages to AG-UI format ### Extraction Helpers - **`extract_response(messages)`** - Extract concatenated AI response text - **`extract_tool_calls(messages)`** - Extract all tool calls from AI messages - **`extract_contexts(messages)`** - Extract tool results/contexts from messages ### Low-Level - **`call_ag_ui_endpoint(endpoint_url, user_input, ...)`** - Call an AG-UI endpoint and collect streaming events - **`AGUIEventCollector`** - Collect and reconstruct messages from streaming events # Create and Evaluate an Amazon Bedrock Agent Integrated with an Amazon Bedrock Knowledge Base and Action Groups In this notebook, you will learn how to evaluate an Amazon Bedrock Agent. The agent we'll evaluate is a restaurant agent that provides clients with information about adult and children's menus and manages the table booking system. This agent is inspired by a [features example notebooks](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group) of [Amazon Bedrock Agents](https://aws.amazon.com/bedrock/agents/) with minor changes. You can learn more about the agent creation process [here](https://github.com/aws-samples/amazon-bedrock-samples/tree/main/agents-and-function-calling/bedrock-agents/features-examples/05-create-agent-with-knowledge-base-and-action-group). The architecture is illustrated below: The steps covered in this notebook include: - Importing necessary libraries - Creating the agent - Defining the Ragas metrics - Evaluating the agent - Cleaning up the created resources Click to View the Agent creation ## Import the needed libraries First step is to install the pre-requisites packages ```python %pip install --upgrade -q boto3 opensearch-py botocore awscli retrying ragas langchain-aws ``` This command will clone the repository containing helper files needed for this tutorial. ```text ! git clone https://huggingface.co/datasets/vibrantlabsai/booking_agent_utils ``` ```python import os import time import boto3 import logging import pprint import json from booking_agent_utils.knowledge_base import BedrockKnowledgeBase from booking_agent_utils.agent import ( create_agent_role_and_policies, create_lambda_role, delete_agent_roles_and_policies, create_dynamodb, create_lambda, clean_up_resources, ) ``` ```python # Clients s3_client = boto3.client("s3") sts_client = boto3.client("sts") session = boto3.session.Session() region = session.region_name account_id = sts_client.get_caller_identity()["Account"] bedrock_agent_client = boto3.client("bedrock-agent") bedrock_agent_runtime_client = boto3.client("bedrock-agent-runtime") logging.basicConfig( format="[%(asctime)s] p%(process)s {%(filename)s:%(lineno)d} %(levelname)s - %(message)s", level=logging.INFO, ) logger = logging.getLogger(__name__) region, account_id ``` ```python suffix = f"{region}-{account_id}" agent_name = "booking-agent" knowledge_base_name = f"{agent_name}-kb" knowledge_base_description = ( "Knowledge Base containing the restaurant menu's collection" ) agent_alias_name = "booking-agent-alias" bucket_name = f"{agent_name}-{suffix}" agent_bedrock_allow_policy_name = f"{agent_name}-ba" agent_role_name = f"AmazonBedrockExecutionRoleForAgents_{agent_name}" agent_foundation_model = "amazon.nova-pro-v1:0" agent_description = "Agent in charge of a restaurants table bookings" agent_instruction = """ You are a restaurant agent responsible for managing clients’ bookings (retrieving, creating, or canceling reservations) and assisting with menu inquiries. When handling menu requests, provide detailed information about the requested items. Offer recommendations only when: 1. The customer explicitly asks for a recommendation, even if the item is available (include complementary dishes). 2. The requested item is unavailableβ€”inform the customer and suggest suitable alternatives. 3. For general menu inquiries, provide the full menu and add a recommendation only if the customer asks for one. In all cases, ensure that any recommended items are present in the menu. Ensure all responses are clear, contextually relevant, and enhance the customer's experience. """ agent_action_group_description = """ Actions for getting table booking information, create a new booking or delete an existing booking""" agent_action_group_name = "TableBookingsActionGroup" ``` ## Setting up Agent ### Create Knowledge Base for Amazon Bedrock Let's start by creating a [Knowledge Base for Amazon Bedrock](https://aws.amazon.com/bedrock/knowledge-bases/) to store the restaurant menus. For this example, we will integrate the knowledge base with Amazon OpenSearch Serverless. ```python knowledge_base = BedrockKnowledgeBase( kb_name=knowledge_base_name, kb_description=knowledge_base_description, data_bucket_name=bucket_name, ) ``` ### Upload the Dataset to Amazon S3 Now that we have created the knowledge base, let's populate it with the restaurant menus dataset. In this example, we will use [boto3 abstraction](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/start_ingestion_job.html) of the API, via our helper classes. Let’s first upload the menu data available in the dataset folder to Amazon S3. ```python def upload_directory(path, bucket_name): for root, dirs, files in os.walk(path): for file in files: file_to_upload = os.path.join(root, file) print(f"uploading file {file_to_upload} to {bucket_name}") s3_client.upload_file(file_to_upload, bucket_name, file) upload_directory("booking_agent_utils/dataset", bucket_name) ``` Now we start the ingestion job ```python # ensure that the kb is available time.sleep(30) # sync knowledge base knowledge_base.start_ingestion_job() ``` Finally we collect the Knowledge Base Id to integrate it with our Agent later on. ```python kb_id = knowledge_base.get_knowledge_base_id() ``` #### Testing Knowledge Base with Retrieve and Generate API First, let’s test the knowledge base using the Retrieve and Generate API to ensure that the knowledge base is functioning correctly. ```python response = bedrock_agent_runtime_client.retrieve_and_generate( input={"text": "Which are the mains available in the childrens menu?"}, retrieveAndGenerateConfiguration={ "type": "KNOWLEDGE_BASE", "knowledgeBaseConfiguration": { "knowledgeBaseId": kb_id, "modelArn": "arn:aws:bedrock:{}::foundation-model/{}".format( region, agent_foundation_model ), "retrievalConfiguration": { "vectorSearchConfiguration": {"numberOfResults": 5} }, }, }, ) print(response["output"]["text"], end="\n" * 2) ``` ### Create the DynamoDB Table We will create a DynamoDB table that contains restaurant booking information. ```python table_name = "restaurant_bookings" create_dynamodb(table_name) ``` ### Create the Lambda Function We will now create a Lambda function that interacts with the DynamoDB table. #### Create the Function Code Create the Lambda function that implements the functions for `get_booking_details`, `create_booking`, and `delete_booking`. ```python %%writefile lambda_function.py import json import uuid import boto3 dynamodb = boto3.resource('dynamodb') table = dynamodb.Table('restaurant_bookings') def get_named_parameter(event, name): """ Get a parameter from the lambda event """ return next(item for item in event['parameters'] if item['name'] == name)['value'] def get_booking_details(booking_id): """ Retrieve details of a restaurant booking Args: booking_id (string): The ID of the booking to retrieve """ try: response = table.get_item(Key={'booking_id': booking_id}) if 'Item' in response: return response['Item'] else: return {'message': f'No booking found with ID {booking_id}'} except Exception as e: return {'error': str(e)} def create_booking(date, name, hour, num_guests): """ Create a new restaurant booking Args: date (string): The date of the booking name (string): Name to idenfity your reservation hour (string): The hour of the booking num_guests (integer): The number of guests for the booking """ try: booking_id = str(uuid.uuid4())[:8] table.put_item( Item={ 'booking_id': booking_id, 'date': date, 'name': name, 'hour': hour, 'num_guests': num_guests } ) return {'booking_id': booking_id} except Exception as e: return {'error': str(e)} def delete_booking(booking_id): """ Delete an existing restaurant booking Args: booking_id (str): The ID of the booking to delete """ try: response = table.delete_item(Key={'booking_id': booking_id}) if response['ResponseMetadata']['HTTPStatusCode'] == 200: return {'message': f'Booking with ID {booking_id} deleted successfully'} else: return {'message': f'Failed to delete booking with ID {booking_id}'} except Exception as e: return {'error': str(e)} def lambda_handler(event, context): # get the action group used during the invocation of the lambda function actionGroup = event.get('actionGroup', '') # name of the function that should be invoked function = event.get('function', '') # parameters to invoke function with parameters = event.get('parameters', []) if function == 'get_booking_details': booking_id = get_named_parameter(event, "booking_id") if booking_id: response = str(get_booking_details(booking_id)) responseBody = {'TEXT': {'body': json.dumps(response)}} else: responseBody = {'TEXT': {'body': 'Missing booking_id parameter'}} elif function == 'create_booking': date = get_named_parameter(event, "date") name = get_named_parameter(event, "name") hour = get_named_parameter(event, "hour") num_guests = get_named_parameter(event, "num_guests") if date and hour and num_guests: response = str(create_booking(date, name, hour, num_guests)) responseBody = {'TEXT': {'body': json.dumps(response)}} else: responseBody = {'TEXT': {'body': 'Missing required parameters'}} elif function == 'delete_booking': booking_id = get_named_parameter(event, "booking_id") if booking_id: response = str(delete_booking(booking_id)) responseBody = {'TEXT': {'body': json.dumps(response)}} else: responseBody = {'TEXT': {'body': 'Missing booking_id parameter'}} else: responseBody = {'TEXT': {'body': 'Invalid function'}} action_response = { 'actionGroup': actionGroup, 'function': function, 'functionResponse': { 'responseBody': responseBody } } function_response = {'response': action_response, 'messageVersion': event['messageVersion']} print("Response: {}".format(function_response)) return function_response ``` #### Create the required permissions ```python lambda_iam_role = create_lambda_role(agent_name, table_name) ``` #### Create the function ```python lambda_function_name = f"{agent_name}-lambda" lambda_function = create_lambda(lambda_function_name, lambda_iam_role) ``` ### Create the IAM Policies Needed for the Agent Now that we have created the Knowledge Base, our DynamoDB table, and the Lambda function to execute the tasks for our Agent, let’s start creating our Agent. ```python agent_role = create_agent_role_and_policies( agent_name, agent_foundation_model, kb_id=kb_id ) ``` ### Create the Agent Now that we have created the necessary IAM role, we can use the [`create_agent`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/create_agent.html) API from boto3 to create a new agent. ```python response = bedrock_agent_client.create_agent( agentName=agent_name, agentResourceRoleArn=agent_role["Role"]["Arn"], description=agent_description, idleSessionTTLInSeconds=1800, foundationModel=agent_foundation_model, instruction=agent_instruction, ) ``` Let's get our Agent ID. It will be important to perform operations with our agent ```python agent_id = response["agent"]["agentId"] print("The agent id is:", agent_id) ``` ### Create the Agent Action Group We will now create an Agent Action Group that uses the Lambda function created earlier. To inform the agent about the capabilities of the action group, we will provide a description outlining its functionalities. To define the functions using a function schema, you need to provide the name, description, and parameters for each function. ```python agent_functions = [ { "name": "get_booking_details", "description": "Retrieve details of a restaurant booking", "parameters": { "booking_id": { "description": "The ID of the booking to retrieve", "required": True, "type": "string", } }, }, { "name": "create_booking", "description": "Create a new restaurant booking", "parameters": { "date": { "description": "The date of the booking", "required": True, "type": "string", }, "name": { "description": "Name to idenfity your reservation", "required": True, "type": "string", }, "hour": { "description": "The hour of the booking", "required": True, "type": "string", }, "num_guests": { "description": "The number of guests for the booking", "required": True, "type": "integer", }, }, }, { "name": "delete_booking", "description": "Delete an existing restaurant booking", "parameters": { "booking_id": { "description": "The ID of the booking to delete", "required": True, "type": "string", } }, }, ] ``` We now use the function schema to create the agent action group using the [`create_agent_action_group`](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-agent/client/create_agent_action_group.html) API ```python # Pause to make sure agent is created time.sleep(30) # Now, we can configure and create an action group here: agent_action_group_response = bedrock_agent_client.create_agent_action_group( agentId=agent_id, agentVersion="DRAFT", actionGroupExecutor={"lambda": lambda_function["FunctionArn"]}, actionGroupName=agent_action_group_name, functionSchema={"functions": agent_functions}, description=agent_action_group_description, ) ``` ### Allow the Agent to invoke the Action Group Lambda ```python # Create allow to invoke permission on lambda lambda_client = boto3.client("lambda") response = lambda_client.add_permission( FunctionName=lambda_function_name, StatementId="allow_bedrock", Action="lambda:InvokeFunction", Principal="bedrock.amazonaws.com", SourceArn=f"arn:aws:bedrock:{region}:{account_id}:agent/{agent_id}", ) ``` ### Associate the Knowledge Base to the agent ```python response = bedrock_agent_client.associate_agent_knowledge_base( agentId=agent_id, agentVersion="DRAFT", description="Access the knowledge base when customers ask about the plates in the menu.", knowledgeBaseId=kb_id, knowledgeBaseState="ENABLED", ) ``` ### Prepare the Agent and create an alias Let's create a DRAFT version of the agent that can be used for internal testing. ```python response = bedrock_agent_client.prepare_agent(agentId=agent_id) print(response) # Pause to make sure agent is prepared time.sleep(30) ``` ```python response = bedrock_agent_client.create_agent_alias( agentAliasName="TestAlias", agentId=agent_id, description="Test alias", ) alias_id = response["agentAlias"]["agentAliasId"] print("The Agent alias is:", alias_id) time.sleep(30) ``` The `invokeAgent` function sends a user query to the Bedrock agent and returns both the agent’s response and trace data. It processes the event stream, capturing trace information for evaluation purposes. ```python def invokeAgent(query, session_id, session_state=dict()): end_session: bool = False # invoke the agent API agentResponse = bedrock_agent_runtime_client.invoke_agent( inputText=query, agentId=agent_id, agentAliasId=alias_id, sessionId=session_id, enableTrace=True, endSession=end_session, sessionState=session_state, ) event_stream = agentResponse["completion"] try: traces = [] for event in event_stream: if "chunk" in event: data = event["chunk"]["bytes"] agent_answer = data.decode("utf8") end_event_received = True return agent_answer, traces # End event indicates that the request finished successfully elif "trace" in event: traces.append(event["trace"]) else: raise Exception("unexpected event.", event) return agent_answer, traces except Exception as e: raise Exception("unexpected event.", e) ``` ## Defining the Ragas metrics Evaluating agents is different from testing traditional software, where you can simply verify whether the output matches expected results. These agents perform complex tasks that often have multiple valid approaches. Given their inherent autonomy, evaluating agents is essential to ensure they function properly. #### Choosing What to Evaluate in Your Agent Selecting evaluation metrics depends entirely on your use case. A good rule of thumb is to select metrics directly tied to user needs or metrics that clearly drive business value. In the restaurant agent example above, we want the agent to fulfill user requests without unnecessary repetition, provide helpful recommendations when appropriate to enhance customer experience, and maintain consistency with the brand tone. We’ll define metrics to evaluate these priorities. Ragas provides several user-defined metrics for evaluations. When defining evaluation criteria, focus on binary decisions or discrete classification scores rather than ambiguous scores. Binary or clear classifications compel you to explicitly define success criteria. Avoid metrics yielding scores between 0 and 100 without clear interpretation, as distinguishing between close scores like 87 and 91 can be challenging, especially when evaluations occur independently. Ragas includes metrics suited to such evaluations, and we will explore some of them in action: - [Aspect Critic Metric](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/aspect_critic/index.md): Evaluates whether a submission follows user-defined criteria by leveraging LLM judgments to yield a binary outcome. - [Rubric Score Metric](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-criteria-scoring): Assesses responses against detailed, user-defined rubrics to consistently assign scores reflecting quality. ```python from langchain_aws import ChatBedrock from ragas.llms import LangchainLLMWrapper model_id = "us.amazon.nova-pro-v1:0" # Choose your desired model region_name = "us-east-1" # Choose your desired AWS region bedrock_llm = ChatBedrock(model_id=model_id, region_name=region_name) evaluator_llm = LangchainLLMWrapper(bedrock_llm) ``` ```python from ragas.metrics import AspectCritic, RubricsScore from ragas.dataset_schema import SingleTurnSample, MultiTurnSample, EvaluationDataset from ragas import evaluate rubrics = { "score-1_description": ( "The item requested by the customer is not present in the menu and no recommendations were made." ), "score0_description": ( "Either the item requested by the customer is present in the menu, or the conversation does not include any food or menu inquiry (e.g., booking, cancellation). This score applies regardless of whether any recommendation was provided." ), "score1_description": ( "The item requested by the customer is not present in the menu and a recommendation was provided." ), } recommendations = RubricsScore(rubrics=rubrics, llm=evaluator_llm, name="Recommendations") # Metric to evaluate if the AI fulfills all human requests completely. request_completeness = AspectCritic( name="Request Completeness", llm=evaluator_llm, definition=( "Return 1 The agent completely fulfills all the user requests with no omissions. " "otherwise, return 0." ), ) # Metric to assess if the AI's communication aligns with the desired brand voice. brand_tone = AspectCritic( name="Brand Voice Metric", llm=evaluator_llm, definition=( "Return 1 if the AI's communication is friendly, approachable, helpful, clear, and concise; " "otherwise, return 0." ), ) ``` ## Evaluating Agent with Ragas In order to perform evaluations using Ragas, the traces need to be converted into the format recognized by Ragas. To convert an Amazon Bedrock agent trace into a format suitable for Ragas evaluation, Ragas provides the function convert_to_ragas_messages, which can be used to transform Amazon Bedrock messages into the format expected by Ragas. You can read more about it [here](https://docs.ragas.io/en/stable/concepts/components/eval_dataset/index.md). ```python %%time import uuid session_id:str = str(uuid.uuid1()) query = "If you have children food then book a table for 2 people at 7pm on the 5th of May 2025." agent_answer, traces_1 = invokeAgent(query, session_id) print(agent_answer) ``` Output ```text Your booking for 2 people at 7pm on the 5th of May 2025 has been successfully created. Your booking ID is ca2fab70. ``` ```python query = "Can you check my previous booking? Can you please delete the booking?" agent_answer, traces_2 = invokeAgent(query, session_id) print(agent_answer) ``` Output ```text Your reservation was found and has been successfully canceled. ``` ```python from ragas.integrations.amazon_bedrock import convert_to_ragas_messages # Convert Amazon Bedrock traces to messages accepted by Ragas. # The convert_to_ragas_messages function transforms Bedrock-specific trace data # into a format that Ragas can process as conversation messages. ragas_messages_trace_1 = convert_to_ragas_messages(traces_1) ragas_messages_trace_2 = convert_to_ragas_messages(traces_2) # Initialize MultiTurnSample objects. # MultiTurnSample is a data type defined in Ragas that encapsulates conversation # data for multi-turn evaluation. This conversion is necessary to perform evaluations. sample_1 = MultiTurnSample(user_input=ragas_messages_trace_1) sample_2 = MultiTurnSample(user_input=ragas_messages_trace_2) result = evaluate( # Create an evaluation dataset from the multi-turn samples dataset=EvaluationDataset(samples=[sample_1, sample_2]), metrics=[request_completeness, brand_tone], ) result.to_pandas() ``` Output ```text Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 4/4 [00:00 export LANGCHAIN_PROJECT= # Defaults to "default" if not set ``` ## Getting the Dataset When creating evaluation dataset or evaluating instance, ensure the terminology matches the schema used in `SingleTurnSample` or `MultiTurnSample`. ```python from ragas import EvaluationDataset dataset = [ { "user_input": "Which CEO is widely recognized for democratizing AI education through platforms like Coursera?", "retrieved_contexts": [ "Andrew Ng, CEO of Landing AI, is known for his pioneering work in deep learning and for democratizing AI education through Coursera." ], "response": "Andrew Ng is widely recognized for democratizing AI education through platforms like Coursera.", "reference": "Andrew Ng, CEO of Landing AI, is known for democratizing AI education through Coursera.", }, { "user_input": "Who is Sam Altman?", "retrieved_contexts": [ "Sam Altman, CEO of OpenAI, has advanced AI research and advocates for safe, beneficial AI technologies." ], "response": "Sam Altman is the CEO of OpenAI and advocates for safe, beneficial AI technologies.", "reference": "Sam Altman, CEO of OpenAI, has advanced AI research and advocates for safe AI.", }, { "user_input": "Who is Demis Hassabis and how did he gain prominence?", "retrieved_contexts": [ "Demis Hassabis, CEO of DeepMind, is known for developing systems like AlphaGo that master complex games." ], "response": "Demis Hassabis is the CEO of DeepMind, known for developing systems like AlphaGo.", "reference": "Demis Hassabis, CEO of DeepMind, is known for developing AlphaGo.", }, { "user_input": "Who is the CEO of Google and Alphabet Inc., praised for leading innovation across Google's product ecosystem?", "retrieved_contexts": [ "Sundar Pichai, CEO of Google and Alphabet Inc., leads innovation across Google's product ecosystem." ], "response": "Sundar Pichai is the CEO of Google and Alphabet Inc., praised for leading innovation across Google's product ecosystem.", "reference": "Sundar Pichai, CEO of Google and Alphabet Inc., leads innovation across Google's product ecosystem.", }, { "user_input": "How did Arvind Krishna transform IBM?", "retrieved_contexts": [ "Arvind Krishna, CEO of IBM, transformed the company by focusing on cloud computing and AI solutions." ], "response": "Arvind Krishna transformed IBM by focusing on cloud computing and AI solutions.", "reference": "Arvind Krishna, CEO of IBM, transformed the company through cloud computing and AI.", }, ] evaluation_dataset = EvaluationDataset.from_list(dataset) ``` ## Tracing ragas metrics Run the Ragas evaluations on your dataset, and the traces will appear in your LangSmith dashboard under the specified project name or "default." ```python from ragas import evaluate from ragas.llms import LangchainLLMWrapper from langchain_openai import ChatOpenAI from ragas.metrics import LLMContextRecall, Faithfulness, FactualCorrectness llm = ChatOpenAI(model="gpt-4o-mini") evaluator_llm = LangchainLLMWrapper(llm) result = evaluate( dataset=evaluation_dataset, metrics=[LLMContextRecall(), Faithfulness(), FactualCorrectness()], llm=evaluator_llm, ) result ``` Output ```text Evaluating: 0%| | 0/15 [00:00 str: """Dummy function to simulate sending an email.""" return f"Successfully sent mail to {to}" llm = OpenAI(model="gpt-4o-mini") ``` ```python from llama_index.core.agent.workflow import FunctionAgent agent = FunctionAgent( tools=[send_message], llm=llm, system_prompt="You are a helpful assistant of Jane", ) ``` ### Agent Goal Accuracy The true value of an AI agent lies in its ability to understand what users want and deliver it effectively. Agent Goal Accuracy serves as a fundamental metric that evaluates whether an agent successfully accomplishes what the user intended. This measurement is crucial as it directly reflects how well the agent interprets user needs and takes appropriate actions to fulfill them. Ragas provides two key variants of this metric: - [AgentGoalAccuracyWithReference](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#with-reference) - A binary assessment (1 or 0) that compares the agent's final outcome against a predefined expected result. - [AgentGoalAccuracyWithoutReference](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#without-reference) - A binary assessment (1 or 0) that evaluates whether the agent achieved the user's goal based on inferred intent rather than predefined expectations. With Reference is ideal for scenarios where the expected outcome is well-defined, such as in controlled testing environments or when testing against ground truth data. ```python from llama_index.core.agent.workflow import ( AgentInput, AgentOutput, AgentStream, ToolCall as LlamaToolCall, ToolCallResult, ) handler = agent.run(user_msg="Send a message to jhon asking for a meeting") events = [] async for ev in handler.stream_events(): if isinstance(ev, (AgentInput, AgentOutput, LlamaToolCall, ToolCallResult)): events.append(ev) elif isinstance(ev, AgentStream): print(f"{ev.delta}", end="", flush=True) elif isinstance(ev, ToolCallResult): print( f"\nCall {ev.tool_name} with {ev.tool_kwargs}\nReturned: {ev.tool_output}" ) response = await handler ``` Output: ```text I have successfully sent a message to Jhon asking for a meeting. ``` ```python from ragas.integrations.llama_index import convert_to_ragas_messages ragas_messages = convert_to_ragas_messages(events) ``` ```python from ragas.metrics import AgentGoalAccuracyWithoutReference from ragas.llms import LlamaIndexLLMWrapper from ragas.dataset_schema import MultiTurnSample from ragas.messages import ToolCall as RagasToolCall evaluator_llm = LlamaIndexLLMWrapper(llm=llm) sample = MultiTurnSample( user_input=ragas_messages, ) agent_goal_accuracy_without_reference = AgentGoalAccuracyWithoutReference(llm=evaluator_llm) await agent_goal_accuracy_without_reference.multi_turn_ascore(sample) ``` Output: ```text 1.0 ``` ```python from ragas.metrics import AgentGoalAccuracyWithReference sample = MultiTurnSample( user_input=ragas_messages, reference="Successfully sent a message to Jhon asking for a meeting" ) agent_goal_accuracy_with_reference = AgentGoalAccuracyWithReference(llm=evaluator_llm) await agent_goal_accuracy_with_reference.multi_turn_ascore(sample) ``` Output: ```text 1.0 ``` ### Tool Call Accuracy In agentic workflows, an AI agent's effectiveness depends heavily on its ability to select and use the right tools at the right time. The Tool Call Accuracy metric evaluates how precisely an agent identifies and invokes appropriate tools in the correct sequence to complete a user's request. This measurement ensures that agents not only understand what tools are available but also how to orchestrate them effectively to achieve the intended outcome. - [ToolCallAccuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#tool-call-accuracy) compares the agent's actual tool usage against a reference sequence of expected tool calls. If the agent's tool selection or sequence differs from the reference, the metric returns a score of 0, indicating a failure to follow the optimal path to task completion. ```python from ragas.metrics import ToolCallAccuracy sample = MultiTurnSample( user_input=ragas_messages, reference_tool_calls=[ RagasToolCall( name="send_message", args={'to': 'jhon', 'content': 'Hi Jhon,\n\nI hope this message finds you well. I would like to schedule a meeting to discuss some important matters. Please let me know your availability.\n\nBest regards,\nJane'}, ), ], ) tool_accuracy_scorer = ToolCallAccuracy() await tool_accuracy_scorer.multi_turn_ascore(sample) ``` Output: ```text 1.0 ``` ## Evaluating LlamaIndex CodeAct Agents LlamaIndex offers a prebuilt CodeAct Agent that can be used to write and execute code, inspired by the original CodeAct paper. The idea is:Β instead of outputting a simple JSON object, a Code Agent generates anΒ executable code blockβ€”typically in a high-level language like Python. Writing actions in code rather than JSON-like snippets provides better: - Composability:Β Code naturally allows nesting and reuse of functions; JSON actions lack this flexibility. - Object management:Β Code elegantly handles operation outputs (image = generate_image()); JSON has no clean equivalent. - Generality:Β Code expresses any computational task; JSON imposes unnecessary constraints. - Representation in LLM training data:Β LLMs already understand code from training data, making it a more natural interface than specialized JSON. Click to View the CodeActAgent Setup ### Defining Functions ```python from llama_index.llms.openai import OpenAI # Configure the LLM llm = OpenAI(model="gpt-4o-mini") # Define a few helper functions def add(a: int, b: int) -> int: """Add two numbers together""" return a + b def subtract(a: int, b: int) -> int: """Subtract two numbers""" return a - b def multiply(a: int, b: int) -> int: """Multiply two numbers""" return a * b def divide(a: int, b: int) -> float: """Divide two numbers""" return a / b ``` ### Create a Code Executor The CodeActAgent will require a specific code_execute_fn to execute the code generated by the agent. ```python from typing import Any, Dict, Tuple import io import contextlib import ast import traceback class SimpleCodeExecutor: """ A simple code executor that runs Python code with state persistence. This executor maintains a global and local state between executions, allowing for variables to persist across multiple code runs. NOTE: not safe for production use! Use with caution. """ def __init__(self, locals: Dict[str, Any], globals: Dict[str, Any]): """ Initialize the code executor. Args: locals: Local variables to use in the execution context globals: Global variables to use in the execution context """ # State that persists between executions self.globals = globals self.locals = locals def execute(self, code: str) -> Tuple[bool, str, Any]: """ Execute Python code and capture output and return values. Args: code: Python code to execute Returns: Dict with keys `success`, `output`, and `return_value` """ # Capture stdout and stderr stdout = io.StringIO() stderr = io.StringIO() output = "" return_value = None try: # Execute with captured output with contextlib.redirect_stdout( stdout ), contextlib.redirect_stderr(stderr): # Try to detect if there's a return value (last expression) try: tree = ast.parse(code) last_node = tree.body[-1] if tree.body else None # If the last statement is an expression, capture its value if isinstance(last_node, ast.Expr): # Split code to add a return value assignment last_line = code.rstrip().split("\n")[-1] exec_code = ( code[: -len(last_line)] + "\n__result__ = " + last_line ) # Execute modified code exec(exec_code, self.globals, self.locals) return_value = self.locals.get("__result__") else: # Normal execution exec(code, self.globals, self.locals) except: # If parsing fails, just execute the code as is exec(code, self.globals, self.locals) # Get output output = stdout.getvalue() if stderr.getvalue(): output += "\n" + stderr.getvalue() except Exception as e: # Capture exception information output = f"Error: {type(e).__name__}: {str(e)}\n" output += traceback.format_exc() if return_value is not None: output += "\n\n" + str(return_value) return output ``` ```python code_executor = SimpleCodeExecutor( # give access to our functions defined above locals={ "add": add, "subtract": subtract, "multiply": multiply, "divide": divide, }, globals={ # give access to all builtins "__builtins__": __builtins__, # give access to numpy "np": __import__("numpy"), }, ) ``` ### Setup the CodeAct Agent ```python from llama_index.core.agent.workflow import CodeActAgent from llama_index.core.workflow import Context agent = CodeActAgent( code_execute_fn=code_executor.execute, llm=llm, tools=[add, subtract, multiply, divide], ) # context to hold the agent's session/state/chat history ctx = Context(agent) ``` ### Running and Evaluating the CodeAct agent ```python from llama_index.core.agent.workflow import ( AgentInput, AgentOutput, AgentStream, ToolCall, ToolCallResult, ) handler = agent.run("Calculate the sum of the first 10 fibonacci numbers", ctx=ctx) events = [] async for event in handler.stream_events(): if isinstance(event, (AgentInput, AgentOutput, ToolCall, ToolCallResult)): events.append(event) elif isinstance(event, AgentStream): print(f"{event.delta}", end="", flush=True) ``` ```text The first 10 Fibonacci numbers are 0, 1, 1, 2, 3, 5, 8, 13, 21, and 34. I will calculate their sum. def fibonacci(n): fib_sequence = [0, 1] for i in range(2, n): next_fib = fib_sequence[-1] + fib_sequence[-2] fib_sequence.append(next_fib) return fib_sequence # Calculate the first 10 Fibonacci numbers first_10_fib = fibonacci(10) # Calculate the sum of the first 10 Fibonacci numbers sum_fib = sum(first_10_fib) print(sum_fib) The sum of the first 10 Fibonacci numbers is 88. ``` ### Extract the ToolCall ```python CodeAct_agent_tool_call = events[2] agent_code = CodeAct_agent_tool_call.tool_kwargs["code"] print(agent_code) ``` Output ```text def fibonacci(n): fib_sequence = [0, 1] for i in range(2, n): next_fib = fib_sequence[-1] + fib_sequence[-2] fib_sequence.append(next_fib) return fib_sequence # Calculate the first 10 Fibonacci numbers first_10_fib = fibonacci(10) # Calculate the sum of the first 10 Fibonacci numbers sum_fib = sum(first_10_fib) print(sum_fib) ``` When assessing CodeAct agents, we can begin with foundational metrics that examine basic functionality, such as code compilability or appropriate argument selection. These straightforward evaluations provide a solid foundation before advancing to more sophisticated assessment approaches. Ragas offers powerful custom metric capabilities that enable increasingly nuanced evaluation as your requirements evolve. - [AspectCritic](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/aspect_critic/index.md) - Provides a binary evaluation (pass/fail) that determines whether an agent's response satisfies specific user-defined criteria, using LLM-based judgment to deliver clear success indicators. - [RubricScoreMetric](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/general_purpose/#rubrics-based-criteria-scoring) - Evaluates agent responses against comprehensive, predefined quality rubrics with discrete scoring levels, enabling consistent performance assessment across multiple dimensions. ```python def is_compilable(code_str: str, mode="exec") -> bool: try: compile(code_str, "", mode) return True except Exception: return False is_compilable(agent_code) ``` Output ```text True ``` ```python from ragas.metrics import AspectCritic from ragas.dataset_schema import SingleTurnSample from ragas.llms import LlamaIndexLLMWrapper llm = OpenAI(model="gpt-4o-mini") evaluator_llm = LlamaIndexLLMWrapper(llm=llm) correct_tool_args = AspectCritic( name="correct_tool_args", llm=evaluator_llm, definition="Score 1 if the tool arguements use in the tool call are correct and 0 otherwise", ) sample = SingleTurnSample( user_input="Calculate the sum of the first 10 fibonacci numbers", response=agent_code, ) await correct_tool_args.single_turn_ascore(sample) ``` Output: ```text 1 ``` ## Evaluating Query Engine Tool When evaluating with Ragas metrics, we need to ensure that our data is formatted suitably for evaluations. When working with a query engine tool within an agentic system, we can approach the evaluation as we would for any retrieval-augmented generation (RAG) system. We will extract all instances where the query engine tool was called during user interactions. Using that, we can construct a Ragas RAG evaluation dataset based on our event stream data. Once the dataset is ready, we can apply the full suite of Ragas evaluation metrics. In this section, we will set up a Functional Agent with Query Engine Tools. The agent has access to two "tools": one to query the 2021 Lyft 10-K and the other to query the 2021 Uber 10-K. Click to View the Agent Setup ### Setting the LLMs ```python from llama_index.llms.openai import OpenAI from llama_index.embeddings.openai import OpenAIEmbedding from llama_index.core import Settings Settings.llm = OpenAI(model="gpt-4o-mini") Settings.embed_model = OpenAIEmbedding(model="text-embedding-3-small") ``` ### Build Query Engine Tools ```python from llama_index.core import StorageContext, load_index_from_storage try: storage_context = StorageContext.from_defaults( persist_dir="./storage/lyft" ) lyft_index = load_index_from_storage(storage_context) storage_context = StorageContext.from_defaults( persist_dir="./storage/uber" ) uber_index = load_index_from_storage(storage_context) index_loaded = True except: index_loaded = False ``` ```python !mkdir -p 'data/10k/' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O 'data/10k/uber_2021.pdf' !wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O 'data/10k/lyft_2021.pdf' ``` ```python from llama_index.core import SimpleDirectoryReader, VectorStoreIndex if not index_loaded: # load data lyft_docs = SimpleDirectoryReader( input_files=["./data/10k/lyft_2021.pdf"] ).load_data() uber_docs = SimpleDirectoryReader( input_files=["./data/10k/uber_2021.pdf"] ).load_data() # build index lyft_index = VectorStoreIndex.from_documents(lyft_docs) uber_index = VectorStoreIndex.from_documents(uber_docs) # persist index lyft_index.storage_context.persist(persist_dir="./storage/lyft") uber_index.storage_context.persist(persist_dir="./storage/uber") ``` ```python lyft_engine = lyft_index.as_query_engine(similarity_top_k=3) uber_engine = uber_index.as_query_engine(similarity_top_k=3) ``` ```python from llama_index.core.tools import QueryEngineTool query_engine_tools = [ QueryEngineTool.from_defaults( query_engine=lyft_engine, name="lyft_10k", description=( "Provides information about Lyft financials for year 2021. " "Use a detailed plain text question as input to the tool." ), ), QueryEngineTool.from_defaults( query_engine=uber_engine, name="uber_10k", description=( "Provides information about Uber financials for year 2021. " "Use a detailed plain text question as input to the tool." ), ), ] ``` ### Agent Setup ```python from llama_index.core.agent.workflow import FunctionAgent, ReActAgent from llama_index.core.workflow import Context agent = FunctionAgent(tools=query_engine_tools, llm=OpenAI(model="gpt-4o-mini")) # context to hold the session/state ctx = Context(agent) ``` ### Running and Evaluating Agents ```python from llama_index.core.agent.workflow import ( AgentInput, AgentOutput, ToolCall, ToolCallResult, AgentStream, ) handler = agent.run("What's the revenue for Lyft in 2021 vs Uber?", ctx=ctx) events = [] async for ev in handler.stream_events(): if isinstance(ev, (AgentInput, AgentOutput, ToolCall, ToolCallResult)): events.append(ev) elif isinstance(ev, AgentStream): print(ev.delta, end="", flush=True) response = await handler ``` Output: ```text In 2021, Lyft generated a total revenue of $3.21 billion, while Uber's total revenue was significantly higher at $17.455 billion. ``` We will extract all instances of `ToolCallResult` where the query engine tool was called during user interactions using that we can construct a proper RAG evaluation dataset based on your event stream data. ```python from ragas.dataset_schema import SingleTurnSample ragas_samples = [] for event in events: if isinstance(event, ToolCallResult): if event.tool_name in ["lyft_10k", "uber_10k"]: sample = SingleTurnSample( user_input=event.tool_kwargs["input"], response=event.tool_output.content, retrieved_contexts=[node.text for node in event.tool_output.raw_output.source_nodes] ) ragas_samples.append(sample) ``` ```python from ragas.dataset_schema import EvaluationDataset dataset = EvaluationDataset(samples=ragas_samples) dataset.to_pandas() ``` Output: | | user_input | retrieved_contexts | response | | --- | ------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------- | | 0 | What was the total revenue for Uber in the yea... | \[Financial and Operational Highlights\\nYear En... | The total revenue for Uber in the year 2021 wa... | | 1 | What was the total revenue for Lyft in the yea... | \[Significant items\\n subject to estimates and ... | The total revenue for Lyft in the year 2021 wa... | The resulting dataset will not include reference answers by default, so we’ll be limited to using metrics that do not require references. However, if you wish to run reference-based evaluations, you can add a reference column to the dataset and then apply the relevant Ragas metrics. ### Evaluating using Ragas RAG Metrics Let's assess the effectiveness of query engines, particularly regarding retrieval quality and hallucination prevention. To accomplish this evaluation, We will employ two key Ragas metrics: faithfulness and context relevance. For more you can visit [here](https://docs.ragas.io/en/stable/howtos/concepts/metrics/available_metrics/index.md). This evaluation approach allows us to identify potential issues with either retrieval quality or response generation that could impact overall system performance. - [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/index.md) - Measures how accurately the generated response adheres to the facts presented in the retrieved context, ensuring claims made by the system can be directly supported by the information provided. - [Context Relevance](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/nvidia_metrics/#context-relevance) - Evaluates how effectively the retrieved information addresses the user's specific query by assessing its pertinence through dual LLM judgment mechanisms. ```python from ragas import evaluate from ragas.metrics import Faithfulness, ContextRelevance from ragas.llms import LlamaIndexLLMWrapper from llama_index.llms.openai import OpenAI llm = OpenAI(model="gpt-4o") evaluator_llm = LlamaIndexLLMWrapper(llm=llm) faithfulness = Faithfulness(llm=evaluator_llm) context_precision = ContextRelevance(llm=evaluator_llm) result = evaluate(dataset, metrics=[faithfulness, context_precision]) ``` ```text Evaluating: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 4/4 [00:03<00:00, 1.19it/s] ``` ```python result.to_pandas() ``` Output: | | user_input | retrieved_contexts | response | faithfulness | nv_context_relevance | | --- | ------------------------------------------------- | --------------------------------------------------- | ------------------------------------------------- | ------------ | -------------------- | | 0 | What was the total revenue for Uber in the yea... | \[Financial and Operational Highlights\\nYear En... | The total revenue for Uber in the year 2021 wa... | 1.0 | 1.0 | | 1 | What was the total revenue for Lyft in the yea... | \[Significant items\\n subject to estimates and ... | The total revenue for Lyft in the year 2021 wa... | 1.0 | 1.0 | # OCI Gen AI Integration This guide shows how to use Oracle Cloud Infrastructure (OCI) Generative AI models with Ragas for evaluation. ## Installation First, install the OCI dependency: ```bash pip install ragas[oci] ``` ## Setup ### 1. Configure OCI Authentication Set up your OCI configuration using one of these methods: #### Option A: OCI CLI Configuration ```bash oci setup config ``` #### Option B: Environment Variables ```bash export OCI_CONFIG_FILE=~/.oci/config export OCI_PROFILE=DEFAULT ``` #### Option C: Manual Configuration ```python config = { "user": "ocid1.user.oc1..example", "key_file": "~/.oci/private_key.pem", "fingerprint": "your_fingerprint", "tenancy": "ocid1.tenancy.oc1..example", "region": "us-ashburn-1" } ``` ### 2. Get Required IDs You'll need: - **Model ID**: The OCI model ID (e.g., `cohere.command`, `meta.llama-3-8b`) - **Compartment ID**: Your OCI compartment OCID - **Endpoint ID** (optional): If using a custom endpoint ## Usage ### Basic Usage ```python from ragas.llms import oci_genai_factory from ragas import evaluate from datasets import Dataset # Initialize OCI Gen AI LLM llm = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example" ) # Your dataset dataset = Dataset.from_dict({ "question": ["What is the capital of France?"], "answer": ["Paris"], "contexts": [["France is a country in Europe. Its capital is Paris."]], "ground_truth": ["Paris"] }) # Evaluate with OCI Gen AI result = evaluate( dataset, llm=llm, embeddings=None # You can use any embedding model ) ``` ### Advanced Configuration ```python from ragas.llms import oci_genai_factory from ragas.run_config import RunConfig # Custom OCI configuration config = { "user": "ocid1.user.oc1..example", "key_file": "~/.oci/private_key.pem", "fingerprint": "your_fingerprint", "tenancy": "ocid1.tenancy.oc1..example", "region": "us-ashburn-1" } # Custom run configuration run_config = RunConfig( timeout=60, max_retries=3 ) # Initialize with custom config and endpoint llm = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example", config=config, endpoint_id="ocid1.endpoint.oc1..example", # Optional run_config=run_config ) ``` ### Using with Different Models ```python # Cohere Command model llm_cohere = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example" ) # Meta Llama model llm_llama = oci_genai_factory( model_id="meta.llama-3-8b", compartment_id="ocid1.compartment.oc1..example" ) # Using with different endpoints llm_endpoint = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example", endpoint_id="ocid1.endpoint.oc1..example" ) ``` ## Available Models OCI Gen AI supports various models including: - **Cohere**: `cohere.command`, `cohere.command-light` - **Meta**: `meta.llama-3-8b`, `meta.llama-3-70b` - **Mistral**: `mistral.mistral-7b-instruct` - **And more**: Check OCI documentation for the latest available models ## Error Handling The OCI Gen AI wrapper includes comprehensive error handling: ```python try: result = evaluate(dataset, llm=llm) except Exception as e: print(f"Evaluation failed: {e}") ``` ## Performance Considerations 1. **Rate Limits**: OCI Gen AI has rate limits. Use appropriate retry configurations. 1. **Timeout**: Set appropriate timeouts for your use case. 1. **Batch Processing**: The wrapper supports batch processing for multiple completions. ## Troubleshooting ### Common Issues 1. **Authentication Errors** ```text Error: OCI SDK authentication failed ``` Solution: Verify your OCI configuration and credentials. 1. **Model Not Found** ```text Error: Model not found in compartment ``` Solution: Check if the model ID exists in your compartment. 1. **Permission Errors** ```text Error: Insufficient permissions ``` Solution: Ensure your user has the necessary IAM policies for Generative AI. ### Debug Mode Enable debug logging to troubleshoot issues: ```python import logging logging.basicConfig(level=logging.DEBUG) # Your OCI Gen AI code here ``` ## Examples ### Complete Evaluation Example ```python from ragas import evaluate from ragas.llms import oci_genai_factory from ragas.metrics import faithfulness, answer_relevancy, context_precision from datasets import Dataset # Initialize OCI Gen AI llm = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example" ) # Create dataset dataset = Dataset.from_dict({ "question": [ "What is the capital of France?", "Who wrote Romeo and Juliet?" ], "answer": [ "Paris is the capital of France.", "William Shakespeare wrote Romeo and Juliet." ], "contexts": [ ["France is a country in Europe. Its capital is Paris."], ["Romeo and Juliet is a play by William Shakespeare."] ], "ground_truth": [ "Paris", "William Shakespeare" ] }) # Evaluate result = evaluate( dataset, metrics=[faithfulness, answer_relevancy, context_precision], llm=llm ) print(result) ``` ### Custom Metrics with OCI Gen AI ```python from ragas.metrics import MetricWithLLM # Create custom metric using OCI Gen AI class CustomMetric(MetricWithLLM): def __init__(self): super().__init__() self.llm = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example" ) # Use in evaluation result = evaluate( dataset, metrics=[CustomMetric()], llm=llm ) ``` ## Best Practices 1. **Use Appropriate Models**: Choose models based on your evaluation needs. 1. **Monitor Costs**: OCI Gen AI usage is billed. Monitor your usage. 1. **Handle Errors**: Implement proper error handling for production use. 1. **Use Caching**: Enable caching for repeated evaluations. 1. **Batch Operations**: Use batch operations when possible for efficiency. ## Support For issues specific to OCI Gen AI integration: - Check OCI documentation: https://docs.oracle.com/en-us/iaas/Content/generative-ai/ - OCI Python SDK: https://docs.oracle.com/en-us/iaas/tools/python/2.160.1/api/generative_ai.html - Ragas GitHub issues: https://github.com/vibrantlabsai/ragas/issues # R2R Integration R2R is an all-in-one solution for AI Retrieval-Augmented Generation (RAG) with production-ready features, including multimodal content ingestion, hybrid search functionality user/document management and many more. ## Overview In this tutorial, we will: - Leverage the `/rag` endpoint from R2R to perform Retrieval-Augmented Generation (RAG) on a small dataset. - Evaluate the generated responses. - Analyze the traces of evaluation. ## R2R Setup #### Installing the Dependencies To begin, install the necessary packages: ```python %pip install r2r -q ``` #### Setting up the local environment Configure the `R2R_API_KEY`, `OPENAI_API_KEY` and `RAGAS_APP_TOKEN`(Optional). ```python from dotenv import load_dotenv load_dotenv() ``` #### Getting the data ```python dataset = [ "OpenAI is one of the most recognized names in the large language model space, known for its GPT series of models. These models excel at generating human-like text and performing tasks like creative writing, answering questions, and summarizing content. GPT-4, their latest release, has set benchmarks in understanding context and delivering detailed responses.", "Anthropic is well-known for its Claude series of language models, designed with a strong focus on safety and ethical AI behavior. Claude is particularly praised for its ability to follow complex instructions and generate text that aligns closely with user intent.", "DeepMind, a division of Google, is recognized for its cutting-edge Gemini models, which are integrated into various Google products like Bard and Workspace tools. These models are renowned for their conversational abilities and their capacity to handle complex, multi-turn dialogues.", "Meta AI is best known for its LLaMA (Large Language Model Meta AI) series, which has been made open-source for researchers and developers. LLaMA models are praised for their ability to support innovation and experimentation due to their accessibility and strong performance.", "Meta AI with it's LLaMA models aims to democratize AI development by making high-quality models available for free, fostering collaboration across industries. Their open-source approach has been a game-changer for researchers without access to expensive resources.", "Microsoft’s Azure AI platform is famous for integrating OpenAI’s GPT models, enabling businesses to use these advanced models in a scalable and secure cloud environment. Azure AI powers applications like Copilot in Office 365, helping users draft emails, generate summaries, and more.", "Amazon’s Bedrock platform is recognized for providing access to various language models, including its own models and third-party ones like Anthropic’s Claude and AI21’s Jurassic. Bedrock is especially valued for its flexibility, allowing users to choose models based on their specific needs.", "Cohere is well-known for its language models tailored for business use, excelling in tasks like search, summarization, and customer support. Their models are recognized for being efficient, cost-effective, and easy to integrate into workflows.", "AI21 Labs is famous for its Jurassic series of language models, which are highly versatile and capable of handling tasks like content creation and code generation. The Jurassic models stand out for their natural language understanding and ability to generate detailed and coherent responses.", "In the rapidly advancing field of artificial intelligence, several companies have made significant contributions with their large language models. Notable players include OpenAI, known for its GPT Series (including GPT-4); Anthropic, which offers the Claude Series; Google DeepMind with its Gemini Models; Meta AI, recognized for its LLaMA Series; Microsoft Azure AI, which integrates OpenAI’s GPT Models; Amazon AWS (Bedrock), providing access to various models including Claude (Anthropic) and Jurassic (AI21 Labs); Cohere, which offers its own models tailored for business use; and AI21 Labs, known for its Jurassic Series. These companies are shaping the landscape of AI by providing powerful models with diverse capabilities.", ] ``` #### Setting up the R2R Client ```python from r2r import R2RClient client = R2RClient() ``` #### Ingesting the Data ```python ingest_response = client.documents.create( chunks=dataset, ) ``` #### Using the `/rag` Endpoint The [`/rag`](https://r2r-docs.sciphi.ai/api-and-sdks/retrieval/rag-app) endpoint facilitate Retrieval-Augmented Generation by integrating search results with language model outputs. The generation process can be customized using the `rag_generation_config` parameter, while the retrieval process can be configured using the `search_settings`. ```python query = "What makes Meta AI’s LLaMA models stand out?" search_settings = { "limit": 2, "graph_settings": {"enabled": False, "limit": 2}, } response = client.retrieval.rag( query=query, search_settings=search_settings ) print(response.results.generated_answer) ``` Output ```text Meta AI’s LLaMA models stand out due to their open-source nature, which supports innovation and experimentation by making high-quality models accessible to researchers and developers [1]. This approach democratizes AI development, fostering collaboration across industries and enabling researchers without access to expensive resources to work with advanced AI models [2]. ``` ## Evaluations #### **Evaluating the `R2R Client` with Ragas** With the `R2R Client` in place, we can use Ragas `r2r` integration for evaluation. This process involves the following key components: - **1. R2R Client and Configurations** The `R2RClient` and `/rag` configurations specifying RAG settings. - **2. Evaluation Dataset** You need a Ragas `EvaluationDataset` that includes all necessary inputs required by Ragas metrics. - **3. Ragas Metrics** Ragas provides various evaluation metrics to assess different aspects of the RAG, such as faithfulness, answer relevance, and context recall. You can explore the full list of available metrics in the [Ragas documentation](https://docs.ragas.io/en/latest/concepts/metrics/available_metrics/). #### Constructing a Ragas EvaluationDataset The [`EvaluationDataset`](https://docs.ragas.io/en/stable/concepts/components/eval_dataset/index.md) is a data type in Ragas designed to represent evaluation samples. You can find more details about its structure and usage in the [core concepts section](https://docs.ragas.io/en/stable/concepts/components/eval_dataset/index.md). We will use the `transform_to_ragas_dataset` function from ragas to get the EvaluationDataset for our data. ```python questions = [ "Who are the major players in the large language model space?", "What is Microsoft’s Azure AI platform known for?", "What kind of models does Cohere provide?", ] references = [ "The major players include OpenAI (GPT Series), Anthropic (Claude Series), Google DeepMind (Gemini Models), Meta AI (LLaMA Series), Microsoft Azure AI (integrating GPT Models), Amazon AWS (Bedrock with Claude and Jurassic), Cohere (business-focused models), and AI21 Labs (Jurassic Series).", "Microsoft’s Azure AI platform is known for integrating OpenAI’s GPT models, enabling businesses to use these models in a scalable and secure cloud environment.", "Cohere provides language models tailored for business use, excelling in tasks like search, summarization, and customer support.", ] r2r_responses = [] search_settings = { "limit": 2, "graph_settings": {"enabled": False, "limit": 2}, } for que in questions: response = client.retrieval.rag(query=que, search_settings=search_settings) r2r_responses.append(response) ``` ```python from ragas.integrations.r2r import transform_to_ragas_dataset ragas_eval_dataset = transform_to_ragas_dataset( user_inputs=questions, r2r_responses=r2r_responses, references=references ) ``` Output ```text EvaluationDataset(features=['user_input', 'retrieved_contexts', 'response', 'reference'], len=3) ``` #### Selecting the Metrics To evaluate our RAG endpoint, we will use the following metrics: - [Response Relevancy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/#response-relevancy): Measures how relevant a response is to the user’s input (query). - [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/): Measures how many of the relevant documents (or pieces of information) were successfully retrieved. - [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/): Measures how factually consistent a response is with the retrieved context. ```python from ragas.metrics import AnswerRelevancy, ContextPrecision, Faithfulness from ragas import evaluate from langchain_openai import ChatOpenAI from ragas.llms import LangchainLLMWrapper llm = ChatOpenAI(model="gpt-4o-mini") evaluator_llm = LangchainLLMWrapper(llm) ragas_metrics = [AnswerRelevancy(llm=evaluator_llm), ContextPrecision(llm=evaluator_llm), Faithfulness(llm=evaluator_llm)] results = evaluate(dataset=ragas_eval_dataset, metrics=ragas_metrics) ``` Output ```text Querying Client: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 3/3 [00:00 None: client = Swarm() print("Starting Swarm CLI 🐝") messages = [] agent = starting_agent while True: user_input = input("User Input: ") if user_input.lower() == "/exit": print("Exiting the loop. Goodbye!") break # Exit the loop messages.append({"role": "user", "content": user_input}) response = client.run( agent=agent, messages=messages, context_variables=context_variables or {}, stream=stream, debug=debug, ) if stream: response = process_and_print_streaming_response(response) else: pretty_print_messages(response.messages) messages.extend(response.messages) agent = response.agent return messages # To access the messages, add this line in your repo or you can redefine this function here. ``` ```python shipment_update_interaction = run_demo_loop(triage_agent) # Messages I used for interacting: # 1. Hi I would like to would like to know where my order is with order number #3000? # 2. That will be all. Thank you! # 3. /exit ``` Output ```text Starting Swarm CLI 🐝 Triage Agent: transfer_to_tracker_agent() Tracker Agent: track_order("order_id"= "3000") Tracker Agent: Woohoo! Your order #3000 is in transit and zooming its way to you! πŸŽ‰ It's expected to make its grand arrival on January 15, 2025. How exciting is that? If you need anything else, feel free to ask! Tracker Agent: case_resolved() Tracker Agent: You're welcome! 🎈 Your case is all wrapped up, and I'm thrilled to have helped. Have a fantastic day! πŸ₯³ Exiting the loop. Goodbye! ``` ### Converting Swarm Messages to Ragas Messages for evaluation The messages exchanged between Swarm agents are stored in the form of dictionaries. However, Ragas requires a different message structure to properly evaluate agent interactions. Therefore, we need to convert Swarm's dictionary-based message objects into the format that Ragas expects. Goal: Convert the list of dictionary-based Swarm messages (e.g., user, assistant, and tool messages) into the format recognized by Ragas, so that Ragas can process and evaluate them using its built-in tools. This conversion ensures that Swarm's message format aligns with the expected structure of Ragas' evaluation framework, enabling seamless integration and evaluation of the agent's interactions. To convert a list of Swarm messages into a format suitable for Ragas evaluation, Ragas provides the function convert_to_ragas_messages, which can be used to transform LangChain messages into the format expected by Ragas. Here's how you can use the function: ```python from ragas.integrations.swarm import convert_to_ragas_messages # Assuming 'result["messages"]' contains the list of LangChain messages shipment_update_ragas_trace = convert_to_ragas_messages(messages=shipment_update_interaction) shipment_update_ragas_trace ``` Output ```text [HumanMessage(content='Hi I would like to would like to know where my order is with order number #3000?', metadata=None, type='human'), AIMessage(content='', metadata=None, type='ai', tool_calls=[ToolCall(name='transfer_to_tracker_agent', args={})]), ToolMessage(content='{"assistant": "Tracker Agent"}', metadata=None, type='tool'), AIMessage(content='', metadata=None, type='ai', tool_calls=[ToolCall(name='track_order', args={'order_id': '3000'})]), ToolMessage(content='{"order_id": "3000", "status": "In Transit", "estimated_delivery": "Jan 15, 2025"}', metadata=None, type='tool'), AIMessage(content="Woohoo! Your order #3000 is in transit and zooming its way to you! πŸŽ‰ It's expected to make its grand arrival on January 15, 2025. How exciting is that? If you need anything else, feel free to ask!", metadata=None, type='ai', tool_calls=[]), HumanMessage(content='That will be all. Thank you!', metadata=None, type='human'), AIMessage(content='', metadata=None, type='ai', tool_calls=[ToolCall(name='case_resolved', args={})]), ToolMessage(content='Case resolved. No further questions.', metadata=None, type='tool'), AIMessage(content="You're welcome! 🎈 Your case is all wrapped up, and I'm thrilled to have helped. Have a fantastic day! πŸ₯³", metadata=None, type='ai', tool_calls=[])] ``` ## Evaluating the Agent's Performance In this tutorial, we will evaluate the Agent using the following metrics: 1. **[Tool Call Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#tool-call-accuracy)**: This metric measures how accurately the Agent identifies and uses the correct tools to complete a task. 1. **[Agent Goal Accuracy](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/agents/#agent-goal-accuracy)**: This binary metric evaluates whether the Agent successfully identifies and achieves the user’s goals. A score of 1 means the goal was achieved, while 0 means it was not. To begin, we will run the Agent with a few sample queries and ensure we have the ground truth labels for these queries. This will allow us to accurately evaluate the Agent’s performance. ### Tool Call Accuracy ```python import os from dotenv import load_dotenv load_dotenv() ``` ```python from pprint import pprint from langchain_openai import ChatOpenAI from ragas.messages import ToolCall from ragas.metrics import ToolCallAccuracy from ragas.dataset_schema import MultiTurnSample # from ragas.integrations.swarm import convert_to_ragas_messages sample = MultiTurnSample( user_input=shipment_update_ragas_trace, reference_tool_calls=[ ToolCall(name="transfer_to_tracker_agent", args={}), ToolCall(name="track_order", args={"order_id": "3000"}), ToolCall(name="case_resolved", args={}), ], ) tool_accuracy_scorer = ToolCallAccuracy() await tool_accuracy_scorer.multi_turn_ascore(sample) ``` Output ```text 1.0 ``` ```python valid_return_interaction = run_demo_loop(triage_agent) # Messages I used for interacting: # 1. I want to return my previous order. # 2. Order ID #4000 # 3. The product I received has expired. # 4. Thankyou very much # 5. /exit ``` Output ```text Starting Swarm CLI 🐝 Triage Agent: transfer_to_return_agent() Return and Refund Agent: I can help you with that. Could you please provide me with the order ID for the order you wish to return? Return and Refund Agent: Thank you for providing the order ID #4000. Could you please let me know the reason you want to return the product? Return and Refund Agent: valid_to_return() Return and Refund Agent: initiate_return() Return and Refund Agent: The return process for your order has been successfully initiated. Is there anything else you need help with? Return and Refund Agent: case_resolved() Return and Refund Agent: You're welcome! If you have any more questions or need assistance in the future, feel free to reach out. Have a great day! Exiting the loop. Goodbye! ``` ```python valid_return_interaction = convert_to_ragas_messages(valid_return_interaction) sample = MultiTurnSample( user_input=valid_return_interaction, reference_tool_calls=[ ToolCall(name="transfer_to_return_agent", args={}), ToolCall(name="valid_to_return", args={}), ToolCall(name="initiate_return", args={}), ToolCall(name="case_resolved", args={}), ], ) tool_accuracy_scorer = ToolCallAccuracy() await tool_accuracy_scorer.multi_turn_ascore(sample) ``` Output ```text 1.0 ``` ### Agent Goal Accuracy ```python invalid_return_interaction = run_demo_loop(triage_agent) # Messages I used for interacting: # 1. I want to return my previous order. # 2. Order ID #4000 # 3. I don't want this product anymore. # 4. /exit ``` Output ```text Starting Swarm CLI 🐝 Triage Agent: transfer_to_return_agent() Return and Refund Agent: Could you please provide the order ID for the product you would like to return? Return and Refund Agent: Thank you for providing your order ID. Could you please let me know the reason you want to return the product? Return and Refund Agent: I understand your situation; however, based on our return policy, the product is only eligible for return if: - You received the wrong shipment. - You received a damaged product. - You received an expired product. Unfortunately, a change of mind does not qualify for a return under our current policy. Is there anything else I can assist you with? Exiting the loop. Goodbye! ``` ```python from ragas.dataset_schema import MultiTurnSample from ragas.metrics import AgentGoalAccuracyWithReference from ragas.llms import LangchainLLMWrapper invalid_return_ragas_trace = convert_to_ragas_messages(invalid_return_interaction) sample = MultiTurnSample( user_input=invalid_return_ragas_trace, reference="The agent should fulfill the user's request.", ) scorer = AgentGoalAccuracyWithReference() evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini")) scorer.llm = evaluator_llm await scorer.multi_turn_ascore(sample) ``` Output ```text 0.0 ``` **Agent Goal Accuracy: 0.0** The **AgentGoalAccuracyWithReference** metric compares the agent's final response to the expected goal. In this case, while the agent’s response follows company policy, it does not fulfill the user’s return request. Since the return request couldn’t be completed due to policy constraints, the reference goal ("successfully resolved the user's request") is not met. As a result, the score is 0.0. ## What’s next πŸŽ‰ Congratulations! We have learned how to evaluate a swarm agent using the Ragas evaluation framework. # API Reference # API References This section contains detailed API documentation for all core components of Ragas. The documentation is organized into the following sections: ## Core Components - [Prompt](https://docs.ragas.io/en/stable/references/prompt/index.md) - Core prompt management and templating - [LLMs](https://docs.ragas.io/en/stable/references/llms/index.md) - Language model interfaces and configurations - [Embeddings](https://docs.ragas.io/en/stable/references/embeddings/index.md) - Embedding model interfaces and utilities - [Tokenizers](https://docs.ragas.io/en/stable/references/tokenizers/index.md) - Tokenizer interfaces for text splitting - [RunConfig](https://docs.ragas.io/en/stable/references/run_config/index.md) - Evaluation runtime configuration - [Executor](https://docs.ragas.io/en/stable/references/executor/index.md) - Execution engine for evaluations - [Cache](https://docs.ragas.io/en/stable/references/cache/index.md) - Caching mechanisms for LLM calls ## Evaluation - [Schemas](https://docs.ragas.io/en/stable/references/evaluation_schema/index.md) - Data structures for evaluation - [Metrics](https://docs.ragas.io/en/stable/references/metrics/index.md) - Available metrics and their implementations - [evaluate()](https://docs.ragas.io/en/stable/references/evaluate/index.md) - Main evaluation function API ## Testset Generation - [Schemas](https://docs.ragas.io/en/stable/references/testset_schema/index.md) - Data structures for test data - [Graph](https://docs.ragas.io/en/stable/references/graph/index.md) - Knowledge graph creation and management - [Transforms](https://docs.ragas.io/en/stable/references/transforms/index.md) - Data transformation utilities - [Synthesizers](https://docs.ragas.io/en/stable/references/synthesizers/index.md) - Test data generation components - [Generation](https://docs.ragas.io/en/stable/references/generate/index.md) - Test data generation API ## Integrations - [Integrations](https://docs.ragas.io/en/stable/references/integrations/index.md) - APIs for external tool integrations # Async Evaluation ## aevaluate() Async version of evaluate that performs evaluation without applying nest_asyncio. This function is the async-first implementation that doesn't patch the event loop, making it safe to use in production async applications. Parameters are identical to evaluate() function. Returns: | Type | Description | | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EvaluationResult or Executor` | If return_executor is False, returns EvaluationResult object containing the scores of each metric. If return_executor is True, returns the Executor instance for cancellable execution. | Examples: ```python import asyncio from ragas import aevaluate async def main(): result = await aevaluate(dataset, metrics) print(result) asyncio.run(main()) ``` Source code in `src/ragas/evaluation.py` ````python async def aevaluate( dataset: t.Union[Dataset, EvaluationDataset], metrics: t.Optional[t.Sequence[Metric]] = None, llm: t.Optional[BaseRagasLLM | InstructorBaseRagasLLM | LangchainLLM] = None, embeddings: t.Optional[ BaseRagasEmbeddings | BaseRagasEmbedding | LangchainEmbeddings ] = None, experiment_name: t.Optional[str] = None, callbacks: Callbacks = None, run_config: t.Optional[RunConfig] = None, token_usage_parser: t.Optional[TokenUsageParser] = None, raise_exceptions: bool = False, column_map: t.Optional[t.Dict[str, str]] = None, show_progress: bool = True, batch_size: t.Optional[int] = None, _run_id: t.Optional[UUID] = None, _pbar: t.Optional[tqdm] = None, return_executor: bool = False, ) -> t.Union[EvaluationResult, Executor]: """ Async version of evaluate that performs evaluation without applying nest_asyncio. This function is the async-first implementation that doesn't patch the event loop, making it safe to use in production async applications. Parameters are identical to evaluate() function. Returns ------- EvaluationResult or Executor If return_executor is False, returns EvaluationResult object containing the scores of each metric. If return_executor is True, returns the Executor instance for cancellable execution. Examples -------- ```python import asyncio from ragas import aevaluate async def main(): result = await aevaluate(dataset, metrics) print(result) asyncio.run(main()) ``` """ warnings.warn( "aevaluate() is deprecated and will be removed in a future version. " "Use the @experiment decorator instead. " "See https://docs.ragas.io/en/latest/concepts/experiment/ for more information.", DeprecationWarning, stacklevel=2, ) column_map = column_map or {} callbacks = callbacks or [] run_config = run_config or RunConfig() if helicone_config.is_enabled: import uuid helicone_config.session_name = "ragas-evaluation" helicone_config.session_id = str(uuid.uuid4()) if dataset is None: raise ValueError("Provide dataset!") # Check metrics are correct type if not isinstance(metrics, (type(None), list)): raise TypeError( "Metrics should be provided in a list, e.g: metrics=[BleuScore()]" ) if isinstance(metrics, list) and any(not isinstance(m, Metric) for m in metrics): raise TypeError( "All metrics must be initialised metric objects, e.g: metrics=[BleuScore(), AspectCritic()]" ) # default metrics if metrics is None: from ragas.metrics._answer_relevance import answer_relevancy from ragas.metrics._context_precision import context_precision from ragas.metrics._context_recall import context_recall from ragas.metrics._faithfulness import faithfulness metrics = [answer_relevancy, context_precision, faithfulness, context_recall] if isinstance(dataset, Dataset): # remap column names from the dataset dataset = remap_column_names(dataset, column_map) dataset = convert_v1_to_v2_dataset(dataset) # validation dataset = EvaluationDataset.from_list(dataset.to_list()) if isinstance(dataset, EvaluationDataset): validate_required_columns(dataset, metrics) validate_supported_metrics(dataset, metrics) # set the llm and embeddings if isinstance(llm, LangchainLLM): llm = LangchainLLMWrapper(llm, run_config=run_config) if isinstance(embeddings, LangchainEmbeddings): embeddings = LangchainEmbeddingsWrapper(embeddings) # init llms and embeddings binary_metrics = [] llm_changed: t.List[int] = [] embeddings_changed: t.List[int] = [] answer_correctness_is_set = -1 # loop through the metrics and perform initializations for i, metric in enumerate(metrics): # set llm and embeddings if not set if isinstance(metric, AspectCritic): binary_metrics.append(metric.name) if isinstance(metric, MetricWithLLM) and metric.llm is None: if llm is None: from openai import OpenAI client = OpenAI() llm = llm_factory("gpt-4o-mini", client=client) metric.llm = t.cast(t.Optional[BaseRagasLLM], llm) llm_changed.append(i) if isinstance(metric, MetricWithEmbeddings) and metric.embeddings is None: if embeddings is None: # Infer embedding provider from LLM if available inferred_provider = _infer_embedding_provider_from_llm(llm) # Extract client from LLM if available for modern embeddings embedding_client = None if hasattr(llm, "client"): embedding_client = getattr(llm, "client") embeddings = embedding_factory( provider=inferred_provider, client=embedding_client ) metric.embeddings = embeddings embeddings_changed.append(i) if isinstance(metric, AnswerCorrectness): if metric.answer_similarity is None: answer_correctness_is_set = i # init all the models metric.init(run_config) executor = Executor( desc="Evaluating", keep_progress_bar=True, raise_exceptions=raise_exceptions, run_config=run_config, show_progress=show_progress, batch_size=batch_size, pbar=_pbar, ) # Ragas Callbacks # init the callbacks we need for various tasks ragas_callbacks: t.Dict[str, BaseCallbackHandler] = {} # Ragas Tracer which traces the run tracer = RagasTracer() ragas_callbacks["tracer"] = tracer # check if cost needs to be calculated if token_usage_parser is not None: from ragas.cost import CostCallbackHandler cost_cb = CostCallbackHandler(token_usage_parser=token_usage_parser) ragas_callbacks["cost_cb"] = cost_cb # append all the ragas_callbacks to the callbacks for cb in ragas_callbacks.values(): if isinstance(callbacks, BaseCallbackManager): callbacks.add_handler(cb) else: callbacks.append(cb) # new evaluation chain row_run_managers = [] evaluation_rm, evaluation_group_cm = new_group( name=experiment_name or RAGAS_EVALUATION_CHAIN_NAME, inputs={}, callbacks=callbacks, metadata={"type": ChainType.EVALUATION}, ) sample_type = dataset.get_sample_type() for i, sample in enumerate(dataset): row = t.cast(t.Dict[str, t.Any], sample.model_dump()) row_rm, row_group_cm = new_group( name=f"row {i}", inputs=row, callbacks=evaluation_group_cm, metadata={"type": ChainType.ROW, "row_index": i}, ) row_run_managers.append((row_rm, row_group_cm)) if sample_type == SingleTurnSample: _ = [ executor.submit( metric.single_turn_ascore, sample, row_group_cm, name=f"{metric.name}-{i}", timeout=run_config.timeout, ) for metric in metrics if isinstance(metric, SingleTurnMetric) ] elif sample_type == MultiTurnSample: _ = [ executor.submit( metric.multi_turn_ascore, sample, row_group_cm, name=f"{metric.name}-{i}", timeout=run_config.timeout, ) for metric in metrics if isinstance(metric, MultiTurnMetric) ] else: raise ValueError(f"Unsupported sample type {sample_type}") # Return executor for cancellable execution if requested if return_executor: return executor scores: t.List[t.Dict[str, t.Any]] = [] try: # get the results using async method results = await executor.aresults() if results == []: raise ExceptionInRunner() # convert results to dataset_like for i, _ in enumerate(dataset): s = {} for j, m in enumerate(metrics): if isinstance(m, ModeMetric): # type: ignore key = f"{m.name}(mode={m.mode})" else: key = m.name s[key] = results[len(metrics) * i + j] scores.append(s) # close the row chain row_rm, row_group_cm = row_run_managers[i] if not row_group_cm.ended: row_rm.on_chain_end(s) # run evaluation task except Exception as e: if not evaluation_group_cm.ended: evaluation_rm.on_chain_error(e) raise e else: # evalution run was successful # now lets process the results cost_cb = ragas_callbacks["cost_cb"] if "cost_cb" in ragas_callbacks else None result = EvaluationResult( scores=scores, dataset=dataset, binary_columns=binary_metrics, cost_cb=t.cast( t.Union["CostCallbackHandler", None], cost_cb, ), ragas_traces=tracer.traces, run_id=_run_id, ) if not evaluation_group_cm.ended: evaluation_rm.on_chain_end({"scores": result.scores}) finally: # reset llms and embeddings if changed for i in llm_changed: t.cast(MetricWithLLM, metrics[i]).llm = None for i in embeddings_changed: t.cast(MetricWithEmbeddings, metrics[i]).embeddings = None if answer_correctness_is_set != -1: t.cast( AnswerCorrectness, metrics[answer_correctness_is_set] ).answer_similarity = None # flush the analytics batcher from ragas._analytics import _analytics_batcher _analytics_batcher.flush() return result ```` ## Async Usage Ragas provides both synchronous and asynchronous evaluation APIs to accommodate different use cases: ### Using aevaluate() (Recommended for Production) For production async applications, use `aevaluate()` to avoid event loop conflicts: ```python import asyncio from ragas import aevaluate async def evaluate_app(): result = await aevaluate(dataset, metrics) return result # In your async application result = await evaluate_app() ``` ### Using evaluate() with Async Control For backward compatibility and Jupyter notebook usage, `evaluate()` provides optional control over `nest_asyncio`: ```python # Default behavior (Jupyter-compatible) result = evaluate(dataset, metrics) # allow_nest_asyncio=True # Production-safe (avoids event loop patching) result = evaluate(dataset, metrics, allow_nest_asyncio=False) ``` ### Migration from nest_asyncio Issues If you're experiencing issues with `nest_asyncio` in production: **Before (problematic):** ```python # This may cause event loop conflicts result = evaluate(dataset, metrics) ``` **After (fixed):** ```python # Option 1: Use async API result = await aevaluate(dataset, metrics) # Option 2: Disable nest_asyncio result = evaluate(dataset, metrics, allow_nest_asyncio=False) ``` ## CacheInterface Bases: `ABC` Abstract base class defining the interface for cache implementations. This class provides a standard interface that all cache implementations must follow. It supports basic cache operations like get, set and key checking. ### get ```python get(key: str) -> Any ``` Retrieve a value from the cache by key. Args: key: The key to look up in the cache. Returns: The cached value associated with the key. Source code in `src/ragas/cache.py` ```python @abstractmethod def get(self, key: str) -> Any: """Retrieve a value from the cache by key. Args: key: The key to look up in the cache. Returns: The cached value associated with the key. """ pass ``` ### set ```python set(key: str, value) -> None ``` Store a value in the cache with the given key. Args: key: The key to store the value under. value: The value to cache. Source code in `src/ragas/cache.py` ```python @abstractmethod def set(self, key: str, value) -> None: """Store a value in the cache with the given key. Args: key: The key to store the value under. value: The value to cache. """ pass ``` ### has_key ```python has_key(key: str) -> bool ``` Check if a key exists in the cache. Args: key: The key to check for. Returns: True if the key exists in the cache, False otherwise. Source code in `src/ragas/cache.py` ```python @abstractmethod def has_key(self, key: str) -> bool: """Check if a key exists in the cache. Args: key: The key to check for. Returns: True if the key exists in the cache, False otherwise. """ pass ``` ## DiskCacheBackend ```python DiskCacheBackend(cache_dir: str = '.cache') ``` Bases: `CacheInterface` A cache implementation that stores data on disk using the diskcache library. This cache backend persists data to disk, allowing it to survive between program runs. It implements the CacheInterface for use with Ragas caching functionality. Args: cache_dir (str, optional): Directory where cache files will be stored. Defaults to ".cache". Source code in `src/ragas/cache.py` ```python def __init__(self, cache_dir: str = ".cache"): try: from diskcache import Cache except ImportError: raise ImportError( "For using the diskcache backend, please install it with `pip install diskcache`." ) self.cache = Cache(cache_dir) ``` ### get ```python get(key: str) -> Any ``` Retrieve a value from the disk cache by key. Args: key: The key to look up in the cache. Returns: The cached value associated with the key, or None if not found. Source code in `src/ragas/cache.py` ```python def get(self, key: str) -> Any: """Retrieve a value from the disk cache by key. Args: key: The key to look up in the cache. Returns: The cached value associated with the key, or None if not found. """ return self.cache.get(key) ``` ### set ```python set(key: str, value) -> None ``` Store a value in the disk cache with the given key. Args: key: The key to store the value under. value: The value to cache. Source code in `src/ragas/cache.py` ```python def set(self, key: str, value) -> None: """Store a value in the disk cache with the given key. Args: key: The key to store the value under. value: The value to cache. """ self.cache.set(key, value) ``` ### has_key ```python has_key(key: str) -> bool ``` Check if a key exists in the disk cache. Args: key: The key to check for. Returns: True if the key exists in the cache, False otherwise. Source code in `src/ragas/cache.py` ```python def has_key(self, key: str) -> bool: """Check if a key exists in the disk cache. Args: key: The key to check for. Returns: True if the key exists in the cache, False otherwise. """ return key in self.cache ``` ## cacher ```python cacher(cache_backend: Optional[CacheInterface] = None) ``` Decorator that adds caching functionality to a function. This decorator can be applied to both synchronous and asynchronous functions to cache their results. If no cache backend is provided, the original function is returned unchanged. Args: cache_backend (Optional[CacheInterface]): The cache backend to use for storing results. If None, caching is disabled. Returns: Callable: A decorated function that implements caching behavior. Source code in `src/ragas/cache.py` ```python def cacher(cache_backend: Optional[CacheInterface] = None): """Decorator that adds caching functionality to a function. This decorator can be applied to both synchronous and asynchronous functions to cache their results. If no cache backend is provided, the original function is returned unchanged. Args: cache_backend (Optional[CacheInterface]): The cache backend to use for storing results. If None, caching is disabled. Returns: Callable: A decorated function that implements caching behavior. """ def decorator(func): if cache_backend is None: return func # hack to make pyright happy backend: CacheInterface = cache_backend is_async = inspect.iscoroutinefunction(func) @functools.wraps(func) async def async_wrapper(*args, **kwargs): cache_key = _generate_cache_key(func, args, kwargs) if backend.has_key(cache_key): logger.debug(f"Cache hit for {cache_key}") return backend.get(cache_key) result = await func(*args, **kwargs) picklable_result = _make_pydantic_picklable(result) backend.set(cache_key, picklable_result) return result @functools.wraps(func) def sync_wrapper(*args, **kwargs): cache_key = _generate_cache_key(func, args, kwargs) if backend.has_key(cache_key): logger.debug(f"Cache hit for {cache_key}") return backend.get(cache_key) result = func(*args, **kwargs) picklable_result = _make_pydantic_picklable(result) backend.set(cache_key, picklable_result) return result return async_wrapper if is_async else sync_wrapper return decorator ``` ## BaseRagasEmbedding ```python BaseRagasEmbedding(cache: Optional[CacheInterface] = None) ``` Bases: `ABC` Modern abstract base class for Ragas embedding implementations. This class provides a consistent interface for embedding text using various providers. Implementations should provide both sync and async methods for embedding single texts, with batch methods automatically provided. Args: cache: Optional cache backend for caching embeddings. Use DiskCacheBackend() for persistent caching. Source code in `src/ragas/embeddings/base.py` ```python def __init__(self, cache: t.Optional[CacheInterface] = None): """Initialize embedding with optional caching. Args: cache: Optional cache backend for caching embeddings. Use DiskCacheBackend() for persistent caching. """ self.cache = cache if self.cache is not None: self.embed_text = cacher(cache_backend=self.cache)(self.embed_text) self.aembed_text = cacher(cache_backend=self.cache)(self.aembed_text) ``` ### embed_text ```python embed_text(text: str, **kwargs: Any) -> List[float] ``` Embed a single text. Args: text: The text to embed \*\*kwargs: Additional arguments for the embedding call Returns: List of floats representing the embedding Source code in `src/ragas/embeddings/base.py` ```python @abstractmethod def embed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Embed a single text. Args: text: The text to embed **kwargs: Additional arguments for the embedding call Returns: List of floats representing the embedding """ pass ``` ### aembed_text ```python aembed_text(text: str, **kwargs: Any) -> List[float] ``` Asynchronously embed a single text. Args: text: The text to embed \*\*kwargs: Additional arguments for the embedding call Returns: List of floats representing the embedding Source code in `src/ragas/embeddings/base.py` ```python @abstractmethod async def aembed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Asynchronously embed a single text. Args: text: The text to embed **kwargs: Additional arguments for the embedding call Returns: List of floats representing the embedding """ pass ``` ### embed_texts ```python embed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Embed multiple texts. Default implementation processes texts individually. Override for batch optimization. Args: texts: List of texts to embed \*\*kwargs: Additional arguments for the embedding calls Returns: List of embeddings, one for each input text Source code in `src/ragas/embeddings/base.py` ```python def embed_texts(self, texts: t.List[str], **kwargs: t.Any) -> t.List[t.List[float]]: """Embed multiple texts. Default implementation processes texts individually. Override for batch optimization. Args: texts: List of texts to embed **kwargs: Additional arguments for the embedding calls Returns: List of embeddings, one for each input text """ texts = validate_texts(texts) return [self.embed_text(text, **kwargs) for text in texts] ``` ### aembed_texts ```python aembed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Asynchronously embed multiple texts. Default implementation processes texts concurrently. Override for batch optimization. Args: texts: List of texts to embed \*\*kwargs: Additional arguments for the embedding calls Returns: List of embeddings, one for each input text Source code in `src/ragas/embeddings/base.py` ```python async def aembed_texts( self, texts: t.List[str], **kwargs: t.Any ) -> t.List[t.List[float]]: """Asynchronously embed multiple texts. Default implementation processes texts concurrently. Override for batch optimization. Args: texts: List of texts to embed **kwargs: Additional arguments for the embedding calls Returns: List of embeddings, one for each input text """ texts = validate_texts(texts) tasks = [self.aembed_text(text, **kwargs) for text in texts] return await asyncio.gather(*tasks) ``` ## BaseRagasEmbeddings ```python BaseRagasEmbeddings(cache: Optional[CacheInterface] = None) ``` Bases: `Embeddings`, `ABC` Abstract base class for Ragas embeddings. This class extends the Embeddings class and provides methods for embedding text and managing run configurations. Attributes: run_config (RunConfig): Configuration for running the embedding operations. Source code in `src/ragas/embeddings/base.py` ```python def __init__(self, cache: t.Optional[CacheInterface] = None): super().__init__() self.cache = cache if self.cache is not None: self.embed_query = cacher(cache_backend=self.cache)(self.embed_query) self.embed_documents = cacher(cache_backend=self.cache)( self.embed_documents ) self.aembed_query = cacher(cache_backend=self.cache)(self.aembed_query) self.aembed_documents = cacher(cache_backend=self.cache)( self.aembed_documents ) ``` ### embed_text ```python embed_text(text: str, is_async=True) -> List[float] ``` Embed a single text string. Source code in `src/ragas/embeddings/base.py` ```python async def embed_text(self, text: str, is_async=True) -> t.List[float]: """ Embed a single text string. """ embs = await self.embed_texts([text], is_async=is_async) return embs[0] ``` ### embed_texts ```python embed_texts(texts: List[str], is_async: bool = True) -> List[List[float]] ``` Embed multiple texts. Source code in `src/ragas/embeddings/base.py` ```python async def embed_texts( self, texts: t.List[str], is_async: bool = True ) -> t.List[t.List[float]]: """ Embed multiple texts. """ if is_async: aembed_documents_with_retry = add_async_retry( self.aembed_documents, self.run_config ) return await aembed_documents_with_retry(texts) else: loop = asyncio.get_event_loop() embed_documents_with_retry = add_retry( self.embed_documents, self.run_config ) return await loop.run_in_executor(None, embed_documents_with_retry, texts) ``` ### set_run_config ```python set_run_config(run_config: RunConfig) ``` Set the run configuration for the embedding operations. Source code in `src/ragas/embeddings/base.py` ```python def set_run_config(self, run_config: RunConfig): """ Set the run configuration for the embedding operations. """ self.run_config = run_config ``` ## HuggingfaceEmbeddings ```python HuggingfaceEmbeddings(cache: Optional[CacheInterface] = None) ``` Bases: `BaseRagasEmbeddings` Hugging Face embeddings class for generating embeddings using pre-trained models. This class provides functionality to load and use Hugging Face models for generating embeddings of text inputs. Parameters: | Name | Type | Description | Default | | --------------- | ------ | ---------------------------------------------------------------------------------------------------- | ---------- | | `model_name` | `str` | Name of the pre-trained model to use, by default DEFAULT_MODEL_NAME. | *required* | | `cache_folder` | `str` | Path to store downloaded models. Can also be set by SENTENCE_TRANSFORMERS_HOME environment variable. | *required* | | `model_kwargs` | `dict` | Additional keyword arguments to pass to the model. | *required* | | `encode_kwargs` | `dict` | Additional keyword arguments to pass to the encoding method. | *required* | Attributes: | Name | Type | Description | | ------------------ | ------------------------------------------ | ----------------------------------------------------- | | `model` | `Union[SentenceTransformer, CrossEncoder]` | The loaded Hugging Face model. | | `is_cross_encoder` | `bool` | Flag indicating whether the model is a cross-encoder. | Methods: | Name | Description | | ----------------- | --------------------------------------------- | | `embed_query` | Embed a single query text. | | `embed_documents` | Embed multiple documents. | | `predict` | Make predictions using a cross-encoder model. | Notes This class requires the `sentence_transformers` and `transformers` packages to be installed. Examples: ```pycon >>> embeddings = HuggingfaceEmbeddings(model_name="bert-base-uncased") >>> query_embedding = embeddings.embed_query("What is the capital of France?") >>> doc_embeddings = embeddings.embed_documents(["Paris is the capital of France.", "London is the capital of the UK."]) ``` Source code in `src/ragas/embeddings/base.py` ```python def __init__(self, cache: t.Optional[CacheInterface] = None): super().__init__() self.cache = cache if self.cache is not None: self.embed_query = cacher(cache_backend=self.cache)(self.embed_query) self.embed_documents = cacher(cache_backend=self.cache)( self.embed_documents ) self.aembed_query = cacher(cache_backend=self.cache)(self.aembed_query) self.aembed_documents = cacher(cache_backend=self.cache)( self.aembed_documents ) ``` ### embed_query ```python embed_query(text: str) -> List[float] ``` Embed a single query text. Source code in `src/ragas/embeddings/base.py` ```python def embed_query(self, text: str) -> t.List[float]: """ Embed a single query text. """ return self.embed_documents([text])[0] ``` ### embed_documents ```python embed_documents(texts: List[str]) -> List[List[float]] ``` Embed multiple documents. Source code in `src/ragas/embeddings/base.py` ```python def embed_documents(self, texts: t.List[str]) -> t.List[t.List[float]]: """ Embed multiple documents. """ from sentence_transformers.SentenceTransformer import SentenceTransformer from torch import Tensor assert isinstance(self.model, SentenceTransformer), ( "Model is not of the type Bi-encoder" ) embeddings = self.model.encode( texts, normalize_embeddings=True, **self.encode_kwargs ) assert isinstance(embeddings, Tensor) return embeddings.tolist() ``` ### predict ```python predict(texts: List[List[str]]) -> List[List[float]] ``` Make predictions using a cross-encoder model. Source code in `src/ragas/embeddings/base.py` ```python def predict(self, texts: t.List[t.List[str]]) -> t.List[t.List[float]]: """ Make predictions using a cross-encoder model. """ from sentence_transformers.cross_encoder import CrossEncoder from torch import Tensor assert isinstance(self.model, CrossEncoder), ( "Model is not of the type CrossEncoder" ) predictions = self.model.predict(texts, **self.encode_kwargs) assert isinstance(predictions, Tensor) return predictions.tolist() ``` ## GoogleEmbeddings ```python GoogleEmbeddings(client: Optional[Any] = None, model: str = 'gemini-embedding-001', use_vertex: bool = False, project_id: Optional[str] = None, location: Optional[str] = 'us-central1', cache: Optional[CacheInterface] = None, **kwargs: Any) ``` Bases: `BaseRagasEmbedding` Google embeddings using Vertex AI or Google AI (Gemini). Supports both Vertex AI and Google AI (Gemini) embedding models. For Vertex AI, requires google-cloud-aiplatform package. For Google AI, supports both: - New SDK (google-genai): Recommended, uses genai.Client() - Old SDK (google-generativeai): Deprecated (support ends Aug 2025) The client parameter is flexible: - For new SDK: genai.Client(api_key="...") instance - For old SDK: None (auto-imports), the genai module, or a GenerativeModel instance - For Vertex: Should be the configured vertex client Note: Unlike LLM generation, embeddings work correctly with both SDKs. The known instructor safety settings issue (github.com/567-labs/instructor/issues/1658) only affects LLM generation, not embeddings. Examples: # New SDK (google-genai) - recommended from google import genai client = genai.Client(api_key="...") embeddings = GoogleEmbeddings(client=client, model="gemini-embedding-001") ```text # Old SDK (google-generativeai) - deprecated import google.generativeai as genai genai.configure(api_key="...") embeddings = GoogleEmbeddings(client=genai, model="text-embedding-004") # Auto-import (tries new SDK first, falls back to old) embeddings = GoogleEmbeddings(model="text-embedding-004") ``` Source code in `src/ragas/embeddings/google_provider.py` ```python def __init__( self, client: t.Optional[t.Any] = None, model: str = "gemini-embedding-001", use_vertex: bool = False, project_id: t.Optional[str] = None, location: t.Optional[str] = "us-central1", cache: t.Optional[CacheInterface] = None, **kwargs: t.Any, ): super().__init__(cache=cache) self._original_client = client self.model = model self.use_vertex = use_vertex self.project_id = project_id self.location = location self.kwargs = kwargs # Track which SDK is being used (new google-genai vs old google-generativeai) self._use_new_sdk = False # Resolve the actual client to use self.client = self._resolve_client(client, use_vertex) ``` ### embed_text ```python embed_text(text: str, **kwargs: Any) -> List[float] ``` Embed a single text using Google's embedding service. Source code in `src/ragas/embeddings/google_provider.py` ```python def embed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Embed a single text using Google's embedding service.""" if self.use_vertex: return self._embed_text_vertex(text, **kwargs) else: return self._embed_text_genai(text, **kwargs) ``` ### aembed_text ```python aembed_text(text: str, **kwargs: Any) -> List[float] ``` Asynchronously embed a single text using Google's embedding service. Google's SDK doesn't provide native async support, so we use ThreadPoolExecutor. Source code in `src/ragas/embeddings/google_provider.py` ```python async def aembed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Asynchronously embed a single text using Google's embedding service. Google's SDK doesn't provide native async support, so we use ThreadPoolExecutor. """ return await run_sync_in_async(self.embed_text, text, **kwargs) ``` ### embed_texts ```python embed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Embed multiple texts using Google's embedding service. Source code in `src/ragas/embeddings/google_provider.py` ```python def embed_texts(self, texts: t.List[str], **kwargs: t.Any) -> t.List[t.List[float]]: """Embed multiple texts using Google's embedding service.""" texts = validate_texts(texts) if not texts: return [] if self.use_vertex: return self._embed_texts_vertex(texts, **kwargs) else: return self._embed_texts_genai(texts, **kwargs) ``` ### aembed_texts ```python aembed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Asynchronously embed multiple texts using Google's embedding service. Source code in `src/ragas/embeddings/google_provider.py` ```python async def aembed_texts( self, texts: t.List[str], **kwargs: t.Any ) -> t.List[t.List[float]]: """Asynchronously embed multiple texts using Google's embedding service.""" texts = validate_texts(texts) if not texts: return [] return await run_sync_in_async(self.embed_texts, texts, **kwargs) ``` ## HaystackEmbeddingsWrapper ```python HaystackEmbeddingsWrapper(embedder: Union[AzureOpenAITextEmbedder, HuggingFaceAPITextEmbedder, OpenAITextEmbedder, SentenceTransformersTextEmbedder], run_config: Optional[RunConfig] = None, cache: Optional[CacheInterface] = None) ``` Bases: `BaseRagasEmbeddings` A wrapper for using Haystack embedders within the Ragas framework. This class allows you to use both synchronous and asynchronous methods (`embed_query`/`embed_documents` and `aembed_query`/`aembed_documents`) for generating embeddings through a Haystack embedder. Parameters: | Name | Type | Description | Default | | ------------ | ------------------------- | ------------------------------------------------------------------------------- | ------------------ | | `embedder` | \`AzureOpenAITextEmbedder | HuggingFaceAPITextEmbedder | OpenAITextEmbedder | | `run_config` | `RunConfig` | A configuration object to manage embedding execution settings, by default None. | `None` | | `cache` | `CacheInterface` | A cache instance for storing and retrieving embedding results, by default None. | `None` | Source code in `src/ragas/embeddings/haystack_wrapper.py` ```python def __init__( self, embedder: t.Union[ "AzureOpenAITextEmbedder", "HuggingFaceAPITextEmbedder", "OpenAITextEmbedder", "SentenceTransformersTextEmbedder", ], run_config: t.Optional[RunConfig] = None, cache: t.Optional[CacheInterface] = None, ): super().__init__(cache=cache) # Lazy Import of required Haystack components try: from haystack import AsyncPipeline from haystack.components.embedders.azure_text_embedder import ( AzureOpenAITextEmbedder, ) from haystack.components.embedders.hugging_face_api_text_embedder import ( HuggingFaceAPITextEmbedder, ) from haystack.components.embedders.openai_text_embedder import ( OpenAITextEmbedder, ) from haystack.components.embedders.sentence_transformers_text_embedder import ( SentenceTransformersTextEmbedder, ) except ImportError as exc: raise ImportError( "Haystack is not installed. Please install it with `pip install haystack-ai`." ) from exc # Validate embedder type if not isinstance( embedder, ( AzureOpenAITextEmbedder, HuggingFaceAPITextEmbedder, OpenAITextEmbedder, SentenceTransformersTextEmbedder, ), ): raise TypeError( "Expected 'embedder' to be one of: AzureOpenAITextEmbedder, " "HuggingFaceAPITextEmbedder, OpenAITextEmbedder, or " f"SentenceTransformersTextEmbedder, but got {type(embedder).__name__}." ) self.embedder = embedder # Initialize an asynchronous pipeline and add the embedder component self.async_pipeline = AsyncPipeline() self.async_pipeline.add_component("embedder", self.embedder) # type: ignore[reportArgumentType] # Set or create the run configuration if run_config is None: run_config = RunConfig() self.set_run_config(run_config) ``` ## HuggingFaceEmbeddings ```python HuggingFaceEmbeddings(model: str, use_api: bool = False, api_key: Optional[str] = None, device: Optional[str] = None, normalize_embeddings: bool = True, batch_size: int = 32, cache: Optional[CacheInterface] = None, **model_kwargs: Any) ``` Bases: `BaseRagasEmbedding` HuggingFace embeddings supporting both local and API-based models. Supports sentence-transformers for local models and HuggingFace API for hosted models. Provides efficient batch processing and caching. Source code in `src/ragas/embeddings/huggingface_provider.py` ```python def __init__( self, model: str, use_api: bool = False, api_key: t.Optional[str] = None, device: t.Optional[str] = None, normalize_embeddings: bool = True, batch_size: int = 32, cache: t.Optional[CacheInterface] = None, **model_kwargs: t.Any, ): super().__init__(cache=cache) self.model = model self.use_api = use_api self.api_key = api_key self.device = device self.normalize_embeddings = normalize_embeddings self.batch_size = batch_size self.model_kwargs = model_kwargs if use_api: self._setup_api_client() else: self._setup_local_model() ``` ### embed_text ```python embed_text(text: str, **kwargs: Any) -> List[float] ``` Embed a single text using HuggingFace. Source code in `src/ragas/embeddings/huggingface_provider.py` ```python def embed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Embed a single text using HuggingFace.""" if self.use_api: return self._embed_text_api(text, **kwargs) else: return self._embed_text_local(text, **kwargs) ``` ### aembed_text ```python aembed_text(text: str, **kwargs: Any) -> List[float] ``` Asynchronously embed a single text using HuggingFace. Source code in `src/ragas/embeddings/huggingface_provider.py` ```python async def aembed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Asynchronously embed a single text using HuggingFace.""" if self.use_api: return await self._aembed_text_api(text, **kwargs) else: return await run_sync_in_async(self._embed_text_local, text, **kwargs) ``` ### embed_texts ```python embed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Embed multiple texts using HuggingFace with batching. Source code in `src/ragas/embeddings/huggingface_provider.py` ```python def embed_texts(self, texts: t.List[str], **kwargs: t.Any) -> t.List[t.List[float]]: """Embed multiple texts using HuggingFace with batching.""" texts = validate_texts(texts) if not texts: return [] if self.use_api: return self._embed_texts_api(texts, **kwargs) else: return self._embed_texts_local(texts, **kwargs) ``` ### aembed_texts ```python aembed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Asynchronously embed multiple texts using HuggingFace. Source code in `src/ragas/embeddings/huggingface_provider.py` ```python async def aembed_texts( self, texts: t.List[str], **kwargs: t.Any ) -> t.List[t.List[float]]: """Asynchronously embed multiple texts using HuggingFace.""" texts = validate_texts(texts) if not texts: return [] if self.use_api: return await run_sync_in_async(self._embed_texts_api, texts, **kwargs) else: return await run_sync_in_async(self._embed_texts_local, texts, **kwargs) ``` ## LiteLLMEmbeddings ```python LiteLLMEmbeddings(model: str, api_key: Optional[str] = None, api_base: Optional[str] = None, api_version: Optional[str] = None, timeout: int = 600, max_retries: int = 3, batch_size: Optional[int] = None, cache: Optional[CacheInterface] = None, **litellm_params: Any) ``` Bases: `BaseRagasEmbedding` Universal embedding interface using LiteLLM. Supports 100+ models across OpenAI, Azure, Google, Cohere, Anthropic, and more. Provides intelligent batching and provider-specific optimizations. Source code in `src/ragas/embeddings/litellm_provider.py` ```python def __init__( self, model: str, api_key: t.Optional[str] = None, api_base: t.Optional[str] = None, api_version: t.Optional[str] = None, timeout: int = 600, max_retries: int = 3, batch_size: t.Optional[int] = None, cache: t.Optional[CacheInterface] = None, **litellm_params: t.Any, ): super().__init__(cache=cache) self.litellm = safe_import("litellm", "litellm") self.model = model self.api_key = api_key self.api_base = api_base self.api_version = api_version self.timeout = timeout self.max_retries = max_retries self.batch_size = batch_size or get_optimal_batch_size("litellm", model) self.litellm_params = litellm_params ``` ### embed_text ```python embed_text(text: str, **kwargs: Any) -> List[float] ``` Embed a single text using LiteLLM. Source code in `src/ragas/embeddings/litellm_provider.py` ```python def embed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Embed a single text using LiteLLM.""" call_kwargs = self._prepare_kwargs(**kwargs) response = self.litellm.embedding(input=[text], **call_kwargs) return response.data[0]["embedding"] ``` ### aembed_text ```python aembed_text(text: str, **kwargs: Any) -> List[float] ``` Asynchronously embed a single text using LiteLLM. Source code in `src/ragas/embeddings/litellm_provider.py` ```python async def aembed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Asynchronously embed a single text using LiteLLM.""" call_kwargs = self._prepare_kwargs(**kwargs) response = await self.litellm.aembedding(input=[text], **call_kwargs) return response.data[0]["embedding"] ``` ### embed_texts ```python embed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Embed multiple texts using LiteLLM with intelligent batching. Source code in `src/ragas/embeddings/litellm_provider.py` ```python def embed_texts(self, texts: t.List[str], **kwargs: t.Any) -> t.List[t.List[float]]: """Embed multiple texts using LiteLLM with intelligent batching.""" texts = validate_texts(texts) if not texts: return [] embeddings = [] batches = batch_texts(texts, self.batch_size) for batch in batches: call_kwargs = self._prepare_kwargs(**kwargs) response = self.litellm.embedding(input=batch, **call_kwargs) embeddings.extend([item["embedding"] for item in response.data]) return embeddings ``` ### aembed_texts ```python aembed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Asynchronously embed multiple texts using LiteLLM with intelligent batching. Source code in `src/ragas/embeddings/litellm_provider.py` ```python async def aembed_texts( self, texts: t.List[str], **kwargs: t.Any ) -> t.List[t.List[float]]: """Asynchronously embed multiple texts using LiteLLM with intelligent batching.""" texts = validate_texts(texts) if not texts: return [] embeddings = [] batches = batch_texts(texts, self.batch_size) for batch in batches: call_kwargs = self._prepare_kwargs(**kwargs) response = await self.litellm.aembedding(input=batch, **call_kwargs) embeddings.extend([item["embedding"] for item in response.data]) return embeddings ``` ## OpenAIEmbeddings ```python OpenAIEmbeddings(client: Any, model: str = 'text-embedding-3-small', cache: Optional[CacheInterface] = None) ``` Bases: `BaseRagasEmbedding` OpenAI embeddings implementation with batch optimization. Supports both sync and async OpenAI clients with automatic detection. Provides optimized batch processing for better performance. Source code in `src/ragas/embeddings/openai_provider.py` ```python def __init__( self, client: t.Any, model: str = "text-embedding-3-small", cache: t.Optional[CacheInterface] = None, ): super().__init__(cache=cache) self.client = client self.model = model self.is_async = self._check_client_async(client) ``` ### embed_text ```python embed_text(text: str, **kwargs: Any) -> List[float] ``` Embed a single text using OpenAI. For async clients, this will run the async method in the appropriate event loop. Source code in `src/ragas/embeddings/openai_provider.py` ```python def embed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Embed a single text using OpenAI. For async clients, this will run the async method in the appropriate event loop. """ if self.is_async: result = self._run_async_in_current_loop(self.aembed_text(text, **kwargs)) else: response = self.client.embeddings.create( input=text, model=self.model, **kwargs ) result = response.data[0].embedding # Track usage track( EmbeddingUsageEvent( provider="openai", model=self.model, embedding_type="modern", num_requests=1, is_async=self.is_async, ) ) return result ``` ### aembed_text ```python aembed_text(text: str, **kwargs: Any) -> List[float] ``` Asynchronously embed a single text using OpenAI. Source code in `src/ragas/embeddings/openai_provider.py` ```python async def aembed_text(self, text: str, **kwargs: t.Any) -> t.List[float]: """Asynchronously embed a single text using OpenAI.""" if not self.is_async: raise TypeError( "Cannot use aembed_text() with a synchronous client. Use embed_text() instead." ) response = await self.client.embeddings.create( input=text, model=self.model, **kwargs ) result = response.data[0].embedding # Track usage track( EmbeddingUsageEvent( provider="openai", model=self.model, embedding_type="modern", num_requests=1, is_async=True, ) ) return result ``` ### embed_texts ```python embed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Embed multiple texts using OpenAI's batch API for optimization. Source code in `src/ragas/embeddings/openai_provider.py` ```python def embed_texts(self, texts: t.List[str], **kwargs: t.Any) -> t.List[t.List[float]]: """Embed multiple texts using OpenAI's batch API for optimization.""" texts = validate_texts(texts) if not texts: return [] if self.is_async: result = self._run_async_in_current_loop(self.aembed_texts(texts, **kwargs)) else: # OpenAI supports batch embedding natively response = self.client.embeddings.create( input=texts, model=self.model, **kwargs ) result = [item.embedding for item in response.data] # Track usage track( EmbeddingUsageEvent( provider="openai", model=self.model, embedding_type="modern", num_requests=len(texts), is_async=self.is_async, ) ) return result ``` ### aembed_texts ```python aembed_texts(texts: List[str], **kwargs: Any) -> List[List[float]] ``` Asynchronously embed multiple texts using OpenAI's batch API. Source code in `src/ragas/embeddings/openai_provider.py` ```python async def aembed_texts( self, texts: t.List[str], **kwargs: t.Any ) -> t.List[t.List[float]]: """Asynchronously embed multiple texts using OpenAI's batch API.""" texts = validate_texts(texts) if not texts: return [] if not self.is_async: raise TypeError( "Cannot use aembed_texts() with a synchronous client. Use embed_texts() instead." ) response = await self.client.embeddings.create( input=texts, model=self.model, **kwargs ) result = [item.embedding for item in response.data] # Track usage track( EmbeddingUsageEvent( provider="openai", model=self.model, embedding_type="modern", num_requests=len(texts), is_async=True, ) ) return result ``` ## batch_texts ```python batch_texts(texts: List[str], batch_size: int) -> List[List[str]] ``` Batch a list of texts into smaller chunks. Args: texts: List of texts to batch batch_size: Size of each batch Returns: List of batches, where each batch is a list of texts Source code in `src/ragas/embeddings/utils.py` ```python def batch_texts(texts: t.List[str], batch_size: int) -> t.List[t.List[str]]: """Batch a list of texts into smaller chunks. Args: texts: List of texts to batch batch_size: Size of each batch Returns: List of batches, where each batch is a list of texts """ if batch_size <= 0: raise ValueError("Batch size must be positive") batches = [] for i in range(0, len(texts), batch_size): batches.append(texts[i : i + batch_size]) return batches ``` ## get_optimal_batch_size ```python get_optimal_batch_size(provider: str, model: str) -> int ``` Get optimal batch size for a provider/model combination. Args: provider: The embedding provider model: The model name Returns: Optimal batch size for the provider/model Source code in `src/ragas/embeddings/utils.py` ```python def get_optimal_batch_size(provider: str, model: str) -> int: """Get optimal batch size for a provider/model combination. Args: provider: The embedding provider model: The model name Returns: Optimal batch size for the provider/model """ provider_lower = provider.lower() # Provider-specific batch sizes if "openai" in provider_lower: return 100 # OpenAI supports large batches elif "cohere" in provider_lower: return 96 # Cohere's documented limit elif "google" in provider_lower or "vertex" in provider_lower: return 5 # Google/Vertex AI is more conservative elif "huggingface" in provider_lower: return 32 # HuggingFace default else: return 10 # Conservative default for unknown providers ``` ## validate_texts ```python validate_texts(texts: Union[str, List[str]]) -> List[str] ``` Validate and normalize text inputs. Args: texts: Single text or list of texts Returns: List of validated texts Raises: ValueError: If texts are invalid Source code in `src/ragas/embeddings/utils.py` ```python def validate_texts(texts: t.Union[str, t.List[str]]) -> t.List[str]: """Validate and normalize text inputs. Args: texts: Single text or list of texts Returns: List of validated texts Raises: ValueError: If texts are invalid """ if isinstance(texts, str): texts = [texts] if not isinstance(texts, list): raise ValueError("Texts must be a string or list of strings") if not texts: raise ValueError("Texts list cannot be empty") for i, text in enumerate(texts): if not isinstance(text, str): raise ValueError(f"Text at index {i} must be a string, got {type(text)}") if not text.strip(): raise ValueError(f"Text at index {i} cannot be empty or whitespace only") return texts ``` ## embedding_factory ```python embedding_factory(*args, **kwargs) ``` Deprecated: Use embedding_factory from base module directly. Source code in `src/ragas/embeddings/__init__.py` ```python def embedding_factory(*args, **kwargs): """Deprecated: Use embedding_factory from base module directly.""" import warnings warnings.warn( "Importing embedding_factory from ragas.embeddings is deprecated. " "Import directly from ragas.embeddings.base or use modern providers: " "from ragas.embeddings import OpenAIEmbeddings, GoogleEmbeddings, HuggingFaceEmbeddings", DeprecationWarning, stacklevel=2, ) return _embedding_factory(*args, **kwargs) ``` # Evaluation ## evaluate() Perform the evaluation on the dataset with different metrics Parameters: | Name | Type | Description | Default | | -------------------- | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `dataset` | `(Dataset, EvaluationDataset)` | The dataset used by the metrics to evaluate the RAG pipeline. | *required* | | `metrics` | `list[Metric]` | List of metrics to use for evaluation. If not provided, ragas will run the evaluation on the best set of metrics to give a complete view. | `None` | | `llm` | `BaseRagasLLM` | The language model (LLM) to use to generate the score for calculating the metrics. If not provided, ragas will use the default language model for metrics that require an LLM. This can be overridden by the LLM specified in the metric level with metric.llm. | `None` | | `embeddings` | `BaseRagasEmbeddings` | The embeddings model to use for the metrics. If not provided, ragas will use the default embeddings for metrics that require embeddings. This can be overridden by the embeddings specified in the metric level with metric.embeddings. | `None` | | `experiment_name` | `str` | The name of the experiment to track. This is used to track the evaluation in the tracing tool. | `None` | | `callbacks` | `Callbacks` | Lifecycle Langchain Callbacks to run during evaluation. Check the Langchain documentation for more information. | `None` | | `run_config` | `RunConfig` | Configuration for runtime settings like timeout and retries. If not provided, default values are used. | `None` | | `token_usage_parser` | `TokenUsageParser` | Parser to get the token usage from the LLM result. If not provided, the cost and total token count will not be calculated. Default is None. | `None` | | `raise_exceptions` | `False` | Whether to raise exceptions or not. If set to True, the evaluation will raise an exception if any of the metrics fail. If set to False, the evaluation will return np.nan for the row that failed. Default is False. | `False` | | `column_map` | `dict[str, str]` | The column names of the dataset to use for evaluation. If the column names of the dataset are different from the default ones, it is possible to provide the mapping as a dictionary here. Example: If the dataset column name is contexts_v1, it is possible to pass column_map as {"contexts": "contexts_v1"}. | `None` | | `show_progress` | `bool` | Whether to show the progress bar during evaluation. If set to False, the progress bar will be disabled. The default is True. | `True` | | `batch_size` | `int` | How large the batches should be. If set to None (default), no batching is done. | `None` | | `return_executor` | `bool` | If True, returns the Executor instance instead of running evaluation. The returned executor can be used to cancel execution by calling executor.cancel(). To get results, call executor.results(). Default is False. | `False` | | `allow_nest_asyncio` | `bool` | Whether to allow nest_asyncio patching for Jupyter compatibility. Set to False in production async applications to avoid event loop conflicts. Default is True. | `True` | Returns: | Type | Description | | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `EvaluationResult or Executor` | If return_executor is False, returns EvaluationResult object containing the scores of each metric. If return_executor is True, returns the Executor instance for cancellable execution. | Raises: | Type | Description | | ------------ | ----------------------------------------------------------------------------------------------------------------------- | | `ValueError` | if validation fails because the columns required for the metrics are missing or if the columns are of the wrong format. | Examples: the basic usage is as follows: ```text from ragas import evaluate >>> dataset Dataset({ features: ['question', 'ground_truth', 'answer', 'contexts'], num_rows: 30 }) >>> result = evaluate(dataset) >>> print(result) {'context_precision': 0.817, 'faithfulness': 0.892, 'answer_relevancy': 0.874} ``` Source code in `src/ragas/evaluation.py` ````python @track_was_completed def evaluate( dataset: t.Union[Dataset, EvaluationDataset], metrics: t.Optional[t.Sequence[Metric]] = None, llm: t.Optional[BaseRagasLLM | LangchainLLM] = None, embeddings: t.Optional[ BaseRagasEmbeddings | BaseRagasEmbedding | LangchainEmbeddings ] = None, experiment_name: t.Optional[str] = None, callbacks: Callbacks = None, run_config: t.Optional[RunConfig] = None, token_usage_parser: t.Optional[TokenUsageParser] = None, raise_exceptions: bool = False, column_map: t.Optional[t.Dict[str, str]] = None, show_progress: bool = True, batch_size: t.Optional[int] = None, _run_id: t.Optional[UUID] = None, _pbar: t.Optional[tqdm] = None, return_executor: bool = False, allow_nest_asyncio: bool = True, ) -> t.Union[EvaluationResult, Executor]: """ Perform the evaluation on the dataset with different metrics Parameters ---------- dataset : Dataset, EvaluationDataset The dataset used by the metrics to evaluate the RAG pipeline. metrics : list[Metric], optional List of metrics to use for evaluation. If not provided, ragas will run the evaluation on the best set of metrics to give a complete view. llm : BaseRagasLLM, optional The language model (LLM) to use to generate the score for calculating the metrics. If not provided, ragas will use the default language model for metrics that require an LLM. This can be overridden by the LLM specified in the metric level with `metric.llm`. embeddings : BaseRagasEmbeddings, optional The embeddings model to use for the metrics. If not provided, ragas will use the default embeddings for metrics that require embeddings. This can be overridden by the embeddings specified in the metric level with `metric.embeddings`. experiment_name : str, optional The name of the experiment to track. This is used to track the evaluation in the tracing tool. callbacks : Callbacks, optional Lifecycle Langchain Callbacks to run during evaluation. Check the [Langchain documentation](https://python.langchain.com/docs/modules/callbacks/) for more information. run_config : RunConfig, optional Configuration for runtime settings like timeout and retries. If not provided, default values are used. token_usage_parser : TokenUsageParser, optional Parser to get the token usage from the LLM result. If not provided, the cost and total token count will not be calculated. Default is None. raise_exceptions : False Whether to raise exceptions or not. If set to True, the evaluation will raise an exception if any of the metrics fail. If set to False, the evaluation will return `np.nan` for the row that failed. Default is False. column_map : dict[str, str], optional The column names of the dataset to use for evaluation. If the column names of the dataset are different from the default ones, it is possible to provide the mapping as a dictionary here. Example: If the dataset column name is `contexts_v1`, it is possible to pass column_map as `{"contexts": "contexts_v1"}`. show_progress : bool, optional Whether to show the progress bar during evaluation. If set to False, the progress bar will be disabled. The default is True. batch_size : int, optional How large the batches should be. If set to None (default), no batching is done. return_executor : bool, optional If True, returns the Executor instance instead of running evaluation. The returned executor can be used to cancel execution by calling executor.cancel(). To get results, call executor.results(). Default is False. allow_nest_asyncio : bool, optional Whether to allow nest_asyncio patching for Jupyter compatibility. Set to False in production async applications to avoid event loop conflicts. Default is True. Returns ------- EvaluationResult or Executor If return_executor is False, returns EvaluationResult object containing the scores of each metric. If return_executor is True, returns the Executor instance for cancellable execution. Raises ------ ValueError if validation fails because the columns required for the metrics are missing or if the columns are of the wrong format. Examples -------- the basic usage is as follows: ``` from ragas import evaluate >>> dataset Dataset({ features: ['question', 'ground_truth', 'answer', 'contexts'], num_rows: 30 }) >>> result = evaluate(dataset) >>> print(result) {'context_precision': 0.817, 'faithfulness': 0.892, 'answer_relevancy': 0.874} ``` """ warnings.warn( "evaluate() is deprecated and will be removed in a future version. " "Use the @experiment decorator instead. " "See https://docs.ragas.io/en/latest/concepts/experiment/ for more information.", DeprecationWarning, stacklevel=2, ) # Create async wrapper for aevaluate async def _async_wrapper(): return await aevaluate( dataset=dataset, metrics=metrics, llm=llm, embeddings=embeddings, experiment_name=experiment_name, callbacks=callbacks, run_config=run_config, token_usage_parser=token_usage_parser, raise_exceptions=raise_exceptions, column_map=column_map, show_progress=show_progress, batch_size=batch_size, _run_id=_run_id, _pbar=_pbar, return_executor=return_executor, ) if not allow_nest_asyncio: # Run without nest_asyncio - creates a new event loop import asyncio return asyncio.run(_async_wrapper()) else: # Default behavior: use nest_asyncio for backward compatibility (Jupyter notebooks) from ragas.async_utils import run return run(_async_wrapper()) ```` ## BaseSample Bases: `BaseModel` Base class for evaluation samples. ### to_dict ```python to_dict() -> Dict ``` Get the dictionary representation of the sample without attributes that are None. Source code in `src/ragas/dataset_schema.py` ```python def to_dict(self) -> t.Dict: """ Get the dictionary representation of the sample without attributes that are None. """ return self.model_dump(exclude_none=True) ``` ### get_features ```python get_features() -> List[str] ``` Get the features of the sample that are not None. Source code in `src/ragas/dataset_schema.py` ```python def get_features(self) -> t.List[str]: """ Get the features of the sample that are not None. """ return list(self.to_dict().keys()) ``` ### to_string ```python to_string() -> str ``` Get the string representation of the sample. Source code in `src/ragas/dataset_schema.py` ```python def to_string(self) -> str: """ Get the string representation of the sample. """ sample_dict = self.to_dict() return "".join(f"\n{key}:\n\t{val}\n" for key, val in sample_dict.items()) ``` ## SingleTurnSample Bases: `BaseSample` Represents evaluation samples for single-turn interactions. Attributes: | Name | Type | Description | | ----------------------- | --------------------------------- | --------------------------------------------------------- | | `user_input` | `Optional[str]` | The input query from the user. | | `retrieved_contexts` | `Optional[List[str]]` | List of contexts retrieved for the query. | | `reference_contexts` | `Optional[List[str]]` | List of reference contexts for the query. | | `retrieved_context_ids` | `Optional[List[Union[str, int]]]` | List of IDs for retrieved contexts. | | `reference_context_ids` | `Optional[List[Union[str, int]]]` | List of IDs for reference contexts. | | `response` | `Optional[str]` | The generated response for the query. | | `multi_responses` | `Optional[List[str]]` | List of multiple responses generated for the query. | | `reference` | `Optional[str]` | The reference answer for the query. | | `rubric` | `Optional[Dict[str, str]]` | Evaluation rubric for the sample. | | `persona_name` | `Optional[str]` | Name of the persona used in query generation. | | `query_style` | `Optional[str]` | Style of the generated query (e.g., formal, casual). | | `query_length` | `Optional[str]` | Length category of the query (e.g., short, medium, long). | ## MultiTurnSample Bases: `BaseSample` Represents evaluation samples for multi-turn interactions. Attributes: | Name | Type | Description | | ---------------------- | --------------------------------------------------- | -------------------------------------------------------------- | | `user_input` | `List[Union[HumanMessage, AIMessage, ToolMessage]]` | A list of messages representing the conversation turns. | | `reference` | `(Optional[str], optional)` | The reference answer or expected outcome for the conversation. | | `reference_tool_calls` | `(Optional[List[ToolCall]], optional)` | A list of expected tool calls for the conversation. | | `rubrics` | `(Optional[Dict[str, str]], optional)` | Evaluation rubrics for the conversation. | | `reference_topics` | `(Optional[List[str]], optional)` | A list of reference topics for the conversation. | ### validate_user_input ```python validate_user_input(messages: List[Union[HumanMessage, AIMessage, ToolMessage]]) -> List[Union[HumanMessage, AIMessage, ToolMessage]] ``` Validates the user input messages. Source code in `src/ragas/dataset_schema.py` ```python @field_validator("user_input") @classmethod def validate_user_input( cls, messages: t.List[t.Union[HumanMessage, AIMessage, ToolMessage]], ) -> t.List[t.Union[HumanMessage, AIMessage, ToolMessage]]: """Validates the user input messages.""" if not all( isinstance(m, (HumanMessage, AIMessage, ToolMessage)) for m in messages ): raise ValueError( "All inputs must be instances of HumanMessage, AIMessage, or ToolMessage." ) has_seen_ai_message = False for i, m in enumerate(messages): if isinstance(m, AIMessage): has_seen_ai_message = True elif isinstance(m, ToolMessage): # Rule 1: ToolMessage must be preceded by an AIMessage somewhere in the conversation if not has_seen_ai_message: raise ValueError( "ToolMessage must be preceded by an AIMessage somewhere in the conversation." ) # Rule 2: ToolMessage must follow an AIMessage or another ToolMessage if i > 0: prev_message = messages[i - 1] if isinstance(prev_message, AIMessage): # Rule 3: If following AIMessage, that message must have tool_calls if not prev_message.tool_calls: raise ValueError( "ToolMessage must follow an AIMessage where tools were called." ) elif not isinstance(prev_message, ToolMessage): # Not following AIMessage or ToolMessage raise ValueError( "ToolMessage must follow an AIMessage or another ToolMessage." ) return messages ``` ### to_messages ```python to_messages() ``` Converts the user input messages to a list of dictionaries. Source code in `src/ragas/dataset_schema.py` ```python def to_messages(self): """Converts the user input messages to a list of dictionaries.""" return [m.model_dump() for m in self.user_input] ``` ### pretty_repr ```python pretty_repr() ``` Returns a pretty string representation of the conversation. Source code in `src/ragas/dataset_schema.py` ```python def pretty_repr(self): """Returns a pretty string representation of the conversation.""" lines = [] for m in self.user_input: lines.append(m.pretty_repr()) return "\n".join(lines) ``` ## RagasDataset ```python RagasDataset(samples: List[Sample]) ``` Bases: `ABC`, `Generic[Sample]` ### to_list ```python to_list() -> List[Dict] ``` Converts the samples to a list of dictionaries. Source code in `src/ragas/dataset_schema.py` ```python @abstractmethod def to_list(self) -> t.List[t.Dict]: """Converts the samples to a list of dictionaries.""" pass ``` ### from_list ```python from_list(data: List[Dict]) -> T ``` Creates an RagasDataset from a list of dictionaries. Source code in `src/ragas/dataset_schema.py` ```python @classmethod @abstractmethod def from_list(cls: t.Type[T], data: t.List[t.Dict]) -> T: """Creates an RagasDataset from a list of dictionaries.""" pass ``` ### validate_samples ```python validate_samples(samples: List[Sample]) -> List[Sample] ``` Validates that all samples are of the same type. Source code in `src/ragas/dataset_schema.py` ```python def validate_samples(self, samples: t.List[Sample]) -> t.List[Sample]: """Validates that all samples are of the same type.""" if len(samples) == 0: return samples first_sample_type = type(samples[0]) for i, sample in enumerate(samples): if not isinstance(sample, first_sample_type): raise ValueError( f"Sample at index {i} is of type {type(sample)}, expected {first_sample_type}" ) return samples ``` ### get_sample_type ```python get_sample_type() -> Type[Sample] ``` Returns the type of the samples in the dataset. Source code in `src/ragas/dataset_schema.py` ```python def get_sample_type(self) -> t.Type[Sample]: """Returns the type of the samples in the dataset.""" return type(self.samples[0]) ``` ### to_hf_dataset ```python to_hf_dataset() -> Dataset ``` Converts the dataset to a Hugging Face Dataset. Source code in `src/ragas/dataset_schema.py` ```python def to_hf_dataset(self) -> HFDataset: """Converts the dataset to a Hugging Face Dataset.""" try: from datasets import Dataset as HFDataset except ImportError: raise ImportError( "datasets is not installed. Please install it to use this function." ) return HFDataset.from_list(self.to_list()) ``` ### from_hf_dataset ```python from_hf_dataset(dataset: Dataset) -> T ``` Creates an EvaluationDataset from a Hugging Face Dataset. Source code in `src/ragas/dataset_schema.py` ```python @classmethod def from_hf_dataset(cls: t.Type[T], dataset: HFDataset) -> T: """Creates an EvaluationDataset from a Hugging Face Dataset.""" return cls.from_list(dataset.to_list()) ``` ### to_pandas ```python to_pandas() -> DataFrame ``` Converts the dataset to a pandas DataFrame. Source code in `src/ragas/dataset_schema.py` ```python def to_pandas(self) -> PandasDataframe: """Converts the dataset to a pandas DataFrame.""" try: import pandas as pd except ImportError: raise ImportError( "pandas is not installed. Please install it to use this function." ) data = self.to_list() return pd.DataFrame(data) ``` ### from_pandas ```python from_pandas(dataframe: DataFrame) ``` Creates an EvaluationDataset from a pandas DataFrame. Source code in `src/ragas/dataset_schema.py` ```python @classmethod def from_pandas(cls, dataframe: PandasDataframe): """Creates an EvaluationDataset from a pandas DataFrame.""" return cls.from_list(dataframe.to_dict(orient="records")) ``` ### features ```python features() ``` Returns the features of the samples. Source code in `src/ragas/dataset_schema.py` ```python def features(self): """Returns the features of the samples.""" return self.samples[0].get_features() ``` ### from_dict ```python from_dict(mapping: Dict) -> T ``` Creates an EvaluationDataset from a dictionary. Source code in `src/ragas/dataset_schema.py` ```python @classmethod def from_dict(cls: t.Type[T], mapping: t.Dict) -> T: """Creates an EvaluationDataset from a dictionary.""" samples = [] if all( "user_input" in item and isinstance(mapping[0]["user_input"], list) for item in mapping ): samples.extend(MultiTurnSample(**sample) for sample in mapping) else: samples.extend(SingleTurnSample(**sample) for sample in mapping) return cls(samples=samples) ``` ### to_csv ```python to_csv(path: Union[str, Path]) ``` Converts the dataset to a CSV file. Source code in `src/ragas/dataset_schema.py` ```python def to_csv(self, path: t.Union[str, Path]): """Converts the dataset to a CSV file.""" import csv data = self.to_list() if not data: return fieldnames = data[0].keys() with open(path, "w", newline="") as csvfile: writer = csv.DictWriter(csvfile, fieldnames=fieldnames) writer.writeheader() for row in data: writer.writerow(row) ``` ### to_jsonl ```python to_jsonl(path: Union[str, Path]) ``` Converts the dataset to a JSONL file. Source code in `src/ragas/dataset_schema.py` ```python def to_jsonl(self, path: t.Union[str, Path]): """Converts the dataset to a JSONL file.""" with open(path, "w") as jsonlfile: for sample in self.to_list(): jsonlfile.write(json.dumps(sample, ensure_ascii=False) + "\n") ``` ### from_jsonl ```python from_jsonl(path: Union[str, Path]) -> T ``` Creates an EvaluationDataset from a JSONL file. Source code in `src/ragas/dataset_schema.py` ```python @classmethod def from_jsonl(cls: t.Type[T], path: t.Union[str, Path]) -> T: """Creates an EvaluationDataset from a JSONL file.""" with open(path, "r") as jsonlfile: data = [json.loads(line) for line in jsonlfile] return cls.from_list(data) ``` ## EvaluationDataset ```python EvaluationDataset(samples: List[Sample], backend: Optional[str] = None, name: Optional[str] = None) ``` Bases: `RagasDataset[SingleTurnSampleOrMultiTurnSample]` Represents a dataset of evaluation samples. Attributes: | Name | Type | Description | | --------- | ------------------ | -------------------------------------------------------------------------------- | | `samples` | `List[BaseSample]` | A list of evaluation samples. | | `backend` | `Optional[str]` | The backend to use for storing the dataset (e.g., "local/csv"). Default is None. | | `name` | `Optional[str]` | The name of the dataset. Default is None. | Methods: | Name | Description | | ------------------ | --------------------------------------------------------- | | `validate_samples` | Validates that all samples are of the same type. | | `get_sample_type` | Returns the type of the samples in the dataset. | | `to_hf_dataset` | Converts the dataset to a Hugging Face Dataset. | | `to_pandas` | Converts the dataset to a pandas DataFrame. | | `features` | Returns the features of the samples. | | `from_list` | Creates an EvaluationDataset from a list of dictionaries. | | `from_dict` | Creates an EvaluationDataset from a dictionary. | | `to_csv` | Converts the dataset to a CSV file. | | `to_jsonl` | Converts the dataset to a JSONL file. | | `from_jsonl` | Creates an EvaluationDataset from a JSONL file. | ## EvaluationResult ```python EvaluationResult(scores: List[Dict[str, Any]], dataset: EvaluationDataset, binary_columns: List[str] = list(), cost_cb: Optional[CostCallbackHandler] = None, traces: List[Dict[str, Any]] = list(), ragas_traces: Dict[str, ChainRun] = dict(), run_id: Optional[UUID] = None) ``` A class to store and process the results of the evaluation. Attributes: | Name | Type | Description | | ---------------- | --------------------------------- | ------------------------------------------------------------------ | | `scores` | `Dataset` | The dataset containing the scores of the evaluation. | | `dataset` | `(Dataset, optional)` | The original dataset used for the evaluation. Default is None. | | `binary_columns` | `list of str, optional` | List of columns that are binary metrics. Default is an empty list. | | `cost_cb` | `(CostCallbackHandler, optional)` | The callback handler for cost computation. Default is None. | ### to_pandas ```python to_pandas(batch_size: int | None = None, batched: bool = False) ``` Convert the result to a pandas DataFrame. Parameters: | Name | Type | Description | Default | | ------------ | ------ | ------------------------------------------------ | ------- | | `batch_size` | `int` | The batch size for conversion. Default is None. | `None` | | `batched` | `bool` | Whether to convert in batches. Default is False. | `False` | Returns: | Type | Description | | ----------- | --------------------------------- | | `DataFrame` | The result as a pandas DataFrame. | Raises: | Type | Description | | ------------ | ------------------------------- | | `ValueError` | If the dataset is not provided. | Source code in `src/ragas/dataset_schema.py` ```python def to_pandas(self, batch_size: int | None = None, batched: bool = False): """ Convert the result to a pandas DataFrame. Parameters ---------- batch_size : int, optional The batch size for conversion. Default is None. batched : bool, optional Whether to convert in batches. Default is False. Returns ------- pandas.DataFrame The result as a pandas DataFrame. Raises ------ ValueError If the dataset is not provided. """ try: import pandas as pd except ImportError: raise ImportError( "pandas is not installed. Please install it to use this function." ) if self.dataset is None: raise ValueError("dataset is not provided for the results class") assert len(self.scores) == len(self.dataset) # convert both to pandas dataframes and concatenate scores_df = pd.DataFrame(self.scores) dataset_df = self.dataset.to_pandas() return pd.concat([dataset_df, scores_df], axis=1) ``` ### total_tokens ```python total_tokens() -> Union[List[TokenUsage], TokenUsage] ``` Compute the total tokens used in the evaluation. Returns: | Type | Description | | ---------------------------------- | ---------------------- | | `list of TokenUsage or TokenUsage` | The total tokens used. | Raises: | Type | Description | | ------------ | --------------------------------------------- | | `ValueError` | If the cost callback handler is not provided. | Source code in `src/ragas/dataset_schema.py` ```python def total_tokens(self) -> t.Union[t.List[TokenUsage], TokenUsage]: """ Compute the total tokens used in the evaluation. Returns ------- list of TokenUsage or TokenUsage The total tokens used. Raises ------ ValueError If the cost callback handler is not provided. """ if self.cost_cb is None: raise ValueError( "The evaluate() run was not configured for computing cost. Please provide a token_usage_parser function to evaluate() to compute cost." ) return self.cost_cb.total_tokens() ``` ### total_cost ```python total_cost(cost_per_input_token: Optional[float] = None, cost_per_output_token: Optional[float] = None, per_model_costs: Dict[str, Tuple[float, float]] = {}) -> float ``` Compute the total cost of the evaluation. Parameters: | Name | Type | Description | Default | | ----------------------- | ------------------------------- | ---------------------------------------------------- | ------- | | `cost_per_input_token` | `float` | The cost per input token. Default is None. | `None` | | `cost_per_output_token` | `float` | The cost per output token. Default is None. | `None` | | `per_model_costs` | `dict of str to tuple of float` | The per model costs. Default is an empty dictionary. | `{}` | Returns: | Type | Description | | ------- | --------------------------------- | | `float` | The total cost of the evaluation. | Raises: | Type | Description | | ------------ | --------------------------------------------- | | `ValueError` | If the cost callback handler is not provided. | Source code in `src/ragas/dataset_schema.py` ```python def total_cost( self, cost_per_input_token: t.Optional[float] = None, cost_per_output_token: t.Optional[float] = None, per_model_costs: t.Dict[str, t.Tuple[float, float]] = {}, ) -> float: """ Compute the total cost of the evaluation. Parameters ---------- cost_per_input_token : float, optional The cost per input token. Default is None. cost_per_output_token : float, optional The cost per output token. Default is None. per_model_costs : dict of str to tuple of float, optional The per model costs. Default is an empty dictionary. Returns ------- float The total cost of the evaluation. Raises ------ ValueError If the cost callback handler is not provided. """ if self.cost_cb is None: raise ValueError( "The evaluate() run was not configured for computing cost. Please provide a token_usage_parser function to evaluate() to compute cost." ) return self.cost_cb.total_cost( cost_per_input_token, cost_per_output_token, per_model_costs ) ``` ## MetricAnnotation Bases: `BaseModel` ### from_json ```python from_json(path: str, metric_name: Optional[str]) -> 'MetricAnnotation' ``` Load annotations from a JSON file Source code in `src/ragas/dataset_schema.py` ```python @classmethod def from_json(cls, path: str, metric_name: t.Optional[str]) -> "MetricAnnotation": """Load annotations from a JSON file""" dataset = json.load(open(path)) return cls._process_dataset(dataset, metric_name) ``` ## SingleMetricAnnotation Bases: `BaseModel` ### train_test_split ```python train_test_split(test_size: float = 0.2, seed: int = 42, stratify: Optional[List[Any]] = None) -> Tuple['SingleMetricAnnotation', 'SingleMetricAnnotation'] ``` Split the dataset into training and testing sets. Parameters: test_size (float): The proportion of the dataset to include in the test split. seed (int): Random seed for reproducibility. stratify (list): The column values to stratify the split on. Source code in `src/ragas/dataset_schema.py` ```python def train_test_split( self, test_size: float = 0.2, seed: int = 42, stratify: t.Optional[t.List[t.Any]] = None, ) -> t.Tuple["SingleMetricAnnotation", "SingleMetricAnnotation"]: """ Split the dataset into training and testing sets. Parameters: test_size (float): The proportion of the dataset to include in the test split. seed (int): Random seed for reproducibility. stratify (list): The column values to stratify the split on. """ raise NotImplementedError ``` ### sample ```python sample(n: int, stratify_key: Optional[str] = None) -> 'SingleMetricAnnotation' ``` Create a subset of the dataset. Parameters: n (int): The number of samples to include in the subset. stratify_key (str): The column to stratify the subset on. Returns: SingleMetricAnnotation: A subset of the dataset with `n` samples. Source code in `src/ragas/dataset_schema.py` ```python def sample( self, n: int, stratify_key: t.Optional[str] = None ) -> "SingleMetricAnnotation": """ Create a subset of the dataset. Parameters: n (int): The number of samples to include in the subset. stratify_key (str): The column to stratify the subset on. Returns: SingleMetricAnnotation: A subset of the dataset with `n` samples. """ if n > len(self.samples): raise ValueError( "Requested sample size exceeds the number of available samples." ) if stratify_key is None: # Simple random sampling sampled_indices = random.sample(range(len(self.samples)), n) sampled_samples = [self.samples[i] for i in sampled_indices] else: # Stratified sampling class_groups = defaultdict(list) for idx, sample in enumerate(self.samples): key = sample[stratify_key] class_groups[key].append(idx) # Determine the proportion of samples to take from each class total_samples = sum(len(indices) for indices in class_groups.values()) proportions = { cls: len(indices) / total_samples for cls, indices in class_groups.items() } sampled_indices = [] for cls, indices in class_groups.items(): cls_sample_count = int(np.round(proportions[cls] * n)) cls_sample_count = min( cls_sample_count, len(indices) ) # Don't oversample sampled_indices.extend(random.sample(indices, cls_sample_count)) # Handle any rounding discrepancies to ensure exactly `n` samples while len(sampled_indices) < n: remaining_indices = set(range(len(self.samples))) - set(sampled_indices) if not remaining_indices: break sampled_indices.append(random.choice(list(remaining_indices))) sampled_samples = [self.samples[i] for i in sampled_indices] return SingleMetricAnnotation(name=self.name, samples=sampled_samples) ``` ### batch ```python batch(batch_size: int, drop_last_batch: bool = False) ``` Create a batch iterator. Parameters: batch_size (int): The number of samples in each batch. stratify (str): The column to stratify the batches on. drop_last_batch (bool): Whether to drop the last batch if it is smaller than the specified batch size. Source code in `src/ragas/dataset_schema.py` ```python def batch( self, batch_size: int, drop_last_batch: bool = False, ): """ Create a batch iterator. Parameters: batch_size (int): The number of samples in each batch. stratify (str): The column to stratify the batches on. drop_last_batch (bool): Whether to drop the last batch if it is smaller than the specified batch size. """ samples = self.samples[:] random.shuffle(samples) all_batches = [ samples[i : i + batch_size] for i in range(0, len(samples), batch_size) if len(samples[i : i + batch_size]) == batch_size or not drop_last_batch ] return all_batches ``` ### stratified_batches ```python stratified_batches(batch_size: int, stratify_key: str, drop_last_batch: bool = False, replace: bool = False) -> List[List[SampleAnnotation]] ``` Create stratified batches based on a specified key, ensuring proportional representation. Parameters: batch_size (int): Number of samples per batch. stratify_key (str): Key in `metric_input` used for stratification (e.g., class labels). drop_last_batch (bool): If True, drops the last batch if it has fewer samples than `batch_size`. replace (bool): If True, allows reusing samples from the same class to fill a batch if necessary. Returns: List\[List[SampleAnnotation]\]: A list of stratified batches, each batch being a list of SampleAnnotation objects. Source code in `src/ragas/dataset_schema.py` ```python def stratified_batches( self, batch_size: int, stratify_key: str, drop_last_batch: bool = False, replace: bool = False, ) -> t.List[t.List[SampleAnnotation]]: """ Create stratified batches based on a specified key, ensuring proportional representation. Parameters: batch_size (int): Number of samples per batch. stratify_key (str): Key in `metric_input` used for stratification (e.g., class labels). drop_last_batch (bool): If True, drops the last batch if it has fewer samples than `batch_size`. replace (bool): If True, allows reusing samples from the same class to fill a batch if necessary. Returns: List[List[SampleAnnotation]]: A list of stratified batches, each batch being a list of SampleAnnotation objects. """ # Group samples based on the stratification key class_groups = defaultdict(list) for sample in self.samples: key = sample[stratify_key] class_groups[key].append(sample) # Shuffle each class group for randomness for group in class_groups.values(): random.shuffle(group) # Determine the number of batches required total_samples = len(self.samples) num_batches = ( np.ceil(total_samples / batch_size).astype(int) if drop_last_batch else np.floor(total_samples / batch_size).astype(int) ) samples_per_class_per_batch = { cls: max(1, len(samples) // num_batches) for cls, samples in class_groups.items() } # Create stratified batches all_batches = [] while len(all_batches) < num_batches: batch = [] for cls, samples in list(class_groups.items()): # Determine the number of samples to take from this class count = min( samples_per_class_per_batch[cls], len(samples), batch_size - len(batch), ) if count > 0: # Add samples from the current class batch.extend(samples[:count]) class_groups[cls] = samples[count:] # Remove used samples elif replace and len(batch) < batch_size: # Reuse samples if `replace` is True batch.extend(random.choices(samples, k=batch_size - len(batch))) # Shuffle the batch to mix classes random.shuffle(batch) if len(batch) == batch_size or not drop_last_batch: all_batches.append(batch) return all_batches ``` ### get_prompt_annotations ```python get_prompt_annotations() -> Dict[str, List[PromptAnnotation]] ``` Get all the prompt annotations for each prompt as a list. Source code in `src/ragas/dataset_schema.py` ```python def get_prompt_annotations(self) -> t.Dict[str, t.List[PromptAnnotation]]: """ Get all the prompt annotations for each prompt as a list. """ prompt_annotations = defaultdict(list) for sample in self.samples: if sample.is_accepted: for prompt_name, prompt_annotation in sample.prompts.items(): prompt_annotations[prompt_name].append(prompt_annotation) return prompt_annotations ``` ## Message Bases: `BaseModel` Represents a generic message. Attributes: | Name | Type | Description | | ---------- | -------------------------------------- | ------------------------------------------------ | | `content` | `str` | The content of the message. | | `metadata` | `(Optional[Dict[str, Any]], optional)` | Additional metadata associated with the message. | ## ToolCall Bases: `BaseModel` Represents a tool call with a name and arguments. Parameters: | Name | Type | Description | Default | | ------ | ---------------- | -------------------------------------------------------------------------------------------------------------------------- | ---------- | | `name` | `str` | The name of the tool being called. | *required* | | `args` | `Dict[str, Any]` | A dictionary of arguments for the tool call, where keys are argument names and values can be strings, integers, or floats. | *required* | ## HumanMessage Bases: `Message` Represents a message from a human user. Attributes: | Name | Type | Description | | ------ | ---------------- | ----------------------------------------------- | | `type` | `Literal[human]` | The type of the message, always set to "human". | Methods: | Name | Description | | ------------- | --------------------------------------------------------------- | | `pretty_repr` | Returns a formatted string representation of the human message. | ### pretty_repr ```python pretty_repr() ``` Returns a formatted string representation of the human message. Source code in `src/ragas/messages.py` ```python def pretty_repr(self): """Returns a formatted string representation of the human message.""" return f"Human: {self.content}" ``` ## ToolMessage Bases: `Message` Represents a message from a tool. Attributes: | Name | Type | Description | | ------ | --------------- | ---------------------------------------------- | | `type` | `Literal[tool]` | The type of the message, always set to "tool". | Methods: | Name | Description | | ------------- | -------------------------------------------------------------- | | `pretty_repr` | Returns a formatted string representation of the tool message. | ### pretty_repr ```python pretty_repr() ``` Returns a formatted string representation of the tool message. Source code in `src/ragas/messages.py` ```python def pretty_repr(self): """Returns a formatted string representation of the tool message.""" return f"ToolOutput: {self.content}" ``` ## AIMessage Bases: `Message` Represents a message from an AI. Attributes: | Name | Type | Description | | ------------ | -------------------------- | --------------------------------------------------- | | `type` | `Literal[ai]` | The type of the message, always set to "ai". | | `tool_calls` | `Optional[List[ToolCall]]` | A list of tool calls made by the AI, if any. | | `metadata` | `Optional[Dict[str, Any]]` | Additional metadata associated with the AI message. | Methods: | Name | Description | | ------------- | ------------------------------------------------------------ | | `dict` | Returns a dictionary representation of the AI message. | | `pretty_repr` | Returns a formatted string representation of the AI message. | ### to_dict ```python to_dict(**kwargs) ``` Returns a dictionary representation of the AI message. Source code in `src/ragas/messages.py` ```python def to_dict(self, **kwargs): """ Returns a dictionary representation of the AI message. """ content = ( self.content if self.tool_calls is None else { "text": self.content, "tool_calls": [tc.dict() for tc in self.tool_calls], } ) return {"content": content, "type": self.type} ``` ### pretty_repr ```python pretty_repr() ``` Returns a formatted string representation of the AI message. Source code in `src/ragas/messages.py` ```python def pretty_repr(self): """ Returns a formatted string representation of the AI message. """ lines = [] if self.content != "": lines.append(f"AI: {self.content}") if self.tool_calls is not None: lines.append("Tools:") for tc in self.tool_calls: lines.append(f" {tc.name}: {tc.args}") return "\n".join(lines) ``` ## ragas.evaluation.EvaluationResult ```python EvaluationResult(scores: List[Dict[str, Any]], dataset: EvaluationDataset, binary_columns: List[str] = list(), cost_cb: Optional[CostCallbackHandler] = None, traces: List[Dict[str, Any]] = list(), ragas_traces: Dict[str, ChainRun] = dict(), run_id: Optional[UUID] = None) ``` A class to store and process the results of the evaluation. Attributes: | Name | Type | Description | | ---------------- | --------------------------------- | ------------------------------------------------------------------ | | `scores` | `Dataset` | The dataset containing the scores of the evaluation. | | `dataset` | `(Dataset, optional)` | The original dataset used for the evaluation. Default is None. | | `binary_columns` | `list of str, optional` | List of columns that are binary metrics. Default is an empty list. | | `cost_cb` | `(CostCallbackHandler, optional)` | The callback handler for cost computation. Default is None. | ### to_pandas ```python to_pandas(batch_size: int | None = None, batched: bool = False) ``` Convert the result to a pandas DataFrame. Parameters: | Name | Type | Description | Default | | ------------ | ------ | ------------------------------------------------ | ------- | | `batch_size` | `int` | The batch size for conversion. Default is None. | `None` | | `batched` | `bool` | Whether to convert in batches. Default is False. | `False` | Returns: | Type | Description | | ----------- | --------------------------------- | | `DataFrame` | The result as a pandas DataFrame. | Raises: | Type | Description | | ------------ | ------------------------------- | | `ValueError` | If the dataset is not provided. | Source code in `src/ragas/dataset_schema.py` ```python def to_pandas(self, batch_size: int | None = None, batched: bool = False): """ Convert the result to a pandas DataFrame. Parameters ---------- batch_size : int, optional The batch size for conversion. Default is None. batched : bool, optional Whether to convert in batches. Default is False. Returns ------- pandas.DataFrame The result as a pandas DataFrame. Raises ------ ValueError If the dataset is not provided. """ try: import pandas as pd except ImportError: raise ImportError( "pandas is not installed. Please install it to use this function." ) if self.dataset is None: raise ValueError("dataset is not provided for the results class") assert len(self.scores) == len(self.dataset) # convert both to pandas dataframes and concatenate scores_df = pd.DataFrame(self.scores) dataset_df = self.dataset.to_pandas() return pd.concat([dataset_df, scores_df], axis=1) ``` ### total_tokens ```python total_tokens() -> Union[List[TokenUsage], TokenUsage] ``` Compute the total tokens used in the evaluation. Returns: | Type | Description | | ---------------------------------- | ---------------------- | | `list of TokenUsage or TokenUsage` | The total tokens used. | Raises: | Type | Description | | ------------ | --------------------------------------------- | | `ValueError` | If the cost callback handler is not provided. | Source code in `src/ragas/dataset_schema.py` ```python def total_tokens(self) -> t.Union[t.List[TokenUsage], TokenUsage]: """ Compute the total tokens used in the evaluation. Returns ------- list of TokenUsage or TokenUsage The total tokens used. Raises ------ ValueError If the cost callback handler is not provided. """ if self.cost_cb is None: raise ValueError( "The evaluate() run was not configured for computing cost. Please provide a token_usage_parser function to evaluate() to compute cost." ) return self.cost_cb.total_tokens() ``` ### total_cost ```python total_cost(cost_per_input_token: Optional[float] = None, cost_per_output_token: Optional[float] = None, per_model_costs: Dict[str, Tuple[float, float]] = {}) -> float ``` Compute the total cost of the evaluation. Parameters: | Name | Type | Description | Default | | ----------------------- | ------------------------------- | ---------------------------------------------------- | ------- | | `cost_per_input_token` | `float` | The cost per input token. Default is None. | `None` | | `cost_per_output_token` | `float` | The cost per output token. Default is None. | `None` | | `per_model_costs` | `dict of str to tuple of float` | The per model costs. Default is an empty dictionary. | `{}` | Returns: | Type | Description | | ------- | --------------------------------- | | `float` | The total cost of the evaluation. | Raises: | Type | Description | | ------------ | --------------------------------------------- | | `ValueError` | If the cost callback handler is not provided. | Source code in `src/ragas/dataset_schema.py` ```python def total_cost( self, cost_per_input_token: t.Optional[float] = None, cost_per_output_token: t.Optional[float] = None, per_model_costs: t.Dict[str, t.Tuple[float, float]] = {}, ) -> float: """ Compute the total cost of the evaluation. Parameters ---------- cost_per_input_token : float, optional The cost per input token. Default is None. cost_per_output_token : float, optional The cost per output token. Default is None. per_model_costs : dict of str to tuple of float, optional The per model costs. Default is an empty dictionary. Returns ------- float The total cost of the evaluation. Raises ------ ValueError If the cost callback handler is not provided. """ if self.cost_cb is None: raise ValueError( "The evaluate() run was not configured for computing cost. Please provide a token_usage_parser function to evaluate() to compute cost." ) return self.cost_cb.total_cost( cost_per_input_token, cost_per_output_token, per_model_costs ) ``` ## Executor ```python Executor(desc: str = 'Evaluating', show_progress: bool = True, keep_progress_bar: bool = True, jobs: List[Any] = list(), raise_exceptions: bool = False, batch_size: Optional[int] = None, run_config: Optional[RunConfig] = None, pbar: Optional[tqdm] = None, _jobs_processed: int = 0, _cancel_event: Event = Event()) ``` Executor class for running asynchronous jobs with progress tracking and error handling. Attributes: | Name | Type | Description | | ----------------------- | ----------- | ------------------------------------------------- | | `desc` | `str` | Description for the progress bar | | `show_progress` | `bool` | Whether to show the progress bar | | `keep_progress_bar` | `bool` | Whether to keep the progress bar after completion | | `jobs` | `List[Any]` | List of jobs to execute | | `raise_exceptions` | `bool` | Whether to raise exceptions or log them | | `batch_size` | `int` | Whether to batch (large) lists of tasks | | `run_config` | `RunConfig` | Configuration for the run | | `_nest_asyncio_applied` | `bool` | Whether nest_asyncio has been applied | | `_cancel_event` | `Event` | Event to signal cancellation | ### cancel ```python cancel() -> None ``` Cancel the execution of all jobs. Source code in `src/ragas/executor.py` ```python def cancel(self) -> None: """Cancel the execution of all jobs.""" self._cancel_event.set() ``` ### is_cancelled ```python is_cancelled() -> bool ``` Check if the execution has been cancelled. Source code in `src/ragas/executor.py` ```python def is_cancelled(self) -> bool: """Check if the execution has been cancelled.""" return self._cancel_event.is_set() ``` ### submit ```python submit(callable: Callable, *args, name: Optional[str] = None, **kwargs) -> None ``` Submit a job to be executed, wrapping the callable with error handling and indexing to keep track of the job index. Source code in `src/ragas/executor.py` ```python def submit( self, callable: t.Callable, *args, name: t.Optional[str] = None, **kwargs, ) -> None: """ Submit a job to be executed, wrapping the callable with error handling and indexing to keep track of the job index. """ # Use _jobs_processed for consistent indexing across multiple runs callable_with_index = self.wrap_callable_with_index( callable, self._jobs_processed ) self.jobs.append((callable_with_index, args, kwargs, name)) self._jobs_processed += 1 ``` ### clear_jobs ```python clear_jobs() -> None ``` Clear all submitted jobs and reset counter. Source code in `src/ragas/executor.py` ```python def clear_jobs(self) -> None: """Clear all submitted jobs and reset counter.""" self.jobs.clear() self._jobs_processed = 0 ``` ### aresults ```python aresults() -> List[Any] ``` Execute all submitted jobs and return their results asynchronously. The results are returned in the order of job submission. This is the async entry point for executing async jobs when already in an async context. Source code in `src/ragas/executor.py` ```python async def aresults(self) -> t.List[t.Any]: """ Execute all submitted jobs and return their results asynchronously. The results are returned in the order of job submission. This is the async entry point for executing async jobs when already in an async context. """ results = await self._process_jobs() sorted_results = sorted(results, key=lambda x: x[0]) return [r[1] for r in sorted_results] ``` ### results ```python results() -> List[Any] ``` Execute all submitted jobs and return their results. The results are returned in the order of job submission. This is the main sync entry point for executing async jobs. Source code in `src/ragas/executor.py` ```python def results(self) -> t.List[t.Any]: """ Execute all submitted jobs and return their results. The results are returned in the order of job submission. This is the main sync entry point for executing async jobs. """ async def _async_wrapper(): return await self.aresults() apply_nest_asyncio() return run(_async_wrapper) ``` ## run_async_batch ```python run_async_batch(desc: str, func: Callable, kwargs_list: List[Dict], batch_size: Optional[int] = None) ``` Provide functionality to run the same async function with different arguments in parallel. Source code in `src/ragas/executor.py` ```python def run_async_batch( desc: str, func: t.Callable, kwargs_list: t.List[t.Dict], batch_size: t.Optional[int] = None, ): """ Provide functionality to run the same async function with different arguments in parallel. """ run_config = RunConfig() executor = Executor( desc=desc, keep_progress_bar=False, raise_exceptions=True, run_config=run_config, batch_size=batch_size, ) for kwargs in kwargs_list: executor.submit(func, **kwargs) return executor.results() ``` ## TestsetGenerator ```python TestsetGenerator(llm: BaseRagasLLM, embedding_model: BaseRagasEmbeddings, knowledge_graph: KnowledgeGraph = KnowledgeGraph(), persona_list: Optional[List[Persona]] = None, llm_context: Optional[str] = None) ``` Generates an evaluation dataset based on given scenarios and parameters. Attributes: | Name | Type | Description | | ----------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | | `llm` | `BaseRagasLLM` | The language model to use for the generation process. | | `knowledge_graph` | `KnowledgeGraph, default empty` | The knowledge graph to use for the generation process. | | `llm_context` | `Optional[str], default None` | Additional context to provide to the LLM when generating responses. This context will be used to guide how the LLM generates queries and answers. | ### from_langchain ```python from_langchain(llm: BaseLanguageModel, embedding_model: Embeddings, knowledge_graph: Optional[KnowledgeGraph] = None, llm_context: Optional[str] = None) -> TestsetGenerator ``` Creates a `TestsetGenerator` from a Langchain LLMs. Source code in `src/ragas/testset/synthesizers/generate.py` ```python @classmethod def from_langchain( cls, llm: LangchainLLM, embedding_model: LangchainEmbeddings, knowledge_graph: t.Optional[KnowledgeGraph] = None, llm_context: t.Optional[str] = None, ) -> TestsetGenerator: """ Creates a `TestsetGenerator` from a Langchain LLMs. """ knowledge_graph = knowledge_graph or KnowledgeGraph() return cls( LangchainLLMWrapper(llm), LangchainEmbeddingsWrapper(embedding_model), knowledge_graph, llm_context=llm_context, ) ``` ### from_llama_index ```python from_llama_index(llm: BaseLLM, embedding_model: BaseEmbedding, knowledge_graph: Optional[KnowledgeGraph] = None, llm_context: Optional[str] = None) -> TestsetGenerator ``` Creates a `TestsetGenerator` from a LlamaIndex LLM and embedding model. Source code in `src/ragas/testset/synthesizers/generate.py` ```python @classmethod def from_llama_index( cls, llm: LlamaIndexLLM, embedding_model: LlamaIndexEmbedding, knowledge_graph: t.Optional[KnowledgeGraph] = None, llm_context: t.Optional[str] = None, ) -> TestsetGenerator: """ Creates a `TestsetGenerator` from a LlamaIndex LLM and embedding model. """ knowledge_graph = knowledge_graph or KnowledgeGraph() return cls( LlamaIndexLLMWrapper(llm), LlamaIndexEmbeddingsWrapper(embedding_model), knowledge_graph, llm_context=llm_context, ) ``` ### generate_with_langchain_docs ```python generate_with_langchain_docs(documents: Sequence[Document], testset_size: int, transforms: Optional[Transforms] = None, transforms_llm: Optional[BaseRagasLLM] = None, transforms_embedding_model: Optional[BaseRagasEmbeddings] = None, query_distribution: Optional[QueryDistribution] = None, run_config: Optional[RunConfig] = None, callbacks: Optional[Callbacks] = None, token_usage_parser: Optional[TokenUsageParser] = None, with_debugging_logs=False, raise_exceptions: bool = True, return_executor: bool = False) -> Union[Testset, Executor] ``` Generates an evaluation dataset based on given Langchain documents and parameters. Parameters: | Name | Type | Description | Default | | ---------------------------- | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `documents` | `Sequence[Document]` | A sequence of Langchain documents to use as source material | *required* | | `testset_size` | `int` | The number of test samples to generate | *required* | | `transforms` | `Optional[Transforms]` | Custom transforms to apply to the documents, by default None | `None` | | `transforms_llm` | `Optional[BaseRagasLLM]` | LLM to use for transforms if different from instance LLM, by default None | `None` | | `transforms_embedding_model` | `Optional[BaseRagasEmbeddings]` | Embedding model to use for transforms if different from instance model, by default None | `None` | | `query_distribution` | `Optional[QueryDistribution]` | Distribution of query types to generate, by default None | `None` | | `run_config` | `Optional[RunConfig]` | Configuration for the generation run, by default None | `None` | | `callbacks` | `Optional[Callbacks]` | Callbacks to use during generation, by default None | `None` | | `token_usage_parser` | `Optional[TokenUsageParser]` | Parse the LLMResult object and return a TokenUsage object. This is used to calculate the cost of the generation process. | `None` | | `with_debugging_logs` | `bool` | Whether to include debug logs, by default False | `False` | | `raise_exceptions` | `bool` | Whether to raise exceptions during generation, by default True | `True` | | `return_executor` | `bool` | If True, returns the Executor instance instead of running generation. The returned executor can be used to cancel execution by calling executor.cancel(). To get results, call executor.results(). Default is False. | `False` | Returns: | Type | Description | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Testset or Executor` | If return_executor is False, returns the generated evaluation dataset. If return_executor is True, returns the Executor instance for cancellable execution. | Raises: | Type | Description | | ------------ | ------------------------------------------------------------------------------------- | | `ValueError` | If no LLM or embedding model is provided either during initialization or as arguments | Source code in `src/ragas/testset/synthesizers/generate.py` ```python def generate_with_langchain_docs( self, documents: t.Sequence[LCDocument], testset_size: int, transforms: t.Optional[Transforms] = None, transforms_llm: t.Optional[BaseRagasLLM] = None, transforms_embedding_model: t.Optional[BaseRagasEmbeddings] = None, query_distribution: t.Optional[QueryDistribution] = None, run_config: t.Optional[RunConfig] = None, callbacks: t.Optional[Callbacks] = None, token_usage_parser: t.Optional[TokenUsageParser] = None, with_debugging_logs=False, raise_exceptions: bool = True, return_executor: bool = False, ) -> t.Union[Testset, Executor]: """ Generates an evaluation dataset based on given Langchain documents and parameters. Parameters ---------- documents : Sequence[LCDocument] A sequence of Langchain documents to use as source material testset_size : int The number of test samples to generate transforms : Optional[Transforms], optional Custom transforms to apply to the documents, by default None transforms_llm : Optional[BaseRagasLLM], optional LLM to use for transforms if different from instance LLM, by default None transforms_embedding_model : Optional[BaseRagasEmbeddings], optional Embedding model to use for transforms if different from instance model, by default None query_distribution : Optional[QueryDistribution], optional Distribution of query types to generate, by default None run_config : Optional[RunConfig], optional Configuration for the generation run, by default None callbacks : Optional[Callbacks], optional Callbacks to use during generation, by default None token_usage_parser : Optional[TokenUsageParser], optional Parse the LLMResult object and return a TokenUsage object. This is used to calculate the cost of the generation process. with_debugging_logs : bool, optional Whether to include debug logs, by default False raise_exceptions : bool, optional Whether to raise exceptions during generation, by default True return_executor : bool, optional If True, returns the Executor instance instead of running generation. The returned executor can be used to cancel execution by calling executor.cancel(). To get results, call executor.results(). Default is False. Returns ------- Testset or Executor If return_executor is False, returns the generated evaluation dataset. If return_executor is True, returns the Executor instance for cancellable execution. Raises ------ ValueError If no LLM or embedding model is provided either during initialization or as arguments """ # force the user to provide an llm and embedding client to prevent use of default LLMs if not self.llm and not transforms_llm: raise ValueError( """An llm client was not provided. Provide an LLM on TestsetGenerator instantiation or as an argument for transforms_llm parameter. Alternatively you can provide your own transforms through the `transforms` parameter.""" ) if not self.embedding_model and not transforms_embedding_model: raise ValueError( """An embedding client was not provided. Provide an embedding through the transforms_embedding_model parameter. Alternatively you can provide your own transforms through the `transforms` parameter.""" ) if not transforms: transforms = default_transforms( documents=list(documents), llm=transforms_llm or self.llm, embedding_model=transforms_embedding_model or self.embedding_model, ) # convert the documents to Ragas nodes nodes = [] for doc in documents: node = Node( type=NodeType.DOCUMENT, properties={ "page_content": doc.page_content, "document_metadata": doc.metadata, }, ) nodes.append(node) kg = KnowledgeGraph(nodes=nodes) # apply transforms and update the knowledge graph apply_transforms(kg, transforms, run_config=run_config or RunConfig()) self.knowledge_graph = kg return self.generate( testset_size=testset_size, query_distribution=query_distribution, run_config=run_config, callbacks=callbacks, token_usage_parser=token_usage_parser, with_debugging_logs=with_debugging_logs, raise_exceptions=raise_exceptions, return_executor=return_executor, ) ``` ### generate_with_llamaindex_docs ```python generate_with_llamaindex_docs(documents: Sequence[Document], testset_size: int, transforms: Optional[Transforms] = None, transforms_llm: Optional[BaseLLM] = None, transforms_embedding_model: Optional[BaseEmbedding] = None, query_distribution: Optional[QueryDistribution] = None, run_config: Optional[RunConfig] = None, callbacks: Optional[Callbacks] = None, token_usage_parser: Optional[TokenUsageParser] = None, with_debugging_logs=False, raise_exceptions: bool = True) ``` Generates an evaluation dataset based on given scenarios and parameters. Source code in `src/ragas/testset/synthesizers/generate.py` ```python def generate_with_llamaindex_docs( self, documents: t.Sequence[LlamaIndexDocument], testset_size: int, transforms: t.Optional[Transforms] = None, transforms_llm: t.Optional[LlamaIndexLLM] = None, transforms_embedding_model: t.Optional[LlamaIndexEmbedding] = None, query_distribution: t.Optional[QueryDistribution] = None, run_config: t.Optional[RunConfig] = None, callbacks: t.Optional[Callbacks] = None, token_usage_parser: t.Optional[TokenUsageParser] = None, with_debugging_logs=False, raise_exceptions: bool = True, ): """ Generates an evaluation dataset based on given scenarios and parameters. """ run_config = run_config or RunConfig() # force the user to provide an llm and embedding client to prevent use of default LLMs if not self.llm and not transforms_llm: raise ValueError( "An llm client was not provided. Provide an LLM on TestsetGenerator instantiation or as an argument for transforms_llm parameter. Alternatively you can provide your own transforms through the `transforms` parameter." ) if not self.embedding_model and not transforms_embedding_model: raise ValueError( "An embedding client was not provided. Provide an embedding through the transforms_embedding_model parameter. Alternatively you can provide your own transforms through the `transforms` parameter." ) if not transforms: # use TestsetGenerator's LLM and embedding model if no transforms_llm or transforms_embedding_model is provided if transforms_llm is None: llm_for_transforms = self.llm else: llm_for_transforms = LlamaIndexLLMWrapper(transforms_llm) if transforms_embedding_model is None: embedding_model_for_transforms = self.embedding_model else: embedding_model_for_transforms = LlamaIndexEmbeddingsWrapper( transforms_embedding_model ) # create the transforms transforms = default_transforms( documents=[LCDocument(page_content=doc.text) for doc in documents], llm=llm_for_transforms, embedding_model=embedding_model_for_transforms, ) # convert the documents to Ragas nodes nodes = [] for doc in documents: if doc.text is not None and doc.text.strip() != "": node = Node( type=NodeType.DOCUMENT, properties={ "page_content": doc.text, "document_metadata": doc.metadata, }, ) nodes.append(node) kg = KnowledgeGraph(nodes=nodes) # apply transforms and update the knowledge graph apply_transforms(kg, transforms, run_config) self.knowledge_graph = kg return self.generate( testset_size=testset_size, query_distribution=query_distribution, run_config=run_config, callbacks=callbacks, token_usage_parser=token_usage_parser, with_debugging_logs=with_debugging_logs, raise_exceptions=raise_exceptions, return_executor=False, # Default value for llamaindex_docs method ) ``` ### generate_with_chunks ```python generate_with_chunks(chunks: Sequence[Union[Document, str]], testset_size: int, transforms: Optional[Transforms] = None, transforms_llm: Optional[BaseRagasLLM] = None, transforms_embedding_model: Optional[BaseRagasEmbeddings] = None, query_distribution: Optional[QueryDistribution] = None, run_config: Optional[RunConfig] = None, callbacks: Optional[Callbacks] = None, token_usage_parser: Optional[TokenUsageParser] = None, with_debugging_logs=False, raise_exceptions: bool = True, return_executor: bool = False) -> Union[Testset, Executor] ``` Generates an evaluation dataset based on provided pre-chunked documents. This method allows users to skip the internal chunking process by providing documents that are already chunked. The input documents are treated as `NodeType.CHUNK` directly. Parameters: | Name | Type | Description | Default | | ---------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------- | | `chunks` | `Sequence[Union[Document, str]]` | A sequence of Langchain documents or strings to use as chunks. Strings will be automatically converted to Documents. | *required* | | `testset_size` | `int` | The number of test samples to generate | *required* | | `transforms` | `Optional[Transforms]` | Custom transforms to apply to the chunks, by default None | `None` | | `transforms_llm` | `Optional[BaseRagasLLM]` | LLM to use for transforms if different from instance LLM, by default None | `None` | | `transforms_embedding_model` | `Optional[BaseRagasEmbeddings]` | Embedding model to use for transforms if different from instance model, by default None | `None` | | `query_distribution` | `Optional[QueryDistribution]` | Distribution of query types to generate, by default None | `None` | | `run_config` | `Optional[RunConfig]` | Configuration for the generation run, by default None | `None` | | `callbacks` | `Optional[Callbacks]` | Callbacks to use during generation, by default None | `None` | | `token_usage_parser` | `Optional[TokenUsageParser]` | Parse the LLMResult object and return a TokenUsage object. | `None` | | `with_debugging_logs` | `bool` | Whether to include debug logs, by default False | `False` | | `raise_exceptions` | `bool` | Whether to raise exceptions during generation, by default True | `True` | | `return_executor` | `bool` | If True, returns the Executor instance instead of running generation. | `False` | Returns: | Type | Description | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------- | | `Testset or Executor` | If return_executor is False, returns the generated evaluation dataset. If return_executor is True, returns the Executor instance. | Source code in `src/ragas/testset/synthesizers/generate.py` ```python def generate_with_chunks( self, chunks: t.Sequence[t.Union[LCDocument, str]], testset_size: int, transforms: t.Optional[Transforms] = None, transforms_llm: t.Optional[BaseRagasLLM] = None, transforms_embedding_model: t.Optional[BaseRagasEmbeddings] = None, query_distribution: t.Optional[QueryDistribution] = None, run_config: t.Optional[RunConfig] = None, callbacks: t.Optional[Callbacks] = None, token_usage_parser: t.Optional[TokenUsageParser] = None, with_debugging_logs=False, raise_exceptions: bool = True, return_executor: bool = False, ) -> t.Union[Testset, Executor]: """ Generates an evaluation dataset based on provided pre-chunked documents. This method allows users to skip the internal chunking process by providing documents that are already chunked. The input documents are treated as `NodeType.CHUNK` directly. Parameters ---------- chunks : Sequence[Union[LCDocument, str]] A sequence of Langchain documents or strings to use as chunks. Strings will be automatically converted to Documents. testset_size : int The number of test samples to generate transforms : Optional[Transforms], optional Custom transforms to apply to the chunks, by default None transforms_llm : Optional[BaseRagasLLM], optional LLM to use for transforms if different from instance LLM, by default None transforms_embedding_model : Optional[BaseRagasEmbeddings], optional Embedding model to use for transforms if different from instance model, by default None query_distribution : Optional[QueryDistribution], optional Distribution of query types to generate, by default None run_config : Optional[RunConfig], optional Configuration for the generation run, by default None callbacks : Optional[Callbacks], optional Callbacks to use during generation, by default None token_usage_parser : Optional[TokenUsageParser], optional Parse the LLMResult object and return a TokenUsage object. with_debugging_logs : bool, optional Whether to include debug logs, by default False raise_exceptions : bool, optional Whether to raise exceptions during generation, by default True return_executor : bool, optional If True, returns the Executor instance instead of running generation. Returns ------- Testset or Executor If return_executor is False, returns the generated evaluation dataset. If return_executor is True, returns the Executor instance. """ # force the user to provide an llm and embedding client if not self.llm and not transforms_llm: raise ValueError( """An llm client was not provided. Provide an LLM on TestsetGenerator instantiation or as an argument for transforms_llm parameter. Alternatively you can provide your own transforms through the `transforms` parameter.""" ) if not self.embedding_model and not transforms_embedding_model: raise ValueError( """An embedding client was not provided. Provide an embedding through the transforms_embedding_model parameter. Alternatively you can provide your own transforms through the `transforms` parameter.""" ) if transforms is None: transforms = default_transforms_for_prechunked( llm=transforms_llm or self.llm, embedding_model=transforms_embedding_model or self.embedding_model, ) # convert the chunks to Ragas nodes nodes = [] for chunk in chunks: if isinstance(chunk, str): page_content = chunk metadata = {} else: page_content = chunk.page_content metadata = chunk.metadata if page_content is not None and page_content.strip() != "": node = Node( type=NodeType.CHUNK, properties={ "page_content": page_content, "document_metadata": metadata, }, ) nodes.append(node) kg = KnowledgeGraph(nodes=nodes) # apply transforms and update the knowledge graph apply_transforms(kg, transforms, run_config=run_config or RunConfig()) self.knowledge_graph = kg return self.generate( testset_size=testset_size, query_distribution=query_distribution, run_config=run_config, callbacks=callbacks, token_usage_parser=token_usage_parser, with_debugging_logs=with_debugging_logs, raise_exceptions=raise_exceptions, return_executor=return_executor, ) ``` ### generate ```python generate(testset_size: int, query_distribution: Optional[QueryDistribution] = None, num_personas: int = 3, run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, callbacks: Optional[Callbacks] = None, token_usage_parser: Optional[TokenUsageParser] = None, with_debugging_logs=False, raise_exceptions: bool = True, return_executor: bool = False) -> Union[Testset, Executor] ``` Generate an evaluation dataset based on given scenarios and parameters. Parameters: | Name | Type | Description | Default | | --------------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `testset_size` | `int` | The number of samples to generate. | *required* | | `query_distribution` | `Optional[QueryDistribution]` | A list of tuples containing scenario simulators and their probabilities. If None, default simulators will be used. | `None` | | `num_personas` | `int` | The number of personas to generate or use from the persona_list. | `3` | | `run_config` | `Optional[RunConfig]` | Configuration for running the generation process. | `None` | | `batch_size` | `Optional[int]` | How large should batches be. If set to None (default), no batching is done. | `None` | | `callbacks` | `Optional[Callbacks]` | Langchain style callbacks to use for the generation process. You can use this to log the generation process or add other metadata. | `None` | | `token_usage_parser` | `Optional[TokenUsageParser]` | Parse the LLMResult object and return a TokenUsage object. This is used to calculate the cost of the generation process. | `None` | | `with_debugging_logs` | `bool` | If True, enable debug logging for various components. | `False` | | `raise_exceptions` | `bool` | If True, raise exceptions during the generation process. | `True` | | `return_executor` | `bool` | If True, returns the Executor instance instead of running generation. The returned executor can be used to cancel execution by calling executor.cancel(). To get results, call executor.results(). | `False` | Returns: | Type | Description | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Testset or Executor` | If return_executor is False, returns a dataset containing the generated TestsetSamples. If return_executor is True, returns the Executor instance for cancellable execution. | Notes This function performs the following steps: 1. Set up scenarios and debug logging if required. 1. Generate scenarios using an Executor. 1. Calculate split values for different scenario types. 1. Generate samples for each scenario. 1. Compile the results into an EvaluationDataset. Source code in `src/ragas/testset/synthesizers/generate.py` ```python def generate( self, testset_size: int, query_distribution: t.Optional[QueryDistribution] = None, num_personas: int = 3, run_config: t.Optional[RunConfig] = None, batch_size: t.Optional[int] = None, callbacks: t.Optional[Callbacks] = None, token_usage_parser: t.Optional[TokenUsageParser] = None, with_debugging_logs=False, raise_exceptions: bool = True, return_executor: bool = False, ) -> t.Union[Testset, Executor]: """ Generate an evaluation dataset based on given scenarios and parameters. Parameters ---------- testset_size : int The number of samples to generate. query_distribution : Optional[QueryDistribution], optional A list of tuples containing scenario simulators and their probabilities. If None, default simulators will be used. num_personas : int, default 3 The number of personas to generate or use from the persona_list. run_config : Optional[RunConfig], optional Configuration for running the generation process. batch_size: int, optional How large should batches be. If set to None (default), no batching is done. callbacks : Optional[Callbacks], optional Langchain style callbacks to use for the generation process. You can use this to log the generation process or add other metadata. token_usage_parser : Optional[TokenUsageParser], optional Parse the LLMResult object and return a TokenUsage object. This is used to calculate the cost of the generation process. with_debugging_logs : bool, default False If True, enable debug logging for various components. raise_exceptions : bool, default True If True, raise exceptions during the generation process. return_executor : bool, default False If True, returns the Executor instance instead of running generation. The returned executor can be used to cancel execution by calling executor.cancel(). To get results, call executor.results(). Returns ------- Testset or Executor If return_executor is False, returns a dataset containing the generated TestsetSamples. If return_executor is True, returns the Executor instance for cancellable execution. Notes ----- This function performs the following steps: 1. Set up scenarios and debug logging if required. 2. Generate scenarios using an Executor. 3. Calculate split values for different scenario types. 4. Generate samples for each scenario. 5. Compile the results into an EvaluationDataset. """ if run_config is not None: # Only BaseRagasLLM has set_run_config method, not InstructorBaseRagasLLM if isinstance(self.llm, BaseRagasLLM): self.llm.set_run_config(run_config) query_distribution = query_distribution or default_query_distribution( self.llm, self.knowledge_graph, self.llm_context ) callbacks = callbacks or [] # dict to store any callbacks we define ragas_callbacks = {} # set the token usage parser if token_usage_parser is not None: from ragas.cost import CostCallbackHandler cost_cb = CostCallbackHandler(token_usage_parser=token_usage_parser) ragas_callbacks["cost_cb"] = cost_cb else: cost_cb = None # append all the ragas_callbacks to the callbacks for cb in ragas_callbacks.values(): if isinstance(callbacks, BaseCallbackManager): callbacks.add_handler(cb) else: callbacks.append(cb) # new group for Testset Generation testset_generation_rm, testset_generation_grp = new_group( name=RAGAS_TESTSET_GENERATION_GROUP_NAME, inputs={"testset_size": testset_size}, callbacks=callbacks, ) if with_debugging_logs: # TODO: Edit this before pre-release from ragas.utils import patch_logger patch_logger("ragas.experimental.testset.synthesizers", logging.DEBUG) patch_logger("ragas.experimental.testset.graph", logging.DEBUG) patch_logger("ragas.experimental.testset.transforms", logging.DEBUG) if self.persona_list is None: self.persona_list = generate_personas_from_kg( llm=self.llm, kg=self.knowledge_graph, num_personas=num_personas, callbacks=callbacks, ) else: random.shuffle(self.persona_list) splits, _ = calculate_split_values( [prob for _, prob in query_distribution], testset_size ) # new group for Generation of Scenarios scenario_generation_rm, scenario_generation_grp = new_group( name="Scenario Generation", inputs={"splits": splits}, callbacks=testset_generation_grp, ) # generate scenarios exec = Executor( desc="Generating Scenarios", raise_exceptions=raise_exceptions, run_config=run_config, keep_progress_bar=False, batch_size=batch_size, ) # generate samples splits, _ = calculate_split_values( [prob for _, prob in query_distribution], testset_size ) for i, (scenario, _) in enumerate(query_distribution): exec.submit( scenario.generate_scenarios, n=splits[i], knowledge_graph=self.knowledge_graph, persona_list=self.persona_list[:num_personas], callbacks=scenario_generation_grp, ) try: scenario_sample_list: t.List[t.List[BaseScenario]] = exec.results() except Exception as e: scenario_generation_rm.on_chain_error(e) raise e else: scenario_generation_rm.on_chain_end( outputs={"scenario_sample_list": scenario_sample_list} ) # new group for Generation of Samples sample_generation_rm, sample_generation_grp = new_group( name="Sample Generation", inputs={"scenario_sample_list": scenario_sample_list}, callbacks=testset_generation_grp, ) exec = Executor( "Generating Samples", raise_exceptions=raise_exceptions, run_config=run_config, keep_progress_bar=True, batch_size=batch_size, ) additional_testset_info: t.List[t.Dict] = [] for i, (synthesizer, _) in enumerate(query_distribution): for sample in scenario_sample_list[i]: exec.submit( synthesizer.generate_sample, scenario=sample, callbacks=sample_generation_grp, ) # fill out the additional info for the TestsetSample additional_testset_info.append( { "synthesizer_name": synthesizer.name, } ) # Return executor for cancellable execution if requested if return_executor: return exec try: eval_samples = exec.results() except Exception as e: sample_generation_rm.on_chain_error(e) raise e else: sample_generation_rm.on_chain_end(outputs={"eval_samples": eval_samples}) # build the testset testsets = [] for sample, additional_info in zip(eval_samples, additional_testset_info): testsets.append(TestsetSample(eval_sample=sample, **additional_info)) testset = Testset(samples=testsets, cost_cb=cost_cb) testset_generation_rm.on_chain_end({"testset": testset}) # tracking how many samples were generated track( TestsetGenerationEvent( event_type="testset_generation", evolution_names=[ e.__class__.__name__.lower() for e, _ in query_distribution ], evolution_percentages=[p for _, p in query_distribution], num_rows=testset_size, language="english", ) ) return testset ``` ## NodeType Bases: `str`, `Enum` Enumeration of node types in the knowledge graph. Currently supported node types are: UNKNOWN, DOCUMENT, CHUNK ## Node Bases: `BaseModel` Represents a node in the knowledge graph. Attributes: | Name | Type | Description | | ------------ | ---------- | -------------------------------------------------- | | `id` | `UUID` | Unique identifier for the node. | | `properties` | `dict` | Dictionary of properties associated with the node. | | `type` | `NodeType` | Type of the node. | ### add_property ```python add_property(key: str, value: Any) ``` Adds a property to the node. Raises: | Type | Description | | ------------ | ------------------------------- | | `ValueError` | If the property already exists. | Source code in `src/ragas/testset/graph.py` ```python def add_property(self, key: str, value: t.Any): """ Adds a property to the node. Raises ------ ValueError If the property already exists. """ if key.lower() in self.properties: raise ValueError(f"Property {key} already exists") self.properties[key.lower()] = value ``` ### get_property ```python get_property(key: str) -> Optional[Any] ``` Retrieves a property value by key. Notes The key is case-insensitive. Source code in `src/ragas/testset/graph.py` ```python def get_property(self, key: str) -> t.Optional[t.Any]: """ Retrieves a property value by key. Notes ----- The key is case-insensitive. """ return self.properties.get(key.lower(), None) ``` ## Relationship Bases: `BaseModel` Represents a relationship between two nodes in a knowledge graph. Attributes: | Name | Type | Description | | --------------- | ------------------ | ------------------------------------------------------------------------------------- | | `id` | `(UUID, optional)` | Unique identifier for the relationship. Defaults to a new UUID. | | `type` | `str` | The type of the relationship. | | `source` | `Node` | The source node of the relationship. | | `target` | `Node` | The target node of the relationship. | | `bidirectional` | `(bool, optional)` | Whether the relationship is bidirectional. Defaults to False. | | `properties` | `(dict, optional)` | Dictionary of properties associated with the relationship. Defaults to an empty dict. | ### get_property ```python get_property(key: str) -> Optional[Any] ``` Retrieves a property value by key. The key is case-insensitive. Source code in `src/ragas/testset/graph.py` ```python def get_property(self, key: str) -> t.Optional[t.Any]: """ Retrieves a property value by key. The key is case-insensitive. """ return self.properties.get(key.lower(), None) ``` ## KnowledgeGraph ```python KnowledgeGraph(nodes: List[Node] = list(), relationships: List[Relationship] = list()) ``` Represents a knowledge graph containing nodes and relationships. Attributes: | Name | Type | Description | | --------------- | -------------------- | --------------------------------------------- | | `nodes` | `List[Node]` | List of nodes in the knowledge graph. | | `relationships` | `List[Relationship]` | List of relationships in the knowledge graph. | ### add ```python add(item: Union[Node, Relationship]) ``` Adds a node or relationship to the knowledge graph. Raises: | Type | Description | | ------------ | --------------------------------------------- | | `ValueError` | If the item type is not Node or Relationship. | Source code in `src/ragas/testset/graph.py` ```python def add(self, item: t.Union[Node, Relationship]): """ Adds a node or relationship to the knowledge graph. Raises ------ ValueError If the item type is not Node or Relationship. """ if isinstance(item, Node): self._add_node(item) elif isinstance(item, Relationship): self._add_relationship(item) else: raise ValueError(f"Invalid item type: {type(item)}") ``` ### save ```python save(path: Union[str, Path]) ``` Saves the knowledge graph to a JSON file. Parameters: | Name | Type | Description | Default | | ------ | ------------------ | ----------------------------------------- | ---------- | | `path` | `Union[str, Path]` | Path where the JSON file should be saved. | *required* | Notes The file is saved using UTF-8 encoding to ensure proper handling of Unicode characters across different platforms. Source code in `src/ragas/testset/graph.py` ```python def save(self, path: t.Union[str, Path]): """Saves the knowledge graph to a JSON file. Parameters ---------- path : Union[str, Path] Path where the JSON file should be saved. Notes ----- The file is saved using UTF-8 encoding to ensure proper handling of Unicode characters across different platforms. """ if isinstance(path, str): path = Path(path) data = { "nodes": [node.model_dump() for node in self.nodes], "relationships": [rel.model_dump() for rel in self.relationships], } with open(path, "w", encoding="utf-8") as f: json.dump(data, f, cls=UUIDEncoder, indent=2, ensure_ascii=False) ``` ### load ```python load(path: Union[str, Path]) -> KnowledgeGraph ``` Loads a knowledge graph from a path. Parameters: | Name | Type | Description | Default | | ------ | ------------------ | ----------------------------------------------------- | ---------- | | `path` | `Union[str, Path]` | Path to the JSON file containing the knowledge graph. | *required* | Returns: | Type | Description | | ---------------- | --------------------------- | | `KnowledgeGraph` | The loaded knowledge graph. | Notes The file is read using UTF-8 encoding to ensure proper handling of Unicode characters across different platforms. Source code in `src/ragas/testset/graph.py` ```python @classmethod def load(cls, path: t.Union[str, Path]) -> "KnowledgeGraph": """Loads a knowledge graph from a path. Parameters ---------- path : Union[str, Path] Path to the JSON file containing the knowledge graph. Returns ------- KnowledgeGraph The loaded knowledge graph. Notes ----- The file is read using UTF-8 encoding to ensure proper handling of Unicode characters across different platforms. """ if isinstance(path, str): path = Path(path) with open(path, "r", encoding="utf-8") as f: data = json.load(f) nodes = [Node(**node_data) for node_data in data["nodes"]] nodes_map = {str(node.id): node for node in nodes} relationships = [ Relationship( id=rel_data["id"], type=rel_data["type"], source=nodes_map[rel_data["source"]], target=nodes_map[rel_data["target"]], bidirectional=rel_data["bidirectional"], properties=rel_data["properties"], ) for rel_data in data["relationships"] ] kg = cls() kg.nodes.extend(nodes) kg.relationships.extend(relationships) return kg ``` ### get_node_by_id ```python get_node_by_id(node_id: Union[UUID, str]) -> Optional[Node] ``` Retrieves a node by its ID. Parameters: | Name | Type | Description | Default | | --------- | ------ | ------------------------------- | ---------- | | `node_id` | `UUID` | The ID of the node to retrieve. | *required* | Returns: | Type | Description | | -------------- | ----------------------------------------------------- | | `Node or None` | The node with the specified ID, or None if not found. | Source code in `src/ragas/testset/graph.py` ```python def get_node_by_id(self, node_id: t.Union[uuid.UUID, str]) -> t.Optional[Node]: """ Retrieves a node by its ID. Parameters ---------- node_id : uuid.UUID The ID of the node to retrieve. Returns ------- Node or None The node with the specified ID, or None if not found. """ if isinstance(node_id, str): node_id = uuid.UUID(node_id) return next(filter(lambda n: n.id == node_id, self.nodes), None) ``` ### find_indirect_clusters ```python find_indirect_clusters(relationship_condition: Callable[[Relationship], bool] = lambda _: True, depth_limit: int = 3) -> List[Set[Node]] ``` Finds "indirect clusters" of nodes in the knowledge graph based on a relationship condition. Uses Leiden algorithm for community detection and identifies unique paths within each cluster. NOTE: "indirect clusters" as used in the method name are "groups of nodes that are not directly connected but share a common relationship through other nodes", while the Leiden algorithm is a "clustering" algorithm that defines neighborhoods of nodes based on their connections -- these definitions of "cluster" are NOT equivalent. Parameters: | Name | Type | Description | Default | | ------------------------ | -------------------------------- | ---------------------------------------------------------------------------------------------- | ---------------- | | `relationship_condition` | `Callable[[Relationship], bool]` | A function that takes a Relationship and returns a boolean, by default lambda \_: True | `lambda _: True` | | `depth_limit` | `int` | The maximum depth of relationships (number of edges) to consider for clustering, by default 3. | `3` | Returns: | Type | Description | | ----------------- | ------------------------------------------------------------------ | | `List[Set[Node]]` | A list of sets, where each set contains nodes that form a cluster. | Source code in `src/ragas/testset/graph.py` ```python def find_indirect_clusters( self, relationship_condition: t.Callable[[Relationship], bool] = lambda _: True, depth_limit: int = 3, ) -> t.List[t.Set[Node]]: """ Finds "indirect clusters" of nodes in the knowledge graph based on a relationship condition. Uses Leiden algorithm for community detection and identifies unique paths within each cluster. NOTE: "indirect clusters" as used in the method name are "groups of nodes that are not directly connected but share a common relationship through other nodes", while the Leiden algorithm is a "clustering" algorithm that defines neighborhoods of nodes based on their connections -- these definitions of "cluster" are NOT equivalent. Parameters ---------- relationship_condition : Callable[[Relationship], bool], optional A function that takes a Relationship and returns a boolean, by default lambda _: True depth_limit : int, optional The maximum depth of relationships (number of edges) to consider for clustering, by default 3. Returns ------- List[Set[Node]] A list of sets, where each set contains nodes that form a cluster. """ import networkx as nx def get_node_clusters( relationships: list[Relationship], ) -> dict[int, set[uuid.UUID]]: """Identify clusters of nodes using Leiden algorithm.""" import numpy as np from sknetwork.clustering import Leiden from sknetwork.data import Dataset as SKDataset, from_edge_list # NOTE: the upstream sknetwork Dataset has some issues with type hints, # so we use type: ignore to bypass them. # Use hex representation to ensure proper UUID strings for clustering graph: SKDataset = from_edge_list( # type: ignore [(rel.source.id.hex, rel.target.id.hex) for rel in relationships], directed=True, ) # Apply Leiden clustering leiden = Leiden(random_state=42) cluster_labels: np.ndarray = leiden.fit_predict(graph["adjacency"]) # Group nodes by cluster clusters: defaultdict[int, set[uuid.UUID]] = defaultdict(set) for label, node_id_hex in zip(cluster_labels, graph["names"]): # node_id_hex is the hex string representation of the UUID clusters[int(label)].add(uuid.UUID(hex=node_id_hex)) return dict(clusters) def to_nx_digraph( nodes: set[uuid.UUID], relationships: list[Relationship] ) -> nx.DiGraph: """Convert a set of nodes and relationships to a directed graph.""" # Create directed subgraph for this cluster graph = nx.DiGraph() for node_id in nodes: graph.add_node( node_id, node_obj=self.get_node_by_id(node_id), ) for rel in relationships: if rel.source.id in nodes and rel.target.id in nodes: graph.add_edge(rel.source.id, rel.target.id, relationship_obj=rel) return graph def max_simple_paths(n: int, k: int = depth_limit) -> int: """Estimate the number of paths up to depth_limit that would exist in a fully-connected graph of size cluster_nodes.""" from math import prod if n - k - 1 <= 0: return 0 return prod(n - i for i in range(k + 1)) def exhaustive_paths( graph: nx.DiGraph, depth_limit: int ) -> list[list[uuid.UUID]]: """Find all simple paths in the subgraph up to depth_limit.""" import itertools # Check if graph has enough nodes for meaningful paths if len(graph) < 2: return [] all_paths: list[list[uuid.UUID]] = [] for source, target in itertools.permutations(graph.nodes(), 2): if not nx.has_path(graph, source, target): continue try: paths = nx.all_simple_paths( graph, source, target, cutoff=depth_limit, ) all_paths.extend(paths) except nx.NetworkXNoPath: continue return all_paths def sample_paths_from_graph( graph: nx.DiGraph, depth_limit: int, sample_size: int = 1000 ) -> list[list[uuid.UUID]]: """Sample random paths in the graph up to depth_limit.""" # we're using a DiGraph, so we need to account for directionality # if a node has no out-paths, then it will cause an error in `generate_random_paths` # Iteratively remove nodes with no out-paths to handle cascading effects while True: nodes_with_no_outpaths = [ n for n in graph.nodes() if graph.out_degree(n) == 0 ] if not nodes_with_no_outpaths: break graph.remove_nodes_from(nodes_with_no_outpaths) # Check if graph is empty after node removal if len(graph) == 0: return [] sampled_paths: list[list[uuid.UUID]] = [] for depth in range(2, depth_limit + 1): # Additional safety check before generating paths if ( len(graph) < depth + 1 ): # Need at least depth+1 nodes for a path of length depth continue paths = nx.generate_random_paths( graph, sample_size=sample_size, path_length=depth, ) sampled_paths.extend(paths) return sampled_paths # depth 2: 3 nodes, 2 edges (A -> B -> C) if depth_limit < 2: raise ValueError("Depth limit must be at least 2") # Filter relationships based on the condition filtered_relationships: list[Relationship] = [] relationship_map: defaultdict[uuid.UUID, set[uuid.UUID]] = defaultdict(set) for rel in self.relationships: if relationship_condition(rel): filtered_relationships.append(rel) relationship_map[rel.source.id].add(rel.target.id) if rel.bidirectional: relationship_map[rel.target.id].add(rel.source.id) if not filtered_relationships: return [] clusters = get_node_clusters(filtered_relationships) # For each cluster, find valid paths up to depth_limit cluster_sets: set[frozenset] = set() for _cluster_label, cluster_nodes in tqdm( clusters.items(), desc="Processing clusters" ): # Skip clusters that are too small to form any meaningful paths (need at least 2 nodes) if len(cluster_nodes) < 2: continue subgraph = to_nx_digraph( nodes=cluster_nodes, relationships=filtered_relationships ) sampled_paths: list[list[uuid.UUID]] = [] # if the expected number of paths is small, use exhaustive search # otherwise sample with random walks if max_simple_paths(n=len(cluster_nodes), k=depth_limit) < 1000: sampled_paths.extend(exhaustive_paths(subgraph, depth_limit)) else: sampled_paths.extend(sample_paths_from_graph(subgraph, depth_limit)) # convert paths (node IDs) to sets of Node objects # and deduplicate for path in sampled_paths: path_nodes = {subgraph.nodes[node_id]["node_obj"] for node_id in path} cluster_sets.add(frozenset(path_nodes)) return [set(path_nodes) for path_nodes in cluster_sets] ``` ### find_n_indirect_clusters ```python find_n_indirect_clusters(n: int, relationship_condition: Callable[[Relationship], bool] = lambda _: True, depth_limit: int = 3) -> List[Set[Node]] ``` Return n indirect clusters of nodes in the knowledge graph based on a relationship condition. Optimized for large datasets by using an adjacency index for lookups and limiting path exploration relative to n. A cluster represents a path through the graph. For example, if A -> B -> C -> D exists in the graph, then {A, B, C, D} forms a cluster. If there's also a path A -> B -> C -> E, it forms a separate cluster. The method returns a list of up to n sets, where each set contains nodes forming a complete path from a starting node to a leaf node or a path segment up to depth_limit nodes long. The result may contain fewer than n clusters if the graph is very sparse or if there aren't enough nodes to form n distinct clusters. To maximize diversity in the results: 1. Random starting nodes are selected 1. Paths from each starting node are grouped 1. Clusters are selected in round-robin fashion from each group until n unique clusters are found 1. Duplicate clusters are eliminated 1. When a superset cluster is found (e.g., {A,B,C,D}), any existing subset clusters (e.g., {A,B,C}) are removed to avoid redundancy Parameters: | Name | Type | Description | Default | | ------------------------ | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ---------------- | | `n` | `int` | Target number of clusters to return. Must be at least 1. Should return n clusters unless the graph is extremely sparse. | *required* | | `relationship_condition` | `Callable[[Relationship], bool]` | A function that takes a Relationship and returns a boolean, by default lambda \_: True | `lambda _: True` | | `depth_limit` | `int` | Maximum depth for path exploration, by default 3. Must be at least 2 to form clusters by definition. | `3` | Returns: | Type | Description | | ----------------- | ------------------------------------------------------------------ | | `List[Set[Node]]` | A list of sets, where each set contains nodes that form a cluster. | Raises: | Type | Description | | ------------ | ---------------------------------------------------------------------------- | | `ValueError` | If depth_limit < 2, n < 1, or no relationships match the provided condition. | Source code in `src/ragas/testset/graph.py` ```python def find_n_indirect_clusters( self, n: int, relationship_condition: t.Callable[[Relationship], bool] = lambda _: True, depth_limit: int = 3, ) -> t.List[t.Set[Node]]: """ Return n indirect clusters of nodes in the knowledge graph based on a relationship condition. Optimized for large datasets by using an adjacency index for lookups and limiting path exploration relative to n. A cluster represents a path through the graph. For example, if A -> B -> C -> D exists in the graph, then {A, B, C, D} forms a cluster. If there's also a path A -> B -> C -> E, it forms a separate cluster. The method returns a list of up to n sets, where each set contains nodes forming a complete path from a starting node to a leaf node or a path segment up to depth_limit nodes long. The result may contain fewer than n clusters if the graph is very sparse or if there aren't enough nodes to form n distinct clusters. To maximize diversity in the results: 1. Random starting nodes are selected 2. Paths from each starting node are grouped 3. Clusters are selected in round-robin fashion from each group until n unique clusters are found 4. Duplicate clusters are eliminated 5. When a superset cluster is found (e.g., {A,B,C,D}), any existing subset clusters (e.g., {A,B,C}) are removed to avoid redundancy Parameters ---------- n : int Target number of clusters to return. Must be at least 1. Should return n clusters unless the graph is extremely sparse. relationship_condition : Callable[[Relationship], bool], optional A function that takes a Relationship and returns a boolean, by default lambda _: True depth_limit : int, optional Maximum depth for path exploration, by default 3. Must be at least 2 to form clusters by definition. Returns ------- List[Set[Node]] A list of sets, where each set contains nodes that form a cluster. Raises ------ ValueError If depth_limit < 2, n < 1, or no relationships match the provided condition. """ if depth_limit < 2: raise ValueError("depth_limit must be at least 2 to form valid clusters") if n < 1: raise ValueError("n must be at least 1") # Filter relationships once upfront filtered_relationships: list[Relationship] = [ rel for rel in self.relationships if relationship_condition(rel) ] if not filtered_relationships: raise ValueError( "No relationships match the provided condition. Cannot form clusters." ) # Build adjacency list for faster neighbor lookup - optimized for large datasets adjacency_list: dict[Node, set[Node]] = {} unique_edges: set[frozenset[Node]] = set() for rel in filtered_relationships: # Lazy initialization since we only care about nodes with relationships if rel.source not in adjacency_list: adjacency_list[rel.source] = set() adjacency_list[rel.source].add(rel.target) unique_edges.add(frozenset({rel.source, rel.target})) if rel.bidirectional: if rel.target not in adjacency_list: adjacency_list[rel.target] = set() adjacency_list[rel.target].add(rel.source) # Aggregate clusters for each start node start_node_clusters: dict[Node, set[frozenset[Node]]] = {} # sample enough starting nodes to handle worst case grouping scenario where nodes are grouped # in independent clusters of size equal to depth_limit. This only surfaces when there are less # unique edges than nodes. connected_nodes: set[Node] = set().union(*unique_edges) sample_size: int = ( (n - 1) * depth_limit + 1 if len(unique_edges) < len(connected_nodes) else max(n, depth_limit, 10) ) def dfs(node: Node, start_node: Node, current_path: t.Set[Node]): # Terminate exploration when max usable clusters is reached so complexity doesn't spiral if len(start_node_clusters.get(start_node, [])) > sample_size: return current_path.add(node) path_length = len(current_path) at_max_depth = path_length >= depth_limit neighbors = adjacency_list.get(node, None) # If this is a leaf node or we've reached depth limit # and we have a valid path of at least 2 nodes, add it as a cluster if path_length > 1 and ( at_max_depth or not neighbors or all(n in current_path for n in neighbors) ): # Lazy initialization of the set for this start node if start_node not in start_node_clusters: start_node_clusters[start_node] = set() start_node_clusters[start_node].add(frozenset(current_path)) elif neighbors: for neighbor in neighbors: # Block cycles if neighbor not in current_path: dfs(neighbor, start_node, current_path) # Backtrack by removing the current node from path current_path.remove(node) # Shuffle nodes for random starting points # Use adjacency list since that has filtered out isolated nodes # Sort by node ID for consistent ordering while maintaining algorithm effectiveness start_nodes = sorted(adjacency_list.keys(), key=lambda n: n.id.hex) # Use a hash-based seed for reproducible but varied shuffling based on the nodes themselves node_ids_str = "".join(n.id.hex for n in start_nodes) node_hash = hashlib.sha256(node_ids_str.encode("utf-8")).hexdigest() rng = random.Random(int(node_hash[:8], 16)) # Use first 8 hex chars as seed rng.shuffle(start_nodes) samples: list[Node] = start_nodes[:sample_size] for start_node in samples: dfs(start_node, start_node, set()) start_node_clusters_list: list[set[frozenset[Node]]] = list( start_node_clusters.values() ) # Iteratively pop from each start_node_clusters until we have n unique clusters # Avoid adding duplicates and subset/superset pairs so we have diversity. We # favor supersets over subsets if we are given a choice. unique_clusters: set[frozenset[Node]] = set() i = 0 while len(unique_clusters) < n and start_node_clusters_list: # Cycle through the start node clusters current_index = i % len(start_node_clusters_list) current_start_node_clusters: set[frozenset[Node]] = ( start_node_clusters_list[current_index] ) cluster: frozenset[Node] = current_start_node_clusters.pop() # Check if the new cluster is a subset of any existing cluster # and collect any existing clusters that are subsets of this cluster is_subset = False subsets_to_remove: set[frozenset[Node]] = set() for existing in unique_clusters: if cluster.issubset(existing): is_subset = True break elif cluster.issuperset(existing): subsets_to_remove.add(existing) # Only add the new cluster if it's not a subset of any existing cluster if not is_subset: # Remove any subsets of the new cluster unique_clusters -= subsets_to_remove unique_clusters.add(cluster) # If this set is now empty, remove it if not current_start_node_clusters: start_node_clusters_list.pop(current_index) # Don't increment i since we removed an element to account for shift else: i += 1 return [set(cluster) for cluster in unique_clusters] ``` ### remove_node ```python remove_node(node: Node, inplace: bool = True) -> Optional[KnowledgeGraph] ``` Removes a node and its associated relationships from the knowledge graph. Parameters: | Name | Type | Description | Default | | --------- | ------ | -------------------------------------------------------------------------------------------------------- | ---------- | | `node` | `Node` | The node to be removed from the knowledge graph. | *required* | | `inplace` | `bool` | If True, modifies the knowledge graph in place. If False, returns a modified copy with the node removed. | `True` | Returns: | Type | Description | | ------------------------ | ---------------------------------------------------------------------------------------------------- | | `KnowledgeGraph or None` | Returns a modified copy of the knowledge graph if inplace is False. Returns None if inplace is True. | Raises: | Type | Description | | ------------ | -------------------------------------------------- | | `ValueError` | If the node is not present in the knowledge graph. | Source code in `src/ragas/testset/graph.py` ```python def remove_node( self, node: Node, inplace: bool = True ) -> t.Optional["KnowledgeGraph"]: """ Removes a node and its associated relationships from the knowledge graph. Parameters ---------- node : Node The node to be removed from the knowledge graph. inplace : bool, optional If True, modifies the knowledge graph in place. If False, returns a modified copy with the node removed. Returns ------- KnowledgeGraph or None Returns a modified copy of the knowledge graph if `inplace` is False. Returns None if `inplace` is True. Raises ------ ValueError If the node is not present in the knowledge graph. """ if node not in self.nodes: raise ValueError("Node is not present in the knowledge graph.") if inplace: # Modify the current instance self.nodes.remove(node) self.relationships = [ rel for rel in self.relationships if rel.source != node and rel.target != node ] else: # Create a deep copy and modify it new_graph = deepcopy(self) new_graph.nodes.remove(node) new_graph.relationships = [ rel for rel in new_graph.relationships if rel.source != node and rel.target != node ] return new_graph ``` ### find_two_nodes_single_rel ```python find_two_nodes_single_rel(relationship_condition: Callable[[Relationship], bool] = lambda _: True) -> List[Tuple[Node, Relationship, Node]] ``` Finds nodes in the knowledge graph based on a relationship condition. (NodeA, NodeB, Rel) triples are considered as multi-hop nodes. Parameters: | Name | Type | Description | Default | | ------------------------ | -------------------------------- | -------------------------------------------------------------------------------------- | ---------------- | | `relationship_condition` | `Callable[[Relationship], bool]` | A function that takes a Relationship and returns a boolean, by default lambda \_: True | `lambda _: True` | Returns: | Type | Description | | ------------------------------------- | ---------------------------------------------------------------------------------------------- | | `List[Set[Node, Relationship, Node]]` | A list of sets, where each set contains two nodes and a relationship forming a multi-hop node. | Source code in `src/ragas/testset/graph.py` ```python def find_two_nodes_single_rel( self, relationship_condition: t.Callable[[Relationship], bool] = lambda _: True ) -> t.List[t.Tuple[Node, Relationship, Node]]: """ Finds nodes in the knowledge graph based on a relationship condition. (NodeA, NodeB, Rel) triples are considered as multi-hop nodes. Parameters ---------- relationship_condition : Callable[[Relationship], bool], optional A function that takes a Relationship and returns a boolean, by default lambda _: True Returns ------- List[Set[Node, Relationship, Node]] A list of sets, where each set contains two nodes and a relationship forming a multi-hop node. """ relationships = [ relationship for relationship in self.relationships if relationship_condition(relationship) ] triplets = set() for relationship in relationships: if relationship.source != relationship.target: node_a = relationship.source node_b = relationship.target # Ensure the smaller ID node is always first if node_a.id < node_b.id: normalized_tuple = (node_a, relationship, node_b) else: normalized_relationship = Relationship( source=node_b, target=node_a, type=relationship.type, properties=relationship.properties, ) normalized_tuple = (node_b, normalized_relationship, node_a) triplets.add(normalized_tuple) return list(triplets) ``` ## ragas.integrations.langchain ### EvaluatorChain ```python EvaluatorChain(metric: Metric, **kwargs: Any) ``` Bases: `Chain`, `RunEvaluator` Wrapper around ragas Metrics to use them with langsmith. Source code in `src/ragas/integrations/langchain.py` ```python def __init__(self, metric: Metric, **kwargs: t.Any): kwargs["metric"] = metric super().__init__(**kwargs) if "run_config" in kwargs: run_config = kwargs["run_config"] else: run_config = RunConfig() if isinstance(self.metric, MetricWithLLM): llm = get_or_init(kwargs, "llm", ChatOpenAI) t.cast(MetricWithLLM, self.metric).llm = LangchainLLMWrapper(llm) if isinstance(self.metric, MetricWithEmbeddings): embeddings = get_or_init(kwargs, "embeddings", OpenAIEmbeddings) t.cast( MetricWithEmbeddings, self.metric ).embeddings = LangchainEmbeddingsWrapper(embeddings) self.metric.init(run_config) assert isinstance(self.metric, SingleTurnMetric), ( "Metric must be SingleTurnMetric" ) ``` #### evaluate_run ```python evaluate_run(run: Run, example: Optional[Example] = None) -> EvaluationResult ``` Evaluate a langsmith run Source code in `src/ragas/integrations/langchain.py` ```python @t.no_type_check def evaluate_run( self, run: Run, example: t.Optional[Example] = None ) -> EvaluationResult: """ Evaluate a langsmith run """ # Moved away from this implementation in LangChain evaluations; # we can safely ignore type checking for this legacy function. self._validate_langsmith_eval(run, example) # this is just to suppress the type checker error # actual check and error message is in the _validate_langsmith_eval assert run.outputs is not None assert example is not None assert example.inputs is not None assert example.outputs is not None chain_eval = run.outputs chain_eval["question"] = example.inputs["question"] if "ground_truth" in get_required_columns_v1(self.metric): if example.outputs is None or "ground_truth" not in example.outputs: raise ValueError("expected `ground_truth` in example outputs.") chain_eval["ground_truth"] = example.outputs["ground_truth"] eval_output = self.invoke(chain_eval, include_run_info=True) evaluation_result = EvaluationResult( key=self.metric.name, score=eval_output[self.metric.name] ) if RUN_KEY in eval_output: evaluation_result.evaluator_info[RUN_KEY] = eval_output[RUN_KEY] return evaluation_result ``` ## ragas.integrations.langsmith ### upload_dataset ```python upload_dataset(dataset: Testset, dataset_name: str, dataset_desc: str = '') -> Dataset ``` Uploads a new dataset to LangSmith, converting it from a TestDataset object to a pandas DataFrame before upload. If a dataset with the specified name already exists, the function raises an error. Parameters: | Name | Type | Description | Default | | -------------- | ------------- | ------------------------------------------------------------------ | ---------- | | `dataset` | `TestDataset` | The dataset to be uploaded. | *required* | | `dataset_name` | `str` | The name for the new dataset in LangSmith. | *required* | | `dataset_desc` | `str` | A description for the new dataset. The default is an empty string. | `''` | Returns: | Type | Description | | --------- | ------------------------------------------------------- | | `Dataset` | The dataset object as stored in LangSmith after upload. | Raises: | Type | Description | | ------------ | ----------------------------------------------------------------- | | `ValueError` | If a dataset with the specified name already exists in LangSmith. | Notes The function attempts to read a dataset by the given name to check its existence. If not found, it proceeds to upload the dataset after converting it to a pandas DataFrame. This involves specifying input and output keys for the dataset being uploaded. Source code in `src/ragas/integrations/langsmith.py` ```python def upload_dataset( dataset: Testset, dataset_name: str, dataset_desc: str = "" ) -> LangsmithDataset: """ Uploads a new dataset to LangSmith, converting it from a TestDataset object to a pandas DataFrame before upload. If a dataset with the specified name already exists, the function raises an error. Parameters ---------- dataset : TestDataset The dataset to be uploaded. dataset_name : str The name for the new dataset in LangSmith. dataset_desc : str, optional A description for the new dataset. The default is an empty string. Returns ------- LangsmithDataset The dataset object as stored in LangSmith after upload. Raises ------ ValueError If a dataset with the specified name already exists in LangSmith. Notes ----- The function attempts to read a dataset by the given name to check its existence. If not found, it proceeds to upload the dataset after converting it to a pandas DataFrame. This involves specifying input and output keys for the dataset being uploaded. """ client = Client() try: # check if dataset exists langsmith_dataset: LangsmithDataset = client.read_dataset( dataset_name=dataset_name ) raise ValueError( f"Dataset {dataset_name} already exists in langsmith. [{langsmith_dataset}]" ) except LangSmithNotFoundError: # if not create a new one with the generated query examples langsmith_dataset: LangsmithDataset = client.upload_dataframe( df=dataset.to_pandas(), name=dataset_name, input_keys=["question"], output_keys=["ground_truth"], description=dataset_desc, ) print( f"Created a new dataset '{langsmith_dataset.name}'. Dataset is accessible at {langsmith_dataset.url}" ) return langsmith_dataset ``` ### evaluate ```python evaluate(dataset_name: str, llm_or_chain_factory: Any, experiment_name: Optional[str] = None, metrics: Optional[list] = None, verbose: bool = False) -> Dict[str, Any] ``` Evaluates a language model or a chain factory on a specified dataset using LangSmith, with the option to customize metrics and verbosity. Parameters: | Name | Type | Description | Default | | ---------------------- | ---------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `dataset_name` | `str` | The name of the dataset to use for evaluation. This dataset must exist in LangSmith. | *required* | | `llm_or_chain_factory` | `Any` | The language model or chain factory to be evaluated. This parameter is flexible and can accept a variety of objects depending on the implementation. | *required* | | `experiment_name` | `Optional[str]` | The name of the experiment. This can be used to categorize or identify the evaluation run within LangSmith. The default is None. | `None` | | `metrics` | `Optional[list]` | A list of custom metrics (functions or evaluators) to be used for the evaluation. If None, a default set of metrics (answer relevancy, context precision, context recall, and faithfulness) are used. The default is None. | `None` | | `verbose` | `bool` | If True, detailed progress and results will be printed during the evaluation process. The default is False. | `False` | Returns: | Type | Description | | ---------------- | ------------------------------------------------------ | | `Dict[str, Any]` | A dictionary containing the results of the evaluation. | Raises: | Type | Description | | ------------ | ----------------------------------------------------- | | `ValueError` | If the specified dataset does not exist in LangSmith. | See Also Client.read_dataset : Method to read an existing dataset. Client.run_on_dataset : Method to run the evaluation on the specified dataset. Examples: ```pycon >>> results = evaluate( ... dataset_name="MyDataset", ... llm_or_chain_factory=my_llm, ... experiment_name="experiment_1_with_vanila_rag", ... verbose=True ... ) >>> print(results) {'evaluation_result': ...} ``` Notes The function initializes a client to interact with LangSmith, validates the existence of the specified dataset, prepares evaluation metrics, and runs the evaluation, returning the results. Custom evaluation metrics can be specified, or a default set will be used if none are provided. Source code in `src/ragas/integrations/langsmith.py` ```python def evaluate( dataset_name: str, llm_or_chain_factory: t.Any, experiment_name: t.Optional[str] = None, metrics: t.Optional[list] = None, verbose: bool = False, ) -> t.Dict[str, t.Any]: """ Evaluates a language model or a chain factory on a specified dataset using LangSmith, with the option to customize metrics and verbosity. Parameters ---------- dataset_name : str The name of the dataset to use for evaluation. This dataset must exist in LangSmith. llm_or_chain_factory : Any The language model or chain factory to be evaluated. This parameter is flexible and can accept a variety of objects depending on the implementation. experiment_name : Optional[str], optional The name of the experiment. This can be used to categorize or identify the evaluation run within LangSmith. The default is None. metrics : Optional[list], optional A list of custom metrics (functions or evaluators) to be used for the evaluation. If None, a default set of metrics (answer relevancy, context precision, context recall, and faithfulness) are used. The default is None. verbose : bool, optional If True, detailed progress and results will be printed during the evaluation process. The default is False. Returns ------- Dict[str, Any] A dictionary containing the results of the evaluation. Raises ------ ValueError If the specified dataset does not exist in LangSmith. See Also -------- Client.read_dataset : Method to read an existing dataset. Client.run_on_dataset : Method to run the evaluation on the specified dataset. Examples -------- >>> results = evaluate( ... dataset_name="MyDataset", ... llm_or_chain_factory=my_llm, ... experiment_name="experiment_1_with_vanila_rag", ... verbose=True ... ) >>> print(results) {'evaluation_result': ...} Notes ----- The function initializes a client to interact with LangSmith, validates the existence of the specified dataset, prepares evaluation metrics, and runs the evaluation, returning the results. Custom evaluation metrics can be specified, or a default set will be used if none are provided. """ # init client and validate dataset client = Client() try: _ = client.read_dataset(dataset_name=dataset_name) except LangSmithNotFoundError: raise ValueError( f"Dataset {dataset_name} not found in langsmith, make sure it exists in langsmith" ) # make config if metrics is None: from ragas.metrics._answer_relevance import answer_relevancy from ragas.metrics._context_precision import context_precision from ragas.metrics._context_recall import context_recall from ragas.metrics._faithfulness import faithfulness metrics = [answer_relevancy, context_precision, faithfulness, context_recall] metrics = [EvaluatorChain(m) for m in metrics] eval_config = RunEvalConfig( custom_evaluators=metrics, ) # run evaluation with langsmith run = client.run_on_dataset( # type: ignore[attr-defined] dataset_name=dataset_name, llm_or_chain_factory=llm_or_chain_factory, evaluation=eval_config, verbose=verbose, # Any experiment metadata can be specified here project_name=experiment_name, ) return run ``` ## ragas.integrations.llama_index ### convert_to_ragas_messages ```python convert_to_ragas_messages(events: List[Event]) -> List[Message] ``` Convert a sequence of LlamIndex agent events into Ragas message objects. This function processes a list of `Event` objects (e.g., `AgentInput`, `AgentOutput`, and `ToolCallResult`) and converts them into a list of `Message` objects (`HumanMessage`, `AIMessage`, and `ToolMessage`) that can be used for evaluation with the Ragas framework. Parameters: | Name | Type | Description | Default | | -------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | | `events` | `List[Event]` | A list of agent events that represent a conversation trace. These can include user inputs (AgentInput), model outputs (AgentOutput), and tool responses (ToolCallResult). | *required* | Returns: | Type | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | `List[Message]` | A list of Ragas Message objects corresponding to the structured conversation. Tool calls are de-duplicated using their tool ID to avoid repeated entries. | Source code in `src/ragas/integrations/llama_index.py` ```python def convert_to_ragas_messages(events: t.List[Event]) -> t.List[Message]: """ Convert a sequence of LlamIndex agent events into Ragas message objects. This function processes a list of `Event` objects (e.g., `AgentInput`, `AgentOutput`, and `ToolCallResult`) and converts them into a list of `Message` objects (`HumanMessage`, `AIMessage`, and `ToolMessage`) that can be used for evaluation with the Ragas framework. Parameters ---------- events : List[Event] A list of agent events that represent a conversation trace. These can include user inputs (`AgentInput`), model outputs (`AgentOutput`), and tool responses (`ToolCallResult`). Returns ------- List[Message] A list of Ragas `Message` objects corresponding to the structured conversation. Tool calls are de-duplicated using their tool ID to avoid repeated entries. """ try: from llama_index.core.agent.workflow import ( AgentInput, AgentOutput, ToolCallResult, ) from llama_index.core.base.llms.types import MessageRole, TextBlock except ImportError: raise ImportError( "Please install the llama_index package to use this function." ) ragas_messages = [] tool_call_ids = set() for event in events: if isinstance(event, AgentInput): last_chat_message = event.input[-1] content = "" if last_chat_message.blocks: content = "\n".join( str(block.text) for block in last_chat_message.blocks if isinstance(block, TextBlock) ) if last_chat_message.role == MessageRole.USER: if ragas_messages and isinstance(ragas_messages[-1], ToolMessage): continue ragas_messages.append(HumanMessage(content=content)) elif isinstance(event, AgentOutput): content = "\n".join( str(block.text) for block in event.response.blocks if isinstance(block, TextBlock) ) ragas_tool_calls = None if hasattr(event, "tool_calls"): raw_tool_calls = event.tool_calls ragas_tool_calls = [] for tc in raw_tool_calls: if tc.tool_id not in tool_call_ids: tool_call_ids.add(tc.tool_id) ragas_tool_calls.append( ToolCall( name=tc.tool_name, args=tc.tool_kwargs, ) ) ragas_messages.append( AIMessage( content=content, tool_calls=ragas_tool_calls if ragas_tool_calls else None, ) ) elif isinstance(event, ToolCallResult): if event.return_direct: ragas_messages.append(AIMessage(content=event.tool_output.content)) else: ragas_messages.append(ToolMessage(content=event.tool_output.content)) return ragas_messages ``` ## ragas.integrations.opik ### OpikTracer Bases: `OpikTracer` Callback for Opik that can be used to log traces and evaluation scores to the Opik platform. Attributes: | Name | Type | Description | | ---------- | -------------- | ------------------------------------------ | | `tags` | `list[string]` | The tags to set on each trace. | | `metadata` | `dict` | Additional metadata to log for each trace. | ## ragas.integrations.helicone ## ragas.integrations.langgraph ### convert_to_ragas_messages ```python convert_to_ragas_messages(messages: List[Union[HumanMessage, SystemMessage, AIMessage, ToolMessage]], metadata: bool = False) -> List[Union[HumanMessage, AIMessage, ToolMessage]] ``` Convert LangChain messages into Ragas messages with metadata for agent evaluation. Parameters: | Name | Type | Description | Default | | ---------- | ------------------------------------------------------------------ | ------------------------------------------------------ | ---------- | | `messages` | `List[Union[HumanMessage, SystemMessage, AIMessage, ToolMessage]]` | List of LangChain message objects to be converted. | *required* | | `metadata` | `(bool, optional(default=False))` | Whether to include metadata in the converted messages. | `False` | Returns: | Type | Description | | --------------------------------------------------- | ---------------------------------------------------------- | | `List[Union[HumanMessage, AIMessage, ToolMessage]]` | List of corresponding Ragas message objects with metadata. | Raises: | Type | Description | | ------------ | ---------------------------------------------- | | `ValueError` | If an unsupported message type is encountered. | | `TypeError` | If message content is not a string. | Notes SystemMessages are skipped in the conversion process. Source code in `src/ragas/integrations/langgraph.py` ```python def convert_to_ragas_messages( messages: List[Union[HumanMessage, SystemMessage, AIMessage, ToolMessage]], metadata: bool = False, ) -> List[Union[r.HumanMessage, r.AIMessage, r.ToolMessage]]: """ Convert LangChain messages into Ragas messages with metadata for agent evaluation. Parameters ---------- messages : List[Union[HumanMessage, SystemMessage, AIMessage, ToolMessage]] List of LangChain message objects to be converted. metadata : bool, optional (default=False) Whether to include metadata in the converted messages. Returns ------- List[Union[r.HumanMessage, r.AIMessage, r.ToolMessage]] List of corresponding Ragas message objects with metadata. Raises ------ ValueError If an unsupported message type is encountered. TypeError If message content is not a string. Notes ----- SystemMessages are skipped in the conversion process. """ def _validate_string_content(message, message_type: str) -> str: if not isinstance(message.content, str): raise TypeError( f"{message_type} content must be a string, got {type(message.content).__name__}. " f"Content: {message.content}" ) return message.content def _extract_metadata(message) -> dict: return {k: v for k, v in message.__dict__.items() if k != "content"} if metadata: MESSAGE_TYPE_MAP = { HumanMessage: lambda m: r.HumanMessage( content=_validate_string_content(m, "HumanMessage"), metadata=_extract_metadata(m), ), ToolMessage: lambda m: r.ToolMessage( content=_validate_string_content(m, "ToolMessage"), metadata=_extract_metadata(m), ), } else: MESSAGE_TYPE_MAP = { HumanMessage: lambda m: r.HumanMessage( content=_validate_string_content(m, "HumanMessage") ), ToolMessage: lambda m: r.ToolMessage( content=_validate_string_content(m, "ToolMessage") ), } def _extract_tool_calls(message: AIMessage) -> List[r.ToolCall]: tool_calls = message.additional_kwargs.get("tool_calls", []) return [ r.ToolCall( name=tool_call["function"]["name"], args=json.loads(tool_call["function"]["arguments"]), ) for tool_call in tool_calls ] def _convert_ai_message(message: AIMessage, metadata: bool) -> r.AIMessage: tool_calls = _extract_tool_calls(message) if message.additional_kwargs else None if metadata: return r.AIMessage( content=_validate_string_content(message, "AIMessage"), tool_calls=tool_calls, metadata=_extract_metadata(message), ) else: return r.AIMessage( content=_validate_string_content(message, "AIMessage"), tool_calls=tool_calls, ) def _convert_message(message, metadata: bool = False): if isinstance(message, SystemMessage): return None # Skip SystemMessages if isinstance(message, AIMessage): return _convert_ai_message(message, metadata) converter = MESSAGE_TYPE_MAP.get(type(message)) if converter is None: raise ValueError(f"Unsupported message type: {type(message).__name__}") return converter(message) return [ converted for message in messages if (converted := _convert_message(message)) is not None ] ``` ## BaseRagasLLM ```python BaseRagasLLM(run_config: RunConfig = RunConfig(), multiple_completion_supported: bool = False, cache: Optional[CacheInterface] = None) ``` Bases: `ABC` ### get_temperature ```python get_temperature(n: int) -> float ``` Return the temperature to use for completion based on n. Source code in `src/ragas/llms/base.py` ```python def get_temperature(self, n: int) -> float: """Return the temperature to use for completion based on n.""" return 0.3 if n > 1 else 0.01 ``` ### is_finished ```python is_finished(response: LLMResult) -> bool ``` Check if the LLM response is finished/complete. Source code in `src/ragas/llms/base.py` ```python @abstractmethod def is_finished(self, response: LLMResult) -> bool: """Check if the LLM response is finished/complete.""" ... ``` ### generate ```python generate(prompt: PromptValue, n: int = 1, temperature: Optional[float] = 0.01, stop: Optional[List[str]] = None, callbacks: Callbacks = None) -> LLMResult ``` Generate text using the given event loop. Source code in `src/ragas/llms/base.py` ```python async def generate( self, prompt: PromptValue, n: int = 1, temperature: t.Optional[float] = 0.01, stop: t.Optional[t.List[str]] = None, callbacks: Callbacks = None, ) -> LLMResult: """Generate text using the given event loop.""" if temperature is None: temperature = self.get_temperature(n) agenerate_text_with_retry = add_async_retry( self.agenerate_text, self.run_config ) result = await agenerate_text_with_retry( prompt=prompt, n=n, temperature=temperature, stop=stop, callbacks=callbacks, ) # check there are no max_token issues if not self.is_finished(result): raise LLMDidNotFinishException() return result ``` ## InstructorBaseRagasLLM Bases: `ABC` Base class for LLMs using the Instructor library pattern. ### generate ```python generate(prompt: str, response_model: Type[InstructorTypeVar]) -> InstructorTypeVar ``` Generate a response using the configured LLM. For async clients, this will run the async method in the appropriate event loop. Source code in `src/ragas/llms/base.py` ```python @abstractmethod def generate( self, prompt: str, response_model: t.Type[InstructorTypeVar] ) -> InstructorTypeVar: """Generate a response using the configured LLM. For async clients, this will run the async method in the appropriate event loop. """ ``` ### agenerate ```python agenerate(prompt: str, response_model: Type[InstructorTypeVar]) -> InstructorTypeVar ``` Asynchronously generate a response using the configured LLM. Source code in `src/ragas/llms/base.py` ```python @abstractmethod async def agenerate( self, prompt: str, response_model: t.Type[InstructorTypeVar], ) -> InstructorTypeVar: """Asynchronously generate a response using the configured LLM.""" ``` ## InstructorLLM ```python InstructorLLM(client: Any, model: str, provider: str, model_args: Optional[InstructorModelArgs] = None, cache: Optional[CacheInterface] = None, **kwargs) ``` Bases: `InstructorBaseRagasLLM` LLM wrapper using the Instructor library for structured outputs. Source code in `src/ragas/llms/base.py` ```python def __init__( self, client: t.Any, model: str, provider: str, model_args: t.Optional[InstructorModelArgs] = None, cache: t.Optional[CacheInterface] = None, **kwargs, ): self.client = client self.model = model self.provider = provider # Use deterministic defaults if no model_args provided if model_args is None: model_args = InstructorModelArgs() # Convert to dict and merge with any additional kwargs self.model_args = {**model_args.model_dump(), **kwargs} # Extract system_prompt separately (not passed to LLM API) self.system_prompt = self.model_args.pop("system_prompt", None) self.cache = cache # Check if client is async-capable at initialization self.is_async = self._check_client_async() if self.cache is not None: self.generate = cacher(cache_backend=self.cache)(self.generate) # type: ignore self.agenerate = cacher(cache_backend=self.cache)(self.agenerate) # type: ignore ``` ### generate ```python generate(prompt: str, response_model: Type[InstructorTypeVar]) -> InstructorTypeVar ``` Generate a response using the configured LLM. For async clients, this will run the async method in the appropriate event loop. Source code in `src/ragas/llms/base.py` ```python def generate( self, prompt: str, response_model: t.Type[InstructorTypeVar] ) -> InstructorTypeVar: """Generate a response using the configured LLM. For async clients, this will run the async method in the appropriate event loop. """ messages = [] if self.system_prompt: messages.append({"role": "system", "content": self.system_prompt}) messages.append({"role": "user", "content": prompt}) # If client is async, use the appropriate method to run it if self.is_async: result = self._run_async_in_current_loop( self.agenerate(prompt, response_model) ) else: # Map parameters based on provider requirements provider_kwargs = self._map_provider_params() if self.provider.lower() == "google": result = self.client.create( model=self.model, messages=messages, response_model=response_model, **provider_kwargs, ) else: # OpenAI, Anthropic, LiteLLM result = self.client.chat.completions.create( model=self.model, messages=messages, response_model=response_model, **provider_kwargs, ) # Track the usage track( LLMUsageEvent( provider=self.provider, model=self.model, llm_type="instructor", num_requests=1, is_async=self.is_async, ) ) return result ``` ### agenerate ```python agenerate(prompt: str, response_model: Type[InstructorTypeVar]) -> InstructorTypeVar ``` Asynchronously generate a response using the configured LLM. Source code in `src/ragas/llms/base.py` ```python async def agenerate( self, prompt: str, response_model: t.Type[InstructorTypeVar], ) -> InstructorTypeVar: """Asynchronously generate a response using the configured LLM.""" messages = [] if self.system_prompt: messages.append({"role": "system", "content": self.system_prompt}) messages.append({"role": "user", "content": prompt}) # If client is not async, raise a helpful error if not self.is_async: raise TypeError( "Cannot use agenerate() with a synchronous client. Use generate() instead." ) # Map parameters based on provider requirements provider_kwargs = self._map_provider_params() if self.provider.lower() == "google": result = await self.client.create( model=self.model, messages=messages, response_model=response_model, **provider_kwargs, ) else: # OpenAI, Anthropic, LiteLLM result = await self.client.chat.completions.create( model=self.model, messages=messages, response_model=response_model, **provider_kwargs, ) # Track the usage track( LLMUsageEvent( provider=self.provider, model=self.model, llm_type="instructor", num_requests=1, is_async=True, ) ) return result ``` ## HaystackLLMWrapper ```python HaystackLLMWrapper(haystack_generator: Union[AzureOpenAIGenerator, HuggingFaceAPIGenerator, HuggingFaceLocalGenerator, OpenAIGenerator], run_config: Optional[RunConfig] = None, cache: Optional[CacheInterface] = None) ``` Bases: `BaseRagasLLM` A wrapper class for using Haystack LLM generators within the Ragas framework. This class integrates Haystack's LLM components (e.g., `OpenAIGenerator`, `HuggingFaceAPIGenerator`, etc.) into Ragas, enabling both synchronous and asynchronous text generation. Parameters: | Name | Type | Description | Default | | -------------------- | ---------------------- | ----------------------------------------------------------------------- | ------------------------- | | `haystack_generator` | \`AzureOpenAIGenerator | HuggingFaceAPIGenerator | HuggingFaceLocalGenerator | | `run_config` | `RunConfig` | Configuration object to manage LLM execution settings, by default None. | `None` | | `cache` | `CacheInterface` | A cache instance for storing results, by default None. | `None` | Source code in `src/ragas/llms/haystack_wrapper.py` ```python def __init__( self, haystack_generator: t.Union[ "AzureOpenAIGenerator", "HuggingFaceAPIGenerator", "HuggingFaceLocalGenerator", "OpenAIGenerator", ], run_config: t.Optional[RunConfig] = None, cache: t.Optional[CacheInterface] = None, ): super().__init__(cache=cache) # Lazy Import of required Haystack components try: from haystack import AsyncPipeline from haystack.components.generators.azure import AzureOpenAIGenerator from haystack.components.generators.hugging_face_api import ( HuggingFaceAPIGenerator, ) from haystack.components.generators.hugging_face_local import ( HuggingFaceLocalGenerator, ) from haystack.components.generators.openai import OpenAIGenerator except ImportError as exc: raise ImportError( "Haystack is not installed. Please install it using `pip install haystack-ai`." ) from exc # Validate haystack_generator type if not isinstance( haystack_generator, ( AzureOpenAIGenerator, HuggingFaceAPIGenerator, HuggingFaceLocalGenerator, OpenAIGenerator, ), ): raise TypeError( "Expected 'haystack_generator' to be one of: " "AzureOpenAIGenerator, HuggingFaceAPIGenerator, " "HuggingFaceLocalGenerator, or OpenAIGenerator, but received " f"{type(haystack_generator).__name__}." ) # Set up Haystack pipeline and generator self.generator = haystack_generator self.async_pipeline = AsyncPipeline() self.async_pipeline.add_component("llm", self.generator) # type: ignore[reportArgumentType] if run_config is None: run_config = RunConfig() self.set_run_config(run_config) ``` ## LiteLLMStructuredLLM ```python LiteLLMStructuredLLM(client: Any, model: str, provider: str, cache: Optional[CacheInterface] = None, system_prompt: Optional[str] = None, **kwargs) ``` Bases: `InstructorBaseRagasLLM` LLM wrapper using LiteLLM for structured outputs. Works with all 100+ LiteLLM-supported providers including Gemini, Ollama, vLLM, Groq, and many others. The LiteLLM client should be initialized with structured output support. Args: client: LiteLLM client instance model: Model name (e.g., "gemini-2.0-flash") provider: Provider name cache: Optional cache backend for caching LLM responses system_prompt: Optional system prompt to prepend to all messages \*\*kwargs: Additional model arguments (temperature, max_tokens, etc.) Source code in `src/ragas/llms/litellm_llm.py` ```python def __init__( self, client: t.Any, model: str, provider: str, cache: t.Optional[CacheInterface] = None, system_prompt: t.Optional[str] = None, **kwargs, ): """ Initialize LiteLLM structured LLM. Args: client: LiteLLM client instance model: Model name (e.g., "gemini-2.0-flash") provider: Provider name cache: Optional cache backend for caching LLM responses system_prompt: Optional system prompt to prepend to all messages **kwargs: Additional model arguments (temperature, max_tokens, etc.) """ self.client = client self.model = model self.provider = provider self.system_prompt = system_prompt self.model_args = kwargs self.cache = cache # Check if client is async-capable at initialization self.is_async = self._check_client_async() if self.cache is not None: self.generate = cacher(cache_backend=self.cache)(self.generate) # type: ignore self.agenerate = cacher(cache_backend=self.cache)(self.agenerate) # type: ignore ``` ### generate ```python generate(prompt: str, response_model: Type[InstructorTypeVar]) -> InstructorTypeVar ``` Generate a response using the configured LLM. For async clients, this will run the async method in the appropriate event loop. Args: prompt: Input prompt response_model: Pydantic model for structured output Returns: Instance of response_model with generated data Source code in `src/ragas/llms/litellm_llm.py` ```python def generate( self, prompt: str, response_model: t.Type[InstructorTypeVar] ) -> InstructorTypeVar: """Generate a response using the configured LLM. For async clients, this will run the async method in the appropriate event loop. Args: prompt: Input prompt response_model: Pydantic model for structured output Returns: Instance of response_model with generated data """ messages = [] if self.system_prompt: messages.append({"role": "system", "content": self.system_prompt}) messages.append({"role": "user", "content": prompt}) # If client is async, use the appropriate method to run it if self.is_async: result = self._run_async_in_current_loop( self.agenerate(prompt, response_model) ) else: # Call LiteLLM with structured output result = self.client.chat.completions.create( model=self.model, messages=messages, response_model=response_model, **self.model_args, ) # Track the usage track( LLMUsageEvent( provider=self.provider, model=self.model, llm_type="litellm", num_requests=1, is_async=self.is_async, ) ) return result ``` ### agenerate ```python agenerate(prompt: str, response_model: Type[InstructorTypeVar]) -> InstructorTypeVar ``` Asynchronously generate a response using the configured LLM. Args: prompt: Input prompt response_model: Pydantic model for structured output Returns: Instance of response_model with generated data Source code in `src/ragas/llms/litellm_llm.py` ```python async def agenerate( self, prompt: str, response_model: t.Type[InstructorTypeVar], ) -> InstructorTypeVar: """Asynchronously generate a response using the configured LLM. Args: prompt: Input prompt response_model: Pydantic model for structured output Returns: Instance of response_model with generated data """ messages = [] if self.system_prompt: messages.append({"role": "system", "content": self.system_prompt}) messages.append({"role": "user", "content": prompt}) # If client is not async, raise a helpful error if not self.is_async: raise TypeError( "Cannot use agenerate() with a synchronous client. Use generate() instead." ) # Call LiteLLM async with structured output result = await self.client.chat.completions.create( model=self.model, messages=messages, response_model=response_model, **self.model_args, ) # Track the usage track( LLMUsageEvent( provider=self.provider, model=self.model, llm_type="litellm", num_requests=1, is_async=True, ) ) return result ``` ## OCIGenAIWrapper ```python OCIGenAIWrapper(model_id: str, compartment_id: str, config: Optional[Dict[str, Any]] = None, endpoint_id: Optional[str] = None, run_config: Optional[RunConfig] = None, cache: Optional[Any] = None, default_system_prompt: Optional[str] = None, client: Optional[Any] = None) ``` Bases: `BaseRagasLLM` OCI Gen AI LLM wrapper for Ragas. This wrapper provides direct integration with Oracle Cloud Infrastructure Generative AI services without requiring LangChain or LlamaIndex. Args: model_id: The OCI model ID to use for generation compartment_id: The OCI compartment ID config: OCI configuration dictionary (optional, uses default if not provided) endpoint_id: Optional endpoint ID for the model run_config: Ragas run configuration cache: Optional cache backend Source code in `src/ragas/llms/oci_genai_wrapper.py` ```python def __init__( self, model_id: str, compartment_id: str, config: t.Optional[t.Dict[str, t.Any]] = None, endpoint_id: t.Optional[str] = None, run_config: t.Optional[RunConfig] = None, cache: t.Optional[t.Any] = None, default_system_prompt: t.Optional[str] = None, client: t.Optional[t.Any] = None, ): """ Initialize OCI Gen AI wrapper. Args: model_id: The OCI model ID to use for generation compartment_id: The OCI compartment ID config: OCI configuration dictionary (optional, uses default if not provided) endpoint_id: Optional endpoint ID for the model run_config: Ragas run configuration cache: Optional cache backend """ super().__init__(cache=cache) self.model_id = model_id self.compartment_id = compartment_id self.endpoint_id = endpoint_id self.default_system_prompt = default_system_prompt # Store client/config; perform lazy initialization to keep import-optional self.client = client self._oci_config = config # If no client and SDK not available and no endpoint fallback, raise early if ( self.client is None and GenerativeAiClient is None and self.endpoint_id is None ): # type: ignore raise ImportError( "OCI SDK not found. Please install it with: pip install oci" ) # Set run config if run_config is None: run_config = RunConfig() self.set_run_config(run_config) # Track initialization track( LLMUsageEvent( provider="oci_genai", model=model_id, llm_type="oci_wrapper", num_requests=1, is_async=False, ) ) ``` ### generate_text ```python generate_text(prompt: PromptValue, n: int = 1, temperature: Optional[float] = 0.01, stop: Optional[List[str]] = None, callbacks: Optional[Any] = None) -> LLMResult ``` Generate text using OCI Gen AI. Source code in `src/ragas/llms/oci_genai_wrapper.py` ```python def generate_text( self, prompt: PromptValue, n: int = 1, temperature: t.Optional[float] = 0.01, stop: t.Optional[t.List[str]] = None, callbacks: t.Optional[t.Any] = None, ) -> LLMResult: """Generate text using OCI Gen AI.""" if temperature is None: temperature = self.get_temperature(n) messages = self._convert_prompt_to_messages(prompt) generations = [] try: for _ in range(n): request = self._create_generation_request( messages, temperature, stop=stop ) response = self._get_client().generate_text(**request) # Extract text from response if hasattr(response.data, "choices") and response.data.choices: text = response.data.choices[0].message.content elif hasattr(response.data, "text"): text = response.data.text else: text = str(response.data) generation = Generation(text=text) generations.append([generation]) # Track usage track( LLMUsageEvent( provider="oci_genai", model=self.model_id, llm_type="oci_wrapper", num_requests=n, is_async=False, ) ) return LLMResult(generations=generations) except Exception as e: logger.error(f"Error generating text with OCI Gen AI: {e}") raise ``` ### agenerate_text ```python agenerate_text(prompt: PromptValue, n: int = 1, temperature: Optional[float] = 0.01, stop: Optional[List[str]] = None, callbacks: Optional[Any] = None) -> LLMResult ``` Generate text asynchronously using OCI Gen AI. Source code in `src/ragas/llms/oci_genai_wrapper.py` ```python async def agenerate_text( self, prompt: PromptValue, n: int = 1, temperature: t.Optional[float] = 0.01, stop: t.Optional[t.List[str]] = None, callbacks: t.Optional[t.Any] = None, ) -> LLMResult: """Generate text asynchronously using OCI Gen AI.""" if temperature is None: temperature = self.get_temperature(n) messages = self._convert_prompt_to_messages(prompt) generations = [] try: # Run synchronous calls in thread pool for async compatibility loop = asyncio.get_event_loop() for _ in range(n): request = self._create_generation_request( messages, temperature, stop=stop ) response = await loop.run_in_executor( None, lambda: self._get_client().generate_text(**request) ) # Extract text from response if hasattr(response.data, "choices") and response.data.choices: text = response.data.choices[0].message.content elif hasattr(response.data, "text"): text = response.data.text else: text = str(response.data) generation = Generation(text=text) generations.append([generation]) # Track usage track( LLMUsageEvent( provider="oci_genai", model=self.model_id, llm_type="oci_wrapper", num_requests=n, is_async=True, ) ) return LLMResult(generations=generations) except Exception as e: logger.error(f"Error generating text with OCI Gen AI: {e}") raise ``` ### is_finished ```python is_finished(response: LLMResult) -> bool ``` Check if the LLM response is finished/complete. Source code in `src/ragas/llms/oci_genai_wrapper.py` ```python def is_finished(self, response: LLMResult) -> bool: """Check if the LLM response is finished/complete.""" # For OCI Gen AI, we assume the response is always finished # unless there's an explicit error or truncation try: for generation_list in response.generations: for generation in generation_list: if not generation.text or generation.text.strip() == "": return False return True except Exception: return False ``` ## llm_factory ```python llm_factory(model: str, provider: str = 'openai', client: Optional[Any] = None, adapter: str = 'auto', cache: Optional[CacheInterface] = None, **kwargs: Any) -> InstructorBaseRagasLLM ``` Create an LLM instance for structured output generation with automatic adapter selection. Supports multiple LLM providers and structured output backends with unified interface for both sync and async operations. Returns instances with .generate() and .agenerate() methods that accept Pydantic models for structured outputs. Auto-detects the best adapter for your provider: - Google Gemini β†’ uses LiteLLM adapter - Other providers β†’ uses Instructor adapter (default) - Explicit control available via adapter parameter Args: model: Model name (e.g., "gpt-4o", "claude-3-sonnet", "gemini-2.0-flash"). provider: LLM provider (default: "openai"). Examples: openai, anthropic, google, groq, mistral, etc. client: Pre-initialized client instance (required). For OpenAI, can be OpenAI(...) or AsyncOpenAI(...). adapter: Structured output adapter to use (default: "auto"). - "auto": Auto-detect based on provider/client (recommended) - "instructor": Use Instructor library - "litellm": Use LiteLLM (supports 100+ providers) cache: Optional cache backend for caching LLM responses. Pass DiskCacheBackend() for persistent caching across runs. Saves costs and speeds up repeated evaluations by 60x. \*\*kwargs: Additional model arguments (temperature, max_tokens, top_p, etc). Returns: InstructorBaseRagasLLM: Instance with generate() and agenerate() methods. Raises: ValueError: If client is missing, provider is unsupported, model is invalid, or adapter initialization fails. Examples: from openai import OpenAI ```text # Basic usage client = OpenAI(api_key="...") llm = llm_factory("gpt-4o-mini", client=client) response = llm.generate(prompt, ResponseModel) # With caching (recommended for experiments) from ragas.cache import DiskCacheBackend cache = DiskCacheBackend() llm = llm_factory("gpt-4o-mini", client=client, cache=cache) # Anthropic from anthropic import Anthropic client = Anthropic(api_key="...") llm = llm_factory("claude-3-sonnet", provider="anthropic", client=client) # Google Gemini (auto-detects litellm adapter) from litellm import OpenAI as LiteLLMClient client = LiteLLMClient(api_key="...", model="gemini-2.0-flash") llm = llm_factory("gemini-2.0-flash", client=client) # Explicit adapter selection llm = llm_factory("gemini-2.0-flash", client=client, adapter="litellm") # Async from openai import AsyncOpenAI client = AsyncOpenAI(api_key="...") llm = llm_factory("gpt-4o-mini", client=client) response = await llm.agenerate(prompt, ResponseModel) ``` Source code in `src/ragas/llms/base.py` ```python def llm_factory( model: str, provider: str = "openai", client: t.Optional[t.Any] = None, adapter: str = "auto", cache: t.Optional[CacheInterface] = None, **kwargs: t.Any, ) -> InstructorBaseRagasLLM: """ Create an LLM instance for structured output generation with automatic adapter selection. Supports multiple LLM providers and structured output backends with unified interface for both sync and async operations. Returns instances with .generate() and .agenerate() methods that accept Pydantic models for structured outputs. Auto-detects the best adapter for your provider: - Google Gemini β†’ uses LiteLLM adapter - Other providers β†’ uses Instructor adapter (default) - Explicit control available via adapter parameter Args: model: Model name (e.g., "gpt-4o", "claude-3-sonnet", "gemini-2.0-flash"). provider: LLM provider (default: "openai"). Examples: openai, anthropic, google, groq, mistral, etc. client: Pre-initialized client instance (required). For OpenAI, can be OpenAI(...) or AsyncOpenAI(...). adapter: Structured output adapter to use (default: "auto"). - "auto": Auto-detect based on provider/client (recommended) - "instructor": Use Instructor library - "litellm": Use LiteLLM (supports 100+ providers) cache: Optional cache backend for caching LLM responses. Pass DiskCacheBackend() for persistent caching across runs. Saves costs and speeds up repeated evaluations by 60x. **kwargs: Additional model arguments (temperature, max_tokens, top_p, etc). Returns: InstructorBaseRagasLLM: Instance with generate() and agenerate() methods. Raises: ValueError: If client is missing, provider is unsupported, model is invalid, or adapter initialization fails. Examples: from openai import OpenAI # Basic usage client = OpenAI(api_key="...") llm = llm_factory("gpt-4o-mini", client=client) response = llm.generate(prompt, ResponseModel) # With caching (recommended for experiments) from ragas.cache import DiskCacheBackend cache = DiskCacheBackend() llm = llm_factory("gpt-4o-mini", client=client, cache=cache) # Anthropic from anthropic import Anthropic client = Anthropic(api_key="...") llm = llm_factory("claude-3-sonnet", provider="anthropic", client=client) # Google Gemini (auto-detects litellm adapter) from litellm import OpenAI as LiteLLMClient client = LiteLLMClient(api_key="...", model="gemini-2.0-flash") llm = llm_factory("gemini-2.0-flash", client=client) # Explicit adapter selection llm = llm_factory("gemini-2.0-flash", client=client, adapter="litellm") # Async from openai import AsyncOpenAI client = AsyncOpenAI(api_key="...") llm = llm_factory("gpt-4o-mini", client=client) response = await llm.agenerate(prompt, ResponseModel) """ if client is None: raise ValueError( "llm_factory() requires a client instance. " "Text-only mode has been removed.\n\n" "To migrate:\n" " from openai import OpenAI\n" " client = OpenAI(api_key='...')\n" " llm = llm_factory('gpt-4o-mini', client=client)\n\n" "For more details: https://docs.ragas.io/en/latest/llm-factory" ) if not model: raise ValueError("model parameter is required") provider_lower = provider.lower() # Auto-detect adapter if needed if adapter == "auto": from ragas.llms.adapters import auto_detect_adapter adapter = auto_detect_adapter(client, provider_lower) # Create LLM using selected adapter from ragas.llms.adapters import get_adapter try: adapter_instance = get_adapter(adapter) llm = adapter_instance.create_llm( client, model, provider_lower, cache=cache, **kwargs ) except ValueError as e: # Re-raise ValueError from get_adapter for unknown adapter names # Also handle adapter initialization failures if "Unknown adapter" in str(e): raise # Adapter-specific failures get wrapped raise ValueError( f"Failed to initialize {provider} client with {adapter} adapter. " f"Ensure you've created a valid {provider} client.\n" f"Error: {str(e)}" ) except Exception as e: raise ValueError( f"Failed to initialize {provider} client with {adapter} adapter. " f"Ensure you've created a valid {provider} client.\n" f"Error: {str(e)}" ) track( LLMUsageEvent( provider=provider, model=model, llm_type="llm_factory", num_requests=1, is_async=False, ) ) return llm ``` ## oci_genai_factory ```python oci_genai_factory(model_id: str, compartment_id: str, config: Optional[Dict[str, Any]] = None, endpoint_id: Optional[str] = None, run_config: Optional[RunConfig] = None, cache: Optional[Any] = None, default_system_prompt: Optional[str] = None, client: Optional[Any] = None) -> OCIGenAIWrapper ``` Factory function to create an OCI Gen AI LLM instance. Args: model_id: The OCI model ID to use for generation compartment_id: The OCI compartment ID config: OCI configuration dictionary (optional) endpoint_id: Optional endpoint ID for the model run_config: Ragas run configuration \*\*kwargs: Additional arguments passed to OCIGenAIWrapper Returns: OCIGenAIWrapper: An instance of the OCI Gen AI LLM wrapper Examples: # Basic usage with default config llm = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example" ) ```text # With custom config llm = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example", config={"user": "user_ocid", "key_file": "~/.oci/private_key.pem"} ) ``` Source code in `src/ragas/llms/oci_genai_wrapper.py` ```python def oci_genai_factory( model_id: str, compartment_id: str, config: t.Optional[t.Dict[str, t.Any]] = None, endpoint_id: t.Optional[str] = None, run_config: t.Optional[RunConfig] = None, cache: t.Optional[t.Any] = None, default_system_prompt: t.Optional[str] = None, client: t.Optional[t.Any] = None, ) -> OCIGenAIWrapper: """ Factory function to create an OCI Gen AI LLM instance. Args: model_id: The OCI model ID to use for generation compartment_id: The OCI compartment ID config: OCI configuration dictionary (optional) endpoint_id: Optional endpoint ID for the model run_config: Ragas run configuration **kwargs: Additional arguments passed to OCIGenAIWrapper Returns: OCIGenAIWrapper: An instance of the OCI Gen AI LLM wrapper Examples: # Basic usage with default config llm = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example" ) # With custom config llm = oci_genai_factory( model_id="cohere.command", compartment_id="ocid1.compartment.oc1..example", config={"user": "user_ocid", "key_file": "~/.oci/private_key.pem"} ) """ return OCIGenAIWrapper( model_id=model_id, compartment_id=compartment_id, config=config, endpoint_id=endpoint_id, run_config=run_config, cache=cache, default_system_prompt=default_system_prompt, client=client, ) ``` ## MetricType Bases: `Enum` Enumeration of metric types in Ragas. Attributes: | Name | Type | Description | | ------------- | ----- | ------------------------------------- | | `SINGLE_TURN` | `str` | Represents a single-turn metric type. | | `MULTI_TURN` | `str` | Represents a multi-turn metric type. | ## Metric ```python Metric(_required_columns: Dict[MetricType, Set[str]] = dict(), name: str = '') ``` Bases: `ABC` Abstract base class for metrics in Ragas. Attributes: | Name | Type | Description | | ------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` | `str` | The name of the metric. | | `required_columns` | `Dict[str, Set[str]]` | A dictionary mapping metric type names to sets of required column names. This is a property and raises ValueError if columns are not in VALID_COLUMNS. | ### init ```python init(run_config: RunConfig) -> None ``` Initialize the metric with the given run configuration. Parameters: | Name | Type | Description | Default | | ------------ | ----------- | ----------------------------------------------------------------------- | ---------- | | `run_config` | `RunConfig` | Configuration for the metric run including timeouts and other settings. | *required* | Source code in `src/ragas/metrics/base.py` ```python @abstractmethod def init(self, run_config: RunConfig) -> None: """ Initialize the metric with the given run configuration. Parameters ---------- run_config : RunConfig Configuration for the metric run including timeouts and other settings. """ ... ``` ## MetricWithLLM ```python MetricWithLLM(_required_columns: Dict[MetricType, Set[str]] = dict(), name: str = '', llm: Optional[BaseRagasLLM] = None, output_type: Optional[MetricOutputType] = None) ``` Bases: `Metric`, `PromptMixin` A metric class that uses a language model for evaluation. Attributes: | Name | Type | Description | | ----- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `llm` | `Optional[BaseRagasLLM]` | The language model used for the metric. Both BaseRagasLLM and InstructorBaseRagasLLM are accepted at runtime via duck typing (both have compatible methods). | ### init ```python init(run_config: RunConfig) -> None ``` Initialize the metric with run configuration and validate LLM is present. Parameters: | Name | Type | Description | Default | | ------------ | ----------- | --------------------------------- | ---------- | | `run_config` | `RunConfig` | Configuration for the metric run. | *required* | Raises: | Type | Description | | ------------ | ------------------------------------ | | `ValueError` | If no LLM is provided to the metric. | Source code in `src/ragas/metrics/base.py` ```python def init(self, run_config: RunConfig) -> None: """ Initialize the metric with run configuration and validate LLM is present. Parameters ---------- run_config : RunConfig Configuration for the metric run. Raises ------ ValueError If no LLM is provided to the metric. """ if self.llm is None: raise ValueError( f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please instantiate the metric with an LLM to run." ) # Only BaseRagasLLM has set_run_config method, not InstructorBaseRagasLLM if isinstance(self.llm, BaseRagasLLM): self.llm.set_run_config(run_config) ``` ### train ```python train(path: str, demonstration_config: Optional[DemonstrationConfig] = None, instruction_config: Optional[InstructionConfig] = None, callbacks: Optional[Callbacks] = None, run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, with_debugging_logs=False, raise_exceptions: bool = True) -> None ``` Train the metric using local JSON data Parameters: | Name | Type | Description | Default | | ---------------------- | --------------------- | -------------------------------------------- | ---------- | | `path` | `str` | Path to local JSON training data file | *required* | | `demonstration_config` | `DemonstrationConfig` | Configuration for demonstration optimization | `None` | | `instruction_config` | `InstructionConfig` | Configuration for instruction optimization | `None` | | `callbacks` | `Callbacks` | List of callback functions | `None` | | `run_config` | `RunConfig` | Run configuration | `None` | | `batch_size` | `int` | Batch size for training | `None` | | `with_debugging_logs` | `bool` | Enable debugging logs | `False` | | `raise_exceptions` | `bool` | Whether to raise exceptions during training | `True` | Raises: | Type | Description | | ------------ | ------------------------------------------ | | `ValueError` | If path is not provided or not a JSON file | Source code in `src/ragas/metrics/base.py` ```python def train( self, path: str, demonstration_config: t.Optional[DemonstrationConfig] = None, instruction_config: t.Optional[InstructionConfig] = None, callbacks: t.Optional[Callbacks] = None, run_config: t.Optional[RunConfig] = None, batch_size: t.Optional[int] = None, with_debugging_logs=False, raise_exceptions: bool = True, ) -> None: """ Train the metric using local JSON data Parameters ---------- path : str Path to local JSON training data file demonstration_config : DemonstrationConfig, optional Configuration for demonstration optimization instruction_config : InstructionConfig, optional Configuration for instruction optimization callbacks : Callbacks, optional List of callback functions run_config : RunConfig, optional Run configuration batch_size : int, optional Batch size for training with_debugging_logs : bool, default=False Enable debugging logs raise_exceptions : bool, default=True Whether to raise exceptions during training Raises ------ ValueError If path is not provided or not a JSON file """ # Validate input parameters if not path: raise ValueError("Path to training data file must be provided") if not path.endswith(".json"): raise ValueError("Train data must be in json format") run_config = run_config or RunConfig() callbacks = callbacks or [] # Load the dataset from JSON file dataset = MetricAnnotation.from_json(path, metric_name=self.name) # only optimize the instruction if instruction_config is provided if instruction_config is not None: self._optimize_instruction( instruction_config=instruction_config, dataset=dataset, callbacks=callbacks, run_config=run_config, batch_size=batch_size, with_debugging_logs=with_debugging_logs, raise_exceptions=raise_exceptions, ) # if demonstration_config is provided, optimize the demonstrations if demonstration_config is not None: self._optimize_demonstration( demonstration_config=demonstration_config, dataset=dataset, ) ``` ## SingleTurnMetric ```python SingleTurnMetric(_required_columns: Dict[MetricType, Set[str]] = dict(), name: str = '') ``` Bases: `Metric` A metric class for evaluating single-turn interactions. This class provides methods to score single-turn samples, both synchronously and asynchronously. ### single_turn_score ```python single_turn_score(sample: SingleTurnSample, callbacks: Callbacks = None) -> float ``` Synchronously score a single-turn sample. May raise ImportError if nest_asyncio is not installed in a Jupyter-like environment. Source code in `src/ragas/metrics/base.py` ```python def single_turn_score( self, sample: SingleTurnSample, callbacks: Callbacks = None, ) -> float: """ Synchronously score a single-turn sample. May raise ImportError if nest_asyncio is not installed in a Jupyter-like environment. """ callbacks = callbacks or [] # only get the required columns sample = self._only_required_columns_single_turn(sample) rm, group_cm = new_group( self.name, inputs=sample.to_dict(), callbacks=callbacks, metadata={"type": ChainType.METRIC}, ) async def _async_wrapper(): try: result = await self._single_turn_ascore( sample=sample, callbacks=group_cm ) except Exception as e: if not group_cm.ended: rm.on_chain_error(e) raise e else: if not group_cm.ended: rm.on_chain_end({"output": result}) return result apply_nest_asyncio() score = run(_async_wrapper) # track the evaluation event _analytics_batcher.add_evaluation( EvaluationEvent( metrics=[self.name], num_rows=1, evaluation_type=MetricType.SINGLE_TURN.name, language=get_metric_language(self), ) ) return score ``` ### single_turn_ascore ```python single_turn_ascore(sample: SingleTurnSample, callbacks: Callbacks = None, timeout: Optional[float] = None) -> float ``` Asynchronously score a single-turn sample with an optional timeout. May raise asyncio.TimeoutError if the scoring process exceeds the specified timeout. Source code in `src/ragas/metrics/base.py` ```python async def single_turn_ascore( self, sample: SingleTurnSample, callbacks: Callbacks = None, timeout: t.Optional[float] = None, ) -> float: """ Asynchronously score a single-turn sample with an optional timeout. May raise asyncio.TimeoutError if the scoring process exceeds the specified timeout. """ callbacks = callbacks or [] # only get the required columns sample = self._only_required_columns_single_turn(sample) rm, group_cm = new_group( self.name, inputs=sample.to_dict(), callbacks=callbacks, metadata={"type": ChainType.METRIC}, ) try: score = await asyncio.wait_for( self._single_turn_ascore(sample=sample, callbacks=group_cm), timeout=timeout, ) except Exception as e: if not group_cm.ended: rm.on_chain_error(e) raise e else: if not group_cm.ended: rm.on_chain_end({"output": score}) # track the evaluation event _analytics_batcher.add_evaluation( EvaluationEvent( metrics=[self.name], num_rows=1, evaluation_type=MetricType.SINGLE_TURN.name, language=get_metric_language(self), ) ) return score ``` ## MultiTurnMetric ```python MultiTurnMetric(_required_columns: Dict[MetricType, Set[str]] = dict(), name: str = '') ``` Bases: `Metric` A metric class for evaluating multi-turn conversations. This class extends the base Metric class to provide functionality for scoring multi-turn conversation samples. ### multi_turn_score ```python multi_turn_score(sample: MultiTurnSample, callbacks: Callbacks = None) -> float ``` Score a multi-turn conversation sample synchronously. May raise ImportError if nest_asyncio is not installed in Jupyter-like environments. Source code in `src/ragas/metrics/base.py` ```python def multi_turn_score( self, sample: MultiTurnSample, callbacks: Callbacks = None, ) -> float: """ Score a multi-turn conversation sample synchronously. May raise ImportError if nest_asyncio is not installed in Jupyter-like environments. """ callbacks = callbacks or [] sample = self._only_required_columns_multi_turn(sample) rm, group_cm = new_group( self.name, inputs=sample.to_dict(), callbacks=callbacks, metadata={"type": ChainType.METRIC}, ) async def _async_wrapper(): try: result = await self._multi_turn_ascore( sample=sample, callbacks=group_cm ) except Exception as e: if not group_cm.ended: rm.on_chain_error(e) raise e else: if not group_cm.ended: rm.on_chain_end({"output": result}) return result apply_nest_asyncio() score = run(_async_wrapper) # track the evaluation event _analytics_batcher.add_evaluation( EvaluationEvent( metrics=[self.name], num_rows=1, evaluation_type=MetricType.SINGLE_TURN.name, language=get_metric_language(self), ) ) return score ``` ### multi_turn_ascore ```python multi_turn_ascore(sample: MultiTurnSample, callbacks: Callbacks = None, timeout: Optional[float] = None) -> float ``` Score a multi-turn conversation sample asynchronously. May raise asyncio.TimeoutError if the scoring process exceeds the specified timeout. Source code in `src/ragas/metrics/base.py` ```python async def multi_turn_ascore( self, sample: MultiTurnSample, callbacks: Callbacks = None, timeout: t.Optional[float] = None, ) -> float: """ Score a multi-turn conversation sample asynchronously. May raise asyncio.TimeoutError if the scoring process exceeds the specified timeout. """ callbacks = callbacks or [] sample = self._only_required_columns_multi_turn(sample) rm, group_cm = new_group( self.name, inputs=sample.to_dict(), callbacks=callbacks, metadata={"type": ChainType.METRIC}, ) try: score = await asyncio.wait_for( self._multi_turn_ascore(sample=sample, callbacks=group_cm), timeout=timeout, ) except Exception as e: if not group_cm.ended: rm.on_chain_error(e) raise e else: if not group_cm.ended: rm.on_chain_end({"output": score}) # track the evaluation event _analytics_batcher.add_evaluation( EvaluationEvent( metrics=[self.name], num_rows=1, evaluation_type=MetricType.SINGLE_TURN.name, language=get_metric_language(self), ) ) return score ``` ## Ensember Combine multiple llm outputs for same input (n>1) to a single output ### from_discrete ```python from_discrete(inputs: list[list[Dict]], attribute: str) -> List[Dict] ``` Simple majority voting for binary values, ie [0,0,1] -> 0 inputs: list of list of dicts each containing verdict for a single input Source code in `src/ragas/metrics/base.py` ```python def from_discrete( self, inputs: list[list[t.Dict]], attribute: str ) -> t.List[t.Dict]: """ Simple majority voting for binary values, ie [0,0,1] -> 0 inputs: list of list of dicts each containing verdict for a single input """ if not isinstance(inputs, list): inputs = [inputs] if not all(len(item) == len(inputs[0]) for item in inputs): logger.warning("All inputs must have the same length") return inputs[0] if not all(attribute in item for input in inputs for item in input): logger.warning(f"All inputs must have {attribute} attribute") return inputs[0] if len(inputs) == 1: return inputs[0] verdict_agg = [] for i in range(len(inputs[0])): item = inputs[0][i] verdicts = [inputs[k][i][attribute] for k in range(len(inputs))] verdict_counts = dict(Counter(verdicts).most_common()) item[attribute] = list(verdict_counts.keys())[0] verdict_agg.append(item) return verdict_agg ``` ## SimpleBaseMetric ```python SimpleBaseMetric(name: str, allowed_values: AllowedValuesType = (lambda: ['pass', 'fail'])()) ``` Bases: `ABC` Base class for simple metrics that return MetricResult objects. This class provides the foundation for metrics that evaluate inputs and return structured MetricResult objects containing scores and reasoning. Attributes: | Name | Type | Description | | ---------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | `str` | The name of the metric. | | `allowed_values` | `AllowedValuesType` | Allowed values for the metric output. Can be a list of strings for discrete metrics, a tuple of floats for numeric metrics, or an integer for ranking metrics. | Examples: ```pycon >>> from ragas.metrics import discrete_metric >>> >>> @discrete_metric(name="sentiment", allowed_values=["positive", "negative"]) >>> def sentiment_metric(user_input: str, response: str) -> str: ... return "positive" if "good" in response else "negative" >>> >>> result = sentiment_metric(user_input="How are you?", response="I'm good!") >>> print(result.value) # "positive" ``` ### score ```python score(**kwargs) -> 'MetricResult' ``` Synchronously calculate the metric score. Parameters: | Name | Type | Description | Default | | ---------- | ------ | ---------------------------------------------------------------- | ------- | | `**kwargs` | `dict` | Input parameters required by the specific metric implementation. | `{}` | Returns: | Type | Description | | -------------- | --------------------------------------------------------- | | `MetricResult` | The evaluation result containing the score and reasoning. | Source code in `src/ragas/metrics/base.py` ```python @abstractmethod def score(self, **kwargs) -> "MetricResult": """ Synchronously calculate the metric score. Parameters ---------- **kwargs : dict Input parameters required by the specific metric implementation. Returns ------- MetricResult The evaluation result containing the score and reasoning. """ pass ``` ### ascore ```python ascore(**kwargs) -> 'MetricResult' ``` Asynchronously calculate the metric score. Parameters: | Name | Type | Description | Default | | ---------- | ------ | ---------------------------------------------------------------- | ------- | | `**kwargs` | `dict` | Input parameters required by the specific metric implementation. | `{}` | Returns: | Type | Description | | -------------- | --------------------------------------------------------- | | `MetricResult` | The evaluation result containing the score and reasoning. | Source code in `src/ragas/metrics/base.py` ```python @abstractmethod async def ascore(self, **kwargs) -> "MetricResult": """ Asynchronously calculate the metric score. Parameters ---------- **kwargs : dict Input parameters required by the specific metric implementation. Returns ------- MetricResult The evaluation result containing the score and reasoning. """ pass ``` ### batch_score ```python batch_score(inputs: List[Dict[str, Any]]) -> List['MetricResult'] ``` Synchronously calculate scores for a batch of inputs. Parameters: | Name | Type | Description | Default | | -------- | ---------------------- | ---------------------------------------------------------------------- | ---------- | | `inputs` | `List[Dict[str, Any]]` | List of input dictionaries, each containing parameters for the metric. | *required* | Returns: | Type | Description | | -------------------- | ----------------------------------------------- | | `List[MetricResult]` | List of evaluation results, one for each input. | Source code in `src/ragas/metrics/base.py` ```python def batch_score( self, inputs: t.List[t.Dict[str, t.Any]], ) -> t.List["MetricResult"]: """ Synchronously calculate scores for a batch of inputs. Parameters ---------- inputs : List[Dict[str, Any]] List of input dictionaries, each containing parameters for the metric. Returns ------- List[MetricResult] List of evaluation results, one for each input. """ return [self.score(**input_dict) for input_dict in inputs] ``` ### abatch_score ```python abatch_score(inputs: List[Dict[str, Any]]) -> List['MetricResult'] ``` Asynchronously calculate scores for a batch of inputs in parallel. Parameters: | Name | Type | Description | Default | | -------- | ---------------------- | ---------------------------------------------------------------------- | ---------- | | `inputs` | `List[Dict[str, Any]]` | List of input dictionaries, each containing parameters for the metric. | *required* | Returns: | Type | Description | | -------------------- | ----------------------------------------------- | | `List[MetricResult]` | List of evaluation results, one for each input. | Source code in `src/ragas/metrics/base.py` ```python async def abatch_score( self, inputs: t.List[t.Dict[str, t.Any]], ) -> t.List["MetricResult"]: """ Asynchronously calculate scores for a batch of inputs in parallel. Parameters ---------- inputs : List[Dict[str, Any]] List of input dictionaries, each containing parameters for the metric. Returns ------- List[MetricResult] List of evaluation results, one for each input. """ async_tasks = [] for input_dict in inputs: # Process input asynchronously async_tasks.append(self.ascore(**input_dict)) # Run all tasks concurrently and return results return await asyncio.gather(*async_tasks) ``` ## SimpleLLMMetric ```python SimpleLLMMetric(name: str, allowed_values: AllowedValuesType = (lambda: ['pass', 'fail'])(), prompt: Optional[Union[str, 'Prompt']] = None) ``` Bases: `SimpleBaseMetric` LLM-based metric that uses prompts to generate structured responses. ### save ```python save(path: Optional[str] = None) -> None ``` Save the metric configuration to a JSON file. Parameters: path : str, optional File path to save to. If not provided, saves to "./{metric.name}.json" Use .gz extension for compression. Note: If the metric has a response_model, its schema will be saved for reference but the model itself cannot be serialized. You'll need to provide it when loading. Examples: All these work: > > > metric.save() # β†’ ./response_quality.json metric.save("custom.json") # β†’ ./custom.json metric.save("/path/to/metrics/") # β†’ /path/to/metrics/response_quality.json metric.save("no_extension") # β†’ ./no_extension.json metric.save("compressed.json.gz") # β†’ ./compressed.json.gz (compressed) Source code in `src/ragas/metrics/base.py` ```python def save(self, path: t.Optional[str] = None) -> None: """ Save the metric configuration to a JSON file. Parameters: ----------- path : str, optional File path to save to. If not provided, saves to "./{metric.name}.json" Use .gz extension for compression. Note: ----- If the metric has a response_model, its schema will be saved for reference but the model itself cannot be serialized. You'll need to provide it when loading. Examples: --------- All these work: >>> metric.save() # β†’ ./response_quality.json >>> metric.save("custom.json") # β†’ ./custom.json >>> metric.save("/path/to/metrics/") # β†’ /path/to/metrics/response_quality.json >>> metric.save("no_extension") # β†’ ./no_extension.json >>> metric.save("compressed.json.gz") # β†’ ./compressed.json.gz (compressed) """ import gzip import json import warnings from pathlib import Path # Handle default path if path is None: # Default to current directory with metric name as filename file_path = Path(f"./{self.name}.json") else: file_path = Path(path) # If path is a directory, append the metric name as filename if file_path.is_dir(): file_path = file_path / f"{self.name}.json" # If path has no extension, add .json elif not file_path.suffix: file_path = file_path.with_suffix(".json") # Collect warning messages for data loss warning_messages = [] if hasattr(self, "_response_model") and self._response_model: # Only warn for custom response models, not auto-generated ones if not getattr(self._response_model, "__ragas_auto_generated__", False): warning_messages.append( "- Custom response_model will be lost (set it manually after loading)" ) # Serialize the prompt (may add embedding_model warning) prompt_data = self._serialize_prompt(warning_messages) # Determine the metric type metric_type = self.__class__.__name__ # Get metric-specific config config = self._get_metric_config() # Emit consolidated warning if there's data loss if warning_messages: warnings.warn( "Some metric components cannot be saved and will be lost:\n" + "\n".join(warning_messages) + "\n\nYou'll need to provide these when loading the metric." ) data = { "format_version": "1.0", "metric_type": metric_type, "name": self.name, "prompt": prompt_data, "config": config, "response_model_info": self._serialize_response_model_info(), } try: if file_path.suffix == ".gz": with gzip.open(file_path, "wt", encoding="utf-8") as f: json.dump(data, f, indent=2) else: with open(file_path, "w", encoding="utf-8") as f: json.dump(data, f, indent=2) except (OSError, IOError) as e: raise ValueError(f"Cannot save metric to {file_path}: {e}") ``` ### load ```python load(path: str, response_model: Optional[Type['BaseModel']] = None, embedding_model: Optional['EmbeddingModelType'] = None) -> 'SimpleLLMMetric' ``` Load a metric from a JSON file. Parameters: path : str File path to load from. Supports .gz compressed files. response_model : Optional\[Type[BaseModel]\] Pydantic model to use for response validation. Required for custom SimpleLLMMetrics. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: SimpleLLMMetric Loaded metric instance Raises: ValueError If file cannot be loaded, is invalid, or missing required models Source code in `src/ragas/metrics/base.py` ```python @classmethod def load( cls, path: str, response_model: t.Optional[t.Type["BaseModel"]] = None, embedding_model: t.Optional["EmbeddingModelType"] = None, ) -> "SimpleLLMMetric": """ Load a metric from a JSON file. Parameters: ----------- path : str File path to load from. Supports .gz compressed files. response_model : Optional[Type[BaseModel]] Pydantic model to use for response validation. Required for custom SimpleLLMMetrics. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: -------- SimpleLLMMetric Loaded metric instance Raises: ------- ValueError If file cannot be loaded, is invalid, or missing required models """ import gzip import json from pathlib import Path file_path = Path(path) # Load JSON data try: if file_path.suffix == ".gz": with gzip.open(file_path, "rt", encoding="utf-8") as f: data = json.load(f) else: with open(file_path, "r", encoding="utf-8") as f: data = json.load(f) except (FileNotFoundError, json.JSONDecodeError, OSError) as e: raise ValueError(f"Cannot load metric from {path}: {e}") # Validate format if data.get("format_version") != "1.0": import warnings warnings.warn( f"Loading metric with format version {data.get('format_version')}, expected 1.0" ) # Reconstruct the prompt prompt = cls._deserialize_prompt(data["prompt"], embedding_model) # Get config config = data.get("config", {}) # Create the metric instance metric = cls(name=data["name"], prompt=prompt, **config) # Set response model if provided if response_model: metric._response_model = response_model return metric ``` ### get_correlation ```python get_correlation(gold_labels: List[str], predictions: List[str]) -> float ``` Calculate the correlation between gold scores and predicted scores. This is a placeholder method and should be implemented based on the specific metric. Source code in `src/ragas/metrics/base.py` ```python @abstractmethod def get_correlation( self, gold_labels: t.List[str], predictions: t.List[str] ) -> float: """ Calculate the correlation between gold scores and predicted scores. This is a placeholder method and should be implemented based on the specific metric. """ pass ``` ### align_and_validate ```python align_and_validate(dataset: 'Dataset', embedding_model: 'EmbeddingModelType', llm: 'BaseRagasLLM', test_size: float = 0.2, random_state: int = 42, **kwargs: Dict[str, Any]) ``` Args: dataset: experiment to align the metric with. embedding_model: The embedding model used for dynamic few-shot prompting. llm: The LLM instance to use for scoring. Align the metric with the specified experiments and validate it against a gold standard experiment. This method combines alignment and validation into a single step. Source code in `src/ragas/metrics/base.py` ```python def align_and_validate( self, dataset: "Dataset", embedding_model: "EmbeddingModelType", llm: "BaseRagasLLM", test_size: float = 0.2, random_state: int = 42, **kwargs: t.Dict[str, t.Any], ): """ Args: dataset: experiment to align the metric with. embedding_model: The embedding model used for dynamic few-shot prompting. llm: The LLM instance to use for scoring. Align the metric with the specified experiments and validate it against a gold standard experiment. This method combines alignment and validation into a single step. """ train_dataset, test_dataset = dataset.train_test_split( test_size=test_size, random_state=random_state ) self.align(train_dataset, embedding_model, **kwargs) # type: ignore return self.validate_alignment(llm, test_dataset) # type: ignore ``` ### align ```python align(train_dataset: 'Dataset', embedding_model: 'EmbeddingModelType', **kwargs: Dict[str, Any]) ``` Args: train_dataset: train_dataset to align the metric with. embedding_model: The embedding model used for dynamic few-shot prompting. Align the metric with the specified experiments by different optimization methods. Source code in `src/ragas/metrics/base.py` ```python def align( self, train_dataset: "Dataset", embedding_model: "EmbeddingModelType", **kwargs: t.Dict[str, t.Any], ): """ Args: train_dataset: train_dataset to align the metric with. embedding_model: The embedding model used for dynamic few-shot prompting. Align the metric with the specified experiments by different optimization methods. """ # get prompt if not self.prompt: raise Exception("prompt not passed") from ragas.prompt.simple_prompt import Prompt self.prompt = ( self.prompt if isinstance(self.prompt, Prompt) else Prompt(self.prompt) ) # Extract specific parameters for from_prompt method max_similar_examples_val = kwargs.get("max_similar_examples", 3) similarity_threshold_val = kwargs.get("similarity_threshold", 0.7) max_similar_examples = ( int(max_similar_examples_val) if isinstance(max_similar_examples_val, (int, str)) else 3 ) similarity_threshold = ( float(similarity_threshold_val) if isinstance(similarity_threshold_val, (int, float, str)) else 0.7 ) # Convert BaseRagasEmbeddings to BaseRagasEmbedding if needed if hasattr(embedding_model, "embed_query"): # For legacy BaseRagasEmbeddings, we need to wrap it # Create a wrapper that implements BaseRagasEmbedding interface class EmbeddingWrapper: def __init__(self, legacy_embedding): self.legacy_embedding = legacy_embedding def embed_text(self, text: str, **kwargs) -> t.List[float]: return self.legacy_embedding.embed_query(text) async def aembed_text(self, text: str, **kwargs) -> t.List[float]: return await self.legacy_embedding.aembed_query(text) actual_embedding_model = EmbeddingWrapper(embedding_model) else: # Already BaseRagasEmbedding actual_embedding_model = embedding_model from ragas.prompt.dynamic_few_shot import DynamicFewShotPrompt self.prompt = DynamicFewShotPrompt.from_prompt( self.prompt, actual_embedding_model, # type: ignore[arg-type] max_similar_examples, similarity_threshold, ) train_dataset.reload() total_items = len(train_dataset) input_vars = self.get_variables() output_vars = [self.name, f"{self.name}_reason"] from rich.progress import Progress with Progress() as progress: task = progress.add_task("Processing examples", total=total_items) for row in train_dataset: inputs = { var: train_dataset.get_row_value(row, var) for var in input_vars } inputs = {k: v for k, v in inputs.items() if v is not None} output = { var: train_dataset.get_row_value(row, var) for var in output_vars } output = {k: v for k, v in output.items() if v is not None} if output: self.prompt.add_example(inputs, output) progress.update(task, advance=1) ``` ### validate_alignment ```python validate_alignment(llm: 'BaseRagasLLM', test_dataset: 'Dataset', mapping: Dict[str, str] = {}) ``` Args: llm: The LLM instance to use for scoring. test_dataset: An Dataset instance containing the gold standard scores. mapping: A dictionary mapping variable names expected by metrics to their corresponding names in the gold experiment. Validate the alignment of the metric by comparing the scores against a gold standard experiment. This method computes the Cohen's Kappa score and agreement rate between the gold standard scores and the predicted scores from the metric. Source code in `src/ragas/metrics/base.py` ```python def validate_alignment( self, llm: "BaseRagasLLM", test_dataset: "Dataset", mapping: t.Dict[str, str] = {}, ): """ Args: llm: The LLM instance to use for scoring. test_dataset: An Dataset instance containing the gold standard scores. mapping: A dictionary mapping variable names expected by metrics to their corresponding names in the gold experiment. Validate the alignment of the metric by comparing the scores against a gold standard experiment. This method computes the Cohen's Kappa score and agreement rate between the gold standard scores and the predicted scores from the metric. """ test_dataset.reload() gold_scores_raw = [ test_dataset.get_row_value(row, self.name) for row in test_dataset ] pred_scores = [] for row in test_dataset: values = { v: ( test_dataset.get_row_value(row, v) if v not in mapping else test_dataset.get_row_value(row, mapping.get(v, v)) ) for v in self.get_variables() } score = self.score(llm=llm, **values) pred_scores.append(score.value) # Convert to strings for correlation calculation, filtering out None values gold_scores = [str(score) for score in gold_scores_raw if score is not None] pred_scores_str = [str(score) for score in pred_scores if score is not None] df = test_dataset.to_pandas() df[f"{self.name}_pred"] = pred_scores correlation = self.get_correlation(gold_scores, pred_scores_str) agreement_rate = sum( x == y for x, y in zip(gold_scores, pred_scores_str) ) / len(gold_scores) return { "correlation": correlation, "agreement_rate": agreement_rate, "df": df, } ``` ## create_auto_response_model ```python create_auto_response_model(name: str, **fields) -> Type['BaseModel'] ``` Create a response model and mark it as auto-generated by Ragas. This function creates a Pydantic model using create_model and marks it with a special attribute to indicate it was auto-generated. This allows the save() method to distinguish between auto-generated models (which are recreated on load) and custom user models. Parameters: | Name | Type | Description | Default | | ---------- | ----- | -------------------------------------------------------------------------------------------------------------- | ---------- | | `name` | `str` | Name for the model class | *required* | | `**fields` | | Field definitions in create_model format. Each field is specified as: field_name=(type, default_or_field_info) | `{}` | Returns: | Type | Description | | ----------------- | --------------------------------------------- | | `Type[BaseModel]` | Pydantic model class marked as auto-generated | Examples: ```pycon >>> from pydantic import Field >>> # Simple model with required fields >>> ResponseModel = create_auto_response_model( ... "ResponseModel", ... value=(str, ...), ... reason=(str, ...) ... ) >>> >>> # Model with Field validators and descriptions >>> ResponseModel = create_auto_response_model( ... "ResponseModel", ... value=(str, Field(..., description="The predicted value")), ... reason=(str, Field(..., description="Reasoning for the prediction")) ... ) ``` Source code in `src/ragas/metrics/base.py` ```python def create_auto_response_model(name: str, **fields) -> t.Type["BaseModel"]: """ Create a response model and mark it as auto-generated by Ragas. This function creates a Pydantic model using create_model and marks it with a special attribute to indicate it was auto-generated. This allows the save() method to distinguish between auto-generated models (which are recreated on load) and custom user models. Parameters ---------- name : str Name for the model class **fields Field definitions in create_model format. Each field is specified as: field_name=(type, default_or_field_info) Returns ------- Type[BaseModel] Pydantic model class marked as auto-generated Examples -------- >>> from pydantic import Field >>> # Simple model with required fields >>> ResponseModel = create_auto_response_model( ... "ResponseModel", ... value=(str, ...), ... reason=(str, ...) ... ) >>> >>> # Model with Field validators and descriptions >>> ResponseModel = create_auto_response_model( ... "ResponseModel", ... value=(str, Field(..., description="The predicted value")), ... reason=(str, Field(..., description="Reasoning for the prediction")) ... ) """ from pydantic import create_model model = create_model(name, **fields) setattr(model, "__ragas_auto_generated__", True) # type: ignore[attr-defined] return model ``` ## Metric ```python Metric(_required_columns: Dict[MetricType, Set[str]] = dict(), name: str = '') ``` Bases: `ABC` Abstract base class for metrics in Ragas. Attributes: | Name | Type | Description | | ------------------ | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | `name` | `str` | The name of the metric. | | `required_columns` | `Dict[str, Set[str]]` | A dictionary mapping metric type names to sets of required column names. This is a property and raises ValueError if columns are not in VALID_COLUMNS. | ### init ```python init(run_config: RunConfig) -> None ``` Initialize the metric with the given run configuration. Parameters: | Name | Type | Description | Default | | ------------ | ----------- | ----------------------------------------------------------------------- | ---------- | | `run_config` | `RunConfig` | Configuration for the metric run including timeouts and other settings. | *required* | Source code in `src/ragas/metrics/base.py` ```python @abstractmethod def init(self, run_config: RunConfig) -> None: """ Initialize the metric with the given run configuration. Parameters ---------- run_config : RunConfig Configuration for the metric run including timeouts and other settings. """ ... ``` ## MetricType Bases: `Enum` Enumeration of metric types in Ragas. Attributes: | Name | Type | Description | | ------------- | ----- | ------------------------------------- | | `SINGLE_TURN` | `str` | Represents a single-turn metric type. | | `MULTI_TURN` | `str` | Represents a multi-turn metric type. | ## MetricWithLLM ```python MetricWithLLM(_required_columns: Dict[MetricType, Set[str]] = dict(), name: str = '', llm: Optional[BaseRagasLLM] = None, output_type: Optional[MetricOutputType] = None) ``` Bases: `Metric`, `PromptMixin` A metric class that uses a language model for evaluation. Attributes: | Name | Type | Description | | ----- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `llm` | `Optional[BaseRagasLLM]` | The language model used for the metric. Both BaseRagasLLM and InstructorBaseRagasLLM are accepted at runtime via duck typing (both have compatible methods). | ### init ```python init(run_config: RunConfig) -> None ``` Initialize the metric with run configuration and validate LLM is present. Parameters: | Name | Type | Description | Default | | ------------ | ----------- | --------------------------------- | ---------- | | `run_config` | `RunConfig` | Configuration for the metric run. | *required* | Raises: | Type | Description | | ------------ | ------------------------------------ | | `ValueError` | If no LLM is provided to the metric. | Source code in `src/ragas/metrics/base.py` ```python def init(self, run_config: RunConfig) -> None: """ Initialize the metric with run configuration and validate LLM is present. Parameters ---------- run_config : RunConfig Configuration for the metric run. Raises ------ ValueError If no LLM is provided to the metric. """ if self.llm is None: raise ValueError( f"Metric '{self.name}' has no valid LLM provided (self.llm is None). Please instantiate the metric with an LLM to run." ) # Only BaseRagasLLM has set_run_config method, not InstructorBaseRagasLLM if isinstance(self.llm, BaseRagasLLM): self.llm.set_run_config(run_config) ``` ### train ```python train(path: str, demonstration_config: Optional[DemonstrationConfig] = None, instruction_config: Optional[InstructionConfig] = None, callbacks: Optional[Callbacks] = None, run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, with_debugging_logs=False, raise_exceptions: bool = True) -> None ``` Train the metric using local JSON data Parameters: | Name | Type | Description | Default | | ---------------------- | --------------------- | -------------------------------------------- | ---------- | | `path` | `str` | Path to local JSON training data file | *required* | | `demonstration_config` | `DemonstrationConfig` | Configuration for demonstration optimization | `None` | | `instruction_config` | `InstructionConfig` | Configuration for instruction optimization | `None` | | `callbacks` | `Callbacks` | List of callback functions | `None` | | `run_config` | `RunConfig` | Run configuration | `None` | | `batch_size` | `int` | Batch size for training | `None` | | `with_debugging_logs` | `bool` | Enable debugging logs | `False` | | `raise_exceptions` | `bool` | Whether to raise exceptions during training | `True` | Raises: | Type | Description | | ------------ | ------------------------------------------ | | `ValueError` | If path is not provided or not a JSON file | Source code in `src/ragas/metrics/base.py` ```python def train( self, path: str, demonstration_config: t.Optional[DemonstrationConfig] = None, instruction_config: t.Optional[InstructionConfig] = None, callbacks: t.Optional[Callbacks] = None, run_config: t.Optional[RunConfig] = None, batch_size: t.Optional[int] = None, with_debugging_logs=False, raise_exceptions: bool = True, ) -> None: """ Train the metric using local JSON data Parameters ---------- path : str Path to local JSON training data file demonstration_config : DemonstrationConfig, optional Configuration for demonstration optimization instruction_config : InstructionConfig, optional Configuration for instruction optimization callbacks : Callbacks, optional List of callback functions run_config : RunConfig, optional Run configuration batch_size : int, optional Batch size for training with_debugging_logs : bool, default=False Enable debugging logs raise_exceptions : bool, default=True Whether to raise exceptions during training Raises ------ ValueError If path is not provided or not a JSON file """ # Validate input parameters if not path: raise ValueError("Path to training data file must be provided") if not path.endswith(".json"): raise ValueError("Train data must be in json format") run_config = run_config or RunConfig() callbacks = callbacks or [] # Load the dataset from JSON file dataset = MetricAnnotation.from_json(path, metric_name=self.name) # only optimize the instruction if instruction_config is provided if instruction_config is not None: self._optimize_instruction( instruction_config=instruction_config, dataset=dataset, callbacks=callbacks, run_config=run_config, batch_size=batch_size, with_debugging_logs=with_debugging_logs, raise_exceptions=raise_exceptions, ) # if demonstration_config is provided, optimize the demonstrations if demonstration_config is not None: self._optimize_demonstration( demonstration_config=demonstration_config, dataset=dataset, ) ``` ## MultiTurnMetric ```python MultiTurnMetric(_required_columns: Dict[MetricType, Set[str]] = dict(), name: str = '') ``` Bases: `Metric` A metric class for evaluating multi-turn conversations. This class extends the base Metric class to provide functionality for scoring multi-turn conversation samples. ### multi_turn_score ```python multi_turn_score(sample: MultiTurnSample, callbacks: Callbacks = None) -> float ``` Score a multi-turn conversation sample synchronously. May raise ImportError if nest_asyncio is not installed in Jupyter-like environments. Source code in `src/ragas/metrics/base.py` ```python def multi_turn_score( self, sample: MultiTurnSample, callbacks: Callbacks = None, ) -> float: """ Score a multi-turn conversation sample synchronously. May raise ImportError if nest_asyncio is not installed in Jupyter-like environments. """ callbacks = callbacks or [] sample = self._only_required_columns_multi_turn(sample) rm, group_cm = new_group( self.name, inputs=sample.to_dict(), callbacks=callbacks, metadata={"type": ChainType.METRIC}, ) async def _async_wrapper(): try: result = await self._multi_turn_ascore( sample=sample, callbacks=group_cm ) except Exception as e: if not group_cm.ended: rm.on_chain_error(e) raise e else: if not group_cm.ended: rm.on_chain_end({"output": result}) return result apply_nest_asyncio() score = run(_async_wrapper) # track the evaluation event _analytics_batcher.add_evaluation( EvaluationEvent( metrics=[self.name], num_rows=1, evaluation_type=MetricType.SINGLE_TURN.name, language=get_metric_language(self), ) ) return score ``` ### multi_turn_ascore ```python multi_turn_ascore(sample: MultiTurnSample, callbacks: Callbacks = None, timeout: Optional[float] = None) -> float ``` Score a multi-turn conversation sample asynchronously. May raise asyncio.TimeoutError if the scoring process exceeds the specified timeout. Source code in `src/ragas/metrics/base.py` ```python async def multi_turn_ascore( self, sample: MultiTurnSample, callbacks: Callbacks = None, timeout: t.Optional[float] = None, ) -> float: """ Score a multi-turn conversation sample asynchronously. May raise asyncio.TimeoutError if the scoring process exceeds the specified timeout. """ callbacks = callbacks or [] sample = self._only_required_columns_multi_turn(sample) rm, group_cm = new_group( self.name, inputs=sample.to_dict(), callbacks=callbacks, metadata={"type": ChainType.METRIC}, ) try: score = await asyncio.wait_for( self._multi_turn_ascore(sample=sample, callbacks=group_cm), timeout=timeout, ) except Exception as e: if not group_cm.ended: rm.on_chain_error(e) raise e else: if not group_cm.ended: rm.on_chain_end({"output": score}) # track the evaluation event _analytics_batcher.add_evaluation( EvaluationEvent( metrics=[self.name], num_rows=1, evaluation_type=MetricType.SINGLE_TURN.name, language=get_metric_language(self), ) ) return score ``` ## BaseMetric ```python BaseMetric(name: str, allowed_values: AllowedValuesType = (lambda: ['pass', 'fail'])()) ``` Bases: `ABC` Base class for simple metrics that return MetricResult objects. This class provides the foundation for metrics that evaluate inputs and return structured MetricResult objects containing scores and reasoning. Attributes: | Name | Type | Description | | ---------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | `str` | The name of the metric. | | `allowed_values` | `AllowedValuesType` | Allowed values for the metric output. Can be a list of strings for discrete metrics, a tuple of floats for numeric metrics, or an integer for ranking metrics. | Examples: ```pycon >>> from ragas.metrics import discrete_metric >>> >>> @discrete_metric(name="sentiment", allowed_values=["positive", "negative"]) >>> def sentiment_metric(user_input: str, response: str) -> str: ... return "positive" if "good" in response else "negative" >>> >>> result = sentiment_metric(user_input="How are you?", response="I'm good!") >>> print(result.value) # "positive" ``` ### score ```python score(**kwargs) -> 'MetricResult' ``` Synchronously calculate the metric score. Parameters: | Name | Type | Description | Default | | ---------- | ------ | ---------------------------------------------------------------- | ------- | | `**kwargs` | `dict` | Input parameters required by the specific metric implementation. | `{}` | Returns: | Type | Description | | -------------- | --------------------------------------------------------- | | `MetricResult` | The evaluation result containing the score and reasoning. | Source code in `src/ragas/metrics/base.py` ```python @abstractmethod def score(self, **kwargs) -> "MetricResult": """ Synchronously calculate the metric score. Parameters ---------- **kwargs : dict Input parameters required by the specific metric implementation. Returns ------- MetricResult The evaluation result containing the score and reasoning. """ pass ``` ### ascore ```python ascore(**kwargs) -> 'MetricResult' ``` Asynchronously calculate the metric score. Parameters: | Name | Type | Description | Default | | ---------- | ------ | ---------------------------------------------------------------- | ------- | | `**kwargs` | `dict` | Input parameters required by the specific metric implementation. | `{}` | Returns: | Type | Description | | -------------- | --------------------------------------------------------- | | `MetricResult` | The evaluation result containing the score and reasoning. | Source code in `src/ragas/metrics/base.py` ```python @abstractmethod async def ascore(self, **kwargs) -> "MetricResult": """ Asynchronously calculate the metric score. Parameters ---------- **kwargs : dict Input parameters required by the specific metric implementation. Returns ------- MetricResult The evaluation result containing the score and reasoning. """ pass ``` ### batch_score ```python batch_score(inputs: List[Dict[str, Any]]) -> List['MetricResult'] ``` Synchronously calculate scores for a batch of inputs. Parameters: | Name | Type | Description | Default | | -------- | ---------------------- | ---------------------------------------------------------------------- | ---------- | | `inputs` | `List[Dict[str, Any]]` | List of input dictionaries, each containing parameters for the metric. | *required* | Returns: | Type | Description | | -------------------- | ----------------------------------------------- | | `List[MetricResult]` | List of evaluation results, one for each input. | Source code in `src/ragas/metrics/base.py` ```python def batch_score( self, inputs: t.List[t.Dict[str, t.Any]], ) -> t.List["MetricResult"]: """ Synchronously calculate scores for a batch of inputs. Parameters ---------- inputs : List[Dict[str, Any]] List of input dictionaries, each containing parameters for the metric. Returns ------- List[MetricResult] List of evaluation results, one for each input. """ return [self.score(**input_dict) for input_dict in inputs] ``` ### abatch_score ```python abatch_score(inputs: List[Dict[str, Any]]) -> List['MetricResult'] ``` Asynchronously calculate scores for a batch of inputs in parallel. Parameters: | Name | Type | Description | Default | | -------- | ---------------------- | ---------------------------------------------------------------------- | ---------- | | `inputs` | `List[Dict[str, Any]]` | List of input dictionaries, each containing parameters for the metric. | *required* | Returns: | Type | Description | | -------------------- | ----------------------------------------------- | | `List[MetricResult]` | List of evaluation results, one for each input. | Source code in `src/ragas/metrics/base.py` ```python async def abatch_score( self, inputs: t.List[t.Dict[str, t.Any]], ) -> t.List["MetricResult"]: """ Asynchronously calculate scores for a batch of inputs in parallel. Parameters ---------- inputs : List[Dict[str, Any]] List of input dictionaries, each containing parameters for the metric. Returns ------- List[MetricResult] List of evaluation results, one for each input. """ async_tasks = [] for input_dict in inputs: # Process input asynchronously async_tasks.append(self.ascore(**input_dict)) # Run all tasks concurrently and return results return await asyncio.gather(*async_tasks) ``` ## LLMMetric ```python LLMMetric(name: str, allowed_values: AllowedValuesType = (lambda: ['pass', 'fail'])(), prompt: Optional[Union[str, 'Prompt']] = None) ``` Bases: `SimpleBaseMetric` LLM-based metric that uses prompts to generate structured responses. ### save ```python save(path: Optional[str] = None) -> None ``` Save the metric configuration to a JSON file. Parameters: path : str, optional File path to save to. If not provided, saves to "./{metric.name}.json" Use .gz extension for compression. Note: If the metric has a response_model, its schema will be saved for reference but the model itself cannot be serialized. You'll need to provide it when loading. Examples: All these work: > > > metric.save() # β†’ ./response_quality.json metric.save("custom.json") # β†’ ./custom.json metric.save("/path/to/metrics/") # β†’ /path/to/metrics/response_quality.json metric.save("no_extension") # β†’ ./no_extension.json metric.save("compressed.json.gz") # β†’ ./compressed.json.gz (compressed) Source code in `src/ragas/metrics/base.py` ```python def save(self, path: t.Optional[str] = None) -> None: """ Save the metric configuration to a JSON file. Parameters: ----------- path : str, optional File path to save to. If not provided, saves to "./{metric.name}.json" Use .gz extension for compression. Note: ----- If the metric has a response_model, its schema will be saved for reference but the model itself cannot be serialized. You'll need to provide it when loading. Examples: --------- All these work: >>> metric.save() # β†’ ./response_quality.json >>> metric.save("custom.json") # β†’ ./custom.json >>> metric.save("/path/to/metrics/") # β†’ /path/to/metrics/response_quality.json >>> metric.save("no_extension") # β†’ ./no_extension.json >>> metric.save("compressed.json.gz") # β†’ ./compressed.json.gz (compressed) """ import gzip import json import warnings from pathlib import Path # Handle default path if path is None: # Default to current directory with metric name as filename file_path = Path(f"./{self.name}.json") else: file_path = Path(path) # If path is a directory, append the metric name as filename if file_path.is_dir(): file_path = file_path / f"{self.name}.json" # If path has no extension, add .json elif not file_path.suffix: file_path = file_path.with_suffix(".json") # Collect warning messages for data loss warning_messages = [] if hasattr(self, "_response_model") and self._response_model: # Only warn for custom response models, not auto-generated ones if not getattr(self._response_model, "__ragas_auto_generated__", False): warning_messages.append( "- Custom response_model will be lost (set it manually after loading)" ) # Serialize the prompt (may add embedding_model warning) prompt_data = self._serialize_prompt(warning_messages) # Determine the metric type metric_type = self.__class__.__name__ # Get metric-specific config config = self._get_metric_config() # Emit consolidated warning if there's data loss if warning_messages: warnings.warn( "Some metric components cannot be saved and will be lost:\n" + "\n".join(warning_messages) + "\n\nYou'll need to provide these when loading the metric." ) data = { "format_version": "1.0", "metric_type": metric_type, "name": self.name, "prompt": prompt_data, "config": config, "response_model_info": self._serialize_response_model_info(), } try: if file_path.suffix == ".gz": with gzip.open(file_path, "wt", encoding="utf-8") as f: json.dump(data, f, indent=2) else: with open(file_path, "w", encoding="utf-8") as f: json.dump(data, f, indent=2) except (OSError, IOError) as e: raise ValueError(f"Cannot save metric to {file_path}: {e}") ``` ### load ```python load(path: str, response_model: Optional[Type['BaseModel']] = None, embedding_model: Optional['EmbeddingModelType'] = None) -> 'SimpleLLMMetric' ``` Load a metric from a JSON file. Parameters: path : str File path to load from. Supports .gz compressed files. response_model : Optional\[Type[BaseModel]\] Pydantic model to use for response validation. Required for custom SimpleLLMMetrics. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: SimpleLLMMetric Loaded metric instance Raises: ValueError If file cannot be loaded, is invalid, or missing required models Source code in `src/ragas/metrics/base.py` ```python @classmethod def load( cls, path: str, response_model: t.Optional[t.Type["BaseModel"]] = None, embedding_model: t.Optional["EmbeddingModelType"] = None, ) -> "SimpleLLMMetric": """ Load a metric from a JSON file. Parameters: ----------- path : str File path to load from. Supports .gz compressed files. response_model : Optional[Type[BaseModel]] Pydantic model to use for response validation. Required for custom SimpleLLMMetrics. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: -------- SimpleLLMMetric Loaded metric instance Raises: ------- ValueError If file cannot be loaded, is invalid, or missing required models """ import gzip import json from pathlib import Path file_path = Path(path) # Load JSON data try: if file_path.suffix == ".gz": with gzip.open(file_path, "rt", encoding="utf-8") as f: data = json.load(f) else: with open(file_path, "r", encoding="utf-8") as f: data = json.load(f) except (FileNotFoundError, json.JSONDecodeError, OSError) as e: raise ValueError(f"Cannot load metric from {path}: {e}") # Validate format if data.get("format_version") != "1.0": import warnings warnings.warn( f"Loading metric with format version {data.get('format_version')}, expected 1.0" ) # Reconstruct the prompt prompt = cls._deserialize_prompt(data["prompt"], embedding_model) # Get config config = data.get("config", {}) # Create the metric instance metric = cls(name=data["name"], prompt=prompt, **config) # Set response model if provided if response_model: metric._response_model = response_model return metric ``` ### get_correlation ```python get_correlation(gold_labels: List[str], predictions: List[str]) -> float ``` Calculate the correlation between gold scores and predicted scores. This is a placeholder method and should be implemented based on the specific metric. Source code in `src/ragas/metrics/base.py` ```python @abstractmethod def get_correlation( self, gold_labels: t.List[str], predictions: t.List[str] ) -> float: """ Calculate the correlation between gold scores and predicted scores. This is a placeholder method and should be implemented based on the specific metric. """ pass ``` ### align_and_validate ```python align_and_validate(dataset: 'Dataset', embedding_model: 'EmbeddingModelType', llm: 'BaseRagasLLM', test_size: float = 0.2, random_state: int = 42, **kwargs: Dict[str, Any]) ``` Args: dataset: experiment to align the metric with. embedding_model: The embedding model used for dynamic few-shot prompting. llm: The LLM instance to use for scoring. Align the metric with the specified experiments and validate it against a gold standard experiment. This method combines alignment and validation into a single step. Source code in `src/ragas/metrics/base.py` ```python def align_and_validate( self, dataset: "Dataset", embedding_model: "EmbeddingModelType", llm: "BaseRagasLLM", test_size: float = 0.2, random_state: int = 42, **kwargs: t.Dict[str, t.Any], ): """ Args: dataset: experiment to align the metric with. embedding_model: The embedding model used for dynamic few-shot prompting. llm: The LLM instance to use for scoring. Align the metric with the specified experiments and validate it against a gold standard experiment. This method combines alignment and validation into a single step. """ train_dataset, test_dataset = dataset.train_test_split( test_size=test_size, random_state=random_state ) self.align(train_dataset, embedding_model, **kwargs) # type: ignore return self.validate_alignment(llm, test_dataset) # type: ignore ``` ### align ```python align(train_dataset: 'Dataset', embedding_model: 'EmbeddingModelType', **kwargs: Dict[str, Any]) ``` Args: train_dataset: train_dataset to align the metric with. embedding_model: The embedding model used for dynamic few-shot prompting. Align the metric with the specified experiments by different optimization methods. Source code in `src/ragas/metrics/base.py` ```python def align( self, train_dataset: "Dataset", embedding_model: "EmbeddingModelType", **kwargs: t.Dict[str, t.Any], ): """ Args: train_dataset: train_dataset to align the metric with. embedding_model: The embedding model used for dynamic few-shot prompting. Align the metric with the specified experiments by different optimization methods. """ # get prompt if not self.prompt: raise Exception("prompt not passed") from ragas.prompt.simple_prompt import Prompt self.prompt = ( self.prompt if isinstance(self.prompt, Prompt) else Prompt(self.prompt) ) # Extract specific parameters for from_prompt method max_similar_examples_val = kwargs.get("max_similar_examples", 3) similarity_threshold_val = kwargs.get("similarity_threshold", 0.7) max_similar_examples = ( int(max_similar_examples_val) if isinstance(max_similar_examples_val, (int, str)) else 3 ) similarity_threshold = ( float(similarity_threshold_val) if isinstance(similarity_threshold_val, (int, float, str)) else 0.7 ) # Convert BaseRagasEmbeddings to BaseRagasEmbedding if needed if hasattr(embedding_model, "embed_query"): # For legacy BaseRagasEmbeddings, we need to wrap it # Create a wrapper that implements BaseRagasEmbedding interface class EmbeddingWrapper: def __init__(self, legacy_embedding): self.legacy_embedding = legacy_embedding def embed_text(self, text: str, **kwargs) -> t.List[float]: return self.legacy_embedding.embed_query(text) async def aembed_text(self, text: str, **kwargs) -> t.List[float]: return await self.legacy_embedding.aembed_query(text) actual_embedding_model = EmbeddingWrapper(embedding_model) else: # Already BaseRagasEmbedding actual_embedding_model = embedding_model from ragas.prompt.dynamic_few_shot import DynamicFewShotPrompt self.prompt = DynamicFewShotPrompt.from_prompt( self.prompt, actual_embedding_model, # type: ignore[arg-type] max_similar_examples, similarity_threshold, ) train_dataset.reload() total_items = len(train_dataset) input_vars = self.get_variables() output_vars = [self.name, f"{self.name}_reason"] from rich.progress import Progress with Progress() as progress: task = progress.add_task("Processing examples", total=total_items) for row in train_dataset: inputs = { var: train_dataset.get_row_value(row, var) for var in input_vars } inputs = {k: v for k, v in inputs.items() if v is not None} output = { var: train_dataset.get_row_value(row, var) for var in output_vars } output = {k: v for k, v in output.items() if v is not None} if output: self.prompt.add_example(inputs, output) progress.update(task, advance=1) ``` ### validate_alignment ```python validate_alignment(llm: 'BaseRagasLLM', test_dataset: 'Dataset', mapping: Dict[str, str] = {}) ``` Args: llm: The LLM instance to use for scoring. test_dataset: An Dataset instance containing the gold standard scores. mapping: A dictionary mapping variable names expected by metrics to their corresponding names in the gold experiment. Validate the alignment of the metric by comparing the scores against a gold standard experiment. This method computes the Cohen's Kappa score and agreement rate between the gold standard scores and the predicted scores from the metric. Source code in `src/ragas/metrics/base.py` ```python def validate_alignment( self, llm: "BaseRagasLLM", test_dataset: "Dataset", mapping: t.Dict[str, str] = {}, ): """ Args: llm: The LLM instance to use for scoring. test_dataset: An Dataset instance containing the gold standard scores. mapping: A dictionary mapping variable names expected by metrics to their corresponding names in the gold experiment. Validate the alignment of the metric by comparing the scores against a gold standard experiment. This method computes the Cohen's Kappa score and agreement rate between the gold standard scores and the predicted scores from the metric. """ test_dataset.reload() gold_scores_raw = [ test_dataset.get_row_value(row, self.name) for row in test_dataset ] pred_scores = [] for row in test_dataset: values = { v: ( test_dataset.get_row_value(row, v) if v not in mapping else test_dataset.get_row_value(row, mapping.get(v, v)) ) for v in self.get_variables() } score = self.score(llm=llm, **values) pred_scores.append(score.value) # Convert to strings for correlation calculation, filtering out None values gold_scores = [str(score) for score in gold_scores_raw if score is not None] pred_scores_str = [str(score) for score in pred_scores if score is not None] df = test_dataset.to_pandas() df[f"{self.name}_pred"] = pred_scores correlation = self.get_correlation(gold_scores, pred_scores_str) agreement_rate = sum( x == y for x, y in zip(gold_scores, pred_scores_str) ) / len(gold_scores) return { "correlation": correlation, "agreement_rate": agreement_rate, "df": df, } ``` ## SingleTurnMetric ```python SingleTurnMetric(_required_columns: Dict[MetricType, Set[str]] = dict(), name: str = '') ``` Bases: `Metric` A metric class for evaluating single-turn interactions. This class provides methods to score single-turn samples, both synchronously and asynchronously. ### single_turn_score ```python single_turn_score(sample: SingleTurnSample, callbacks: Callbacks = None) -> float ``` Synchronously score a single-turn sample. May raise ImportError if nest_asyncio is not installed in a Jupyter-like environment. Source code in `src/ragas/metrics/base.py` ```python def single_turn_score( self, sample: SingleTurnSample, callbacks: Callbacks = None, ) -> float: """ Synchronously score a single-turn sample. May raise ImportError if nest_asyncio is not installed in a Jupyter-like environment. """ callbacks = callbacks or [] # only get the required columns sample = self._only_required_columns_single_turn(sample) rm, group_cm = new_group( self.name, inputs=sample.to_dict(), callbacks=callbacks, metadata={"type": ChainType.METRIC}, ) async def _async_wrapper(): try: result = await self._single_turn_ascore( sample=sample, callbacks=group_cm ) except Exception as e: if not group_cm.ended: rm.on_chain_error(e) raise e else: if not group_cm.ended: rm.on_chain_end({"output": result}) return result apply_nest_asyncio() score = run(_async_wrapper) # track the evaluation event _analytics_batcher.add_evaluation( EvaluationEvent( metrics=[self.name], num_rows=1, evaluation_type=MetricType.SINGLE_TURN.name, language=get_metric_language(self), ) ) return score ``` ### single_turn_ascore ```python single_turn_ascore(sample: SingleTurnSample, callbacks: Callbacks = None, timeout: Optional[float] = None) -> float ``` Asynchronously score a single-turn sample with an optional timeout. May raise asyncio.TimeoutError if the scoring process exceeds the specified timeout. Source code in `src/ragas/metrics/base.py` ```python async def single_turn_ascore( self, sample: SingleTurnSample, callbacks: Callbacks = None, timeout: t.Optional[float] = None, ) -> float: """ Asynchronously score a single-turn sample with an optional timeout. May raise asyncio.TimeoutError if the scoring process exceeds the specified timeout. """ callbacks = callbacks or [] # only get the required columns sample = self._only_required_columns_single_turn(sample) rm, group_cm = new_group( self.name, inputs=sample.to_dict(), callbacks=callbacks, metadata={"type": ChainType.METRIC}, ) try: score = await asyncio.wait_for( self._single_turn_ascore(sample=sample, callbacks=group_cm), timeout=timeout, ) except Exception as e: if not group_cm.ended: rm.on_chain_error(e) raise e else: if not group_cm.ended: rm.on_chain_end({"output": score}) # track the evaluation event _analytics_batcher.add_evaluation( EvaluationEvent( metrics=[self.name], num_rows=1, evaluation_type=MetricType.SINGLE_TURN.name, language=get_metric_language(self), ) ) return score ``` ## DiscreteMetric ```python DiscreteMetric(name: str, allowed_values: List[str] = (lambda: ['pass', 'fail'])(), prompt: Optional[Union[str, 'Prompt']] = None) ``` Bases: `SimpleLLMMetric`, `DiscreteValidator` Metric for categorical/discrete evaluations with predefined allowed values. This class is used for metrics that output categorical values like "pass/fail", "good/bad/excellent", or custom discrete categories. Uses the instructor library for structured LLM outputs. Attributes: | Name | Type | Description | | ---------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | | `allowed_values` | `List[str]` | List of allowed categorical values the metric can output. Default is ["pass", "fail"]. | | `prompt` | `Optional[Union[str, Prompt]]` | The prompt template for the metric. Should contain placeholders for evaluation inputs that will be formatted at runtime. | Examples: ```pycon >>> from ragas.metrics import DiscreteMetric >>> from ragas.llms import llm_factory >>> from openai import OpenAI >>> >>> # Create an LLM instance >>> client = OpenAI(api_key="your-api-key") >>> llm = llm_factory("gpt-4o-mini", client=client) >>> >>> # Create a custom discrete metric >>> metric = DiscreteMetric( ... name="quality_check", ... prompt="Check the quality of the response: {response}. Return 'excellent', 'good', or 'poor'.", ... allowed_values=["excellent", "good", "poor"] ... ) >>> >>> # Score with the metric >>> result = metric.score( ... llm=llm, ... response="This is a great response!" ... ) >>> print(result.value) # Output: "excellent" or similar ``` ### get_correlation ```python get_correlation(gold_labels: List[str], predictions: List[str]) -> float ``` Calculate the correlation between gold labels and predictions. This is a placeholder method and should be implemented based on the specific metric. Source code in `src/ragas/metrics/discrete.py` ```python def get_correlation( self, gold_labels: t.List[str], predictions: t.List[str] ) -> float: """ Calculate the correlation between gold labels and predictions. This is a placeholder method and should be implemented based on the specific metric. """ try: from sklearn.metrics import cohen_kappa_score except ImportError: raise ImportError( "scikit-learn is required for correlation calculation. " "Please install it with `pip install scikit-learn`." ) return cohen_kappa_score(gold_labels, predictions) ``` ### load ```python load(path: str, embedding_model: Optional[EmbeddingModelType] = None) -> DiscreteMetric ``` Load a DiscreteMetric from a JSON file. Parameters: path : str File path to load from. Supports .gz compressed files. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: DiscreteMetric Loaded metric instance Raises: ValueError If file cannot be loaded or is not a DiscreteMetric Source code in `src/ragas/metrics/discrete.py` ```python @classmethod def load( cls, path: str, embedding_model: t.Optional["EmbeddingModelType"] = None ) -> "DiscreteMetric": """ Load a DiscreteMetric from a JSON file. Parameters: ----------- path : str File path to load from. Supports .gz compressed files. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: -------- DiscreteMetric Loaded metric instance Raises: ------- ValueError If file cannot be loaded or is not a DiscreteMetric """ # Validate metric type before loading cls._validate_metric_type(path) # Load using parent class method metric = super().load(path, embedding_model=embedding_model) # Additional type check for safety if not isinstance(metric, cls): raise ValueError(f"Loaded metric is not a {cls.__name__}") return metric ``` ## NumericMetric ```python NumericMetric(name: str, allowed_values: Union[Tuple[float, float], range] = (0.0, 1.0), prompt: Optional[Union[str, 'Prompt']] = None) ``` Bases: `SimpleLLMMetric`, `NumericValidator` Metric for continuous numeric evaluations within a specified range. This class is used for metrics that output numeric scores within a defined range, such as 0.0 to 1.0 for similarity scores or 1-10 ratings. Uses the instructor library for structured LLM outputs. Attributes: | Name | Type | Description | | ---------------- | ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | `allowed_values` | `Union[Tuple[float, float], range]` | The valid range for metric outputs. Can be a tuple of (min, max) floats or a range object. Default is (0.0, 1.0). | | `llm` | `Optional[BaseRagasLLM]` | The language model instance for evaluation. Can be created using llm_factory(). | | `prompt` | `Optional[Union[str, Prompt]]` | The prompt template for the metric. Should contain placeholders for evaluation inputs that will be formatted at runtime. | Examples: ```pycon >>> from ragas.metrics import NumericMetric >>> from ragas.llms import llm_factory >>> from openai import OpenAI >>> >>> # Create an LLM instance >>> client = OpenAI(api_key="your-api-key") >>> llm = llm_factory("gpt-4o-mini", client=client) >>> >>> # Create a custom numeric metric with 0-10 range >>> metric = NumericMetric( ... name="quality_score", ... llm=llm, ... prompt="Rate the quality of this response on a scale of 0-10: {response}", ... allowed_values=(0.0, 10.0) ... ) >>> >>> # Score with the metric >>> result = metric.score( ... llm=llm, ... response="This is a great response!" ... ) >>> print(result.value) # Output: a float between 0.0 and 10.0 ``` ### get_correlation ```python get_correlation(gold_labels: List[str], predictions: List[str]) -> float ``` Calculate the correlation between gold labels and predictions. This is a placeholder method and should be implemented based on the specific metric. Source code in `src/ragas/metrics/numeric.py` ```python def get_correlation( self, gold_labels: t.List[str], predictions: t.List[str] ) -> float: """ Calculate the correlation between gold labels and predictions. This is a placeholder method and should be implemented based on the specific metric. """ try: from scipy.stats import pearsonr except ImportError: raise ImportError( "scipy is required for correlation calculation. " "Please install it with `pip install scipy`." ) # Convert strings to floats for correlation calculation gold_floats = [float(x) for x in gold_labels] pred_floats = [float(x) for x in predictions] result = pearsonr(gold_floats, pred_floats) # pearsonr returns (correlation, p-value) tuple correlation = t.cast(float, result[0]) return correlation ``` ### load ```python load(path: str, embedding_model: Optional[EmbeddingModelType] = None) -> NumericMetric ``` Load a NumericMetric from a JSON file. Parameters: path : str File path to load from. Supports .gz compressed files. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: NumericMetric Loaded metric instance Raises: ValueError If file cannot be loaded or is not a NumericMetric Source code in `src/ragas/metrics/numeric.py` ```python @classmethod def load( cls, path: str, embedding_model: t.Optional["EmbeddingModelType"] = None ) -> "NumericMetric": """ Load a NumericMetric from a JSON file. Parameters: ----------- path : str File path to load from. Supports .gz compressed files. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: -------- NumericMetric Loaded metric instance Raises: ------- ValueError If file cannot be loaded or is not a NumericMetric """ # Validate metric type before loading cls._validate_metric_type(path) # Load using parent class method metric = super().load(path, embedding_model=embedding_model) # Additional type check for safety if not isinstance(metric, cls): raise ValueError(f"Loaded metric is not a {cls.__name__}") # Convert allowed_values back to tuple if it's a list (due to JSON serialization) if hasattr(metric, "allowed_values") and isinstance( metric.allowed_values, list ): # Ensure it's a 2-element tuple for NumericMetric if len(metric.allowed_values) == 2: metric.allowed_values = ( metric.allowed_values[0], metric.allowed_values[1], ) else: metric.allowed_values = tuple(metric.allowed_values) # type: ignore return metric ``` ## RankingMetric ```python RankingMetric(name: str, allowed_values: int = 2, prompt: Optional[Union[str, 'Prompt']] = None) ``` Bases: `SimpleLLMMetric`, `RankingValidator` Metric for evaluations that produce ranked lists of items. This class is used for metrics that output ordered lists, such as ranking search results, prioritizing features, or ordering responses by relevance. Uses the instructor library for structured LLM outputs. Attributes: | Name | Type | Description | | ---------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ | | `allowed_values` | `int` | Expected number of items in the ranking list. Default is 2. | | `llm` | `Optional[BaseRagasLLM]` | The language model instance for evaluation. Can be created using llm_factory(). | | `prompt` | `Optional[Union[str, Prompt]]` | The prompt template for the metric. Should contain placeholders for evaluation inputs that will be formatted at runtime. | Examples: ```pycon >>> from ragas.metrics import RankingMetric >>> from ragas.llms import llm_factory >>> from openai import OpenAI >>> >>> # Create an LLM instance >>> client = OpenAI(api_key="your-api-key") >>> llm = llm_factory("gpt-4o-mini", client=client) >>> >>> # Create a ranking metric that returns top 3 items >>> metric = RankingMetric( ... name="relevance_ranking", ... llm=llm, ... prompt="Rank these results by relevance: {results}", ... allowed_values=3 ... ) >>> >>> # Score with the metric >>> result = metric.score( ... llm=llm, ... results="result1, result2, result3" ... ) >>> print(result.value) # Output: a list of 3 ranked items ``` ### get_correlation ```python get_correlation(gold_labels: List[str], predictions: List[str]) -> float ``` Calculate the correlation between gold labels and predictions. This is a placeholder method and should be implemented based on the specific metric. Source code in `src/ragas/metrics/ranking.py` ```python def get_correlation( self, gold_labels: t.List[str], predictions: t.List[str] ) -> float: """ Calculate the correlation between gold labels and predictions. This is a placeholder method and should be implemented based on the specific metric. """ try: from sklearn.metrics import cohen_kappa_score except ImportError: raise ImportError( "scikit-learn is required for correlation calculation. " "Please install it with `pip install scikit-learn`." ) kappa_scores = [] for gold_item, prediction in zip(gold_labels, predictions): kappa = cohen_kappa_score(gold_item, prediction, weights="quadratic") kappa_scores.append(kappa) return sum(kappa_scores) / len(kappa_scores) if kappa_scores else 0.0 ``` ### load ```python load(path: str, embedding_model: Optional[EmbeddingModelType] = None) -> RankingMetric ``` Load a RankingMetric from a JSON file. Parameters: path : str File path to load from. Supports .gz compressed files. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: RankingMetric Loaded metric instance Raises: ValueError If file cannot be loaded or is not a RankingMetric Source code in `src/ragas/metrics/ranking.py` ```python @classmethod def load( cls, path: str, embedding_model: t.Optional["EmbeddingModelType"] = None ) -> "RankingMetric": """ Load a RankingMetric from a JSON file. Parameters: ----------- path : str File path to load from. Supports .gz compressed files. embedding_model : Optional[Any] Embedding model for DynamicFewShotPrompt. Required if the original used one. Returns: -------- RankingMetric Loaded metric instance Raises: ------- ValueError If file cannot be loaded or is not a RankingMetric """ # Validate metric type before loading cls._validate_metric_type(path) # Load using parent class method metric = super().load(path, embedding_model=embedding_model) # Additional type check for safety if not isinstance(metric, cls): raise ValueError(f"Loaded metric is not a {cls.__name__}") return metric ``` ## MetricResult ```python MetricResult(value: Any, reason: Optional[str] = None, traces: Optional[Dict[str, Any]] = None) ``` Class to hold the result of a metric evaluation. This class behaves like its underlying result value but still provides access to additional metadata like reasoning. Works with: - DiscreteMetrics (string results) - NumericMetrics (float/int results) - RankingMetrics (list results) Source code in `src/ragas/metrics/result.py` ```python def __init__( self, value: t.Any, reason: t.Optional[str] = None, traces: t.Optional[t.Dict[str, t.Any]] = None, ): if traces is not None: invalid_keys = [ key for key in traces.keys() if key not in {"input", "output"} ] if invalid_keys: raise ValueError( f"Invalid keys in traces: {invalid_keys}. Allowed keys are 'input' and 'output'." ) self._value = value self.reason = reason self.traces = traces ``` ### value ```python value ``` Get the raw result value. ### to_dict ```python to_dict() ``` Convert the result to a dictionary. Source code in `src/ragas/metrics/result.py` ```python def to_dict(self): """Convert the result to a dictionary.""" return {"result": self._value, "reason": self.reason} ``` ### validate ```python validate(value: Any, info: ValidationInfo) ``` Provide compatibility with older Pydantic versions. Source code in `src/ragas/metrics/result.py` ```python @classmethod def validate(cls, value: t.Any, info: ValidationInfo): """Provide compatibility with older Pydantic versions.""" if isinstance(value, MetricResult): return value return cls(value=value) ``` ## discrete_metric ```python discrete_metric(*, name: Optional[str] = None, allowed_values: Optional[List[str]] = None, **metric_params: Any) -> Callable[[Callable[..., Any]], DiscreteMetricProtocol] ``` Decorator for creating discrete/categorical metrics. This decorator transforms a regular function into a DiscreteMetric instance that can be used for evaluation with predefined categorical outputs. Parameters: | Name | Type | Description | Default | | ----------------- | ----------- | -------------------------------------------------------------------------------------- | ------- | | `name` | `str` | Name for the metric. If not provided, uses the function name. | `None` | | `allowed_values` | `List[str]` | List of allowed categorical values for the metric output. Default is ["pass", "fail"]. | `None` | | `**metric_params` | `Any` | Additional parameters to pass to the metric initialization. | `{}` | Returns: | Type | Description | | -------------------------------------------------------- | ---------------------------------------------------------------------- | | `Callable[[Callable[..., Any]], DiscreteMetricProtocol]` | A decorator that transforms a function into a DiscreteMetric instance. | Examples: ```pycon >>> from ragas.metrics import discrete_metric >>> >>> @discrete_metric(name="sentiment", allowed_values=["positive", "neutral", "negative"]) >>> def sentiment_analysis(user_input: str, response: str) -> str: ... '''Analyze sentiment of the response.''' ... if "great" in response.lower() or "good" in response.lower(): ... return "positive" ... elif "bad" in response.lower() or "poor" in response.lower(): ... return "negative" ... return "neutral" >>> >>> result = sentiment_analysis( ... user_input="How was your day?", ... response="It was great!" ... ) >>> print(result.value) # "positive" ``` Source code in `src/ragas/metrics/discrete.py` ```python def discrete_metric( *, name: t.Optional[str] = None, allowed_values: t.Optional[t.List[str]] = None, **metric_params: t.Any, ) -> t.Callable[[t.Callable[..., t.Any]], DiscreteMetricProtocol]: """ Decorator for creating discrete/categorical metrics. This decorator transforms a regular function into a DiscreteMetric instance that can be used for evaluation with predefined categorical outputs. Parameters ---------- name : str, optional Name for the metric. If not provided, uses the function name. allowed_values : List[str], optional List of allowed categorical values for the metric output. Default is ["pass", "fail"]. **metric_params : Any Additional parameters to pass to the metric initialization. Returns ------- Callable[[Callable[..., Any]], DiscreteMetricProtocol] A decorator that transforms a function into a DiscreteMetric instance. Examples -------- >>> from ragas.metrics import discrete_metric >>> >>> @discrete_metric(name="sentiment", allowed_values=["positive", "neutral", "negative"]) >>> def sentiment_analysis(user_input: str, response: str) -> str: ... '''Analyze sentiment of the response.''' ... if "great" in response.lower() or "good" in response.lower(): ... return "positive" ... elif "bad" in response.lower() or "poor" in response.lower(): ... return "negative" ... return "neutral" >>> >>> result = sentiment_analysis( ... user_input="How was your day?", ... response="It was great!" ... ) >>> print(result.value) # "positive" """ if allowed_values is None: allowed_values = ["pass", "fail"] decorator_factory = create_metric_decorator() return decorator_factory(name=name, allowed_values=allowed_values, **metric_params) # type: ignore[return-value] ``` ## numeric_metric ```python numeric_metric(*, name: Optional[str] = None, allowed_values: Optional[Union[Tuple[float, float], range]] = None, **metric_params: Any) -> Callable[[Callable[..., Any]], NumericMetricProtocol] ``` Decorator for creating numeric/continuous metrics. This decorator transforms a regular function into a NumericMetric instance that outputs continuous values within a specified range. Parameters: | Name | Type | Description | Default | | ----------------- | ----------------------------------- | ---------------------------------------------------------------------------------------------- | ------- | | `name` | `str` | Name for the metric. If not provided, uses the function name. | `None` | | `allowed_values` | `Union[Tuple[float, float], range]` | The valid range for metric outputs as (min, max) tuple or range object. Default is (0.0, 1.0). | `None` | | `**metric_params` | `Any` | Additional parameters to pass to the metric initialization. | `{}` | Returns: | Type | Description | | ------------------------------------------------------- | --------------------------------------------------------------------- | | `Callable[[Callable[..., Any]], NumericMetricProtocol]` | A decorator that transforms a function into a NumericMetric instance. | Examples: ```pycon >>> from ragas.metrics import numeric_metric >>> >>> @numeric_metric(name="relevance_score", allowed_values=(0.0, 1.0)) >>> def calculate_relevance(user_input: str, response: str) -> float: ... '''Calculate relevance score between 0 and 1.''' ... # Simple word overlap example ... user_words = set(user_input.lower().split()) ... response_words = set(response.lower().split()) ... if not user_words: ... return 0.0 ... overlap = len(user_words & response_words) ... return overlap / len(user_words) >>> >>> result = calculate_relevance( ... user_input="What is Python?", ... response="Python is a programming language" ... ) >>> print(result.value) # Numeric score between 0.0 and 1.0 ``` Source code in `src/ragas/metrics/numeric.py` ```python def numeric_metric( *, name: t.Optional[str] = None, allowed_values: t.Optional[t.Union[t.Tuple[float, float], range]] = None, **metric_params: t.Any, ) -> t.Callable[[t.Callable[..., t.Any]], NumericMetricProtocol]: """ Decorator for creating numeric/continuous metrics. This decorator transforms a regular function into a NumericMetric instance that outputs continuous values within a specified range. Parameters ---------- name : str, optional Name for the metric. If not provided, uses the function name. allowed_values : Union[Tuple[float, float], range], optional The valid range for metric outputs as (min, max) tuple or range object. Default is (0.0, 1.0). **metric_params : Any Additional parameters to pass to the metric initialization. Returns ------- Callable[[Callable[..., Any]], NumericMetricProtocol] A decorator that transforms a function into a NumericMetric instance. Examples -------- >>> from ragas.metrics import numeric_metric >>> >>> @numeric_metric(name="relevance_score", allowed_values=(0.0, 1.0)) >>> def calculate_relevance(user_input: str, response: str) -> float: ... '''Calculate relevance score between 0 and 1.''' ... # Simple word overlap example ... user_words = set(user_input.lower().split()) ... response_words = set(response.lower().split()) ... if not user_words: ... return 0.0 ... overlap = len(user_words & response_words) ... return overlap / len(user_words) >>> >>> result = calculate_relevance( ... user_input="What is Python?", ... response="Python is a programming language" ... ) >>> print(result.value) # Numeric score between 0.0 and 1.0 """ if allowed_values is None: allowed_values = (0.0, 1.0) decorator_factory = create_metric_decorator() return decorator_factory(name=name, allowed_values=allowed_values, **metric_params) # type: ignore[return-value] ``` ## ranking_metric ```python ranking_metric(*, name: Optional[str] = None, allowed_values: Optional[int] = None, **metric_params: Any) -> Callable[[Callable[..., Any]], RankingMetricProtocol] ``` Decorator for creating ranking/ordering metrics. This decorator transforms a regular function into a RankingMetric instance that outputs ordered lists of items. Parameters: | Name | Type | Description | Default | | ----------------- | ----- | ------------------------------------------------------------- | ------- | | `name` | `str` | Name for the metric. If not provided, uses the function name. | `None` | | `allowed_values` | `int` | Expected number of items in the ranking list. Default is 2. | `None` | | `**metric_params` | `Any` | Additional parameters to pass to the metric initialization. | `{}` | Returns: | Type | Description | | ------------------------------------------------------- | --------------------------------------------------------------------- | | `Callable[[Callable[..., Any]], RankingMetricProtocol]` | A decorator that transforms a function into a RankingMetric instance. | Examples: ```pycon >>> from ragas.metrics import ranking_metric >>> >>> @ranking_metric(name="priority_ranker", allowed_values=3) >>> def rank_by_urgency(user_input: str, responses: list) -> list: ... '''Rank responses by urgency keywords.''' ... urgency_keywords = ["urgent", "asap", "critical"] ... scored = [] ... for resp in responses: ... score = sum(kw in resp.lower() for kw in urgency_keywords) ... scored.append((score, resp)) ... # Sort by score descending and return top items ... ranked = sorted(scored, key=lambda x: x[0], reverse=True) ... return [item[1] for item in ranked[:3]] >>> >>> result = rank_by_urgency( ... user_input="What should I do first?", ... responses=["This is urgent", "Take your time", "Critical issue!"] ... ) >>> print(result.value) # Ranked list of responses ``` Source code in `src/ragas/metrics/ranking.py` ```python def ranking_metric( *, name: t.Optional[str] = None, allowed_values: t.Optional[int] = None, **metric_params: t.Any, ) -> t.Callable[[t.Callable[..., t.Any]], RankingMetricProtocol]: """ Decorator for creating ranking/ordering metrics. This decorator transforms a regular function into a RankingMetric instance that outputs ordered lists of items. Parameters ---------- name : str, optional Name for the metric. If not provided, uses the function name. allowed_values : int, optional Expected number of items in the ranking list. Default is 2. **metric_params : Any Additional parameters to pass to the metric initialization. Returns ------- Callable[[Callable[..., Any]], RankingMetricProtocol] A decorator that transforms a function into a RankingMetric instance. Examples -------- >>> from ragas.metrics import ranking_metric >>> >>> @ranking_metric(name="priority_ranker", allowed_values=3) >>> def rank_by_urgency(user_input: str, responses: list) -> list: ... '''Rank responses by urgency keywords.''' ... urgency_keywords = ["urgent", "asap", "critical"] ... scored = [] ... for resp in responses: ... score = sum(kw in resp.lower() for kw in urgency_keywords) ... scored.append((score, resp)) ... # Sort by score descending and return top items ... ranked = sorted(scored, key=lambda x: x[0], reverse=True) ... return [item[1] for item in ranked[:3]] >>> >>> result = rank_by_urgency( ... user_input="What should I do first?", ... responses=["This is urgent", "Take your time", "Critical issue!"] ... ) >>> print(result.value) # Ranked list of responses """ if allowed_values is None: allowed_values = 2 decorator_factory = create_metric_decorator() return decorator_factory(name=name, allowed_values=allowed_values, **metric_params) # type: ignore[return-value] ``` # Optimizers API Reference Ragas provides optimizers to improve metric prompts through automated optimization. This page documents the available optimizer classes and their configuration. ## Overview Optimizers use annotated datasets with ground truth scores to refine metric prompts, improving accuracy through: - **Instruction optimization**: Finding better prompt wording - **Demonstration optimization**: Selecting effective few-shot examples - **Search strategies**: Exploring the prompt space efficiently ## Core Classes ## Optimizer ```python Optimizer(metric: Optional[MetricWithLLM] = None, llm: Optional[BaseRagasLLM] = None) ``` Bases: `ABC` Abstract base class for all optimizers. ### optimize ```python optimize(dataset: SingleMetricAnnotation, loss: Loss, config: Dict[Any, Any], run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, callbacks: Optional[Callbacks] = None, with_debugging_logs=False, raise_exceptions: bool = True) -> Dict[str, str] ``` Optimizes the prompts for the given metric. Parameters: | Name | Type | Description | Default | | ------------ | ------------------- | --------------------------- | ---------- | | `metric` | `MetricWithLLM` | The metric to optimize. | *required* | | `train_data` | `Any` | The training data. | *required* | | `config` | `InstructionConfig` | The training configuration. | *required* | Returns: | Type | Description | | ---------------- | -------------------------------------- | | `Dict[str, str]` | The optimized prompts for given chain. | Source code in `src/ragas/optimizers/base.py` ```python @abstractmethod def optimize( self, dataset: SingleMetricAnnotation, loss: Loss, config: t.Dict[t.Any, t.Any], run_config: t.Optional[RunConfig] = None, batch_size: t.Optional[int] = None, callbacks: t.Optional[Callbacks] = None, with_debugging_logs=False, raise_exceptions: bool = True, ) -> t.Dict[str, str]: """ Optimizes the prompts for the given metric. Parameters ---------- metric : MetricWithLLM The metric to optimize. train_data : Any The training data. config : InstructionConfig The training configuration. Returns ------- Dict[str, str] The optimized prompts for given chain. """ raise NotImplementedError("The method `optimize` must be implemented.") ``` ## GeneticOptimizer ```python GeneticOptimizer(metric: Optional[MetricWithLLM] = None, llm: Optional[BaseRagasLLM] = None) ``` Bases: `Optimizer` A genetic algorithm optimizer that balances exploration and exploitation. ## DSPyOptimizer ```python DSPyOptimizer(metric: Optional[MetricWithLLM] = None, llm: Optional[BaseRagasLLM] = None, num_candidates: int = 10, max_bootstrapped_demos: int = 5, max_labeled_demos: int = 5, init_temperature: float = 1.0, auto: Optional[Literal['light', 'medium', 'heavy']] = 'light', num_threads: Optional[int] = None, max_errors: Optional[int] = None, seed: int = 9, verbose: bool = False, track_stats: bool = True, log_dir: Optional[str] = None, metric_threshold: Optional[float] = None, cache: Optional[CacheInterface] = None) ``` Bases: `Optimizer` Advanced prompt optimizer using DSPy's MIPROv2. MIPROv2 performs sophisticated prompt optimization by combining: - Instruction optimization (prompt engineering) - Demonstration optimization (few-shot examples) - Combined search over both spaces Requires: pip install dspy-ai or uv add ragas[dspy] Parameters: | Name | Type | Description | Default | | ------------------------ | ---------------- | -------------------------------------------------------------------------------------------------------- | --------- | | `num_candidates` | `int` | Number of prompt variants to try during optimization. | `10` | | `max_bootstrapped_demos` | `int` | Maximum number of auto-generated examples to use. | `5` | | `max_labeled_demos` | `int` | Maximum number of human-annotated examples to use. | `5` | | `init_temperature` | `float` | Exploration temperature for optimization. | `1.0` | | `auto` | `str` | Automatic configuration level: 'light', 'medium', or 'heavy'. Controls the depth of optimization search. | `'light'` | | `num_threads` | `int` | Number of parallel threads for optimization. | `None` | | `max_errors` | `int` | Maximum errors tolerated during optimization before stopping. | `None` | | `seed` | `int` | Random seed for reproducibility. | `9` | | `verbose` | `bool` | Enable verbose logging during optimization. | `False` | | `track_stats` | `bool` | Track and report optimization statistics. | `True` | | `log_dir` | `str` | Directory for saving optimization logs and progress. | `None` | | `metric_threshold` | `float` | Minimum acceptable metric value to achieve. | `None` | | `cache` | `CacheInterface` | Cache backend for storing optimization results. | `None` | ### optimize ```python optimize(dataset: SingleMetricAnnotation, loss: Loss, config: Dict[Any, Any], run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, callbacks: Optional[Callbacks] = None, with_debugging_logs: bool = False, raise_exceptions: bool = True) -> Dict[str, str] ``` Optimize metric prompts using DSPy MIPROv2. Steps: 1. Convert Ragas PydanticPrompt to DSPy Signature 1. Create DSPy Module with signature 1. Convert dataset to DSPy Examples 1. Run MIPROv2 optimization 1. Extract optimized prompts 1. Convert back to Ragas format Parameters: | Name | Type | Description | Default | | --------------------- | ------------------------ | ------------------------------------------------ | ---------- | | `dataset` | `SingleMetricAnnotation` | Annotated dataset with ground truth scores. | *required* | | `loss` | `Loss` | Loss function to optimize. | *required* | | `config` | `Dict[Any, Any]` | Additional configuration parameters. | *required* | | `run_config` | `RunConfig` | Runtime configuration. | `None` | | `batch_size` | `int` | Batch size for evaluation. | `None` | | `callbacks` | `Callbacks` | Langchain callbacks for tracking. | `None` | | `with_debugging_logs` | `bool` | Enable debug logging. | `False` | | `raise_exceptions` | `bool` | Whether to raise exceptions during optimization. | `True` | Returns: | Type | Description | | ---------------- | --------------------------------------- | | `Dict[str, str]` | Optimized prompts for each prompt name. | Source code in `src/ragas/optimizers/dspy_optimizer.py` ```python def optimize( self, dataset: SingleMetricAnnotation, loss: Loss, config: t.Dict[t.Any, t.Any], run_config: t.Optional[RunConfig] = None, batch_size: t.Optional[int] = None, callbacks: t.Optional[Callbacks] = None, with_debugging_logs: bool = False, raise_exceptions: bool = True, ) -> t.Dict[str, str]: """ Optimize metric prompts using DSPy MIPROv2. Steps: 1. Convert Ragas PydanticPrompt to DSPy Signature 2. Create DSPy Module with signature 3. Convert dataset to DSPy Examples 4. Run MIPROv2 optimization 5. Extract optimized prompts 6. Convert back to Ragas format Parameters ---------- dataset : SingleMetricAnnotation Annotated dataset with ground truth scores. loss : Loss Loss function to optimize. config : Dict[Any, Any] Additional configuration parameters. run_config : RunConfig, optional Runtime configuration. batch_size : int, optional Batch size for evaluation. callbacks : Callbacks, optional Langchain callbacks for tracking. with_debugging_logs : bool Enable debug logging. raise_exceptions : bool Whether to raise exceptions during optimization. Returns ------- Dict[str, str] Optimized prompts for each prompt name. """ if self.metric is None: raise ValueError("No metric provided for optimization.") if self.llm is None: raise ValueError("No llm provided for optimization.") if self._dspy is None: raise RuntimeError("DSPy module not loaded.") if self.cache is not None: cache_key = self._generate_cache_key(dataset, loss, config) if self.cache.has_key(cache_key): logger.info( f"Cache hit for DSPy optimization of metric: {self.metric.name}" ) return self.cache.get(cache_key) logger.info(f"Starting DSPy optimization for metric: {self.metric.name}") from ragas.optimizers.dspy_adapter import ( create_dspy_metric, pydantic_prompt_to_dspy_signature, ragas_dataset_to_dspy_examples, setup_dspy_llm, ) setup_dspy_llm(self._dspy, self.llm) prompts = self.metric.get_prompts() optimized_prompts = {} for prompt_name, prompt in prompts.items(): logger.info(f"Optimizing prompt: {prompt_name}") signature = pydantic_prompt_to_dspy_signature(prompt) module = self._dspy.Predict(signature) examples = ragas_dataset_to_dspy_examples(dataset, prompt_name) teleprompter = self._dspy.MIPROv2( num_candidates=self.num_candidates, max_bootstrapped_demos=self.max_bootstrapped_demos, max_labeled_demos=self.max_labeled_demos, init_temperature=self.init_temperature, auto=self.auto, num_threads=self.num_threads, max_errors=self.max_errors, seed=self.seed, verbose=self.verbose, track_stats=self.track_stats, log_dir=self.log_dir, metric_threshold=self.metric_threshold, ) metric_fn = create_dspy_metric(loss, dataset.name) optimized = teleprompter.compile( module, trainset=examples, metric=metric_fn, ) optimized_instruction = self._extract_instruction(optimized) optimized_prompts[prompt_name] = optimized_instruction logger.info( f"Optimized prompt for {prompt_name}: {optimized_instruction[:100]}..." ) if self.cache is not None: cache_key = self._generate_cache_key(dataset, loss, config) self.cache.set(cache_key, optimized_prompts) logger.info("Cached optimization results") return optimized_prompts ``` ## GeneticOptimizer Simple evolutionary optimizer for prompt instructions. ### Parameters | Parameter | Type | Default | Description | | ----------------- | ------- | ------- | ------------------------------ | | `max_steps` | `int` | 50 | Maximum evolution steps | | `population_size` | `int` | 10 | Population size per generation | | `mutation_rate` | `float` | 0.2 | Probability of mutation | ### Usage ```python from ragas.optimizers import GeneticOptimizer from ragas.config import InstructionConfig optimizer = GeneticOptimizer( max_steps=50, population_size=10, ) config = InstructionConfig(llm=llm, optimizer=optimizer) metric.optimize_prompts(dataset, config) ``` ### How it Works 1. Generates population of prompt variations 1. Evaluates each on annotated dataset 1. Selects best performers 1. Creates next generation via crossover and mutation 1. Repeats for max_steps iterations **Pros**: Simple, works with limited data **Cons**: Slower convergence, instruction-only ## DSPyOptimizer Advanced optimizer using DSPy's [MIPROv2](https://dspy.ai/api/optimizers/MIPROv2/) algorithm. ### Parameters | Parameter | Type | Default | Description | | ------------------------ | ------- | ------- | --------------------------------- | | `num_candidates` | `int` | 10 | Number of prompt variants to try | | `max_bootstrapped_demos` | `int` | 5 | Max auto-generated examples | | `max_labeled_demos` | `int` | 5 | Max human-annotated examples | | `init_temperature` | `float` | 1.0 | Exploration temperature (0.0-2.0) | ### Usage ```python from ragas.optimizers import DSPyOptimizer from ragas.config import InstructionConfig optimizer = DSPyOptimizer( num_candidates=10, max_bootstrapped_demos=5, max_labeled_demos=5, ) config = InstructionConfig(llm=llm, optimizer=optimizer) metric.optimize_prompts(dataset, config) ``` ### How it Works 1. Generates candidate prompt instructions 1. Bootstraps few-shot demonstrations from data 1. Selects best human-annotated examples 1. Evaluates all combinations on dataset 1. Returns best-performing configuration Learn more about DSPy concepts: - [Signatures](https://dspy.ai/learn/programming/signatures/) - DSPy's approach to defining input/output specifications - [Optimizers](https://dspy.ai/learn/optimization/optimizers/) - Algorithms for improving prompts and LM weights - [Modules](https://dspy.ai/learn/programming/modules/) - Building blocks for LLM programs **Pros**: Better results, combines instructions + demos **Cons**: Requires DSPy installation, more LLM calls ### Installation [DSPy](https://dspy.ai/) is an optional dependency: ```bash # Using uv (recommended) uv add "ragas[dspy]" # Using pip pip install "ragas[dspy]" ``` ### Cost Estimation Approximate LLM calls per optimization: ```text Total calls β‰ˆ num_candidates Γ— 30 + max_bootstrapped_demos Γ— 7 ``` Examples: - Default config (10, 5, 5): ~335 calls - Budget config (5, 2, 3): ~164 calls - Aggressive config (20, 10, 10): ~670 calls ## Optimizer Base Class Bases: `ABC` Abstract base class for all optimizers. ## optimize ```python optimize(dataset: SingleMetricAnnotation, loss: Loss, config: Dict[Any, Any], run_config: Optional[RunConfig] = None, batch_size: Optional[int] = None, callbacks: Optional[Callbacks] = None, with_debugging_logs=False, raise_exceptions: bool = True) -> Dict[str, str] ``` Optimizes the prompts for the given metric. Parameters: | Name | Type | Description | Default | | ------------ | ------------------- | --------------------------- | ---------- | | `metric` | `MetricWithLLM` | The metric to optimize. | *required* | | `train_data` | `Any` | The training data. | *required* | | `config` | `InstructionConfig` | The training configuration. | *required* | Returns: | Type | Description | | ---------------- | -------------------------------------- | | `Dict[str, str]` | The optimized prompts for given chain. | ## Configuration Both optimizers are used with `InstructionConfig`: ```python from ragas.config import InstructionConfig config = InstructionConfig( llm=llm, # LLM for optimization optimizer=optimizer_instance, # Optimizer to use ) # Use with metric metric.optimize_prompts(dataset, config) ``` ## Dataset Format Optimizers require annotated datasets with ground truth scores: ```python from ragas.dataset_schema import ( PromptAnnotation, SampleAnnotation, SingleMetricAnnotation ) # Create annotated sample prompt_annotation = PromptAnnotation( prompt_input={"user_input": "...", "response": "..."}, prompt_output={"score": 0.9}, edited_output=None, # Optional: corrected output ) sample = SampleAnnotation( metric_input={"user_input": "...", "response": "..."}, metric_output=0.9, # Ground truth score prompts={"metric_prompt": prompt_annotation}, is_accepted=True, # Include in optimization ) # Create dataset dataset = SingleMetricAnnotation( name="metric_name", samples=[sample, ...] # 20-50+ samples recommended ) ``` ## Loss Functions Optimizers use loss functions to evaluate prompt quality: ```python from ragas.losses import MSELoss, HuberLoss # Mean Squared Error (default) loss = MSELoss() # Huber Loss (robust to outliers) loss = HuberLoss(delta=1.0) # Use with config config = InstructionConfig(llm=llm, optimizer=optimizer, loss=loss) ``` ## Comparison | Feature | GeneticOptimizer | DSPyOptimizer | | -------------------- | ------------------ | ---------------------- | | Installation | Built-in | Requires `ragas[dspy]` | | Optimization Target | Instructions only | Instructions + Demos | | Min Dataset Size | 10+ samples | 20+ samples | | Typical LLM Calls | 100-500 | 200-700 | | Accuracy Improvement | +5-8% | +8-12% | | Best For | Quick optimization | Production metrics | ## See Also - [DSPy Optimizer Guide](https://docs.ragas.io/en/stable/references/howtos/customizations/optimizers/dspy-optimizer.md) - Detailed usage - [Metric Customization](https://docs.ragas.io/en/stable/references/howtos/customizations/metrics/custom-metrics.md) - Creating metrics - [Prompt API Reference](https://docs.ragas.io/en/stable/references/prompt/index.md) - Understanding prompts ## Additional Resources **DSPy Documentation:** - [DSPy Official Documentation](https://dspy.ai/) - Complete guide to DSPy - [MIPROv2 API Reference](https://dspy.ai/api/optimizers/MIPROv2/) - Detailed MIPROv2 documentation - [DSPy Optimizers Overview](https://dspy.ai/learn/optimization/optimizers/) - Guide to all DSPy optimizers - [DSPy GitHub Repository](https://github.com/stanfordnlp/dspy) - Source code and examples **Research Papers:** - [Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs](https://arxiv.org/abs/2406.11695) - MIPROv2 paper # Prompt API Reference The prompt system in Ragas provides a flexible and type-safe way to define prompts for LLM-based metrics and other components. This page documents the core prompt classes and their usage. ## Overview Ragas uses a modular prompt architecture based on the `BasePrompt` class. Prompts can be: - **Input/Output Models**: Pydantic BaseModel classes that define the structure of prompt inputs and outputs - **Prompt Classes**: Inherit from `BasePrompt` to define instructions, examples, and prompt generation logic - **String Prompts**: Simple text-based prompts for backward compatibility ## Core Classes ## InputModel ```python InputModel = TypeVar('InputModel', bound=BaseModel) ``` ## OutputModel ```python OutputModel = TypeVar('OutputModel', bound=BaseModel) ``` ## BasePrompt ```python BasePrompt(name: Optional[str] = None, language: str = 'english', original_hash: Optional[str] = None) ``` Bases: `ABC` Source code in `src/ragas/prompt/base.py` ```python def __init__( self, name: t.Optional[str] = None, language: str = "english", original_hash: t.Optional[str] = None, ): if name is None: self.name = camel_to_snake(self.__class__.__name__) self.language = language self.original_hash = original_hash ``` ### generate ```python generate(llm: BaseRagasLLM, data: Any, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Callbacks = []) -> Any ``` Generate a single completion from the prompt. Source code in `src/ragas/prompt/base.py` ```python @abstractmethod async def generate( self, llm: BaseRagasLLM, data: t.Any, temperature: t.Optional[float] = None, stop: t.Optional[t.List[str]] = None, callbacks: Callbacks = [], ) -> t.Any: """ Generate a single completion from the prompt. """ pass ``` ### generate_multiple ```python generate_multiple(llm: BaseRagasLLM, data: Any, n: int = 1, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Callbacks = []) -> Any ``` Generate multiple completions from the prompt. Source code in `src/ragas/prompt/base.py` ```python @abstractmethod def generate_multiple( self, llm: BaseRagasLLM, data: t.Any, n: int = 1, temperature: t.Optional[float] = None, stop: t.Optional[t.List[str]] = None, callbacks: Callbacks = [], ) -> t.Any: """ Generate multiple completions from the prompt. """ pass ``` ### save ```python save(file_path: str) ``` Save the prompt to a file. Source code in `src/ragas/prompt/base.py` ```python def save(self, file_path: str): """ Save the prompt to a file. """ data = { "ragas_version": __version__, "language": self.language, "original_hash": self.original_hash, } if os.path.exists(file_path): raise FileExistsError(f"The file '{file_path}' already exists.") with open(file_path, "w", encoding="utf-8") as f: json.dump(data, f, indent=2, ensure_ascii=False) print(f"Prompt saved to {file_path}") ``` ### load ```python load(file_path: str) -> 'BasePrompt' ``` Load the prompt from a file. Source code in `src/ragas/prompt/base.py` ```python @classmethod def load(cls, file_path: str) -> "BasePrompt": """ Load the prompt from a file. """ with open(file_path, "r", encoding="utf-8") as f: data = json.load(f) ragas_version = data.get("ragas_version") if ragas_version != __version__: logger.warning( "Prompt was saved with Ragas v%s, but you are loading it with Ragas v%s. " "There might be incompatibilities.", ragas_version, __version__, ) prompt = cls( language=data.get("language", "english"), original_hash=data.get("original_hash"), ) return prompt ``` ## StringPrompt ```python StringPrompt(name: Optional[str] = None, language: str = 'english', original_hash: Optional[str] = None) ``` Bases: `BasePrompt` A simple prompt that can be formatted with additional data using f-string syntax. This prompt is a simpler alternative to PydanticPrompt for those who prefer a more flexible approach without the need for a Pydantic model. Parameters: | Name | Type | Description | Default | | ------------- | ----- | ------------------------------------------------------------------ | ---------- | | `instruction` | `str` | The instruction string that can be formatted with additional data. | *required* | Examples: ```pycon >>> from ragas.prompt import string_prompt >>> await prompt.generate(llm=llm, data={"category": "commerce"}) ``` Source code in `src/ragas/prompt/base.py` ```python def __init__( self, name: t.Optional[str] = None, language: str = "english", original_hash: t.Optional[str] = None, ): if name is None: self.name = camel_to_snake(self.__class__.__name__) self.language = language self.original_hash = original_hash ``` ### generate ```python generate(llm: BaseRagasLLM, data: str, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Callbacks = []) -> str ``` Generate text based on the instruction and provided data. Parameters: | Name | Type | Description | Default | | ------------- | -------------------------- | ----------------------------------------------------------- | ---------- | | `llm` | `BaseRagasLLM` | The language model to use for text generation. | *required* | | `data` | `Optional[Dict[str, Any]]` | The data to format the instruction with, by default None. | *required* | | `n` | `int` | The number of completions to generate, by default 1. | *required* | | `temperature` | `Optional[float]` | The temperature for text generation, by default None. | `None` | | `stop` | `Optional[List[str]]` | The stop sequences for text generation, by default None. | `None` | | `callbacks` | `Callbacks` | The callbacks to use during text generation, by default []. | `[]` | Returns: | Type | Description | | ----- | ------------------- | | `str` | The generated text. | Source code in `src/ragas/prompt/base.py` ```python async def generate( self, llm: BaseRagasLLM, data: str, temperature: t.Optional[float] = None, stop: t.Optional[t.List[str]] = None, callbacks: Callbacks = [], ) -> str: """ Generate text based on the instruction and provided data. Parameters ---------- llm : BaseRagasLLM The language model to use for text generation. data : Optional[Dict[str, Any]], optional The data to format the instruction with, by default None. n : int, optional The number of completions to generate, by default 1. temperature : Optional[float], optional The temperature for text generation, by default None. stop : Optional[List[str]], optional The stop sequences for text generation, by default None. callbacks : Callbacks, optional The callbacks to use during text generation, by default []. Returns ------- str The generated text. """ llm_result = await llm.agenerate_text( StringPromptValue(text=data), n=1, temperature=temperature, stop=stop, callbacks=callbacks, ) return llm_result.generations[0][0].text ``` ### generate_multiple ```python generate_multiple(llm: BaseRagasLLM, data: str, n: int = 1, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Callbacks = []) -> List[str] ``` Generate multiple distinct text outputs based on the instruction and provided data. Parameters: | Name | Type | Description | Default | | ------------- | --------------------- | ------------------------------------------------------- | ---------- | | `llm` | `BaseRagasLLM` | The language model to use for text generation. | *required* | | `data` | `str` | The data to format the instruction with. | *required* | | `n` | `int` | The number of completions to generate, by default 1. | `1` | | `temperature` | `Optional[float]` | The temperature for text generation, by default None. | `None` | | `stop` | `Optional[List[str]]` | Stop sequences for text generation, by default None. | `None` | | `callbacks` | `Callbacks` | Callbacks to use during text generation, by default []. | `[]` | Returns: | Type | Description | | ----------- | -------------------------------------- | | `List[str]` | A list containing n generated outputs. | Notes - When caching is enabled, each output is uniquely cached to prevent duplicates. - This ensures that multiple outputs for the same input are distinct. - Previous issues where caching returned duplicate outputs have been fixed. Source code in `src/ragas/prompt/base.py` ```python async def generate_multiple( self, llm: BaseRagasLLM, data: str, n: int = 1, temperature: t.Optional[float] = None, stop: t.Optional[t.List[str]] = None, callbacks: Callbacks = [], ) -> t.List[str]: """ Generate multiple distinct text outputs based on the instruction and provided data. Parameters ---------- llm : BaseRagasLLM The language model to use for text generation. data : str The data to format the instruction with. n : int, optional The number of completions to generate, by default 1. temperature : Optional[float], optional The temperature for text generation, by default None. stop : Optional[List[str]], optional Stop sequences for text generation, by default None. callbacks : Callbacks, optional Callbacks to use during text generation, by default []. Returns ------- List[str] A list containing `n` generated outputs. Notes ----- - When caching is enabled, each output is uniquely cached to prevent duplicates. - This ensures that multiple outputs for the same input are distinct. - Previous issues where caching returned duplicate outputs have been fixed. """ llm_result = await llm.agenerate_text( StringPromptValue(text=data), n=n, temperature=temperature, stop=stop, callbacks=callbacks, ) # flatten the generations return [gen.text for gen in llm_result.generations[0]] ``` ## PydanticPrompt ```python PydanticPrompt(name: Optional[str] = None, language: str = 'english', original_hash: Optional[str] = None) ``` Bases: `BasePrompt`, `Generic[InputModel, OutputModel]` Source code in `src/ragas/prompt/base.py` ```python def __init__( self, name: t.Optional[str] = None, language: str = "english", original_hash: t.Optional[str] = None, ): if name is None: self.name = camel_to_snake(self.__class__.__name__) self.language = language self.original_hash = original_hash ``` ### generate ```python generate(llm: Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel], data: InputModel, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Optional[Callbacks] = None, retries_left: int = 3) -> OutputModel ``` Generate a single output using the provided language model and input data. This method is a special case of `generate_multiple` where only one output is generated. Parameters: | Name | Type | Description | Default | | -------------- | -------------- | ------------------------------------------------------------------- | ---------- | | `llm` | `BaseRagasLLM` | The language model to use for generation. | *required* | | `data` | `InputModel` | The input data for generation. | *required* | | `temperature` | `float` | The temperature parameter for controlling randomness in generation. | `None` | | `stop` | `List[str]` | A list of stop sequences to end generation. | `None` | | `callbacks` | `Callbacks` | Callback functions to be called during the generation process. | `None` | | `retries_left` | `int` | Number of retry attempts for an invalid LLM response | `3` | Returns: | Type | Description | | ------------- | --------------------- | | `OutputModel` | The generated output. | Notes This method internally calls `generate_multiple` with `n=1` and returns the first (and only) result. Source code in `src/ragas/prompt/pydantic_prompt.py` ```python async def generate( self, llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel], data: InputModel, temperature: t.Optional[float] = None, stop: t.Optional[t.List[str]] = None, callbacks: t.Optional[Callbacks] = None, retries_left: int = 3, ) -> OutputModel: """ Generate a single output using the provided language model and input data. This method is a special case of `generate_multiple` where only one output is generated. Parameters ---------- llm : BaseRagasLLM The language model to use for generation. data : InputModel The input data for generation. temperature : float, optional The temperature parameter for controlling randomness in generation. stop : List[str], optional A list of stop sequences to end generation. callbacks : Callbacks, optional Callback functions to be called during the generation process. retries_left : int, optional Number of retry attempts for an invalid LLM response Returns ------- OutputModel The generated output. Notes ----- This method internally calls `generate_multiple` with `n=1` and returns the first (and only) result. """ callbacks = callbacks or [] # this is just a special case of generate_multiple output_single = await self.generate_multiple( llm=llm, data=data, n=1, temperature=temperature, stop=stop, callbacks=callbacks, retries_left=retries_left, ) return output_single[0] ``` ### generate_multiple ```python generate_multiple(llm: Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel], data: InputModel, n: int = 1, temperature: Optional[float] = None, stop: Optional[List[str]] = None, callbacks: Optional[Callbacks] = None, retries_left: int = 3) -> List[OutputModel] ``` Generate multiple outputs using the provided language model and input data. Parameters: | Name | Type | Description | Default | | -------------- | -------------- | ------------------------------------------------------------------- | ---------- | | `llm` | `BaseRagasLLM` | The language model to use for generation. | *required* | | `data` | `InputModel` | The input data for generation. | *required* | | `n` | `int` | The number of outputs to generate. Default is 1. | `1` | | `temperature` | `float` | The temperature parameter for controlling randomness in generation. | `None` | | `stop` | `List[str]` | A list of stop sequences to end generation. | `None` | | `callbacks` | `Callbacks` | Callback functions to be called during the generation process. | `None` | | `retries_left` | `int` | Number of retry attempts for an invalid LLM response | `3` | Returns: | Type | Description | | ------------------- | ---------------------------- | | `List[OutputModel]` | A list of generated outputs. | Raises: | Type | Description | | ---------------------------- | --------------------------------------- | | `RagasOutputParserException` | If there's an error parsing the output. | Source code in `src/ragas/prompt/pydantic_prompt.py` ```python async def generate_multiple( self, llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM, BaseLanguageModel], data: InputModel, n: int = 1, temperature: t.Optional[float] = None, stop: t.Optional[t.List[str]] = None, callbacks: t.Optional[Callbacks] = None, retries_left: int = 3, ) -> t.List[OutputModel]: """ Generate multiple outputs using the provided language model and input data. Parameters ---------- llm : BaseRagasLLM The language model to use for generation. data : InputModel The input data for generation. n : int, optional The number of outputs to generate. Default is 1. temperature : float, optional The temperature parameter for controlling randomness in generation. stop : List[str], optional A list of stop sequences to end generation. callbacks : Callbacks, optional Callback functions to be called during the generation process. retries_left : int, optional Number of retry attempts for an invalid LLM response Returns ------- List[OutputModel] A list of generated outputs. Raises ------ RagasOutputParserException If there's an error parsing the output. """ callbacks = callbacks or [] processed_data = self.process_input(data) prompt_rm, prompt_cb = new_group( name=self.name, inputs={"data": processed_data}, callbacks=callbacks, metadata={"type": ChainType.RAGAS_PROMPT}, ) prompt_value = PromptValue(text=self.to_string(processed_data)) # Handle different LLM types with different interfaces # 1. LangChain LLMs have agenerate_prompt() for async with specific signature # 2. BaseRagasLLM have generate() with n, temperature, stop, callbacks # 3. InstructorLLM has generate()/agenerate() with only prompt and response_model if is_langchain_llm(llm): # This is a LangChain LLM - use agenerate_prompt() with batch for multiple generations langchain_llm = t.cast(BaseLanguageModel, llm) # LangChain doesn't support n parameter directly, so we batch multiple prompts prompts = t.cast(t.List[t.Any], [prompt_value for _ in range(n)]) resp = await langchain_llm.agenerate_prompt( prompts, stop=stop, callbacks=prompt_cb, ) elif isinstance(llm, InstructorBaseRagasLLM): # This is an InstructorLLM - use its generate()/agenerate() method # InstructorLLM.generate()/agenerate() only takes prompt and response_model parameters from ragas.llms.base import InstructorLLM instructor_llm = t.cast(InstructorLLM, llm) if instructor_llm.is_async: result = await llm.agenerate( prompt=prompt_value.text, response_model=self.output_model, ) else: result = llm.generate( prompt=prompt_value.text, response_model=self.output_model, ) # Wrap the single response in an LLMResult-like structure for consistency from langchain_core.outputs import Generation, LLMResult generation = Generation(text=result.model_dump_json()) resp = LLMResult(generations=[[generation]]) else: # This is a standard BaseRagasLLM - use generate() ragas_llm = t.cast(BaseRagasLLM, llm) resp = await ragas_llm.generate( prompt_value, n=n, temperature=temperature, stop=stop, callbacks=prompt_cb, ) output_models = [] parser = RagasOutputParser(pydantic_object=self.output_model) # Handle cases where LLM returns fewer generations than requested if is_langchain_llm(llm) or isinstance(llm, InstructorBaseRagasLLM): available_generations = len(resp.generations) else: available_generations = len(resp.generations[0]) if resp.generations else 0 actual_n = min(n, available_generations) if actual_n == 0: logger.error( f"LLM returned no generations when {n} were requested. Cannot proceed." ) raise ValueError(f"LLM returned no generations when {n} were requested") if actual_n < n: logger.warning( f"LLM returned {actual_n} generations instead of requested {n}. " f"Proceeding with {actual_n} generations." ) for i in range(actual_n): if is_langchain_llm(llm) or isinstance(llm, InstructorBaseRagasLLM): # For LangChain LLMs and InstructorLLM, each generation is in a separate batch result output_string = resp.generations[i][0].text else: # For Ragas LLMs, all generations are in the first batch output_string = resp.generations[0][i].text try: # For the parser, we need a BaseRagasLLM, so if it's a LangChain LLM, we need to handle this if is_langchain_llm(llm) or isinstance(llm, InstructorBaseRagasLLM): # Skip parsing retry for LangChain LLMs since parser expects BaseRagasLLM answer = self.output_model.model_validate_json(output_string) else: ragas_llm = t.cast(BaseRagasLLM, llm) answer = await parser.parse_output_string( output_string=output_string, prompt_value=prompt_value, llm=ragas_llm, callbacks=prompt_cb, retries_left=retries_left, ) processed_output = self.process_output(answer, data) # type: ignore output_models.append(processed_output) except RagasOutputParserException as e: prompt_rm.on_chain_error(error=e) logger.error("Prompt %s failed to parse output: %s", self.name, e) raise e prompt_rm.on_chain_end({"output": output_models}) # Track prompt usage track( PromptUsageEvent( prompt_type="pydantic", has_examples=len(self.examples) > 0, num_examples=len(self.examples), has_response_model=True, # PydanticPrompt always has response model language=self.language, ) ) return output_models ``` ### adapt ```python adapt(target_language: str, llm: Union[BaseRagasLLM, InstructorBaseRagasLLM], adapt_instruction: bool = False) -> 'PydanticPrompt[InputModel, OutputModel]' ``` Adapt the prompt to a new language. Source code in `src/ragas/prompt/pydantic_prompt.py` ```python async def adapt( self, target_language: str, llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM], adapt_instruction: bool = False, ) -> "PydanticPrompt[InputModel, OutputModel]": """ Adapt the prompt to a new language. """ strings = get_all_strings(self.examples) translated_strings = await translate_statements_prompt.generate( llm=llm, data=ToTranslate(target_language=target_language, statements=strings), ) translated_examples = update_strings( obj=self.examples, old_strings=strings, new_strings=translated_strings.statements, ) new_prompt = copy.deepcopy(self) new_prompt.examples = translated_examples new_prompt.language = target_language if adapt_instruction: translated_instruction = await translate_statements_prompt.generate( llm=llm, data=ToTranslate( target_language=target_language, statements=[self.instruction] ), ) new_prompt.instruction = translated_instruction.statements[0] new_prompt.original_hash = hash(new_prompt) return new_prompt ``` ### save ```python save(file_path: str) ``` Save the prompt to a file. Source code in `src/ragas/prompt/pydantic_prompt.py` ```python def save(self, file_path: str): """ Save the prompt to a file. """ data = { "ragas_version": __version__, "original_hash": ( hash(self) if self.original_hash is None else self.original_hash ), "language": self.language, "instruction": self.instruction, "examples": [ {"input": example[0].model_dump(), "output": example[1].model_dump()} for example in self.examples ], } if os.path.exists(file_path): raise FileExistsError(f"The file '{file_path}' already exists.") with open(file_path, "w", encoding="utf-8") as f: json.dump(data, f, indent=2, ensure_ascii=False) print(f"Prompt saved to {file_path}") ``` ## BoolIO Bases: `BaseModel` ## StringIO Bases: `BaseModel` ## PromptMixin Mixin class for classes that have prompts. eg: BaseSynthesizer, MetricWithLLM ### get_prompts ```python get_prompts() -> Dict[str, PydanticPrompt] ``` Returns a dictionary of prompts for the class. Source code in `src/ragas/prompt/mixin.py` ```python def get_prompts(self) -> t.Dict[str, PydanticPrompt]: """ Returns a dictionary of prompts for the class. """ prompts = {} for _, value in self._get_prompts().items(): prompts.update({value.name: value}) return prompts ``` ### set_prompts ```python set_prompts(**prompts) ``` Sets the prompts for the class. Raises: | Type | Description | | ------------ | --------------------------------------------------- | | `ValueError` | If the prompt is not an instance of PydanticPrompt. | Source code in `src/ragas/prompt/mixin.py` ```python def set_prompts(self, **prompts): """ Sets the prompts for the class. Raises ------ ValueError If the prompt is not an instance of `PydanticPrompt`. """ available_prompts = self.get_prompts() name_to_var = {v.name: k for k, v in self._get_prompts().items()} for key, value in prompts.items(): if key not in available_prompts: raise ValueError( f"Prompt with name '{key}' does not exist. Use get_prompts() to see available prompts." ) if not isinstance(value, PydanticPrompt): raise ValueError( f"Prompt with name '{key}' must be an instance of 'ragas.prompt.PydanticPrompt'" ) setattr(self, name_to_var[key], value) ``` ### adapt_prompts ```python adapt_prompts(language: str, llm: Union[BaseRagasLLM, InstructorBaseRagasLLM], adapt_instruction: bool = False) -> Dict[str, PydanticPrompt] ``` Adapts the prompts in the class to the given language and using the given LLM. Notes Make sure you use the best available LLM for adapting the prompts and then save and load the prompts using save_prompts and load_prompts methods. Source code in `src/ragas/prompt/mixin.py` ```python async def adapt_prompts( self, language: str, llm: t.Union[BaseRagasLLM, InstructorBaseRagasLLM], adapt_instruction: bool = False, ) -> t.Dict[str, PydanticPrompt]: """ Adapts the prompts in the class to the given language and using the given LLM. Notes ----- Make sure you use the best available LLM for adapting the prompts and then save and load the prompts using [save_prompts][ragas.prompt.mixin.PromptMixin.save_prompts] and [load_prompts][ragas.prompt.mixin.PromptMixin.load_prompts] methods. """ prompts = self.get_prompts() adapted_prompts = {} for name, prompt in prompts.items(): adapted_prompt = await prompt.adapt(language, llm, adapt_instruction) adapted_prompts[name] = adapted_prompt return adapted_prompts ``` ### save_prompts ```python save_prompts(path: str) ``` Saves the prompts to a directory in the format of {name}\_{language}.json Source code in `src/ragas/prompt/mixin.py` ```python def save_prompts(self, path: str): """ Saves the prompts to a directory in the format of {name}_{language}.json """ # check if path is valid if not os.path.exists(path): raise ValueError(f"Path {path} does not exist") prompts = self.get_prompts() for prompt_name, prompt in prompts.items(): # hash_hex = f"0x{hash(prompt) & 0xFFFFFFFFFFFFFFFF:016x}" if self.name == "": file_name = os.path.join(path, f"{prompt_name}_{prompt.language}.json") else: file_name = os.path.join( path, f"{self.name}_{prompt_name}_{prompt.language}.json" ) prompt.save(file_name) ``` ### load_prompts ```python load_prompts(path: str, language: Optional[str] = None) ``` Loads the prompts from a path. File should be in the format of {name}\_{language}.json Source code in `src/ragas/prompt/mixin.py` ```python def load_prompts(self, path: str, language: t.Optional[str] = None): """ Loads the prompts from a path. File should be in the format of {name}_{language}.json """ # check if path is valid if not os.path.exists(path): raise ValueError(f"Path {path} does not exist") # check if language is supported, defaults to english if language is None: language = "english" logger.info( "Language not specified, loading prompts for default language: %s", language, ) loaded_prompts = {} for prompt_name, prompt in self.get_prompts().items(): if self.name == "": file_name = os.path.join(path, f"{prompt_name}_{language}.json") else: file_name = os.path.join( path, f"{self.name}_{prompt_name}_{language}.json" ) loaded_prompt = prompt.__class__.load(file_name) loaded_prompts[prompt_name] = loaded_prompt return loaded_prompts ``` ## Metrics Collections Prompts Modern metrics in Ragas use specialized prompt classes. Each metric module contains: - **Input Model**: Defines what data the prompt needs (e.g., `FaithfulnessInput`) - **Output Model**: Defines the expected LLM response structure (e.g., `FaithfulnessOutput`) - **Prompt Class**: Inherits from `BasePrompt` to generate the prompt string with examples and instructions ### Example: Faithfulness Metric Prompts ```python from ragas.metrics.collections.faithfulness.util import ( FaithfulnessPrompt, FaithfulnessInput, FaithfulnessOutput, ) # The prompt class combines input/output models with instructions and examples prompt = FaithfulnessPrompt() # Create input data input_data = FaithfulnessInput( response="The capital of France is Paris.", context="Paris is the capital and most populous city of France." ) # Generate the prompt string for the LLM prompt_string = prompt.to_string(input_data) # The output will be structured according to FaithfulnessOutput model ``` ### Available Metric Prompts See the individual metric documentation for details on their prompts: - [Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/index.md) - [Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/index.md) - [Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/index.md) - [Answer Correctness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_correctness/index.md) - [Factual Correctness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/factual_correctness/index.md) - [Noise Sensitivity](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/noise_sensitivity/index.md) ## Customization For detailed guidance on customizing prompts for metrics, see [Modifying prompts in metrics](https://docs.ragas.io/en/stable/howtos/customizations/metrics/modifying-prompts-metrics/index.md). ## RunConfig ```python RunConfig(timeout: int = 180, max_retries: int = 10, max_wait: int = 60, max_workers: int = 16, exception_types: Union[Type[BaseException], Tuple[Type[BaseException], ...]] = (Exception,), log_tenacity: bool = False, seed: int = 42) ``` Configuration for a timeouts, retries and seed for Ragas operations. Parameters: | Name | Type | Description | Default | | ----------------- | ------------------------------------------------------------- | ------------------------------------------------------------------------- | -------------- | | `timeout` | `int` | Maximum time (in seconds) to wait for a single operation, by default 180. | `180` | | `max_retries` | `int` | Maximum number of retry attempts, by default 10. | `10` | | `max_wait` | `int` | Maximum wait time (in seconds) between retries, by default 60. | `60` | | `max_workers` | `int` | Maximum number of concurrent workers, by default 16. | `16` | | `exception_types` | `Union[Type[BaseException], Tuple[Type[BaseException], ...]]` | Exception types to catch and retry on, by default (Exception,). | `(Exception,)` | | `log_tenacity` | `bool` | Whether to log retry attempts using tenacity, by default False. | `False` | | `seed` | `int` | Random seed for reproducibility, by default 42. | `42` | Attributes: | Name | Type | Description | | ----- | ----------- | ------------------------------------------------------------ | | `rng` | `Generator` | Random number generator initialized with the specified seed. | Notes The `__post_init__` method initializes the `rng` attribute as a numpy random number generator using the specified seed. ## add_retry ```python add_retry(fn: WrappedFn, run_config: RunConfig) -> WrappedFn ``` Adds retry functionality to a given function using the provided RunConfig. This function wraps the input function with retry logic using the tenacity library. It configures the retry behavior based on the settings in the RunConfig. Notes - If log_tenacity is enabled in the RunConfig, it sets up logging for retry attempts. - The retry logic uses exponential backoff with random jitter for wait times. - The number of retry attempts and exception types to retry on are configured based on the RunConfig. Source code in `src/ragas/run_config.py` ```python def add_retry(fn: WrappedFn, run_config: RunConfig) -> WrappedFn: """ Adds retry functionality to a given function using the provided RunConfig. This function wraps the input function with retry logic using the tenacity library. It configures the retry behavior based on the settings in the RunConfig. Notes ----- - If log_tenacity is enabled in the RunConfig, it sets up logging for retry attempts. - The retry logic uses exponential backoff with random jitter for wait times. - The number of retry attempts and exception types to retry on are configured based on the RunConfig. """ # configure tenacity's after section wtih logger if run_config.log_tenacity is not None: logger = logging.getLogger(f"ragas.retry.{fn.__name__}") tenacity_logger = after_log(logger, logging.DEBUG) else: tenacity_logger = after_nothing r = Retrying( wait=wait_random_exponential(multiplier=1, max=run_config.max_wait), stop=stop_after_attempt(run_config.max_retries), retry=retry_if_exception_type(run_config.exception_types), reraise=True, after=tenacity_logger, ) return r.wraps(fn) ``` ## add_async_retry ```python add_async_retry(fn: WrappedFn, run_config: RunConfig) -> WrappedFn ``` Decorator for retrying a function if it fails. Source code in `src/ragas/run_config.py` ```python def add_async_retry(fn: WrappedFn, run_config: RunConfig) -> WrappedFn: """ Decorator for retrying a function if it fails. """ # configure tenacity's after section wtih logger if run_config.log_tenacity is not None: logger = logging.getLogger(f"TENACITYRetry[{fn.__name__}]") tenacity_logger = after_log(logger, logging.DEBUG) else: tenacity_logger = after_nothing r = AsyncRetrying( wait=wait_random_exponential(multiplier=1, max=run_config.max_wait), stop=stop_after_attempt(run_config.max_retries), retry=retry_if_exception_type(run_config.exception_types), reraise=True, after=tenacity_logger, ) return r.wraps(fn) ``` ## BaseSynthesizer ```python BaseSynthesizer(name: str = '', llm: Union[BaseRagasLLM, 'InstructorBaseRagasLLM'] = _default_llm_factory(), llm_context: Optional[str] = None) ``` Bases: `ABC`, `Generic[Scenario]`, `PromptMixin` Base class for synthesizing scenarios and samples. ## default_query_distribution ```python default_query_distribution(llm: Union[BaseRagasLLM, InstructorBaseRagasLLM], kg: Optional[KnowledgeGraph] = None, llm_context: Optional[str] = None) -> QueryDistribution ``` Source code in `src/ragas/testset/synthesizers/__init__.py` ```python def default_query_distribution( llm: t.Union[BaseRagasLLM, "InstructorBaseRagasLLM"], kg: t.Optional[KnowledgeGraph] = None, llm_context: t.Optional[str] = None, ) -> QueryDistribution: """ """ default_queries = [ SingleHopSpecificQuerySynthesizer(llm=llm, llm_context=llm_context), MultiHopAbstractQuerySynthesizer(llm=llm, llm_context=llm_context), MultiHopSpecificQuerySynthesizer(llm=llm, llm_context=llm_context), ] if kg is not None: available_queries = [] for query in default_queries: try: if query.get_node_clusters(kg): available_queries.append(query) except Exception as e: # Keep broad catch minimal for resilience; log and skip. logger.warning( "Skipping %s due to unexpected error: %s", getattr(query, "name", type(query).__name__), e, ) continue if not available_queries: raise ValueError( "No compatible query synthesizers for the provided KnowledgeGraph." ) else: available_queries = default_queries return [(query, 1 / len(available_queries)) for query in available_queries] ``` ## TestsetSample Bases: `BaseSample` Represents a sample in a test set. Attributes: | Name | Type | Description | | ------------------ | ------------------------------------------ | ------------------------------------------------------------------------------ | | `eval_sample` | `Union[SingleTurnSample, MultiTurnSample]` | The evaluation sample, which can be either a single-turn or multi-turn sample. | | `synthesizer_name` | `str` | The name of the synthesizer used to generate this sample. | ## TestsetPacket Bases: `BaseModel` A packet of testset samples to be uploaded to the server. ## Testset ```python Testset(samples: List[TestsetSample], run_id: str = (lambda: str(uuid4()))(), cost_cb: Optional[CostCallbackHandler] = None) ``` Bases: `RagasDataset[TestsetSample]` Represents a test set containing multiple test samples. Attributes: | Name | Type | Description | | --------- | --------------------- | ------------------------------------------------------------------------- | | `samples` | `List[TestsetSample]` | A list of TestsetSample objects representing the samples in the test set. | ### to_evaluation_dataset ```python to_evaluation_dataset() -> EvaluationDataset ``` Converts the Testset to an EvaluationDataset. Source code in `src/ragas/testset/synthesizers/testset_schema.py` ```python def to_evaluation_dataset(self) -> EvaluationDataset: """ Converts the Testset to an EvaluationDataset. """ return EvaluationDataset( samples=[sample.eval_sample for sample in self.samples] ) ``` ### to_list ```python to_list() -> List[Dict] ``` Converts the Testset to a list of dictionaries. Source code in `src/ragas/testset/synthesizers/testset_schema.py` ```python def to_list(self) -> t.List[t.Dict]: """ Converts the Testset to a list of dictionaries. """ list_dict = [] for sample in self.samples: sample_dict = sample.eval_sample.model_dump(exclude_none=True) sample_dict["synthesizer_name"] = sample.synthesizer_name list_dict.append(sample_dict) return list_dict ``` ### from_list ```python from_list(data: List[Dict]) -> Testset ``` Converts a list of dictionaries to a Testset. Source code in `src/ragas/testset/synthesizers/testset_schema.py` ```python @classmethod def from_list(cls, data: t.List[t.Dict]) -> Testset: """ Converts a list of dictionaries to a Testset. """ # first create the samples samples = [] for sample in data: synthesizer_name = sample["synthesizer_name"] # remove the synthesizer name from the sample sample.pop("synthesizer_name") # the remaining sample is the eval_sample eval_sample = sample # if user_input is a list it is MultiTurnSample if "user_input" in eval_sample and not isinstance( eval_sample.get("user_input"), list ): eval_sample = SingleTurnSample(**eval_sample) else: eval_sample = MultiTurnSample(**eval_sample) samples.append( TestsetSample( eval_sample=eval_sample, synthesizer_name=synthesizer_name ) ) # then create the testset return Testset(samples=samples) ``` ### total_tokens ```python total_tokens() -> Union[List[TokenUsage], TokenUsage] ``` Compute the total tokens used in the evaluation. Source code in `src/ragas/testset/synthesizers/testset_schema.py` ```python def total_tokens(self) -> t.Union[t.List[TokenUsage], TokenUsage]: """ Compute the total tokens used in the evaluation. """ if self.cost_cb is None: raise ValueError( "The Testset was not configured for computing cost. Please provide a token_usage_parser function to TestsetGenerator to compute cost." ) return self.cost_cb.total_tokens() ``` ### total_cost ```python total_cost(cost_per_input_token: Optional[float] = None, cost_per_output_token: Optional[float] = None) -> float ``` Compute the total cost of the evaluation. Source code in `src/ragas/testset/synthesizers/testset_schema.py` ```python def total_cost( self, cost_per_input_token: t.Optional[float] = None, cost_per_output_token: t.Optional[float] = None, ) -> float: """ Compute the total cost of the evaluation. """ if self.cost_cb is None: raise ValueError( "The Testset was not configured for computing cost. Please provide a token_usage_parser function to TestsetGenerator to compute cost." ) return self.cost_cb.total_cost( cost_per_input_token=cost_per_input_token, cost_per_output_token=cost_per_output_token, ) ``` ### from_annotated ```python from_annotated(path: str) -> Testset ``` Loads a testset from an annotated JSON file. Source code in `src/ragas/testset/synthesizers/testset_schema.py` ```python @classmethod def from_annotated(cls, path: str) -> Testset: """ Loads a testset from an annotated JSON file. """ import json with open(path, "r") as f: annotated_testset = json.load(f) samples = [] for sample in annotated_testset: if sample["approval_status"] == "approved": samples.append(TestsetSample(**sample)) return cls(samples=samples) ``` ## QueryLength Bases: `str`, `Enum` Enumeration of query lengths. Available options are: LONG, MEDIUM, SHORT ## QueryStyle Bases: `str`, `Enum` Enumeration of query styles. Available options are: MISSPELLED, PERFECT_GRAMMAR, POOR_GRAMMAR, WEB_SEARCH_LIKE ## BaseScenario Bases: `BaseModel` Base class for representing a scenario for generating test samples. Attributes: | Name | Type | Description | | --------- | ------------- | --------------------------------------- | | `nodes` | `List[Node]` | List of nodes involved in the scenario. | | `style` | `QueryStyle` | The style of the query. | | `length` | `QueryLength` | The length of the query. | | `persona` | `Persona` | A persona associated with the scenario. | ## SingleHopSpecificQuerySynthesizer ```python SingleHopSpecificQuerySynthesizer(name: str = 'single_hop_specific_query_synthesizer', llm: Union[BaseRagasLLM, 'InstructorBaseRagasLLM'] = _default_llm_factory(), llm_context: Optional[str] = None, generate_query_reference_prompt: PydanticPrompt = QueryAnswerGenerationPrompt(), theme_persona_matching_prompt: PydanticPrompt = ThemesPersonasMatchingPrompt(), property_name: str = 'entities') ``` Bases: `SingleHopQuerySynthesizer` ## MultiHopSpecificQuerySynthesizer ```python MultiHopSpecificQuerySynthesizer(name: str = 'multi_hop_specific_query_synthesizer', llm: Union[BaseRagasLLM, 'InstructorBaseRagasLLM'] = _default_llm_factory(), llm_context: Optional[str] = None, generate_query_reference_prompt: PydanticPrompt = QueryAnswerGenerationPrompt(), property_name: str = 'entities', relation_type: str = 'entities_overlap', relation_overlap_property: str = 'overlapped_items', theme_persona_matching_prompt: PydanticPrompt = ThemesPersonasMatchingPrompt()) ``` Bases: `MultiHopQuerySynthesizer` Synthesize multi-hop queries based on a chunk cluster defined by entity overlap. ### get_node_clusters ```python get_node_clusters(knowledge_graph: KnowledgeGraph) -> List[Tuple] ``` Identify clusters of nodes based on the specified relationship condition. Source code in `src/ragas/testset/synthesizers/multi_hop/specific.py` ```python def get_node_clusters(self, knowledge_graph: KnowledgeGraph) -> t.List[t.Tuple]: """Identify clusters of nodes based on the specified relationship condition.""" node_clusters = knowledge_graph.find_two_nodes_single_rel( relationship_condition=lambda rel: rel.type == self.relation_type ) logger.info("found %d clusters", len(node_clusters)) return node_clusters ``` # Tokenizers Ragas supports multiple tokenizer implementations for text splitting during knowledge graph operations and test data generation. ## Overview When extracting properties from knowledge graph nodes, text is split into chunks based on token limits. By default, Ragas uses tiktoken (OpenAI's tokenizer), but you can also use HuggingFace tokenizers for better compatibility with open-source models. ## Available Tokenizers ### TiktokenWrapper Wrapper for OpenAI's tiktoken tokenizers. This is the default tokenizer. ```python from ragas import TiktokenWrapper # Using default encoding (o200k_base) tokenizer = TiktokenWrapper() # Using a specific encoding tokenizer = TiktokenWrapper(encoding_name="cl100k_base") # Using encoding for a specific model tokenizer = TiktokenWrapper(model_name="gpt-4") ``` ### HuggingFaceTokenizer Wrapper for HuggingFace transformers tokenizers. Use this when working with open-source models. ```python from ragas import HuggingFaceTokenizer # Load tokenizer for a specific model tokenizer = HuggingFaceTokenizer(model_name="meta-llama/Llama-2-7b-hf") # Use a pre-initialized tokenizer from transformers import AutoTokenizer hf_tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1") tokenizer = HuggingFaceTokenizer(tokenizer=hf_tokenizer) ``` **Note:** HuggingFace tokenizers require the `transformers` package. Install it with: ```sh pip install transformers # or uv add transformers ``` ### Factory Function Use `get_tokenizer()` for a simple way to create tokenizers: ```python from ragas import get_tokenizer # Default tiktoken tokenizer tokenizer = get_tokenizer() # Tiktoken for a specific model tokenizer = get_tokenizer("tiktoken", model_name="gpt-4") # HuggingFace tokenizer tokenizer = get_tokenizer("huggingface", model_name="meta-llama/Llama-2-7b-hf") ``` ## Using Custom Tokenizers ### With LLM-based Extractors All LLM-based extractors accept a `tokenizer` parameter: ```python from ragas import HuggingFaceTokenizer from ragas.testset.transforms import ( SummaryExtractor, KeyphrasesExtractor, HeadlinesExtractor, ) # Create a HuggingFace tokenizer for your model tokenizer = HuggingFaceTokenizer(model_name="meta-llama/Llama-2-7b-hf") # Use it with extractors summary_extractor = SummaryExtractor(llm=your_llm, tokenizer=tokenizer) keyphrase_extractor = KeyphrasesExtractor(llm=your_llm, tokenizer=tokenizer) headlines_extractor = HeadlinesExtractor(llm=your_llm, tokenizer=tokenizer) ``` ### Custom Tokenizer Implementation You can create your own tokenizer by extending `BaseTokenizer`: ```python from ragas.tokenizers import BaseTokenizer class MyCustomTokenizer(BaseTokenizer): def __init__(self, ...): # Initialize your tokenizer pass def encode(self, text: str) -> list[int]: # Return token IDs pass def decode(self, tokens: list[int]) -> str: # Return decoded text pass ``` ## API Reference Tokenizer abstractions for Ragas. This module provides a unified interface for different tokenizer implementations, supporting both tiktoken (OpenAI) and HuggingFace tokenizers. ## BaseTokenizer Bases: `ABC` Abstract base class for tokenizers. ### encode ```python encode(text: str) -> List[int] ``` Encode text into token IDs. Source code in `src/ragas/tokenizers.py` ```python @abstractmethod def encode(self, text: str) -> t.List[int]: """Encode text into token IDs.""" pass ``` ### decode ```python decode(tokens: List[int]) -> str ``` Decode token IDs back into text. Source code in `src/ragas/tokenizers.py` ```python @abstractmethod def decode(self, tokens: t.List[int]) -> str: """Decode token IDs back into text.""" pass ``` ### count_tokens ```python count_tokens(text: str) -> int ``` Count the number of tokens in text. Source code in `src/ragas/tokenizers.py` ```python def count_tokens(self, text: str) -> int: """Count the number of tokens in text.""" return len(self.encode(text)) ``` ## TiktokenWrapper ```python TiktokenWrapper(encoding: Optional[Encoding] = None, model_name: Optional[str] = None, encoding_name: Optional[str] = None) ``` Bases: `BaseTokenizer` Wrapper for tiktoken encodings (OpenAI tokenizers). Parameters: | Name | Type | Description | Default | | --------------- | ---------- | ---------------------------------------------------------------- | ---------- | | `encoding` | `Encoding` | A pre-initialized tiktoken encoding. | `None` | | `model_name` | `str` | Model name to get encoding for (e.g., "gpt-4", "gpt-3.5-turbo"). | `None` | | `encoding_name` | `str` | Encoding name (e.g., "cl100k_base", "o200k_base"). | `None` | | `If` | | | *required* | Source code in `src/ragas/tokenizers.py` ```python def __init__( self, encoding: t.Optional[tiktoken.Encoding] = None, model_name: t.Optional[str] = None, encoding_name: t.Optional[str] = None, ): """ Initialize TiktokenWrapper. Parameters ---------- encoding : tiktoken.Encoding, optional A pre-initialized tiktoken encoding. model_name : str, optional Model name to get encoding for (e.g., "gpt-4", "gpt-3.5-turbo"). encoding_name : str, optional Encoding name (e.g., "cl100k_base", "o200k_base"). If none provided, defaults to "o200k_base" encoding. """ if encoding is not None: self._encoding = encoding elif model_name is not None: self._encoding = tiktoken.encoding_for_model(model_name) elif encoding_name is not None: self._encoding = tiktoken.get_encoding(encoding_name) else: self._encoding = tiktoken.get_encoding("o200k_base") ``` ### encoding ```python encoding: Encoding ``` Access the underlying tiktoken encoding. ## HuggingFaceTokenizer ```python HuggingFaceTokenizer(tokenizer: Optional[Any] = None, model_name: Optional[str] = None) ``` Bases: `BaseTokenizer` Wrapper for HuggingFace tokenizers. Parameters: | Name | Type | Description | Default | | ------------ | ------------------------------------------------ | -------------------------------------------------------------------------- | ---------- | | `tokenizer` | `PreTrainedTokenizer or PreTrainedTokenizerFast` | A pre-initialized HuggingFace tokenizer. | `None` | | `model_name` | `str` | Model name or path to load tokenizer from (e.g., "meta-llama/Llama-2-7b"). | `None` | | `One` | | | *required* | Source code in `src/ragas/tokenizers.py` ```python def __init__( self, tokenizer: t.Optional[t.Any] = None, model_name: t.Optional[str] = None, ): """ Initialize HuggingFaceTokenizer. Parameters ---------- tokenizer : PreTrainedTokenizer or PreTrainedTokenizerFast, optional A pre-initialized HuggingFace tokenizer. model_name : str, optional Model name or path to load tokenizer from (e.g., "meta-llama/Llama-2-7b"). One of tokenizer or model_name must be provided. """ if tokenizer is not None: self._tokenizer = tokenizer elif model_name is not None: try: from transformers import AutoTokenizer except ImportError: raise ImportError( "transformers package is required for HuggingFace tokenizers. " "Install it with: pip install transformers" ) self._tokenizer = AutoTokenizer.from_pretrained(model_name) else: raise ValueError("Either tokenizer or model_name must be provided") ``` ### tokenizer ```python tokenizer: Any ``` Access the underlying HuggingFace tokenizer. ## get_default_tokenizer ```python get_default_tokenizer() -> TiktokenWrapper ``` Get the default tokenizer, creating it lazily on first access. Source code in `src/ragas/tokenizers.py` ```python def get_default_tokenizer() -> TiktokenWrapper: """Get the default tokenizer, creating it lazily on first access.""" global _default_tokenizer if _default_tokenizer is None: _default_tokenizer = TiktokenWrapper(encoding_name="o200k_base") return _default_tokenizer ``` ## get_tokenizer ```python get_tokenizer(tokenizer_type: str = 'tiktoken', model_name: Optional[str] = None, encoding_name: Optional[str] = None) -> BaseTokenizer ``` Factory function to get a tokenizer instance. Parameters: | Name | Type | Description | Default | | ---------------- | ----- | ----------------------------------------------- | ------------ | | `tokenizer_type` | `str` | Type of tokenizer: "tiktoken" or "huggingface". | `'tiktoken'` | | `model_name` | `str` | Model name for the tokenizer. | `None` | | `encoding_name` | `str` | Encoding name (only for tiktoken). | `None` | Returns: | Type | Description | | --------------- | --------------------- | | `BaseTokenizer` | A tokenizer instance. | Examples: ```pycon >>> # Get default tiktoken tokenizer >>> tokenizer = get_tokenizer() ``` ```pycon >>> # Get tiktoken for a specific model >>> tokenizer = get_tokenizer("tiktoken", model_name="gpt-4") ``` ```pycon >>> # Get HuggingFace tokenizer >>> tokenizer = get_tokenizer("huggingface", model_name="meta-llama/Llama-2-7b") ``` Source code in `src/ragas/tokenizers.py` ```python def get_tokenizer( tokenizer_type: str = "tiktoken", model_name: t.Optional[str] = None, encoding_name: t.Optional[str] = None, ) -> BaseTokenizer: """ Factory function to get a tokenizer instance. Parameters ---------- tokenizer_type : str Type of tokenizer: "tiktoken" or "huggingface". model_name : str, optional Model name for the tokenizer. encoding_name : str, optional Encoding name (only for tiktoken). Returns ------- BaseTokenizer A tokenizer instance. Examples -------- >>> # Get default tiktoken tokenizer >>> tokenizer = get_tokenizer() >>> # Get tiktoken for a specific model >>> tokenizer = get_tokenizer("tiktoken", model_name="gpt-4") >>> # Get HuggingFace tokenizer >>> tokenizer = get_tokenizer("huggingface", model_name="meta-llama/Llama-2-7b") """ if tokenizer_type == "tiktoken": return TiktokenWrapper(model_name=model_name, encoding_name=encoding_name) elif tokenizer_type == "huggingface": if model_name is None: raise ValueError("model_name is required for HuggingFace tokenizers") return HuggingFaceTokenizer(model_name=model_name) else: raise ValueError(f"Unknown tokenizer type: {tokenizer_type}") ``` ## BaseGraphTransformation ```python BaseGraphTransformation(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)()) ``` Bases: `ABC` Abstract base class for graph transformations on a KnowledgeGraph. ### transform ```python transform(kg: KnowledgeGraph) -> Any ``` Abstract method to transform the KnowledgeGraph. Transformations should be idempotent, meaning that applying the transformation multiple times should yield the same result as applying it once. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | -------------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be transformed. | *required* | Returns: | Type | Description | | ----- | -------------------------------- | | `Any` | The transformed knowledge graph. | Source code in `src/ragas/testset/transforms/base.py` ```python @abstractmethod async def transform(self, kg: KnowledgeGraph) -> t.Any: """ Abstract method to transform the KnowledgeGraph. Transformations should be idempotent, meaning that applying the transformation multiple times should yield the same result as applying it once. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be transformed. Returns ------- t.Any The transformed knowledge graph. """ pass ``` ### filter ```python filter(kg: KnowledgeGraph) -> KnowledgeGraph ``` Filters the KnowledgeGraph and returns the filtered graph. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | ----------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be filtered. | *required* | Returns: | Type | Description | | ---------------- | ----------------------------- | | `KnowledgeGraph` | The filtered knowledge graph. | Source code in `src/ragas/testset/transforms/base.py` ```python def filter(self, kg: KnowledgeGraph) -> KnowledgeGraph: """ Filters the KnowledgeGraph and returns the filtered graph. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be filtered. Returns ------- KnowledgeGraph The filtered knowledge graph. """ logger.debug("Filtering KnowledgeGraph with %s", self.filter_nodes.__name__) filtered_nodes = [node for node in kg.nodes if self.filter_nodes(node)] node_ids = {node.id for node in filtered_nodes} filtered_relationships = [ rel for rel in kg.relationships if (rel.source.id in node_ids) and (rel.target.id in node_ids) ] logger.debug( "Filter reduced KnowledgeGraph by %d/%d nodes and %d/%d relationships", len(kg.nodes) - len(filtered_nodes), len(kg.nodes), len(kg.relationships) - len(filtered_relationships), len(kg.relationships), ) return KnowledgeGraph( nodes=filtered_nodes, relationships=filtered_relationships, ) ``` ### generate_execution_plan ```python generate_execution_plan(kg: KnowledgeGraph) -> Sequence[Coroutine] ``` Generates a sequence of coroutines to be executed in sequence by the Executor. This coroutine will, upon execution, write the transformation into the KnowledgeGraph. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | -------------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be transformed. | *required* | Returns: | Type | Description | | --------------------- | ---------------------------------------------------- | | `Sequence[Coroutine]` | A sequence of coroutines to be executed in parallel. | Source code in `src/ragas/testset/transforms/base.py` ```python @abstractmethod def generate_execution_plan(self, kg: KnowledgeGraph) -> t.Sequence[t.Coroutine]: """ Generates a sequence of coroutines to be executed in sequence by the Executor. This coroutine will, upon execution, write the transformation into the KnowledgeGraph. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be transformed. Returns ------- t.Sequence[t.Coroutine] A sequence of coroutines to be executed in parallel. """ pass ``` ## Extractor ```python Extractor(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)()) ``` Bases: `BaseGraphTransformation` Abstract base class for extractors that transform a KnowledgeGraph by extracting specific properties from its nodes. Methods: | Name | Description | | ----------- | ---------------------------------------------------------------------- | | `transform` | Transforms the KnowledgeGraph by extracting properties from its nodes. | | `extract` | Abstract method to extract a specific property from a node. | ### transform ```python transform(kg: KnowledgeGraph) -> List[Tuple[Node, Tuple[str, Any]]] ``` Transforms the KnowledgeGraph by extracting properties from its nodes. Uses the `filter` method to filter the graph and the `extract` method to extract properties from each node. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | -------------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be transformed. | *required* | Returns: | Type | Description | | ------------------------------------ | ----------------------------------------------------------------------------- | | `List[Tuple[Node, Tuple[str, Any]]]` | A list of tuples where each tuple contains a node and the extracted property. | Examples: ```pycon >>> kg = KnowledgeGraph(nodes=[Node(id=1, properties={"name": "Node1"}), Node(id=2, properties={"name": "Node2"})]) >>> extractor = SomeConcreteExtractor() >>> extractor.transform(kg) [(Node(id=1, properties={"name": "Node1"}), ("property_name", "extracted_value")), (Node(id=2, properties={"name": "Node2"}), ("property_name", "extracted_value"))] ``` Source code in `src/ragas/testset/transforms/base.py` ```python async def transform( self, kg: KnowledgeGraph ) -> t.List[t.Tuple[Node, t.Tuple[str, t.Any]]]: """ Transforms the KnowledgeGraph by extracting properties from its nodes. Uses the `filter` method to filter the graph and the `extract` method to extract properties from each node. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be transformed. Returns ------- t.List[t.Tuple[Node, t.Tuple[str, t.Any]]] A list of tuples where each tuple contains a node and the extracted property. Examples -------- >>> kg = KnowledgeGraph(nodes=[Node(id=1, properties={"name": "Node1"}), Node(id=2, properties={"name": "Node2"})]) >>> extractor = SomeConcreteExtractor() >>> extractor.transform(kg) [(Node(id=1, properties={"name": "Node1"}), ("property_name", "extracted_value")), (Node(id=2, properties={"name": "Node2"}), ("property_name", "extracted_value"))] """ filtered = self.filter(kg) return [(node, await self.extract(node)) for node in filtered.nodes] ``` ### extract ```python extract(node: Node) -> Tuple[str, Any] ``` Abstract method to extract a specific property from a node. Parameters: | Name | Type | Description | Default | | ------ | ------ | -------------------------------------------- | ---------- | | `node` | `Node` | The node from which to extract the property. | *required* | Returns: | Type | Description | | ----------------- | ------------------------------------------------------------- | | `Tuple[str, Any]` | A tuple containing the property name and the extracted value. | Source code in `src/ragas/testset/transforms/base.py` ```python @abstractmethod async def extract(self, node: Node) -> t.Tuple[str, t.Any]: """ Abstract method to extract a specific property from a node. Parameters ---------- node : Node The node from which to extract the property. Returns ------- t.Tuple[str, t.Any] A tuple containing the property name and the extracted value. """ pass ``` ### generate_execution_plan ```python generate_execution_plan(kg: KnowledgeGraph) -> Sequence[Coroutine] ``` Generates a sequence of coroutines to be executed in parallel by the Executor. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | -------------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be transformed. | *required* | Returns: | Type | Description | | --------------------- | ---------------------------------------------------- | | `Sequence[Coroutine]` | A sequence of coroutines to be executed in parallel. | Source code in `src/ragas/testset/transforms/base.py` ```python def generate_execution_plan(self, kg: KnowledgeGraph) -> t.Sequence[t.Coroutine]: """ Generates a sequence of coroutines to be executed in parallel by the Executor. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be transformed. Returns ------- t.Sequence[t.Coroutine] A sequence of coroutines to be executed in parallel. """ async def apply_extract(node: Node): property_name, property_value = await self.extract(node) if node.get_property(property_name) is None: node.add_property(property_name, property_value) else: logger.warning( "Property '%s' already exists in node '%.6s'. Skipping!", property_name, node.id, ) filtered = self.filter(kg) plan = [apply_extract(node) for node in filtered.nodes] logger.debug( "Created %d coroutines for %s", len(plan), self.__class__.__name__, ) return plan ``` ## NodeFilter ```python NodeFilter(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)()) ``` Bases: `BaseGraphTransformation` ### custom_filter ```python custom_filter(node: Node, kg: KnowledgeGraph) -> bool ``` Abstract method to filter a node based on a prompt. Parameters: | Name | Type | Description | Default | | ------ | ------ | ------------------------ | ---------- | | `node` | `Node` | The node to be filtered. | *required* | Returns: | Type | Description | | ------ | --------------------------------------------------------- | | `bool` | A boolean indicating whether the node should be filtered. | Source code in `src/ragas/testset/transforms/base.py` ```python @abstractmethod async def custom_filter(self, node: Node, kg: KnowledgeGraph) -> bool: """ Abstract method to filter a node based on a prompt. Parameters ---------- node : Node The node to be filtered. Returns ------- bool A boolean indicating whether the node should be filtered. """ pass ``` ### generate_execution_plan ```python generate_execution_plan(kg: KnowledgeGraph) -> Sequence[Coroutine] ``` Generates a sequence of coroutines to be executed Source code in `src/ragas/testset/transforms/base.py` ```python def generate_execution_plan(self, kg: KnowledgeGraph) -> t.Sequence[t.Coroutine]: """ Generates a sequence of coroutines to be executed """ async def apply_filter(node: Node): if await self.custom_filter(node, kg): kg.remove_node(node) filtered = self.filter(kg) plan = [apply_filter(node) for node in filtered.nodes] logger.debug( "Created %d coroutines for %s", len(plan), self.__class__.__name__, ) return plan ``` ## RelationshipBuilder ```python RelationshipBuilder(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)()) ``` Bases: `BaseGraphTransformation` Abstract base class for building relationships in a KnowledgeGraph. Methods: | Name | Description | | ----------- | -------------------------------------------------------- | | `transform` | Transforms the KnowledgeGraph by building relationships. | ### transform ```python transform(kg: KnowledgeGraph) -> List[Relationship] ``` Transforms the KnowledgeGraph by building relationships. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | -------------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be transformed. | *required* | Returns: | Type | Description | | -------------------- | ---------------------------- | | `List[Relationship]` | A list of new relationships. | Source code in `src/ragas/testset/transforms/base.py` ```python @abstractmethod async def transform(self, kg: KnowledgeGraph) -> t.List[Relationship]: """ Transforms the KnowledgeGraph by building relationships. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be transformed. Returns ------- t.List[Relationship] A list of new relationships. """ pass ``` ### generate_execution_plan ```python generate_execution_plan(kg: KnowledgeGraph) -> Sequence[Coroutine] ``` Generates a sequence of coroutines to be executed in parallel by the Executor. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | -------------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be transformed. | *required* | Returns: | Type | Description | | --------------------- | ---------------------------------------------------- | | `Sequence[Coroutine]` | A sequence of coroutines to be executed in parallel. | Source code in `src/ragas/testset/transforms/base.py` ```python def generate_execution_plan(self, kg: KnowledgeGraph) -> t.Sequence[t.Coroutine]: """ Generates a sequence of coroutines to be executed in parallel by the Executor. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be transformed. Returns ------- t.Sequence[t.Coroutine] A sequence of coroutines to be executed in parallel. """ async def apply_build_relationships( filtered_kg: KnowledgeGraph, original_kg: KnowledgeGraph ): relationships = await self.transform(filtered_kg) original_kg.relationships.extend(relationships) filtered_kg = self.filter(kg) plan = [apply_build_relationships(filtered_kg=filtered_kg, original_kg=kg)] logger.debug( "Created %d coroutines for %s", len(plan), self.__class__.__name__, ) return plan ``` ## Splitter ```python Splitter(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)()) ``` Bases: `BaseGraphTransformation` Abstract base class for splitters that transform a KnowledgeGraph by splitting its nodes into smaller chunks. Methods: | Name | Description | | ----------- | ------------------------------------------------------------------------- | | `transform` | Transforms the KnowledgeGraph by splitting its nodes into smaller chunks. | | `split` | Abstract method to split a node into smaller chunks. | ### transform ```python transform(kg: KnowledgeGraph) -> Tuple[List[Node], List[Relationship]] ``` Transforms the KnowledgeGraph by splitting its nodes into smaller chunks. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | -------------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be transformed. | *required* | Returns: | Type | Description | | --------------------------------------- | ----------------------------------------------------------------------- | | `Tuple[List[Node], List[Relationship]]` | A tuple containing a list of new nodes and a list of new relationships. | Source code in `src/ragas/testset/transforms/base.py` ```python async def transform( self, kg: KnowledgeGraph ) -> t.Tuple[t.List[Node], t.List[Relationship]]: """ Transforms the KnowledgeGraph by splitting its nodes into smaller chunks. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be transformed. Returns ------- t.Tuple[t.List[Node], t.List[Relationship]] A tuple containing a list of new nodes and a list of new relationships. """ filtered = self.filter(kg) all_nodes = [] all_relationships = [] for node in filtered.nodes: nodes, relationships = await self.split(node) all_nodes.extend(nodes) all_relationships.extend(relationships) return all_nodes, all_relationships ``` ### split ```python split(node: Node) -> Tuple[List[Node], List[Relationship]] ``` Abstract method to split a node into smaller chunks. Parameters: | Name | Type | Description | Default | | ------ | ------ | --------------------- | ---------- | | `node` | `Node` | The node to be split. | *required* | Returns: | Type | Description | | --------------------------------------- | ----------------------------------------------------------------------- | | `Tuple[List[Node], List[Relationship]]` | A tuple containing a list of new nodes and a list of new relationships. | Source code in `src/ragas/testset/transforms/base.py` ```python @abstractmethod async def split(self, node: Node) -> t.Tuple[t.List[Node], t.List[Relationship]]: """ Abstract method to split a node into smaller chunks. Parameters ---------- node : Node The node to be split. Returns ------- t.Tuple[t.List[Node], t.List[Relationship]] A tuple containing a list of new nodes and a list of new relationships. """ pass ``` ### generate_execution_plan ```python generate_execution_plan(kg: KnowledgeGraph) -> Sequence[Coroutine] ``` Generates a sequence of coroutines to be executed in parallel by the Executor. Parameters: | Name | Type | Description | Default | | ---- | ---------------- | -------------------------------------- | ---------- | | `kg` | `KnowledgeGraph` | The knowledge graph to be transformed. | *required* | Returns: | Type | Description | | --------------------- | ---------------------------------------------------- | | `Sequence[Coroutine]` | A sequence of coroutines to be executed in parallel. | Source code in `src/ragas/testset/transforms/base.py` ```python def generate_execution_plan(self, kg: KnowledgeGraph) -> t.Sequence[t.Coroutine]: """ Generates a sequence of coroutines to be executed in parallel by the Executor. Parameters ---------- kg : KnowledgeGraph The knowledge graph to be transformed. Returns ------- t.Sequence[t.Coroutine] A sequence of coroutines to be executed in parallel. """ async def apply_split(node: Node): nodes, relationships = await self.split(node) kg.nodes.extend(nodes) kg.relationships.extend(relationships) filtered = self.filter(kg) plan = [apply_split(node) for node in filtered.nodes] logger.debug( "Created %d coroutines for %s", len(plan), self.__class__.__name__, ) return plan ``` ## Parallel ```python Parallel(*transformations: Union[BaseGraphTransformation, 'Parallel']) ``` Collection of transformations to be applied in parallel. Examples: ```pycon >>> Parallel(HeadlinesExtractor(), SummaryExtractor()) ``` Source code in `src/ragas/testset/transforms/engine.py` ```python def __init__(self, *transformations: t.Union[BaseGraphTransformation, "Parallel"]): self.transformations = list(transformations) ``` ## EmbeddingExtractor ```python EmbeddingExtractor(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), property_name: str = 'embedding', embed_property_name: str = 'page_content', embedding_model: Union[BaseRagasEmbeddings, BaseRagasEmbedding] = embedding_factory()) ``` Bases: `Extractor` A class for extracting embeddings from nodes in a knowledge graph. Attributes: | Name | Type | Description | | --------------------- | ------------------------------------------- | ----------------------------------------------------- | | `property_name` | `str` | The name of the property to store the embedding | | `embed_property_name` | `str` | The name of the property containing the text to embed | | `embedding_model` | `BaseRagasEmbeddings or BaseRagasEmbedding` | The embedding model used for generating embeddings | ### extract ```python extract(node: Node) -> Tuple[str, Any] ``` Extracts the embedding for a given node. Raises: | Type | Description | | ------------ | ----------------------------------------------- | | `ValueError` | If the property to be embedded is not a string. | Source code in `src/ragas/testset/transforms/extractors/embeddings.py` ```python async def extract(self, node: Node) -> t.Tuple[str, t.Any]: """ Extracts the embedding for a given node. Raises ------ ValueError If the property to be embedded is not a string. """ text = node.get_property(self.embed_property_name) if not isinstance(text, str): raise ValueError( f"node.property('{self.embed_property_name}') must be a string, found '{type(text)}'" ) # Handle both modern (BaseRagasEmbedding) and legacy (BaseRagasEmbeddings) interfaces if hasattr(self.embedding_model, "aembed_text"): # Modern interface (BaseRagasEmbedding) # Check if the client supports async operations by checking if is_async exists and is True if hasattr(self.embedding_model, "is_async") and getattr( self.embedding_model, "is_async", False ): embedding = await self.embedding_model.aembed_text(text) # type: ignore[attr-defined] else: # For sync clients, use the sync method wrapped in thread executor to avoid blocking warnings.warn( f"Using sync embedding model {self.embedding_model.__class__.__name__} " f"in async context. This may impact performance. " f"Consider using an async-compatible embedding model for better performance.", UserWarning, stacklevel=2, ) embedding = await run_sync_in_async( self.embedding_model.embed_text, text ) # type: ignore[attr-defined] else: # Legacy interface (BaseRagasEmbeddings) embedding = await self.embedding_model.embed_text(text) # type: ignore[misc] return self.property_name, embedding ``` ## HeadlinesExtractor ```python HeadlinesExtractor(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), llm: Union[BaseRagasLLM, InstructorBaseRagasLLM] = _default_llm_factory(), merge_if_possible: bool = True, max_token_limit: int = 32000, tokenizer: BaseTokenizer = (lambda: DEFAULT_TOKENIZER)(), property_name: str = 'headlines', prompt: HeadlinesExtractorPrompt = HeadlinesExtractorPrompt(), max_num: int = 5) ``` Bases: `LLMBasedExtractor` Extracts the headlines from the given text. Attributes: | Name | Type | Description | | --------------- | -------------------------- | ------------------------------------ | | `property_name` | `str` | The name of the property to extract. | | `prompt` | `HeadlinesExtractorPrompt` | The prompt used for extraction. | ## KeyphrasesExtractor ```python KeyphrasesExtractor(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), llm: Union[BaseRagasLLM, InstructorBaseRagasLLM] = _default_llm_factory(), merge_if_possible: bool = True, max_token_limit: int = 32000, tokenizer: BaseTokenizer = (lambda: DEFAULT_TOKENIZER)(), property_name: str = 'keyphrases', prompt: KeyphrasesExtractorPrompt = KeyphrasesExtractorPrompt(), max_num: int = 5) ``` Bases: `LLMBasedExtractor` Extracts top keyphrases from the given text. Attributes: | Name | Type | Description | | --------------- | --------------------------- | ------------------------------------ | | `property_name` | `str` | The name of the property to extract. | | `prompt` | `KeyphrasesExtractorPrompt` | The prompt used for extraction. | ## SummaryExtractor ```python SummaryExtractor(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), llm: Union[BaseRagasLLM, InstructorBaseRagasLLM] = _default_llm_factory(), merge_if_possible: bool = True, max_token_limit: int = 32000, tokenizer: BaseTokenizer = (lambda: DEFAULT_TOKENIZER)(), property_name: str = 'summary', prompt: SummaryExtractorPrompt = SummaryExtractorPrompt()) ``` Bases: `LLMBasedExtractor` Extracts a summary from the given text. Attributes: | Name | Type | Description | | --------------- | ------------------------ | ------------------------------------ | | `property_name` | `str` | The name of the property to extract. | | `prompt` | `SummaryExtractorPrompt` | The prompt used for extraction. | ## TitleExtractor ```python TitleExtractor(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), llm: Union[BaseRagasLLM, InstructorBaseRagasLLM] = _default_llm_factory(), merge_if_possible: bool = True, max_token_limit: int = 32000, tokenizer: BaseTokenizer = (lambda: DEFAULT_TOKENIZER)(), property_name: str = 'title', prompt: TitleExtractorPrompt = TitleExtractorPrompt()) ``` Bases: `LLMBasedExtractor` Extracts the title from the given text. Attributes: | Name | Type | Description | | --------------- | ---------------------- | ------------------------------------ | | `property_name` | `str` | The name of the property to extract. | | `prompt` | `TitleExtractorPrompt` | The prompt used for extraction. | ## CustomNodeFilter ```python CustomNodeFilter(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), llm: Union[BaseRagasLLM, InstructorBaseRagasLLM] = _default_llm_factory(), scoring_prompt: PydanticPrompt = QuestionPotentialPrompt(), min_score: int = 2, rubrics: Dict[str, str] = (lambda: DEFAULT_RUBRICS)()) ``` Bases: `LLMBasedNodeFilter` returns True if the score is less than min_score ## CosineSimilarityBuilder ```python CosineSimilarityBuilder(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), property_name: str = 'embedding', new_property_name: str = 'cosine_similarity', threshold: float = 0.9, block_size: int = 1024) ``` Bases: `RelationshipBuilder` ### generate_execution_plan ```python generate_execution_plan(kg: KnowledgeGraph) -> List[Coroutine] ``` Generates a coroutine task for finding similar embedding pairs, which can be scheduled/executed by an Executor. Source code in `src/ragas/testset/transforms/relationship_builders/cosine.py` ```python def generate_execution_plan(self, kg: KnowledgeGraph) -> t.List[t.Coroutine]: """ Generates a coroutine task for finding similar embedding pairs, which can be scheduled/executed by an Executor. """ filtered_kg = self.filter(kg) embeddings = [] for node in filtered_kg.nodes: embedding = node.get_property(self.property_name) if embedding is None: raise ValueError(f"Node {node.id} has no {self.property_name}") embeddings.append(embedding) self._validate_embedding_shapes(embeddings) async def find_and_add_relationships(): similar_pairs = self._find_similar_embedding_pairs( np.array(embeddings), self.threshold ) for i, j, similarity_float in similar_pairs: rel = Relationship( source=filtered_kg.nodes[i], target=filtered_kg.nodes[j], type=self.new_property_name, properties={self.new_property_name: similarity_float}, bidirectional=True, ) kg.relationships.append(rel) return [find_and_add_relationships()] ``` ## SummaryCosineSimilarityBuilder ```python SummaryCosineSimilarityBuilder(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), property_name: str = 'summary_embedding', new_property_name: str = 'summary_cosine_similarity', threshold: float = 0.1, block_size: int = 1024) ``` Bases: `CosineSimilarityBuilder` ## JaccardSimilarityBuilder ```python JaccardSimilarityBuilder(name: str = '', filter_nodes: Callable[[Node], bool] = (lambda: default_filter)(), property_name: str = 'entities', key_name: Optional[str] = None, new_property_name: str = 'jaccard_similarity', threshold: float = 0.5) ``` Bases: `RelationshipBuilder` ### generate_execution_plan ```python generate_execution_plan(kg: KnowledgeGraph) -> List[Coroutine] ``` Generates a coroutine task for finding similar pairs, which can be scheduled/executed by an Executor. Source code in `src/ragas/testset/transforms/relationship_builders/traditional.py` ```python def generate_execution_plan(self, kg: KnowledgeGraph) -> t.List[t.Coroutine]: """ Generates a coroutine task for finding similar pairs, which can be scheduled/executed by an Executor. """ async def find_and_add_relationships(): similar_pairs = self._find_similar_embedding_pairs(kg) for i, j, similarity_float in similar_pairs: rel = Relationship( source=kg.nodes[i], target=kg.nodes[j], type=self.new_property_name, properties={self.new_property_name: similarity_float}, bidirectional=True, ) kg.relationships.append(rel) return [find_and_add_relationships()] ``` ## default_transforms ```python default_transforms(documents: List[Document], llm: Union[BaseRagasLLM, 'InstructorBaseRagasLLM'], embedding_model: BaseRagasEmbeddings) -> 'Transforms' ``` Creates and returns a default set of transforms for processing a knowledge graph. This function defines a series of transformation steps to be applied to a knowledge graph, including extracting summaries, keyphrases, titles, headlines, and embeddings, as well as building similarity relationships between nodes. Returns: | Type | Description | | ------------ | -------------------------------------------------------------------- | | `Transforms` | A list of transformation steps to be applied to the knowledge graph. | Source code in `src/ragas/testset/transforms/default.py` ```python def default_transforms( documents: t.List[LCDocument], llm: t.Union[BaseRagasLLM, "InstructorBaseRagasLLM"], embedding_model: BaseRagasEmbeddings, ) -> "Transforms": """ Creates and returns a default set of transforms for processing a knowledge graph. This function defines a series of transformation steps to be applied to a knowledge graph, including extracting summaries, keyphrases, titles, headlines, and embeddings, as well as building similarity relationships between nodes. Returns ------- Transforms A list of transformation steps to be applied to the knowledge graph. """ def count_doc_length_bins(documents, bin_ranges): data = [num_tokens_from_string(doc.page_content) for doc in documents] bins = {f"{start}-{end}": 0 for start, end in bin_ranges} for num in data: for start, end in bin_ranges: if start <= num <= end: bins[f"{start}-{end}"] += 1 break # Move to the next number once it’s placed in a bin return bins def filter_doc_with_num_tokens(node, min_num_tokens=500): return ( node.type == NodeType.DOCUMENT and num_tokens_from_string(node.properties["page_content"]) > min_num_tokens ) def filter_docs(node): return node.type == NodeType.DOCUMENT def filter_chunks(node): return node.type == NodeType.CHUNK bin_ranges = [(0, 100), (101, 500), (501, float("inf"))] result = count_doc_length_bins(documents, bin_ranges) result = {k: v / len(documents) for k, v in result.items()} transforms = [] if result["501-inf"] >= 0.25: headline_extractor = HeadlinesExtractor( llm=llm, filter_nodes=lambda node: filter_doc_with_num_tokens(node) ) splitter = HeadlineSplitter(min_tokens=500) summary_extractor = SummaryExtractor( llm=llm, filter_nodes=lambda node: filter_doc_with_num_tokens(node) ) theme_extractor = ThemesExtractor( llm=llm, filter_nodes=lambda node: filter_chunks(node) ) ner_extractor = NERExtractor( llm=llm, filter_nodes=lambda node: filter_chunks(node) ) summary_emb_extractor = EmbeddingExtractor( embedding_model=embedding_model, property_name="summary_embedding", embed_property_name="summary", filter_nodes=lambda node: filter_doc_with_num_tokens(node), ) cosine_sim_builder = CosineSimilarityBuilder( property_name="summary_embedding", new_property_name="summary_similarity", threshold=0.7, filter_nodes=lambda node: filter_doc_with_num_tokens(node), ) ner_overlap_sim = OverlapScoreBuilder( threshold=0.01, filter_nodes=lambda node: filter_chunks(node) ) node_filter = CustomNodeFilter( llm=llm, filter_nodes=lambda node: filter_chunks(node) ) transforms = [ headline_extractor, splitter, summary_extractor, node_filter, Parallel(summary_emb_extractor, theme_extractor, ner_extractor), Parallel(cosine_sim_builder, ner_overlap_sim), ] elif result["101-500"] >= 0.25: summary_extractor = SummaryExtractor( llm=llm, filter_nodes=lambda node: filter_doc_with_num_tokens(node, 100) ) summary_emb_extractor = EmbeddingExtractor( embedding_model=embedding_model, property_name="summary_embedding", embed_property_name="summary", filter_nodes=lambda node: filter_doc_with_num_tokens(node, 100), ) cosine_sim_builder = CosineSimilarityBuilder( property_name="summary_embedding", new_property_name="summary_similarity", threshold=0.5, filter_nodes=lambda node: filter_doc_with_num_tokens(node, 100), ) ner_extractor = NERExtractor(llm=llm) ner_overlap_sim = OverlapScoreBuilder(threshold=0.01) theme_extractor = ThemesExtractor( llm=llm, filter_nodes=lambda node: filter_docs(node) ) node_filter = CustomNodeFilter(llm=llm) transforms = [ summary_extractor, node_filter, Parallel(summary_emb_extractor, theme_extractor, ner_extractor), Parallel(cosine_sim_builder, ner_overlap_sim), ] else: raise ValueError( "Documents appears to be too short (ie 100 tokens or less). Please provide longer documents." ) return transforms ``` ## default_transforms_for_prechunked ```python default_transforms_for_prechunked(llm: Union[BaseRagasLLM, 'InstructorBaseRagasLLM'], embedding_model: BaseRagasEmbeddings) -> 'Transforms' ``` Creates and returns a default set of transforms for processing a knowledge graph containing pre-chunked documents. This ignores the splitting step and directly applies extractors and relationship builders to the chunks. Source code in `src/ragas/testset/transforms/default.py` ```python def default_transforms_for_prechunked( llm: t.Union[BaseRagasLLM, "InstructorBaseRagasLLM"], embedding_model: BaseRagasEmbeddings, ) -> "Transforms": """ Creates and returns a default set of transforms for processing a knowledge graph containing pre-chunked documents. This ignores the splitting step and directly applies extractors and relationship builders to the chunks. """ def filter_chunks(node): return node.type == NodeType.CHUNK summary_extractor = SummaryExtractor(llm=llm, filter_nodes=filter_chunks) summary_emb_extractor = EmbeddingExtractor( embedding_model=embedding_model, property_name="summary_embedding", embed_property_name="summary", filter_nodes=filter_chunks, ) theme_extractor = ThemesExtractor(llm=llm, filter_nodes=filter_chunks) ner_extractor = NERExtractor(llm=llm, filter_nodes=filter_chunks) cosine_sim_builder = CosineSimilarityBuilder( property_name="summary_embedding", new_property_name="summary_similarity", threshold=0.7, filter_nodes=filter_chunks, ) ner_overlap_sim = OverlapScoreBuilder(threshold=0.01, filter_nodes=filter_chunks) node_filter = CustomNodeFilter(llm=llm, filter_nodes=filter_chunks) return [ summary_extractor, node_filter, Parallel(summary_emb_extractor, theme_extractor, ner_extractor), Parallel(cosine_sim_builder, ner_overlap_sim), ] ``` ## apply_transforms ```python apply_transforms(kg: KnowledgeGraph, transforms: Transforms, run_config: RunConfig = RunConfig(), callbacks: Optional[Callbacks] = None) ``` Recursively apply transformations to a knowledge graph in place. Source code in `src/ragas/testset/transforms/engine.py` ```python def apply_transforms( kg: KnowledgeGraph, transforms: Transforms, run_config: RunConfig = RunConfig(), callbacks: t.Optional[Callbacks] = None, ): """ Recursively apply transformations to a knowledge graph in place. """ # apply nest_asyncio to fix the event loop issue in jupyter apply_nest_asyncio() max_workers = getattr(run_config, "max_workers", -1) if isinstance(transforms, t.Sequence): for transform in transforms: apply_transforms(kg, transform, run_config, callbacks) elif isinstance(transforms, Parallel): apply_transforms(kg, transforms.transformations, run_config, callbacks) elif isinstance(transforms, BaseGraphTransformation): logger.debug( f"Generating execution plan for transformation {transforms.__class__.__name__}" ) coros = transforms.generate_execution_plan(kg) desc = get_desc(transforms) run_async_tasks( coros, batch_size=None, show_progress=True, progress_bar_desc=desc, max_workers=max_workers, ) else: raise ValueError( f"Invalid transforms type: {type(transforms)}. Expects a sequence of BaseGraphTransformations or a Parallel instance." ) logger.debug("All transformations applied successfully.") ``` ## rollback_transforms ```python rollback_transforms(kg: KnowledgeGraph, transforms: Transforms) ``` Rollback a sequence of transformations from a knowledge graph. Note This is not yet implemented. Please open an issue if you need this feature. Source code in `src/ragas/testset/transforms/engine.py` ```python def rollback_transforms(kg: KnowledgeGraph, transforms: Transforms): """ Rollback a sequence of transformations from a knowledge graph. Note ---- This is not yet implemented. Please open an issue if you need this feature. """ # this will allow you to roll back the transformations raise NotImplementedError ```