Skip to content

Generate Synthetic Testset for RAG

Testset Generation for RAG

This simple guide will help you generate a testset for evaluating your RAG pipeline using your own documents.

Load Sample Documents

For the sake of this tutorial we will use sample documents from this repository. You can replace this with your own documents.

git clone https://huggingface.co/datasets/explodinggradients/Sample_Docs_Markdown

Load documents

Now we will load the documents from the sample dataset using DirectoryLoader, which is one of document loaders from langchain_community. You may also use any loaders from llama_index

from langchain_community.document_loaders import DirectoryLoader

path = "Sample_Docs_Markdown/"
loader = DirectoryLoader(path, glob="**/*.md")
docs = loader.load()

Choose your LLM

You may choose to use any LLM of your choice

This guide utilizes OpenAI for running some metrics, so ensure you have your OpenAI key ready and available in your environment.

import os
os.environ["OPENAI_API_KEY"] = "your-openai-key"
Wrapp the LLMs in LangchainLLMWrapper 
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))

First you have to set your AWS credentials and configurations

config = {
    "credentials_profile_name": "your-profile-name",  # E.g "default"
    "region_name": "your-region-name",  # E.g. "us-east-1"
    "llm": "your-llm-model-id",  # E.g "anthropic.claude-3-5-sonnet-20240620-v1:0"
    "embeddings": "your-embedding-model-id",  # E.g "amazon.titan-embed-text-v2:0"
    "temperature": 0.4,
}
define you LLMs 
from langchain_aws import ChatBedrockConverse
from ragas.llms import LangchainLLMWrapper
evaluator_llm = LangchainLLMWrapper(ChatBedrockConverse(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
    base_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    model=config["llm"],
    temperature=config["temperature"],
))

Generate Testset

Now we will run the test generation using the loaded documents and the LLM setup. If you have used llama_index to load documents, please use generate_with_llama_index_docs method instead.

from ragas.testset import TestsetGenerator

generator = TestsetGenerator(llm=generator_llm)
dataset = generator.generate_with_langchain_docs(docs, testset_size=10)

Export

You may now export and inspect the generated testset.

dataset.to_pandas()