Max Workers, Timeouts, Retries and more with RunConfig

The RunConfig allows you to pass in the run parameters to functions like evaluate() and TestsetGenerator.generate(). Depending on your LLM providers rate limits, SLAs and traffic, controlling these parameters can improve the speed and reliablility of Ragas runs.

Let’s explore some basics

Rate Limits

Ragas leverages parallelism with Async in python but the RunConfig has a field called max_workers which control the number of concurent requests allowed together. You adjust this to get the maximum concurency your provider allows

from ragas.metrics import faithfulness
from ragas import evaluate

# load the dataset
from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

# configure RunConfig
from ragas.run_config import RunConfig

_ = evaluate(
    dataset=amnesty_qa["eval"],
    metrics=[faithfulness],
    run_config=RunConfig(max_workers=64),  # increasing max_workers from default 16
)

Repo card metadata block was not found. Setting CardData to empty.

Now lets try with less concurrent workers

_ = evaluate(
    dataset=amnesty_qa["eval"],
    metrics=[faithfulness],
    run_config=RunConfig(max_workers=2),  # increasing max_workers from default 16
)

Similarly you can control adjust various parameters. Check the reference for more