Evaluating LlamaIndex

LlamaIndex is a data framework for LLM applications to ingest, structure, and access private or domain-specific data. Makes it super easy to connect LLMs with your own data. But in order to figure out the best configuration for llamaIndex and your data you need a object measure of the performance. This is where ragas comes in. Ragas will help you evaluate your QueryEngine and gives you the confidence to tweak the configuration to get hightest score.

This guide assumes you have familarity with the LlamaIndex framework.

# attach to the same event-loop
import nest_asyncio

nest_asyncio.apply()

Building the VectorStoreIndex and QueryEngine

To start lets build an VectorStoreIndex over the New York Citie’s wikipedia page as an example and use ragas to evaluate it.

from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext

import pandas as pd

load the data, build the VectorStoreIndex and create the QueryEngine.

documents = SimpleDirectoryReader("./nyc_wikipedia/").load_data()
vector_index = VectorStoreIndex.from_documents(
    documents, service_context=ServiceContext.from_defaults(chunk_size=512)
)

query_engine = vector_index.as_query_engine()

Lets try an sample question to see if it is working

response_vector = query_engine.query("How did New York City get its name?")

print(response_vector)

New York City was named in honor of the Duke of York, who would become King James II of England. In 1664, King Charles II appointed the Duke as proprietor of the former territory of New Netherland, including the city of New Amsterdam, when England seized it from Dutch control. The city was then renamed New York in his honor.

Evaluating with Ragas

Now that we have a QueryEngine for the VectorStoreIndex we can use the llama_index integration Ragas has to evaluate it.

In order to run an evaluation with Ragas and LlamaIndex you need 3 things

  1. LlamaIndex QueryEngine: what we will be evaluating

  2. Metrics: Ragas defines a set of metrics that can measure different aspects of the QueryEngine. The available metrics and their meaning can be found here

  3. Questions: A list of questions that ragas will test the QueryEngine against.

first lets generate the questions. Ideally you should use that you see in production so that the distribution of question with which we evaluate matches the distribution of questions seen in production. This ensures that the scores reflect the performance seen in production but to start off we’ll be using a few example question.

eval_questions = [
    "What is the population of New York City as of 2020?",
    "Which borough of New York City has the highest population?",
    "What is the economic significance of New York City?",
    "How did New York City get its name?",
    "What is the significance of the Statue of Liberty in New York City?",
]

eval_answers = [
    "8,804,000",  # incorrect answer
    "Queens",  # incorrect answer
    "New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.",
    "New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.",
    "The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.",
]

eval_answers = [[a] for a in eval_answers]

Now lets import the metrics we will be using to evaluate

from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from ragas.metrics.critique import harmfulness

metrics = [
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
    harmfulness,
]

Finally lets run the evaluation

from ragas.llama_index import evaluate

result = evaluate(query_engine, metrics, eval_questions, eval_answers)

evaluating with [faithfulness]

100%|█████████████████████████████████████████████████████████████| 1/1 [00:51<00:00, 51.40s/it]

evaluating with [answer_relevancy]

100%|█████████████████████████████████████████████████████████████| 1/1 [00:09<00:00,  9.64s/it]

evaluating with [context_precision]

100%|█████████████████████████████████████████████████████████████| 1/1 [00:41<00:00, 41.21s/it]

evaluating with [context_recall]

100%|█████████████████████████████████████████████████████████████| 1/1 [00:34<00:00, 34.97s/it]

evaluating with [harmfulness]

100%|█████████████████████████████████████████████████████████████| 1/1 [00:13<00:00, 13.98s/it]

# final scores
print(result)

{faithfulness': 0.7000, 'answer_relevancy': 0.9550, 'context_precision': 0.2335, 'context_recall': 0.9800, 'harmfulness': 0.0000}

You can convert into a pandas dataframe to run more analysis on it.

result.to_pandas()
question contexts answer ground_truth faithfulness answer_relevancy context_precision context_recall harmfulness
0 What is the population of New York City as of ... [Aeromedical Staging Squadron, and a military ... \nThe population of New York City as of 2020 i... [8,804,000] 1.0 0.999999 0.320000 1.0 0
1 Which borough of New York City has the highest... [co-extensive with New York County, the boroug... \nThe borough of Manhattan has the highest pop... [Queens] 0.0 0.998524 0.038462 0.9 0
2 What is the economic significance of New York ... [health care and life sciences, medical techno... \nNew York City is a major global economic cen... [New York City's economic significance is vast... 1.0 0.904272 0.423077 1.0 0
3 How did New York City get its name? [a US$1 billion research and education center ... \nNew York City was named in honor of the Duke... [New York City got its name when it came under... 1.0 0.929719 0.333333 1.0 0
4 What is the significance of the Statue of Libe... [(stylized I ❤ NY) is both a logo and a song t... \nThe Statue of Liberty is a symbol of the Uni... [The Statue of Liberty in New York City holds ... 0.5 0.942676 0.052632 1.0 0