Evaluating LlamaIndexΒΆ
LlamaIndex is a data framework for LLM applications to ingest, structure, and access private or domain-specific data. Makes it super easy to connect LLMs with your own data. But in order to figure out the best configuration for llamaIndex and your data you need a object measure of the performance. This is where ragas comes in. Ragas will help you evaluate your QueryEngine
and gives you the confidence to tweak the configuration to get hightest score.
This guide assumes you have familarity with the LlamaIndex framework.
# attach to the same event-loop
import nest_asyncio
nest_asyncio.apply()
Building the VectorStoreIndex
and QueryEngine
ΒΆ
To start lets build an VectorStoreIndex
over the New York Citieβs wikipedia page as an example and use ragas to evaluate it.
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
import pandas as pd
load the data, build the VectorStoreIndex
and create the QueryEngine
.
documents = SimpleDirectoryReader("./nyc_wikipedia/").load_data()
vector_index = VectorStoreIndex.from_documents(
documents, service_context=ServiceContext.from_defaults(chunk_size=512)
)
query_engine = vector_index.as_query_engine()
Lets try an sample question to see if it is working
response_vector = query_engine.query("How did New York City get its name?")
print(response_vector)
New York City was named in honor of the Duke of York, who would become King James II of England. In 1664, King Charles II appointed the Duke as proprietor of the former territory of New Netherland, including the city of New Amsterdam, when England seized it from Dutch control. The city was then renamed New York in his honor.
Evaluating with RagasΒΆ
Now that we have a QueryEngine
for the VectorStoreIndex
we can use the llama_index integration Ragas has to evaluate it.
In order to run an evaluation with Ragas and LlamaIndex you need 3 things
LlamaIndex
QueryEngine
: what we will be evaluatingMetrics: Ragas defines a set of metrics that can measure different aspects of the
QueryEngine
. The available metrics and their meaning can be found hereQuestions: A list of questions that ragas will test the
QueryEngine
against.
first lets generate the questions. Ideally you should use that you see in production so that the distribution of question with which we evaluate matches the distribution of questions seen in production. This ensures that the scores reflect the performance seen in production but to start off weβll be using a few example question.
eval_questions = [
"What is the population of New York City as of 2020?",
"Which borough of New York City has the highest population?",
"What is the economic significance of New York City?",
"How did New York City get its name?",
"What is the significance of the Statue of Liberty in New York City?",
]
eval_answers = [
"8,804,000", # incorrect answer
"Queens", # incorrect answer
"New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.",
"New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.",
"The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.",
]
eval_answers = [[a] for a in eval_answers]
Now lets import the metrics we will be using to evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_precision,
context_recall,
)
from ragas.metrics.critique import harmfulness
metrics = [
faithfulness,
answer_relevancy,
context_precision,
context_recall,
harmfulness,
]
Finally lets run the evaluation
from ragas.llama_index import evaluate
result = evaluate(query_engine, metrics, eval_questions, eval_answers)
evaluating with [faithfulness]
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1/1 [00:51<00:00, 51.40s/it]
evaluating with [answer_relevancy]
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1/1 [00:09<00:00, 9.64s/it]
evaluating with [context_precision]
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1/1 [00:41<00:00, 41.21s/it]
evaluating with [context_recall]
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1/1 [00:34<00:00, 34.97s/it]
evaluating with [harmfulness]
100%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 1/1 [00:13<00:00, 13.98s/it]
# final scores
print(result)
{faithfulness': 0.7000, 'answer_relevancy': 0.9550, 'context_precision': 0.2335, 'context_recall': 0.9800, 'harmfulness': 0.0000}
You can convert into a pandas dataframe to run more analysis on it.
result.to_pandas()
question | contexts | answer | ground_truth | faithfulness | answer_relevancy | context_precision | context_recall | harmfulness | |
---|---|---|---|---|---|---|---|---|---|
0 | What is the population of New York City as of ... | [Aeromedical Staging Squadron, and a military ... | \nThe population of New York City as of 2020 i... | [8,804,000] | 1.0 | 0.999999 | 0.320000 | 1.0 | 0 |
1 | Which borough of New York City has the highest... | [co-extensive with New York County, the boroug... | \nThe borough of Manhattan has the highest pop... | [Queens] | 0.0 | 0.998524 | 0.038462 | 0.9 | 0 |
2 | What is the economic significance of New York ... | [health care and life sciences, medical techno... | \nNew York City is a major global economic cen... | [New York City's economic significance is vast... | 1.0 | 0.904272 | 0.423077 | 1.0 | 0 |
3 | How did New York City get its name? | [a US$1 billion research and education center ... | \nNew York City was named in honor of the Duke... | [New York City got its name when it came under... | 1.0 | 0.929719 | 0.333333 | 1.0 | 0 |
4 | What is the significance of the Statue of Libe... | [(stylized I β€ NY) is both a logo and a song t... | \nThe Statue of Liberty is a symbol of the Uni... | [The Statue of Liberty in New York City holds ... | 0.5 | 0.942676 | 0.052632 | 1.0 | 0 |