Evaluating Langchain QA Chains¶

LangChain is a framework for developing applications powered by language models. It can also be used to create RAG systems (or QA systems as they are reffered to in langchain). If you want to know more about creating RAG systems with langchain you can check the docs.

With this integration you can easily evaluate your QA chains with the metrics offered in ragas

# attach to the existing event loop when using jupyter notebooks
import nest_asyncio

nest_asyncio.apply()

First lets load the dataset. We are going to build a generic QA system over the NYC wikipedia page. Load the dataset and create the VectorstoreIndex and the RetrievalQA from it.

from langchain.document_loaders import TextLoader
from langchain.indexes import VectorstoreIndexCreator
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

loader = TextLoader("./nyc_wikipedia/nyc_text.txt")
index = VectorstoreIndexCreator().from_loaders([loader])


llm = ChatOpenAI(temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm,
    retriever=index.vectorstore.as_retriever(),
    return_source_documents=True,
)

# testing it out

question = "How did New York City get its name?"
result = qa_chain({"query": question})
result["result"]

'New York City was originally named New Amsterdam by Dutch colonists in 1626. However, in 1664, the city came under British control and was renamed New York after King Charles II of England granted the lands to his brother, the Duke of York. The city has been continuously named New York since November 1674.'

Now in order to evaluate the qa system we generated a few relevant questions. We’ve generated a few question for you but feel free to add any you want.

eval_questions = [
    "What is the population of New York City as of 2020?",
    "Which borough of New York City has the highest population?",
    "What is the economic significance of New York City?",
    "How did New York City get its name?",
    "What is the significance of the Statue of Liberty in New York City?",
]

eval_answers = [
    "8,804,190",
    "Brooklyn",
    "New York City's economic significance is vast, as it serves as the global financial capital, housing Wall Street and major financial institutions. Its diverse economy spans technology, media, healthcare, education, and more, making it resilient to economic fluctuations. NYC is a hub for international business, attracting global companies, and boasts a large, skilled labor force. Its real estate market, tourism, cultural industries, and educational institutions further fuel its economic prowess. The city's transportation network and global influence amplify its impact on the world stage, solidifying its status as a vital economic player and cultural epicenter.",
    "New York City got its name when it came under British control in 1664. King Charles II of England granted the lands to his brother, the Duke of York, who named the city New York in his own honor.",
    "The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.",
]

examples = [
    {"query": q, "ground_truths": [eval_answers[i]]}
    for i, q in enumerate(eval_questions)
]

Introducing `RagasEvaluatorChain`¶

RagasEvaluatorChain creates a wrapper around the metrics ragas provides (documented here), making it easier to run these evaluation with langchain and langsmith.

The evaluator chain has the following APIs

__call__(): call the RagasEvaluatorChain directly on the result of a QA chain.
evaluate(): evaluate on a list of examples (with the input queries) and predictions (outputs from the QA chain).
evaluate_run(): method implemented that is called by langsmith evaluators to evaluate langsmith datasets.

lets see each of them in action to learn more.

result = qa_chain({"query": eval_questions[1]})
result["result"]

'The borough of Brooklyn (Kings County) has the highest population in New York City.'

result = qa_chain(examples[4])
result["result"]

'The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants as they arrived in the U.S. in the late 19th and early 20th centuries, representing hope and opportunity. It has become an iconic landmark and a representation of freedom and cultural diversity.'

from ragas.langchain.evalchain import RagasEvaluatorChain
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

# create evaluation chains
faithfulness_chain = RagasEvaluatorChain(metric=faithfulness)
answer_rel_chain = RagasEvaluatorChain(metric=answer_relevancy)
context_rel_chain = RagasEvaluatorChain(metric=context_precision)
context_recall_chain = RagasEvaluatorChain(metric=context_recall)

__call__()

Directly run the evaluation chain with the results from the QA chain. Do note that metrics like context_precision and faithfulness require the source_documents to be present.

# Recheck the result that we are going to validate.
result

{'query': 'What is the significance of the Statue of Liberty in New York City?',
 'ground_truths': ['The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants who arrived in the U.S. by ship in the late 19th and early 20th centuries, representing hope and freedom for those seeking a better life. It has since become an iconic landmark and a global symbol of cultural diversity and freedom.'],
 'result': 'The Statue of Liberty in New York City holds great significance as a symbol of the United States and its ideals of liberty and peace. It greeted millions of immigrants as they arrived in the U.S. in the late 19th and early 20th centuries, representing hope and opportunity. It has become an iconic landmark and a representation of freedom and cultural diversity.',
 'source_documents': [Document(page_content='from 1785 until 1790, and has been the largest U.S. city since 1790. The Statue of Liberty greeted millions of immigrants as they came to the U.S. by ship in the late 19th and early 20th centuries, and is a symbol of the U.S. and its ideals of liberty and peace. In the 21st century, New York City has emerged as a global node of creativity, entrepreneurship, and as a symbol of freedom and cultural diversity. The New York Times has won the most Pulitzer Prizes for journalism and remains the U.S. media\'s "newspaper of record". In 2019, New York City was voted the greatest city in the world in a survey of over 30,000 people from 48 cities worldwide, citing its cultural diversity.Many districts and monuments in New York City are major landmarks, including three of the world\'s ten most visited tourist attractions in 2013. A record 66.6 million tourists visited New York City in 2019. Times Square is the brightly illuminated hub of the Broadway Theater District, one of the world\'s busiest', metadata={'source': './nyc_wikipedia/nyc_text.txt'}),
  Document(page_content="The Statue of Liberty National Monument and Ellis Island Immigration Museum are managed by the National Park Service and are in both New York and New Jersey. They are joined in the harbor by Governors Island National Monument. Historic sites under federal management on Manhattan Island include Stonewall National Monument; Castle Clinton National Monument; Federal Hall National Memorial; Theodore Roosevelt Birthplace National Historic Site; General Grant National Memorial (Grant's Tomb); African Burial Ground National Monument; and Hamilton Grange National Memorial. Hundreds of properties are listed on the National Register of Historic Places or as a National Historic Landmark.", metadata={'source': './nyc_wikipedia/nyc_text.txt'}),
  Document(page_content="The majority of the most high-profile tourist destinations to the city are situated in Manhattan. These include Times Square; Broadway theater productions; the Empire State Building; the Statue of Liberty; Ellis Island; the United Nations headquarters; the World Trade Center (including the National September 11 Memorial & Museum and One World Trade Center); the art museums along Museum Mile; green spaces such as Central Park, Washington Square Park, the High Line, and the medieval gardens of The Cloisters; the Stonewall Inn; Rockefeller Center; ethnic enclaves including the Manhattan Chinatown, Koreatown, Curry Hill, Harlem, Spanish Harlem, Little Italy, and Little Australia; luxury shopping along Fifth and Madison Avenues; and events such as the Halloween Parade in Greenwich Village; the Brooklyn Bridge (shared with Brooklyn); the Macy's Thanksgiving Day Parade; the lighting of the Rockefeller Center Christmas Tree; the St. Patrick's Day Parade; seasonal activities such as ice", metadata={'source': './nyc_wikipedia/nyc_text.txt'}),
  Document(page_content="Tourism is a vital industry for New York City, and NYC & Company represents the city's official bureau of tourism. New York has witnessed a growing combined volume of international and domestic tourists, reflecting over 60 million visitors to the city per year, the world's busiest tourist destination. Approximately 12 million visitors to New York City have been from outside the United States, with the highest numbers from the United Kingdom, Canada, Brazil, and China. Multiple sources have called New York the most photographed city in the world.I Love New York (stylized I ❤ NY) is both a logo and a song that are the basis of an advertising campaign and have been used since 1977 to promote tourism in New York City, and later to promote New York State as well. The trademarked logo, owned by New York State Empire State Development, appears in souvenir shops and brochures throughout the city and state, some licensed, many not. The song is the state song of New York.", metadata={'source': './nyc_wikipedia/nyc_text.txt'})]}

Faithfulness

eval_result = faithfulness_chain(result)
eval_result["faithfulness_score"]

0.8

High faithfulness_score means that there are exact consistency between the source documents and the answer.

You can check lower faithfulness scores by changing the result (answer from LLM) or source_documents to something else.

fake_result = result.copy()
fake_result["result"] = "we are the champions"
eval_result = faithfulness_chain(fake_result)
eval_result["faithfulness_score"]

0.0

Context Recall

eval_result = context_recall_chain(result)
eval_result["context_recall_score"]

1.0

High context_recall_score means that the ground truth is present in the source documents.

You can check lower context recall scores by changing the source_documents to something else.

from langchain.schema import Document

fake_result = result.copy()
fake_result["source_documents"] = [Document(page_content="I love christmas")]
eval_result = context_recall_chain(fake_result)
eval_result["context_recall_score"]

0.0

evaluate()

Evaluate a list of inputs/queries and the outputs/predictions from the QA chain.

# run the queries as a batch for efficiency
predictions = qa_chain.batch(examples)

# evaluate
print("evaluating...")
r = faithfulness_chain.evaluate(examples, predictions)
r

evaluating...

100%|█████████████████████████████████████████████████████████████| 1/1 [00:49<00:00, 49.13s/it]

[{'faithfulness_score': 1.0},
 {'faithfulness_score': 0.5},
 {'faithfulness_score': 1.0},
 {'faithfulness_score': 1.0},
 {'faithfulness_score': 0.8}]

# evaluate context recall
print("evaluating...")
r = context_recall_chain.evaluate(examples, predictions)
r

evaluating...

100%|█████████████████████████████████████████████████████████████| 1/1 [00:36<00:00, 36.18s/it]

[{'context_recall_score': 1.0},
 {'context_recall_score': 0.1},
 {'context_recall_score': 1.0},
 {'context_recall_score': 1.0},
 {'context_recall_score': 1.0}]

Evaluate with langsmith¶

Langsmith is a platform that helps to debug, test, evaluate and monitor chains and agents built on any LLM framework. It also seamlessly integrates with LangChain.

Langsmith also has a tools to build a testing dataset and run evaluations against them and with RagasEvaluatorChain you can use the ragas metrics for running langsmith evaluations as well. To know more about langsmith evaluations checkout the quickstart.

Lets start of creating the dataset with the NYC questions listed in eval_questions. Create a new langsmith dataset and upload the questions.

# dataset creation

from langsmith import Client
from langsmith.utils import LangSmithError

client = Client()
dataset_name = "NYC test"

try:
    # check if dataset exists
    dataset = client.read_dataset(dataset_name=dataset_name)
    print("using existing dataset: ", dataset.name)
except LangSmithError:
    # if not create a new one with the generated query examples
    dataset = client.create_dataset(
        dataset_name=dataset_name, description="NYC test dataset"
    )
    for e in examples:
        client.create_example(
            inputs={"query": e["query"]},
            outputs={"ground_truths": e["ground_truths"]},
            dataset_id=dataset.id,
        )

    print("Created a new dataset: ", dataset.name)

Created a new dataset:  NYC test

As you can see the questions have been uploaded. Now you can run your QA chain against this test dataset and compare the results in the langchain platform.

Before you call run_on_dataset you need a factory function which creates a new instance of the QA chain you want to test. This is so that the internal state is not reused when running against each example.

# factory function that return a new qa chain
def create_qa_chain(return_context=True):
    qa_chain = RetrievalQA.from_chain_type(
        llm,
        retriever=index.vectorstore.as_retriever(),
        return_source_documents=return_context,
    )
    return qa_chain

Now lets run the evaluation

from langchain.smith import RunEvalConfig, run_on_dataset

evaluation_config = RunEvalConfig(
    custom_evaluators=[
        faithfulness_chain,
        answer_rel_chain,
        context_rel_chain,
        context_recall_chain,
    ],
    prediction_key="result",
)

result = run_on_dataset(
    client,
    dataset_name,
    create_qa_chain,
    evaluation=evaluation_config,
    input_mapper=lambda x: x,
)

View the evaluation results for project '2023-09-26-16-02-59-RetrievalQA' at:
https://smith.langchain.com/projects/p/624cd93e-7310-46c5-9156-6dcb46e92c5b?eval=true

You can follow the link to open the result for the run in langsmith. Check out the scores for each example too

Now if you want to dive more into the reasons for the scores and how to improve them, click on any example and open the feedback tab. This will show you each scores.

You can also see the curresponding RagasEvaluatorChain trace too to figure out why ragas scored the way it did.

Evaluating LlamaIndex

Langsmith Integrations

Evaluating Langchain QA Chains¶

Introducing RagasEvaluatorChain¶

Evaluate with langsmith¶

Introducing `RagasEvaluatorChain`¶