Using different Embedding Models

Ragas allows users to change the default embedding model used in the evaluation task.

This guide will show you how to use different embedding models for evaluation in Ragas.

Evaluating with Azure Open AI Embeddings

Ragas uses open-ai embeddings by default. In this example we can use Azure Open AI Embeddings from langchain with the embedding model text-embedding-ada-002. We will be using gpt-35-turbo-16k from Azure OpenAI as the llm for evaluation and AnswerSimilarity as the metric

To start-off, we initialise the gpt-35-turbo-16k from Azure and create a chat_model using langchain

import os
from langchain_openai.chat_models import AzureChatOpenAI

os.environ["OPENAI_API_VERSION"] = "2023-05-15"
os.environ["OPENAI_API_KEY"] = "your-openai-key"

azure_model = AzureChatOpenAI(
    deployment_name="your-deployment-name",
    model="gpt-35-turbo-16k",
    azure_endpoint="https://your-endpoint.openai.azure.com/",
    openai_api_type="azure",
)

In order to use the Azure Open AI embedding, we have to instantiate an object of the AzureOpenAIEmbeddings class from Langchain.

from langchain_openai.embeddings import AzureOpenAIEmbeddings

azure_embeddings = AzureOpenAIEmbeddings(
    deployment="your-deployment-name",
    model="text-embedding-ada-002",
    azure_endpoint="https://your-endpoint.openai.azure.com/",
    openai_api_type="azure",
)
# dataset
from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english")
amnesty_qa
DatasetDict({
    baseline: Dataset({
        features: ['question', 'ground_truths', 'answer', 'contexts'],
        num_rows: 30
    })
})

To use the AzureOpenAIEmbeddings with the AnswerSimilarity metric, we first import the answer_similarity metric and pass in the embeddings in the evaluate function.

from ragas.metrics import answer_similarity
from ragas import evaluate

result = evaluate(
    amnesty_qa["eval"].select(range(5)),  # showing only 5 for demonstration
    metrics=[answer_similarity],
    llm=azure_model,
    embeddings=azure_embeddings,
)
result
Evaluating: 100%|██████████| 5/5 [00:19<00:00,  3.86s/it]
{'answer_similarity': 0.8494}

That’s it! answer_similarity will now be using AzureOpenAIEmbeddings under the hood for evaluations.

Evaluating with FastEmbed Embeddings

FastEmbed is a Python library built for embedding generation and has support for popular text models. FastEmbed models can be used by instantiating an object of the FastEmbedEmbeddings class from Langchain. More information regarding FastEmbed and supported models can be found here.

from langchain_community.embeddings.fastembed import FastEmbedEmbeddings

fast_embeddings = FastEmbedEmbeddings(model_name="BAAI/bge-base-en")
100%|██████████| 252M/252M [00:09<00:00, 25.7MiB/s] 

Now lets change the embedding model used for AnswerSimilarity by passing the embedding as the FastEmbedEmbeddings object that we created in the evaluation function.

result2 = evaluate(
    amnesty_qa["eval"].select(range(5)),  # showing only 5 for demonstration
    metrics=[answer_similarity],
    llm=azure_model,
    embeddings=fast_embeddings,
)

result2
Evaluating: 100%|██████████| 5/5 [00:21<00:00,  4.24s/it]
{'answer_similarity': 0.8494}

Evaluating with HuggingFace Embeddings

Ragas has support for using embedding models using HuggingFace. Using the HuggingfaceEmbeddings class in Ragas, the embedding models supported by HuggingFace can directly be used for the evaluation task.

# To use embedding models from HuggingFace, you need to install the following
%pip install sentence-transformers
from ragas.embeddings import HuggingfaceEmbeddings

hf_embeddings = HuggingfaceEmbeddings(model_name="BAAI/bge-small-en")

Now we follow the same steps as above to use the HuggingFace Embeddings in the evaluate function with the metrics.

result3 = evaluate(
    amnesty_qa["eval"].select(range(5)),  # showing only 5 for demonstration
    metrics=[answer_similarity],
    llm=azure_model,
    embeddings=hf_embeddings,
)

result3
Evaluating:   0%|          | 0/5 [00:00<?, ?it/s]
Evaluating: 100%|██████████| 5/5 [00:21<00:00,  4.24s/it]
{'answer_similarity': 0.8494}