Domain-Specific and Multilingual Embeddings

Capsule overview

"General" embeddings (OpenAI, SBERT) work well for standard texts, but specialized domains (legal, medical, code) and multiple languages require specific models. When should you use specialized embeddings? When is fine-tuning worth it?

In this capsule you'll learn about domain-specific embeddings (legal, medical, financial, code), multilingual embeddings (mBERT, XLM-R, BGE-M3), when to fine-tune vs zero-shot, and trade-offs. You'll also see code to evaluate embeddings in your domain.

By the end, you'll be able to choose specialized embeddings when necessary.

Domain-Specific Embeddings

Why domain matters:

# General embedding (SBERT):
query = "consideration"
doc_1 = "He showed consideration for others"  # Social context
doc_2 = "In consideration of $100, parties agree..."  # Legal context

# Problem: a general embedding may not differentiate well
# "consideration" has a VERY different meaning in legal vs social

Solution: Embeddings trained on a legal corpus.

Main domains:

1. Legal:

# Models:
# - Law-BERT
# - Legal-SBERT

# Trained on:
# - Case law
# - Contracts
# - Legal documents

# Improvement: +10-15% MTEB on legal tasks vs general models

2. Medical:

# Models:
# - BioBERT
# - PubMedBERT
# - Clinical-BERT

# Trained on:
# - PubMed papers
# - Clinical notes
# - Medical terminology

# Improvement: +20% on medical tasks

3. Financial:

# Models:
# - FinBERT

# Trained on:
# - Financial reports
# - News articles
# - SEC filings

# Improvement: +15% on financial sentiment analysis

4. Code:

# Models:
# - CodeBERT
# - GraphCodeBERT

# Trained on:
# - GitHub repos
# - Stack Overflow

# Improvement: +30% on code search

Multilingual Embeddings

Why multiple languages:

# Problem: SBERT (all-MiniLM-L6-v2) is English-only
# Texts in Spanish, Chinese, Arabic have suboptimal embeddings

# Solution: Multilingual embeddings
# Trained on 100+ languages simultaneously

Multilingual models:

1. mBERT (multilingual BERT):

Languages: 104
Dimensions: 768
MTEB (avg 100+ languages): ~55
Use: General purpose multilingual

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('distiluse-base-multilingual-cased-v2')

2. XLM-RoBERTa:

Languages: 100
Dimensions: 768
MTEB: ~58
Use: Better performance than mBERT

model = SentenceTransformer('xlm-r-100langs-bert-base-nli-stsb-mean-tokens')

3. BGE-M3 (BAAI Multilingual):

Languages: 100+
Dimensions: 1024
MTEB: ~64 (on average)
Use: State of the art multilingual

model = SentenceTransformer('BAAI/bge-m3')

4. Cohere Multilingual (API):

Languages: 100+
Dimensions: 1024
MTEB: ~62
Cost: $0.10/1M tokens

# Usage with the Cohere API
import cohere
co = cohere.Client('YOUR_API_KEY')
response = co.embed(texts=["Hello world"], model="embed-multilingual-v3.0")

Zero-Shot vs Fine-Tuning

Zero-Shot (use a pre-trained model without modifying it):

# Advantages:
✅ Doesn't require training data
✅ Works "out-of-the-box"
✅ Zero effort

# Disadvantages:
❌ Suboptimal performance in a specific domain
❌ Doesn't capture the unique vocabulary of your domain

When to use:

• Prototyping
• General domain (e-commerce, FAQ)
• You don't have training data

Fine-Tuning (re-train embeddings on your domain):

# Advantages:
✅ Performance +10-30% in your domain
✅ Captures specific vocabulary
✅ Learns relationships from your data

# Disadvantages:
❌ Requires training data (1K-10K pairs)
❌ Training time (hours-days)
❌ ML expertise required

When to use:

• Very specific domain (legal, medical)
• You have training data
• Critical performance (worth the effort)

Comparison table:

Practical examples

1. Multilingual embeddings:

from sentence_transformers import SentenceTransformer
import numpy as np

# Load a multilingual model
model = SentenceTransformer('distiluse-base-multilingual-cased-v2')

# Texts in different languages (same meaning)
texts = [
    "The cat is sleeping",  # English
    "El gato está durmiendo",  # Spanish
    "Le chat dort",  # French
    "猫在睡觉",  # Chinese
    "القط نائم"  # Arabic
]

# Generate embeddings
embeddings = model.encode(texts)

# Calculate similarities
def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

# Compare English with other languages
en_emb = embeddings[0]

for i, lang in enumerate(["Spanish", "French", "Chinese", "Arabic"]):
    sim = cosine_similarity(en_emb, embeddings[i+1])
    print(f"English vs {lang}: {sim:.4f}")

Expected output:

English vs Spanish: 0.9743
English vs French: 0.9173
English vs Chinese: 0.9573
English vs Arabic: 0.9236

Embeddings capture cross-lingual meaning.

2. Domain-specific (code search):

from sentence_transformers import SentenceTransformer

# General model
model_general = SentenceTransformer('all-MiniLM-L6-v2')

# Code-specific model (if it exists)
# model_code = SentenceTransformer('microsoft/codebert-base')

query = "function to sort array"
docs = [
    "def sort_array(arr): return sorted(arr)",
    "function sortArray(arr) { return arr.sort(); }",
    "The weather is nice today"
]

# Generate embeddings
query_emb = model_general.encode([query])[0]
docs_embs = model_general.encode(docs)

# Similarities
sims = [cosine_similarity(query_emb, doc_emb) for doc_emb in docs_embs]

for doc, sim in zip(docs, sims):
    print(f"{sim:.4f} | {doc[:50]}...")

Output (general model):

0.6713 | def sort_array(arr): return sorted(arr)...
0.7465 | function sortArray(arr) { return arr.sort(); }...
0.1279 | The weather is nice today...

With a code-specific model (CodeBERT), scores would be higher (~0.85).

Evaluate embeddings in your domain

Create an evaluation dataset:

# Format:
evaluation_pairs = [
    {
        "query": "How to install Python?",
        "doc_relevant": "Download Python from python.org",
        "doc_irrelevant": "JavaScript is a web language"
    },
    # ... more pairs
]

# Minimum: 100 pairs (the more, the better)

Evaluate a model:

from sentence_transformers import SentenceTransformer
import numpy as np

def evaluate_model(model_name, eval_pairs):
    """
    Evaluate a model on a custom dataset
    
    Returns:
        Accuracy (% of times doc_relevant > doc_irrelevant)
    """
    model = SentenceTransformer(model_name)
    
    correct = 0
    
    for pair in eval_pairs:
        query_emb = model.encode([pair['query']])[0]
        rel_emb = model.encode([pair['doc_relevant']])[0]
        irrel_emb = model.encode([pair['doc_irrelevant']])[0]
        
        sim_rel = np.dot(query_emb, rel_emb)
        sim_irrel = np.dot(query_emb, irrel_emb)
        
        if sim_rel > sim_irrel:
            correct += 1
    
    accuracy = correct / len(eval_pairs)
    return accuracy

# Test
evaluation_pairs = [
    {
        "query": "Python installation",
        "doc_relevant": "Download from python.org",
        "doc_irrelevant": "JavaScript is popular"
    },
    # ... more pairs
]

accuracy = evaluate_model("all-MiniLM-L6-v2", evaluation_pairs)
print(f"Accuracy: {accuracy:.2%}")

When NOT to fine-tune

Cases where zero-shot is enough:

✅ General domain (e-commerce, FAQ)
✅ You don't have training data
✅ Limited budget/time
✅ The general model already has >85% accuracy on your eval

Cases where fine-tuning is worth it:

✅ Very specific domain (legal, medical, code)
✅ You have 1K+ training pairs
✅ Critical performance (every % of accuracy matters)
✅ The general model has <80% accuracy
✅ Budget for ML Engineering

Exercises

Exercise 1: Multilingual embeddings

Use distiluse-base-multilingual-cased-v2 to compare:

# Texts:
# - "Hello world" (English)
# - "Hola mundo" (Spanish)
# - "Bonjour le monde" (French)

# Which are most similar to each other?

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer('distiluse-base-multilingual-cased-v2')

texts = [
    "Hello world",
    "Hola mundo",
    "Bonjour le monde"
]

embeddings = model.encode(texts)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

print("Similarities:")
print(f"EN vs ES: {cosine_similarity(embeddings[0], embeddings[1]):.4f}")
print(f"EN vs FR: {cosine_similarity(embeddings[0], embeddings[2]):.4f}")
print(f"ES vs FR: {cosine_similarity(embeddings[1], embeddings[2]):.4f}")

Similarities:
EN vs ES: 0.9823  ← Most similar (same meaning)
EN vs FR: 0.9515
ES vs FR: 0.9478

Exercise 2: Evaluate on a domain

Create 5 evaluation pairs for your domain and evaluate all-MiniLM-L6-v2:

# Implement the evaluate_model() function
# Create evaluation_pairs (minimum 5)
# Calculate accuracy

evaluation_pairs = [
    {
        "query": "Python installation",
        "doc_relevant": "Download Python from python.org",
        "doc_irrelevant": "JavaScript is a web language"
    },
    {
        "query": "List comprehension syntax",
        "doc_relevant": "[x for x in range(10)]",
        "doc_irrelevant": "Java uses for loops"
    },
    {
        "query": "Django tutorial",
        "doc_relevant": "Django is a Python web framework",
        "doc_irrelevant": "React is a JavaScript library"
    },
    {
        "query": "NumPy array",
        "doc_relevant": "np.array([1, 2, 3])",
        "doc_irrelevant": "Arrays in C are different"
    },
    {
        "query": "Virtual environment",
        "doc_relevant": "Use venv or virtualenv in Python",
        "doc_irrelevant": "Docker containers isolate apps"
    }
]

accuracy = evaluate_model("all-MiniLM-L6-v2", evaluation_pairs)
print(f"Accuracy in the Python domain: {accuracy:.2%}")

Summary

What you learned:

• ✅ Domain-specific: Legal, medical, code (improvement +10-30%)
• ✅ Multilingual: mBERT, XLM-R, BGE-M3 (100+ languages)
• ✅ Zero-shot: Use pre-trained (easy, enough if >85% accuracy)
• ✅ Fine-tuning: Re-train (high effort, improvement +10-30%)
• ✅ Custom evaluation: Create a dataset for your domain

Key concepts:

1. A specific domain improves performance significantly
2. Multilingual is critical for non-English
3. Fine-tune only if it's worth the effort (>$500, weeks)

Additional resources

1. BioBERT - Medical embeddings
2. Legal-BERT - Legal embeddings
3. CodeBERT - Code embeddings
4. BGE-M3 - Multilingual SOTA

In the next capsule

Capsule 08: Mini-Project - Benchmark Framework

You'll build:

• Compare 3+ models
• Metrics: MTEB, latency, cost
• Automatic reports
• Final recommendation

From theory to a complete project.

Module 3 - Embeddings Deep Dive Guide Domain-specific and multilingual: when a generalist isn't enough