Nano Language Models

How to measure what really matters when working with small language models.

🎯 Introduction — Beyond “It Works”

You’ve fine-tuned or quantized your Small Language Model (SLM). It runs, it talks, it answers.

But does it perform well?

In AI development, “it works” isn’t enough.

You need to evaluate how well it performs — across accuracy, speed, efficiency, and cost.

This article will teach you how to test SLMs properly using practical metrics, benchmarks, and Python tools — all optimized for developers who value efficiency.

🧠 Step 1: The Four Pillars of SLM Evaluation

Every SLM can be evaluated across four key categories:

For SLMs, performance isn’t only about output quality — it’s about running smoothly on limited hardware while maintaining task accuracy.

⚙️ Step 2: Text Quality Metrics

Let’s start with the most common quality metrics for NLP tasks.

🧩 Perplexity (Language Modeling)

Perplexity measures how confidently a model predicts the next word.
Lower = better.

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch, math

model_id = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

test = load_dataset("wikitext", "wikitext-2-raw-v1", split="test")
enc = tok("\n\n".join(test["text"]), return_tensors="pt")
loss = model(**enc, labels=enc["input_ids"]).loss
print("Perplexity:", math.exp(loss))

Typical ranges:

• Excellent: < 20
• Good: 20–50
• Weak: > 100

🧩 ROUGE & BLEU (Summarization/Translation)

For generation tasks (like summaries or translations), use ROUGE and BLEU to compare generated text vs reference output.

from datasets import load_metric
metric = load_metric("rouge")

pred = ["The cat sat on the mat."]
ref = ["The cat is sitting on the mat."]
score = metric.compute(predictions=pred, references=ref)
print(score)

• ROUGE-1: overlap of unigrams (words)
• ROUGE-L: longest common subsequence

Scores are between 0 and 1 (higher = better).

🧩 Accuracy and F1 (Classification)

If your fine-tuned SLM does classification (like sentiment or topic tagging):

from sklearn.metrics import accuracy_score, f1_score

y_true = [1, 0, 1, 1]
y_pred = [1, 0, 0, 1]
print("Accuracy:", accuracy_score(y_true, y_pred))
print("F1 Score:", f1_score(y_true, y_pred))

Even though SLMs are text generators, adapter-based models often perform structured classification through prompt engineering.

⚡ Step 3: Efficiency Metrics

Evaluating small models means measuring how much you get per watt, per second, or per GB.

Example test:

import time, torch
start = time.time()
output = model.generate(**enc, max_new_tokens=100)
torch.cuda.synchronize()
print("Latency:", time.time() - start, "s")

🧩 Step 4: Benchmarking Frameworks

Here are the top tools used to benchmark SLMs today.

🧪 1. EleutherAI LM Evaluation Harness

Industry standard for evaluating language models on 200+ datasets.

pip install lm-eval
lm-eval --model hf --model_args pretrained=TinyLlama/TinyLlama-1.1B-Chat-v1.0 --tasks wikitext,lambada_openai

Outputs:

| Task        | Accuracy | Perplexity |
|--------------|-----------|-------------|
| wikitext     | 0.91      | 18.4        |
| lambada_openai | 0.82    | 25.3        |

✅ Supports all major open SLMs (TinyLlama, Phi-3, Gemma, Mistral).
✅ Gives direct leaderboard-style results.

⚙️ 2. Hugging Face Evaluate

Easy to use within notebooks for specific metrics.

from evaluate import load
metric = load("bleu")
print(metric.compute(predictions=["A fast small model."], references=["A quick little model."]))

⚡ 3. Torch Profiler

For advanced users tracking performance bottlenecks.

import torch.profiler as profiler

with profiler.profile(record_shapes=True) as prof:
    model(**enc)
print(prof.key_averages().table(sort_by="cpu_time_total"))

🧩 Step 5: Comparative Testing

Benchmarking only makes sense in context.
Let’s compare three models on local inference:

📊 Interpretation:

• TinyLlama is fastest — perfect for dashboards and edge devices.
• Mistral offers best quality but slower throughput.
• Phi-3 Mini balances both.

⚙️ Step 6: Latency Testing with Locust

For API-based models (e.g., FastAPI deployment), test concurrency and response time with Locust.

Install:

pip install locust

Then create a locustfile.py:

from locust import HttpUser, task

class SLMUser(HttpUser):
    @task
    def generate(self):
        self.client.post("/generate", json={"text": "Summarize this: The AI market is booming."})

Run:

locust -f locustfile.py

Monitor how your SLM handles concurrent requests — a key performance factor for real deployments.

🧠 Step 7: Visualization and Reports

You can visualize metrics directly in Python using:

import matplotlib.pyplot as plt

models = ["TinyLlama", "Phi-3 Mini", "Mistral 7B"]
speed = [31, 24, 20]
quality = [0.9, 0.95, 1.0]

plt.bar(models, speed, color='teal')
plt.title("Inference Speed (tokens/sec)")
plt.show()

Charts make your findings digestible — especially for Medium or GitHub writeups.

🔍 Step 8: Defining “Good Enough”

For most real-world SLM applications:

• Perplexity < 30 → fluent generation
• Latency < 3s → responsive apps
• RAM < 8 GB → deployable on laptops
• Accuracy > 0.85 (task-based) → production-ready

You don’t need GPT-4-level benchmarks to achieve business value — just consistent local performance.

🔮 The Future of SLM Evaluation

Expect more energy-aware metrics (like “tokens per watt”) and adaptive testing frameworks that automatically balance precision and efficiency.

In the Nano AI era, evaluation isn’t just about what your model says — it’s about how well and where it says it.

The next frontier? Dynamic benchmarking — models that report their own performance as they run.

Follow NanoLanguageModels.com for tutorials on benchmarking, quantization, and building efficient AI systems that balance accuracy, speed, and scale. ⚙️