Choosing the Right AI Model for My FAQ Chatbot: From TF-IDF to Transformers

Introduction

description: "What I learned about LLMs and how I plan to improve my NLP FAQ chatbot by comparing traditional text matching, sentence embeddings, and open-weight language models."

While studying Natural Language Processing (NLP), I learned that building a chatbot is not only about collecting questions and answers. It is also about helping a computer identify what a user means, even when the question is written differently from the examples it has seen.

In my FAQ chatbot project, I explored TF-IDF and word embeddings as approaches to text representation. Learning about Large Language Models (LLMs) made me interested in finding a better way to handle questions that do not exactly match the wording stored in the system.

This raised an important question for me: Should I improve my existing text-matching approach, use a sentence-embedding model, or introduce an LLM?

In this article, I compare three approaches and explain which one I would test first.

1. Understanding the Problem

Imagine that a chatbot's FAQ contains this question:

"What are the requirements for registration?"

A user might instead ask:

"What do I need to sign up?"

A basic keyword-matching system may struggle if the two questions use different words, even though their meanings are similar.

This is where the difference between matching words and representing meaning becomes important.

My goal is to make the chatbot identify relevant questions more reliably, return useful answers, and avoid generating information that is not supported by its FAQ data.

2. Three Approaches Worth Comparing

Approach 1: TF-IDF

TF-IDF stands for Term Frequency–Inverse Document Frequency. It converts text into numerical features based on how important words are within a document and across a collection of documents.

For an FAQ chatbot, it can help rank stored questions according to their similarity to a user's question.

Advantages

• Lightweight and relatively simple to implement.
• Fast for small FAQ collections.
• Does not require a large language model.
• Provides a useful baseline for evaluating improvements.

Limitations

TF-IDF primarily represents word importance. Questions that use different vocabulary can receive low similarity scores even when their meanings are closely related.

For example, "How can I register?" and "What is the sign-up procedure?" may be difficult to match if they share few important terms.

TF-IDF is therefore a useful starting point, but I want to investigate whether semantic representations can improve the results.

Reference: Scikit-learn TF-IDF documentation

Approach 2: Sentence Transformers

Instead of relying mainly on individual word frequencies, sentence-embedding models convert entire sentences into numerical vectors that represent their semantic features.

One model I would consider is sentence-transformers/all-MiniLM-L6-v2.

It is designed to produce sentence and paragraph embeddings that can be compared using similarity measures such as cosine similarity. This makes it suitable for exploring semantic search in a question-and-answer system.

Advantages

• Can identify similarities between differently worded sentences.
• Suitable for searching a collection of stored FAQ questions.
• More lightweight than using a large generative language model.
• Can be used locally without sending every question to a paid API.

Limitations

• Similarity scores do not guarantee that two questions have exactly the same meaning.
• The model may not perform equally well across all languages and domains.
• The chatbot still needs a rule for deciding when a match is too weak to return.

For these reasons, I see sentence embeddings as a promising way to improve question retrieval while keeping the system relatively simple.

Reference: Sentence Transformers documentation

Approach 3: Qwen3-4B-Instruct-2507

The LLM lecture introduced me to open-weight models, including Qwen3-4B-Instruct-2507.

Unlike TF-IDF and sentence-embedding models, this is a generative language model. It can follow instructions and generate responses based on the prompt and context it receives.

Its model card lists 4 billion parameters and a native context length of 262,144 tokens.

Advantages

• Can generate answers in natural language.
• Can interpret questions using surrounding context.
• Can be instructed to answer using retrieved FAQ information.
• Can be run using tools such as Hugging Face Transformers, subject to available hardware and the model's licence.

Limitations

• Requires substantially more computing memory than a small embedding model.
• Can generate incorrect information if it is not properly constrained.
• A large context window does not guarantee correct answers.
• It may be unnecessarily complex for a small FAQ collection where retrieving an existing answer is sufficient.

For my project, I would not automatically replace the existing system with an LLM. I would first determine whether the main problem is retrieving the right FAQ or generating a more natural response.

Reference: Qwen3-4B-Instruct-2507 model card

3. Comparing the Approaches

This comparison highlights an important lesson: these approaches do not perform exactly the same job. TF-IDF and sentence embeddings can retrieve relevant questions, while an LLM can generate a response from the information provided to it.

A fair comparison should therefore evaluate each approach according to its intended role.

4. My Choice: Sentence Transformers First

For the first improvement to my FAQ chatbot, I would choose sentence-transformers/all-MiniLM-L6-v2 for semantic retrieval.

My main reason is that the chatbot's first responsibility is to find the correct answer in its FAQ collection. If the system retrieves the wrong information, generating a fluent response will not solve the underlying problem.

Sentence embeddings offer a practical middle ground between traditional keyword matching and a larger generative model.

I would keep TF-IDF as a baseline so that I can measure whether the new approach actually improves retrieval. If semantic retrieval performs well, I could later experiment with Qwen3 to generate more conversational answers using the retrieved FAQ information.

This is a proposed model choice, not a claim that I have already tested it or proven that it is the most accurate option for my dataset.

5. How I Would Evaluate the Improvement

Before deciding whether to keep the new model, I would prepare a small test dataset containing real or representative FAQ questions and their expected answers.

I would then compare TF-IDF and sentence embeddings using the same questions.

My evaluation plan would include:

1. Correct retrieval: Does the system retrieve the FAQ entry containing the expected answer?
2. Paraphrase handling: Can it match questions that express the same meaning using different words?
3. Response time: How quickly does it return a result?
4. Unanswered questions: Does it avoid selecting an unrelated FAQ when no suitable answer exists?
5. Resource usage: Can it run efficiently on the hardware available to me?

I would record the results for both approaches instead of relying only on a few successful examples.

If I later add Qwen3, I would also check whether its generated responses remain faithful to the retrieved FAQ content. The system should be able to say that it does not have enough information instead of inventing an answer.

6. What I Learned About LLMs

Studying LLMs helped me understand that these systems process text through tokenization, embeddings, Transformer layers, and next-token prediction.

I also learned that model size, context windows, decoding settings, and computational resources influence how a model can be used.

Most importantly, I learned that the most powerful model is not always the best model for a particular problem. The right choice depends on the task, the available data, the required accuracy, and the resources available.

For my FAQ chatbot, improving retrieval may be more valuable as a first step than immediately introducing a large generative model.

Conclusion

My exploration of TF-IDF, sentence embeddings, and LLMs has helped me think more carefully about how to improve my FAQ chatbot.

TF-IDF provides a simple baseline, Sentence Transformers offer a promising way to retrieve questions based on semantic similarity, and Qwen3 provides an option for generating more natural responses when needed.

My next step is to compare TF-IDF with sentence embeddings using the same test questions and measure the results. I would then decide whether adding an LLM provides enough value to justify its additional complexity.

For me, this is the most useful lesson from studying NLP: choosing a model should begin with understanding the problem, not simply selecting the largest or newest technology.

References

• Scikit-learn: Text Feature Extraction
• Sentence Transformers: Pretrained Models
• Sentence Transformers: Semantic Similarity
• Qwen3-4B-Instruct-2507: Model Card

AI #NLP #MachineLearning #Python #LearningInPublic