I Built a Simple RAG App with LangChain, OpenAI, and Pinecone

Large language models know a lot, but they do not automatically know the contents of our private files, company documents, notes, or recently written articles.

This is where Retrieval-Augmented Generation, commonly called RAG, becomes useful.

In this project, I built a small RAG application that:

Reads a text document.
Breaks the document into smaller sections.
Converts those sections into numerical representations called embeddings.
Stores the embeddings in Pinecone.
Finds the most relevant sections when a user asks a question.
Gives those sections to an AI model so it can produce a grounded answer. A grounded answer is an answer based on the retrieved source material. RAG reduces unsupported answers, but it does not guarantee that every response will be correct
I used:

Python for the application
LangChain to connect the different steps
OpenAI embeddings to represent text as numbers
Pinecone to store and search those embeddings
uv to manage the Python project and dependencies
In this article, I will explain how the application works, how the different components fit together, and what I learned while debugging it.

What Is RAG?
Press enter or click to view image in full size

RAG Pipeline
Imagine that you are taking an open-book examination.

Without a book, you answer only from memory. You may know the answer, but you may also forget something or provide incorrect information.

Press enter or click to view image in full size

Debugging using LangSmith
With an open book, you can:

Read the question.
Search for the relevant page.
Use that information to write the answer.
A RAG application works in a similar way.

The language model is the person answering the question. The vector database acts like a searchable book. Before answering, the application finds the most relevant information and gives it to the model.

The model does not need to memorize the entire document. It only receives the sections that are likely to contain the answer.

The Two Main Parts of a RAG Application
A basic RAG application has two phases.

Phase 1: Ingestion
Ingestion means preparing the documents so that they can be searched later.

Text document
↓
Load the document
↓
Split it into smaller chunks
↓
Convert each chunk into an embedding
↓
Store the embeddings in Pinecone
This process usually runs when documents are added or updated.

Phase 2: Retrieval and Answer Generation
This phase runs whenever a user asks a question.

User question
↓
Convert the question into an embedding
↓
Search Pinecone for similar document chunks
↓
Add those chunks to the prompt
↓
Ask the language model to answer
Keeping these phases separate makes the application easier to understand.

Ingestion prepares the knowledge. Retrieval uses that knowledge.

What Is an Embedding?
Press enter or click to view image in full size

Computers cannot directly compare the meaning of sentences in the same way people do.

An embedding model converts text into a list of numbers called a vector.

For example:

"How do I reset my password?"
might become something that looks like:

[0.12, -0.48, 0.77, 0.03, ...]
The actual vector contains hundreds or thousands of numbers.

Sentences with similar meanings usually receive vectors that are close to each other. This allows the application to search by meaning rather than only by exact keywords.

Press enter or click to view image in full size

How embeddings work in real.
LangChain provides a standard interface for embedding models, while OpenAI’s text-embedding-3 models allow developers to control the number of dimensions returned.

What Is Pinecone?
Press enter or click to view image in full size

Pinecone is a vector database.

A normal database is excellent at questions such as:

Find the user whose ID is 123.
A vector database is useful for questions such as:

Find the document sections whose meaning is closest to this question.
Pinecone stores the document vectors and searches for the vectors that are closest to the user’s question.

The Pinecone index and the embedding model must use the same vector dimension. Otherwise, Pinecone cannot store or compare the vectors.

Project Structure
Now that the basic idea is clear, let us look at how the application is structured and how each part of the pipeline works.

My project contains two main Python files:

rag-project/
├── rag-gist/
│   ├── ingest.py
│   ├── main.py
│   └── mediumblog.txt
├── .env
├── .gitignore
└── pyproject.toml

Here is what each file does:

mediumblog.txt contains the source information.
ingest.py loads and stores the document in Pinecone.
main.py accepts a question and generates an answer.
.env stores API keys.
.gitignore prevents secrets from being committed.
pyproject.toml contains the project dependencies.

The project contains separate scripts for document ingestion and question answering.
Step 1: Install the Dependencies
I used uv to manage the project.

The required packages can be installed with:

uv add langchain \
    langchain-community \
    langchain-openai \
    langchain-pinecone \
    langchain-text-splitters \
    python-dotenv

The main packages have different responsibilities:

langchain-openai connects LangChain to OpenAI models.
langchain-pinecone connects LangChain to Pinecone.
langchain-community provides document loaders.
langchain-text-splitters provides tools for dividing documents into chunks.
python-dotenv loads environment variables from a .env file.
LLM libraries change quickly, so it is better to keep the generated lock file in the repository instead of relying only on version numbers written in an article.

Step 2: Add the API Keys
Create a .env file in the project root:

OPENAI_API_KEY=your_openai_api_key
PINECONE_API_KEY=your_pinecone_api_key

Never upload this file to GitHub.

Add it to .gitignore:

.env
.venv/
__pycache__/

You can also create a safe example file called .env.example:

OPENAI_API_KEY=
PINECONE_API_KEY=

This tells other developers which variables are required without exposing the real values.

Step 3: Create the Pinecone Index
Before running the ingestion script, create a dense Pinecone index.

Pinecone does not generate the answer. Its job is to find the document chunks that are most relevant to the user’s question.

For this project, I used:

Index name: medium-blogs-embedding-index
Dimension: 512
Metric: cosine
The dimension is important.

The embedding model and the Pinecone index must use the same number of dimensions. Otherwise, Pinecone cannot store or compare the vectors.

OpenAI embeddings can be used for semantic search by comparing the similarity between document and query vectors.

Press enter or click to view image in full size

The Pinecone index uses 512 dimensions, matching the embedding configuration in the Python code
Step 4: Load and Store the Document
The ingestion script reads the text file, divides it into chunks, converts each chunk into an embedding, and stores the result in Pinecone.

Here is the ingestion script:

from pathlib import Path
from dotenv import load_dotenv
from langchain_community.document_loaders import TextLoader
from langchain_openai import OpenAIEmbeddings
from langchain_pinecone import PineconeVectorStore
from langchain_text_splitters import CharacterTextSplitter
load_dotenv()
INDEX_NAME = "medium-blogs-embedding-index"
EMBEDDING_MODEL = "text-embedding-3-small"
EMBEDDING_DIMENSION = 512
def main() -> None:
    print("Ingesting...")
    file_path = Path(__file__).with_name("mediumblog.txt")
    if not file_path.exists():
        raise FileNotFoundError(
            f"Knowledge file was not found: {file_path}"
        )
    print("Loading document...")
    loader = TextLoader(str(file_path), encoding="utf-8")
    documents = loader.load()
    text_splitter = CharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=0,
    )
    chunks = text_splitter.split_documents(documents)
    print(f"Created {len(chunks)} text chunks")
    embeddings = OpenAIEmbeddings(
        model=EMBEDDING_MODEL,
        dimensions=EMBEDDING_DIMENSION,
    )
    PineconeVectorStore.from_documents(
        documents=chunks,
        embedding=embeddings,
        index_name=INDEX_NAME,
    )
    print("Ingestion complete.")
if __name__ == "__main__":
    main()

Run the script with:

uv run python rag-gist/ingest.py

The output should look similar to this:

Ingesting...
Loading document...
Created 31 text chunks
Ingestion complete.

Press enter or click to view image in full size

The ingestion script loaded the document, created 31 chunks, and stored them in Pinecone.
Understanding the Ingestion Code
Let us look at what the script is doing.

Loading the Document

loader = TextLoader(str(file_path), encoding="utf-8")
documents = loader.load()

The loader turns the text file into LangChain Document objects.

A document usually contains:

page_content: the actual text
metadata: information such as the source filename
Splitting the Document

text_splitter = CharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=0,
)

Instead of embedding the entire document as one large block, we divide it into smaller chunks.

Press enter or click to view image in full size

Chunking is essential for context window : and token optimization. Instead of sending the entire document to the language model, the application sends only the sections that are most relevant to the question.
This is important because a user’s question may only be related to one small part of the document.

A chunk size of 1000 means that each section contains approximately 1,000 characters.

For this first version, I used no overlap. In a stronger version, I would add some overlap so that information near the end of one chunk is also available at the beginning of the next chunk.

Creating Embeddings

embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",
    dimensions=512,
)

This code tells OpenAI to return embeddings with 512 dimensions.

The dimensions option is supported by OpenAI’s text-embedding-3 family of models.

Storing the Chunks

PineconeVectorStore.from_documents(
    documents=chunks,
    embedding=embeddings,
    index_name=INDEX_NAME,
)

This method:

Reads the text from each document chunk.
Sends the text to the embedding model.
Receives a vector for each chunk.
Stores the vector and document metadata in Pinecone.
Step 5: Retrieve Information and Generate an Answer
After ingestion is complete, we can ask questions.

Here is the retrieval script:

from dotenv import load_dotenv
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_pinecone import PineconeVectorStore
load_dotenv()
INDEX_NAME = "medium-blogs-embedding-index"
EMBEDDING_MODEL = "text-embedding-3-small"
EMBEDDING_DIMENSION = 512
embeddings = OpenAIEmbeddings(
    model=EMBEDDING_MODEL,
    dimensions=EMBEDDING_DIMENSION,
)
# This was the model used in my original project.
# Replace it when necessary with a model available to your account.
llm = ChatOpenAI(
    model="gpt-4o-mini",
    temperature=0,
)
vectorstore = PineconeVectorStore(
    index_name=INDEX_NAME,
    embedding=embeddings,
)
retriever = vectorstore.as_retriever(
    search_kwargs={"k": 5}
)
prompt_template = ChatPromptTemplate.from_template(
    """
You are a helpful assistant.
Answer the question using only the context provided below.
If the answer is not present in the context, say:
"I could not find that information in the provided document."
Context:
{context}
Question:
{question}
Answer:
"""
)
def format_documents(documents) -> str:
    return "\n\n".join(
        document.page_content for document in documents
    )
def answer_question(question: str) -> str:
    documents = retriever.invoke(question)
    context = format_documents(documents)
    messages = prompt_template.format_messages(
        context=context,
        question=question,
    )
    response = llm.invoke(messages)
    return response.content
if __name__ == "__main__":
    user_question = input("Ask a question: ").strip()
    if not user_question:
        raise ValueError("The question cannot be empty.")
    answer = answer_question(user_question)
    print("\nAnswer:\n")
    print(answer)

Run it with:

uv run python rag-gist/main.py

The program asks for a question:

Ask a question: What is the main idea of the document?

It then searches Pinecone and sends the most relevant sections to the language model.

Press enter or click to view image in full size

Query for simple RAG System
Caption: The application retrieved relevant document sections and used them to generate an answer.

Why Use a Retriever?
The vector store handles storage and similarity search.

A retriever provides a simpler interface for requesting relevant documents:

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 5}
)

The value k=5 means:

Return the five document chunks that are most closely related to the question.

When this line runs:

documents = retriever.invoke(question)

the application:

Converts the question into an embedding,
Searches Pinecone,
Finds the closest vectors.
Returns their original text.
How the Prompt Reduces Unsupported Answers
The prompt contains an important instruction:

Answer the question using only the context provided below.

It also tells the model what to do when the answer is missing:

If the answer is not present in the context, say:
"I could not find that information in the provided document."

This does not guarantee perfect accuracy, but it gives the model a clear boundary.

A RAG system should not only retrieve information. It should also tell the model how that information must be used.

What Is a LangChain Runnable?
LangChain uses a common interface called a Runnable.

A runnable is something that accepts input and produces output.

For example:

documents = retriever.invoke(question)

and:

response = llm.invoke(messages)

Both the retriever and the language model support .invoke().

That common interface makes it easier to connect prompts, retrievers, models, and custom functions into larger chains.

Problems I Encountered
The most useful part of building this project was understanding the errors.

1. Wrong Pinecone Index Name

The error said:

Index 'mediumblog' not found.
Did you mean: medium-blogs-embedding-index

The name in the Python script did not match the name in Pinecone.

I fixed the constant:

INDEX_NAME = "medium-blogs-embedding-index"

A useful improvement is to keep shared settings in one configuration file so that ingestion and retrieval cannot accidentally use different index names.

2. Vector Dimension Mismatch

I then received:

Vector dimension 1536 does not match
the dimension of the index 512

The Pinecone index expected vectors containing 512 values, while the application was sending vectors of a different size.

I fixed it by explicitly setting:

dimensions=512

The same embedding model and dimension must be used during both ingestion and retrieval.

3. Using the Wrong Constructor Parameter

I initially used:

PineconeVectorStore(
    index_name=INDEX_NAME,
    embeddings=embeddings,
)

However, the constructor expected embedding, singular:

PineconeVectorStore(
    index_name=INDEX_NAME,
    embedding=embeddings,
)

Current LangChain-Pinecone examples also initialize the vector store with an embedding object.

This was a small naming difference, but it prevented the application from starting.

4. Prompt Variable Mismatch

My prompt used this placeholder:

{question}

But the code passed:

query=query

LangChain raised:

KeyError: 'question'

The variable name passed to the template must exactly match the placeholder:

messages = prompt_template.format_messages(
    context=context,
    question=query,
)

This error also reminded me to keep naming consistent throughout the application.

What I Would Improve Next
This project is a good learning version, but it is not yet a production-ready RAG system.

My next improvements would be:

Use a Better Text Splitter
I would replace CharacterTextSplitter with RecursiveCharacterTextSplitter.

It tries to keep paragraphs and sentences together instead of cutting the text only by character count.

Add Chunk Overlap
For example:

chunk_size=1000
chunk_overlap=150

The overlap helps preserve information that appears near chunk boundaries.

Prevent Duplicate Ingestion
Running the ingestion script repeatedly can store the same content more than once.

A production version should create stable document IDs and update existing records instead of always inserting new ones.

Return Source Information
The answer should show which chunks or source files were used.

For example:

Source: mediumblog.txt, chunk 8

This would make the application easier to trust and debug.

Add a Relevance Threshold
The application currently returns the top five chunks even when none of them is strongly related to the question.

A similarity threshold could help reject unrelated results.

Add Logging and Tracing
Tracing tools can show:

which question was asked
which chunks were retrieved
how long each step took
how many tokens were used
what answer was generated
Create an Evaluation Set
I would prepare a small collection of:

test questions
expected sources
expected answer points
This would make it easier to compare changes to chunk size, retrieval count, prompts, and embedding models.

What I Learned
Before building this project, RAG looked like one complicated AI feature.

After implementing it manually, I understood that it is simply a pipeline:

Documents become vectors.
Questions become vectors.
Vector search finds related text.
The language model turns that text into an answer.
The difficult part is not calling one magical RAG function.

The important decisions are:

how the document is divided,
which embedding model is used
whether vector dimensions match
how many chunks are retrieved
how irrelevant results are handled
what instructions are given to the language model
how sources are shown to the user
One of the most useful lessons from this project was to inspect the retrieved chunks before changing the prompt.
If the correct information is not retrieved, the language model cannot use it, no matter how well the prompt is written.
Building each step separately made the architecture much easier to understand.

Final Thoughts
A small RAG application is a useful project for learning how modern AI applications work with external knowledge.

The best way to learn RAG is to begin with one document, one question, and one successful retrieval.

It teaches more than prompt writing. It introduces document processing, embeddings, vector search, retrieval, prompt design, configuration management, and debugging.

The first version does not need to be perfect.

Start with one document, one ingestion script, one retrieval script, and one question.

Once the basic pipeline works reliably, improve one part at a time and observe how each change affects retrieval quality.