AI TOOLS / JUNE 2026
Local LLM and RAG Experiments
Two small projects exploring how local models and retrieval work.
Built on my own.
- MY ROLE
- Solo project
- KEY RESULT
- Runs on your own computer, so the PDF never leaves it
Built with
- Python
- Ollama
- LangChain
- ChromaDB
What it is
These are two small experiments with AI models that run on my own computer. The first sorts a messy grocery list into sections. The second answers questions about a PDF.
For the PDF, the AI does not just guess. The program first finds the few parts of the document most related to your question and hands only those to the AI, so the answer comes from the document.
What it can do
Grocery list sorter
Give it a messy list and it sorts the items into sections such as produce, dairy, and drinks.
Ask questions about a PDF
Ask a question in plain words and get an answer based on the document.
Answers only from the document
The AI is told to use only what it was handed, and to say “I cannot find that information” when the answer is not there.
Remembers the document
The prepared document is saved, so it is only processed once.
How a request flows
- PDF loader
- Chunks + embeddings
- ChromaDB
- Ollama
How it works
- 01
The program reads the PDF.
- 02
It cuts the text into overlapping pieces, each about 1,200 characters long.
- 03
It turns each piece into a set of numbers that capture its meaning, and stores them in a small local database.
- 04
When you ask a question, it finds the 5 pieces whose meaning is closest to your question.
- 05
It gives those pieces to the AI with strict instructions to answer only from them.
Illustration
Why
I wanted to see how retrieval-augmented generation works under the hood, using models on my own machine instead of a hosted API.
What I built
- 01
Wrote categorizer.py, which sends a grocery list to a local model and organizes the items into sections.
- 02
Wrote pdf-rag.py: load a PDF, split it into chunks, embed them with nomic-embed-text, store them in ChromaDB, retrieve relevant context, and answer with llama3.2.
The code
def create_chain(retriever, llm):
prompt = ChatPromptTemplate.from_template(
"""
You are a helpful assistant.
Answer ONLY using the provided context.
If the answer is not in the context, say:
"I cannot find that information in the provided documents."
Context:
{context}
Question:
{question}
"""
)
chain = (
{
"context": retriever | format_docs,
"question": RunnablePassthrough(),
}
| prompt
| llm
| StrOutputParser()
)
logging.info("Chain created.")
return chainOutcome
Two working scripts, one for structured output and one for asking questions about a document.
The trade-off
Everything runs locally, so there is no API cost and no data leaves the machine, but answer quality is limited by the size of the model I can run.
What I learned
Most of a RAG system’s quality (RAG means the AI looks up the relevant part of a document before it answers) comes from how the document is chunked and retrieved, not from the model.
What I'd fix next
Add citations that point to the retrieved chunk and a small set of test questions to compare chunk sizes.
Working on something like Local LLM and RAG Experiments?
Tell me what you're building and I'll reply within two business days.