ai-chat

Self-hosted chat assistant on vLLM serving a web UI. Grew into a coding agent, then back into a chat assistant.

View project on GitHub

Problem

I want to learn how AI actually works. I had watched Claude write HTML pages with JavaScript programs, Python scripts, and more. But at the start of this project, I had no hands on application development experience and Claude could just move much faster than I could follow or understand. Using some of the Ai products available could lead to confusion because it worked faster than I could understand and the output often summarized many steps of effort that I could not reasonably audit. And the total output was a massive software project that I had no business navigating.

I decided the best way to know more about what was going on was to create my own clone Claude / ChatGPT application. At least a mini toy version.

Below, I share more details on the project including what worked / did not work and lessons learned.

Constraints

A ChatGPT/Claude clone can get expensive, so I set one rule: no new costs. I already had a PC with a high end GPU, some AI subscriptions and time. Also electricity and internet.

Key decisions

Model

Models under 30B parameters at int4 fit in ~20 GB. Whatever VRAM is left is the context window. Chose early on to focus on int4 quantized models only to get 24-27B param options. There is some variance in availability between GGUF and AWQ format so vllm/ollama can matter for model selection. After these details fixed, every model choice was a trade between output quality and context length.

There are different kinds of models, like coding, mixture-of-experts, and instruct.

Gemma 4 26B-A4B won in the end. It's mixture-of-experts with ~4B active parameters, so it's fast, and at 4-bit it fits under 20 GB with room for the full 262k context. It can read a document and respond to the user. I wouldn't trust it to review my taxes, and frankly I wouldn't ask it to summarize some pdf either. It's mostly just fun to mess with, and can't do very much useful work at all.

Runner-up: Qwen's latest 27B. Better at code, less fun to chat with. Almost actually usable for agentic work but gets derailed too frequently and hallucinates a bunch of junk. In apocalypse, or if the internet was down, might be a reasonable resource for information and support.

Infrastructure

vLLM over Ollama. It batches requests, serves multiple users and is very configurable. It also lets you attempt impossible configurations, so you can spend hours watching a server try to start that was always going to run out of memory.

Tailscale for remote access. Rolling my own WireGuard setup looked like a security nightmare. Tailscale just works.

The rest: Docker for some reason, I don't think I needed this, but every google search result told me i needed this for gpu/ai work. FastAPI, SQLite. Obviously there is a lot more thought that can go into the chat history storage, compaction, memory etc. to improve this kind of application, particularly one with many multiple users. Embedding / compressing chat streams is one little optimization. But simple local db worked for my purposes and improved on the JSON that my Claude tried before I asked for a db.

How it evolved

Three phases, each undoing part of the last.

Nov 2025 – Mar 2026: chat UI

Started as a plain chat page over a small Qwen model. Piled on features (model switching, dashboards, themes) until the main file hit 4k lines, then cut most of it back out.

Apr – May 2026: coding agent

Got ambitious. Rebuilt it as a coding agent with tools, a workspace, planning and an eval loop. Briefly tried a mascot and voice too. A lot of the machinery was there to work around a 4k–8k context window.

Oct 2026: back to simple

Gemma 4 made 262k context fit on one card, which made most of that machinery pointless. Replaced it with a lean helper app: drop in files, chat, remember things. The agent version is still there behind a flag.

Outcome

A lean helper app on Gemma 4 26B-A4B with 262k context and image input, on one RTX 4090.

Main lesson: I spent a lot of time building features and then deleting them. Most of the complexity was a workaround for small context. Once the context got big, simple won.

How good is it?

It works OK. I don't run it full time. It can't compete with any tier of ChatGPT or Claude, and getting close to what max plans offer would take a lot of money.

Good for: decent first drafts of code, and questions about books in its training data. Fast, and in some niche cases better than a Google search.

Not good for: detailed solutions. Ask for high-level guidance and it's useful. Agentic coding gets you a basic function that looks like a stock example, and that's about it.

What I'd do differently

  1. Pick a narrow problem. Instead of cloning a product the big labs already do well, choose one task and a small model and actually tune it: weights, serving settings, the stack. Go deeper on fewer things.
  2. Rent more compute. Run the same app on a cloud GPU (Blackwell or newer) with a bigger model.
  3. Use the small model for small jobs. I have editor macros that call the local model for things one step fuzzier than a docs lookup: explain this line, write a function that does X. That's where a 27B model is potentially useful.
  4. Tiny model for autocomplete. Quick predictions in the editor. Noisy, and big models do it better, but it saves some typing on repeated patterns.

Ultimately it was a fun project to work on and learn from.

Doubtful I was that my ai chat app would be good in any particular way compared to frontier products, I hoped to learn a lot. I used cursor and claude code together for most of the development, this allowed me to write & read code in cursor's editor software and use claude code on the server to debug runtime issues.

Building a productive development loop between my laptop and server encouraged me to use multiple agents and handle them with helpful tools like ssh, tmux and rsync. I found a loop that allowed me to develop and run an application while monitoring the production system and observing the changes required in various components of the codebase. This effort forced me to construct helpful mental models of application development in general and specifically for dev/prod systems, frontend/backend work split, and version control across multiple workers. Specifically w.r.t. AI chatbots and coding sandboxes I learned about certain biases of LLMs and what tools are available to developers to offer the benefits and mitigate the hazards of enabling users to interact with AI.

In the future I'd like to learn more about evals and other modern AI engineering concepts with a practice project.