myrssfeed
Self-hosted RSS reader on a Raspberry Pi with feed scoring, theme labeling, newsletter ingestion and a web UI.
Problem
A friend sent me a Raspberry Pi 5 and I wanted to put something useful on it. Inspired by idea of a wholly owned CDN, asked myself what do I want to read on my private feed?
I like reading news from a lot of sources: Reuters, AP, WSJ, NYT and more. Podcasts kept advertising Ground News, which gave me the idea to build a news feed aggregator. Having used aggregators to review news in banking, I had vague notions of powerful search queries like Factiva and timely updates from Bloomberg. But I just wanted to make an MVP to understand how the pieces fit together in an application.
The result is was to create a rss feed collection backend on the Pi that hosted a JS frontend to private network, serving on LAN. It pulls RSS feeds and collects email newsletters into one library I can browse from any device.
Constraints
- Weak hardware. Not much RAM. This shaped the project more than anything else.
- Free data only. RSS feeds and newsletters. Paywalled sources are out of reach.
- Broad source list. Discovery for available rss feeds... there are some channels that still operate to this day.
- Runs on its own. Always on, refreshes on a schedule, and one part crashing can't take down the rest.
Key decisions
Where to run it
I was building on my laptop, but didn't want a pile of news files accumulating there. Although I had vague understanding of a server, setting up the Pi as one seemed interesting. It also seemed cheaper than renting, since I already had the hardware. And I understood this model to be relatively performant. I thought that meant it would be simple to set up.
Boy was I wrong. Electronics are finicky. They get hot and things break. You need a special power adapter to keep the Pi alive long term, cooling for consistent performance, a case so nothing touches it, and storage that won't wear out. I gave it an SSD to avoid microSD write errors and a little box with a fan.
This enabled the computer to function more or less 99.9999%, although none of these improvements enhanced the capabilities of the underlying chips. A fan may give linear improvement to speed of processing, but it will not scale down effort by multiples like bigger/better chips do. Laws of physics are such that running hardware sucks and a tiny Pi that can take all RSS feeds can't simultaneously do very much processing over the information contained in the RSS feeds.
Stack
FastAPI + SQLite. Easy Python, one process, no database server to babysit. Jinja for the HTML templates.
APScheduler for the background refreshes, nginx in front. I'd read about nginx years ago and figured that meant it was good.
Containing memory
At some point I thought having an LLM read and categorize every RSS item was a good idea. Most of the effort after that went into keeping it from taking the Pi down: running it in a separate capped process, writing data to disk instead of holding it in RAM, a lock so two refreshes can't run at once, and memory limits on the whole service.
I learned here it can be easier to buld one big process and make it cheaper / run it frequently rather than to manage many separate processes for separate events.
How it evolved
Early March: build big
Topic clustering, an LLM digest, topic maps, a full-article scraper. Then the reader: feed catalog, settings, mobile layout, scheduler controls.
Mid March: cut back
Deleted all the fancy stuff. Replaced it with simple rules for quality scores and theme labels. The reader now shows what the feed gives you, plus a link to the original.
April: clean up
Split the one big file into a proper package and added tests.
Outcome
A working reader in web app. Subscribe to feeds, browse a catalog, filter by date, quality, source or theme, mark things read or liked, and optionally pull in email newsletters. Refreshes are cheap and responsive.
The real problem: data quality
I tinkered with ML scoring, ranking, quality and source bias. But the main problem was the data.
RSS feeds are low quality. Of course they are: they're free and have been around for decades. Channels get stuffed with junk because it's cheap and easy to do.
No doubt, there are good RSS feed use cases, e.g. sources like EDGAR filings, podcasts, github repos, some blogs -- but not all of these benefit from a 'reader' application. Financial data can be compiled and analyzed across many APIs and RSS feeds but a better output is algorithmic trade execution or consolidated analysis with timely updates rather than a generic reader app.
Other good sources, like WSJ or the Financial Times, are behind paywalls. Having the computer pull articles automatically with a user account may be outside of ToS of an account. Why try to recreate the database that is already offered for a reasonable price from the publisher. Unless again the application goes beyond reader which the native websites do better than a weekend app.
Reading RSS is also a solved problem. Most readers work like email: mark as read, save for later, share. Mine was hard to get running on real hardware, worked fine as a reader, but wasn't much to look at. I didn't love reading rss so there was not a point in optimizing the experience of a reader.
What I'd do differently
- Start smaller and perfect a core loop. ML tricks did not improve the application. Simple rules for scoring and labels were good enough. They should have been the first version, with ML only if they fell short if provided lots of data. Instead I wasted time and tokens pursuing a matchbox magic trick.
- Size features to the hardware. Most of March went into building ML features and then reigning them in. A quick memory check up front would have ruled them out. I am personally in favor of building 100%, cutting 80%, bringing back 20%-style development to learn what is essential and make systems modular and robust to change. Putting that effort in the right direction before expending work and time and budget would be a step worth taking. And writing goal->plan->outcome loops can help ensure current work is focused properly on the immediate priority.
- Organize early. Splitting my big monofile project came late. Knowing where to split ideas across files, directories, worktrees, etc. requires judgement from experience. Somewhere between 1 and 10 files in a project it becomes useful to organize concepts across files in a familiar fashion. I should have kept this top of mind from the jump. Struggling with what backend really was became useful once I acknowledged that backed was just a computer humming along which let me bring to bear old knowledge to employ judgement for the new situation.