Nirpekha Barta: AI Unbiased News Aggregation

What I chose, and what I rejected

The ordering of the pipeline is the design. Everything else follows from it.

Why clustering has to come before summarisation

It would have been easier to summarise each article as it arrived and group the summaries afterwards. One less stage, no clustering step on raw text, and the model cost spread evenly through the day instead of spiking.

That approach quietly destroys the product. Once an article has been reduced to a summary, the differences in framing are already gone — summarisation is lossy in exactly the dimension this project is about. Two papers describing the same event with opposite emphasis produce two summaries that sound alike, because the summariser is normalising tone as a side effect of its job.

Worse, clustering summaries groups articles that sound similar rather than ones that describe the same event. Two unrelated stories written in the same register cluster together; the same story told angrily and told neutrally drifts apart. The signal you are clustering on stops being the event.

So clustering happens on raw text, before any model touches it for summarisation. It is the more expensive ordering and it is the only one that produces the thing I set out to build.

Bias scored per source, not per summary

The lazy version of this product asserts its own neutrality: here is the neutral summary, trust us.

That claim is unfalsifiable, and a reader has no way to check it — which is precisely the position they were already in with any single newspaper. Swapping one unverifiable authority for another is not an improvement.

Scoring each contributing source independently and showing those scores next to the summary inverts it. The reader is not asked to trust that the summary is neutral; they can see the spread it was drawn from, and how far each source sits from it. The transparency is the argument, not the neutrality claim. If they disagree with the summary, they can go read the source that leans the way they suspect.

It also keeps the system honest about its own limits. A bias score is a model output with error bars, not a fact. Presented as one signal among several visible ones, that is fine. Presented as the basis for a hidden decision about what you get to read, it would not be.

Bangla is the hard part, and it is the point

Most of the retrieval and summarisation tooling that works well is tuned for English. Bangla has less training data, fewer quality embeddings, and tokenisers that fragment it more aggressively — which means more tokens per sentence, tighter context budgets, and worse similarity behaviour at the clustering stage.

That is also why the project is worth doing. An unbiased-news aggregator for English is a crowded space with good tooling; for Bangladeshi newspapers it does not exist, and the reason it does not exist is this difficulty. The constraint is the moat.

Async because it has to survive without me

Running unattended on a VPS was a constraint rather than a goal, and it is what forced clean stage boundaries.

Each stage — collect, cluster, bias-score, summarise, publish — is a separate Celery task, so a slow scrape cannot block publishing and a failing source cannot take the run down. Sources are unreliable in uninteresting ways: a site is down, a layout changed, a page times out. If collection were synchronous with publication, one bad newspaper would mean no news that day.

Redis carries the queue and Postgres holds the stories and scores, which splits ephemeral coordination from durable state — the queue can be lost and rebuilt without losing a published story.

The property that matters is that anything requiring a manual retry would have stalled the whole thing. A pipeline that needs a human is a pipeline that stops the first week you are busy. Designing so every stage retries independently is what makes “runs unattended” true rather than aspirational.

What I would do differently

Clustering on raw text is correct and expensive. The stage that most limits quality is bias scoring, because it is the least verifiable: a summary can be checked against its sources, but a bias score has no ground truth to check against, only inter-rater agreement I do not have raters for. If I extended this, that is where I would start — not with better summaries, which are already adequate, but with making the scoring auditable.