About this dashboard

Every week, Halcyon publishes What's Happening in Energy — a digest of the biggest stories in U.S. power, policy, and cleantech. This site reads every issue, breaks each one into the individual stories inside it, and plots how the themes covered shift week to week.

The goal isn't to replace the newsletter — it's to answer the question the newsletter can't: what's actually trending across the last month, quarter, or year of coverage?

A Node script scrapes each newsletter article, extracts the title, author, and publish date, and converts the body from HTML to Markdown. Articles are hashed so re-imports don't accidentally clobber reviewed data.

A versioned splitter then chops the Markdown into paragraphs. The splitter is versioned because every paragraph's position is persisted — changing the algorithm for an existing article would orphan all the downstream work. New articles use v3, which keeps bullet lists attached to their introducing line and handles the publisher's odd whitespace markers.

Each paragraph is sent to Claude with a short prompt and a few examples. The model decides whether the paragraph is a discrete story worth tracking, whether it bundles with neighbouring paragraphs (a story can span an intro line plus a bullet list, or wrap around an embedded chart), and tags it with keywords (companies, technologies, projects) and jurisdictions (the states, regions, or countries the story is about).

The output is a draft. A reviewer opens the article in an internal admin UI, sees the AI's highlighted phrases overlaid on the text, corrects anything that's wrong, and approves. Only approved bundles — called excerpts — make it into the charts.

Each approved excerpt is embedded (a 1,536-dimensional vector via OpenAI's text-embedding-3-small) and stored in Postgres with pgvector. From there, three things happen offline:

• Theme scoring.Every excerpt is scored 0–100 against eight curated themes (large loads, storage, transmission, permitting, and so on). The highest-scoring theme above a floor becomes the excerpt's primary theme; everything else is implicitly “misc”.
• Sub-theme scoring.Within each primary theme, excerpts are scored against ~4–5 sub-themes — finer-grained pockets like “battery storage” or “interconnection reform”.
• Clustering.A Python pipeline runs BERTopic over the embeddings (after projecting out the jurisdiction axis with LEACE, so clusters aren't dominated by geography), and HDBSCAN groups the survivors into tight, story-shaped clusters. Each cluster gets a short LLM-written label, cached forever by member hash so re-runs are cheap.

The bullets at the top of the home page and each topic page aren't hand-written. For the window you've selected, the system finds the clusters whose weekly counts are growing fastest (a linear regression slope over the window), feeds the cluster labels and representative excerpts to Claude, and asks for a three-or-four-bullet summary in the newsletter's own voice. The result is cached per (window, prompt version, model) so the same view doesn't pay the LLM cost twice.

If the clustering pipeline doesn't cover a window, the system falls back to a simpler “biggest movers vs. the prior period” query. Same prompt, same cache, just a different payload shape.

The dashboard itself is Next.js (App Router, React 19, React Compiler) on top of Supabase. The six charts are powered by Postgres functions that aggregate excerpts inside the selected window; everything date-driven is just a URL parameter, so links are shareable. Charts use Recharts; the heatmaps are plain CSS grids — quicker to render, easier to make accessible.

The public site has no auth and only reads. The reviewer UI behind /admin is gated by a Supabase magic-link session against a short email allowlist.

• App. Next.js 16 (App Router) · React 19 · TypeScript · Tailwind v4 · Recharts
• Data. Supabase (Postgres + pgvector + RLS) · @supabase/ssr
• AI. OpenRouter for embeddings, classification, cluster labelling, and trend bullets — currently Claude Sonnet 4.6 and OpenAI text-embedding-3-small
• Topic pipeline. Python · BERTopic · UMAP · HDBSCAN · concept-erasure (LEACE) for jurisdiction de-biasing
• Ingest. Node CLI · Cheerio · Turndown

Built by Matan Nice. Source content remains the property of Halcyon. Back to the dashboard →