# How the Thread-Surfacing Algorithm Works — and How It Could Be Better

---

## Part 1: How It Works Today

The algorithm answers one question: **"What needs Pavel's attention right now?"** It runs every time you open the dashboard (GET `/api/now`) and produces the Activity feed — the ordered list of threads on your NowDashboard.

The pipeline has six stages. Here's what each one does.

---

### Stage 1 — Collect threads from three sources

The system pulls threads from three independent places and combines them into one pool.

**Source A: Your MEMORY files** ([scanWorkspace](file:///Users/pavelracu/personal/codebase/head-of-product-ui-v2/server/lib/now.ts#L240-L309))

The system reads every `.md` file in your MEMORY directory — `MEMORY.md`, any `project_*.md`, and any `decision_*.md`. For each file, it scans every line looking for two things:

- **Date + status keyword together**: If a line contains both a parseable date (ISO format or "May 5, 2026" style) and a status keyword (`due`, `blocked`, `awaiting`, `decision needed`, `follow-up`, `must`, `owe`, `ships`, `launch`, etc.), the system classifies it by comparing the date to today:
  - Date in the past → `overdue`
  - Date is today → `today`
  - Date within 14 days → `upcoming`
  - Date further than 14 days → ignored entirely

- **Status keyword without a date**: Lines with a status keyword but no parseable date become `open` items — open loops that need attention but have no firm deadline.

- **Stalled projects**: Any `project_*.md` file that hasn't been modified in 14+ days gets a `stalled` entry. The system uses the file's actual filesystem modification time to detect this.

**Source B: Manual tasks** ([manual-tasks.ts](file:///Users/pavelracu/personal/codebase/head-of-product-ui-v2/server/lib/manual-tasks.ts))

Threads you've added manually via the "+ new thread" button. These live in a JSON file and follow the same classification logic — if they have a due date, they're bucketed by how that date compares to today. If not, they're `open`.

**Source C: Discovered threads** ([discovered-threads.ts](file:///Users/pavelracu/personal/codebase/head-of-product-ui-v2/server/lib/discovered-threads.ts))

This is the most interesting source. When you press "Scan now," a Claude agent runs and searches your active sources — Linear, Notion, claude-output files, recent chats, PostHog, Outlook/Teams — looking for work items that need your attention. It returns a JSON array of up to 30 items, each classified by kind.

These discovered threads are **persisted in a cache file** (`discovered_threads.json`). Each run upserts into the cache — it adds new items and refreshes existing ones, but never deletes. This means a single failed scan doesn't wipe your Activity feed. Items that haven't been re-surfaced by the agent in 14 days get flagged as `stale` and are dimmed in the UI but never auto-deleted.

The three sources are concatenated: `[...scanned, ...manualItems, ...discoveredItems]`.

---

### Stage 2 — Filter out things you've already dealt with

Four suppression filters run in sequence, removing items that shouldn't be shown:

1. **Resolved filter**: Items whose key appears in `resolutions.md` (the append-only log of how you resolved things) are hidden permanently.

2. **Active postpone filter**: Items with an unexpired snooze entry in `postpones.json` are hidden until the snooze expires.

3. **CLOSE verdict suppression**: If the auto-check agent gave an item a CLOSE verdict within the last 7 days, it's hidden. The idea is that the agent judged it "not actionable," so you shouldn't be bothered with it — but the suppression expires after 7 days so it can resurface if conditions change.

4. **Snooze fatigue**: If you've postponed the same item 2 or more times in the last 30 days, it's hidden. The reasoning: repeatedly snoozing something signals "stop showing me this for a while." This required a schema change from overwriting postpone entries to appending them, so the system can count how many times you snoozed each item.

**How items are identified across all these filters**: Every item has a key constructed as `file:line|first-80-chars-of-text`. This key is what links an item to its resolutions, postpones, check history, and linked chats. More on why this matters later.

---

### Stage 3 — Deduplicate across sources

The same work item might appear from multiple sources — a Linear ticket `AWS-456` mentioned in your MEMORY file AND surfaced by the discovery agent from Linear MCP. Without dedup, you'd see it twice.

[mergeThreads](file:///Users/pavelracu/personal/codebase/head-of-product-ui-v2/server/lib/thread-merge.ts#L34-L48) does two passes:

1. **Exact key match**: Same `file:line|text-prefix` → drop the duplicate.
2. **Cross-source reference match**: Regex extracts Linear-style issue IDs (`AWS-456`, `FE-12`), Notion page IDs from URLs, and Linear issue URLs. If two items share any of these references, only the first one seen survives.

Because the input is ordered as `[MEMORY items, manual items, discovered items]`, MEMORY items win the dedup — so your hand-written annotations are preserved over the agent's paraphrased versions.

---

### Stage 4 — Score and rank within each kind

Each surviving item gets a **weight** computed by [computeWeight](file:///Users/pavelracu/personal/codebase/head-of-product-ui-v2/server/lib/now.ts#L210-L222):

```
weight = (days_overdue × source_coefficient) + attention_signals
```

Where:
- **days_overdue** = `max(0, -daysOff)` — only positive for overdue items (0 for everything else)
- **source_coefficient** = 1.0 for all sources (uniform, waiting for calibration data)
- **attention_signals** = sum of three binary/count signals:
  - `pavel_mention`: +1 if the text contains "pavel" (case-insensitive)
  - `status_keywords`: count of distinct status words in the text (due, blocked, awaiting, etc.)
  - `escalate`: +1 if the latest auto-check verdict was ESCALATE

Items are then sorted **within each kind** by weight descending, and capped:
- overdue: top 20
- today: top 16
- upcoming: top 16
- stalled: top 16
- open: top 30

---

### Stage 5 — Decorate with metadata

Each item gets enriched with:
- **Check history**: The latest auto-check verdict (CLOSE/KEEP/ESCALATE) plus the full history of past checks
- **Linked chats**: Any conversations started via "Work on this" that are bound to this thread
- **threadKey**: The stable identifier used for all cross-references

---

### Stage 6 — Render the feed

The frontend ([NowDashboard](file:///Users/pavelracu/personal/codebase/head-of-product-ui-v2/src/components/NowDashboard.tsx#L214-L220)) receives the decorated items and displays them in a flat list, ordered by kind: all overdue items first, then today, then upcoming, then stalled, then open. Within each kind, they appear in the weight-sorted order from Stage 4.

Each row shows:
- A colored status dot (red for overdue, yellow for today, accent for upcoming, grey for stalled/open)
- Date, status label, timing marker ("3d late", "in 5d"), and source
- The weight as a small `w<N>` debug badge (the EXPERIMENT(weighting) Phase 2 output)
- The latest auto-check verdict if one exists
- Hover actions: resolve, check progress, postpone

---

## Part 2: What Could Be Better

Grounded in research on Gmail Priority Inbox, PagerDuty AIOps, GitHub Notifications, and prioritization literature (WSJF / Cost of Delay).

---

### The weight formula has a dead zone

The biggest concrete problem today is in `computeWeight`. The urgency component is `max(0, -daysOff)` — which means **only overdue items have non-zero urgency**. An item due today scores 0 urgency. An item due in 3 days scores 0 urgency. An item due in 14 days scores 0 urgency.

The attention signals (pavel_mention + status_keywords + escalate) range from 0 to maybe 3-4. So within the `today`, `upcoming`, and `open` kinds, ranking is entirely driven by "how many status buzzwords does the text contain." That's not a meaningful prioritization signal.

Meanwhile, a `stalled` project file untouched for 45 days gets `daysOverdue = 45` — dominating the score. An old project file that nobody's touched will always outrank a Linear ticket that's 2 days overdue.

**How it should work**: Cost of Delay / WSJF frameworks use a *continuous* urgency function that increases as a deadline approaches, not a cliff that activates only after the deadline passes. An item due tomorrow should score higher than one due in 12 days. Exponential decay (`e^(-daysOff/τ)`) is the standard shape — used by Gmail, Meta feeds, and scheduling literature — because it provides smooth urgency gradients without hard cutoffs.

---

### The most universal ranking signal is missing: recency

Every production system I researched uses time-since-last-touch as a primary ranking signal. Gmail uses it. Meta's feed ranking uses it. PagerDuty uses it. Linear's default sort uses it. Your own research doc says "recency wins in every system surveyed."

But `computeWeight` doesn't use any recency signal at all. The `mtime` field exists on stalled items but is never fed into the weight. Discovered items have `lastSeenAt` but it's only used for the binary stale/not-stale flag.

**The consequence**: A thread actively discussed yesterday and one that hasn't been touched in 13 days (just under the stale threshold) look identical to the ranker. They'll be sorted by status-keyword count, which tells you nothing about which one actually needs attention right now.

---

### Item identity is fragile

Every thread's identity across the system — resolve, postpone, check history, linked chats — is based on `file:line|first-80-chars-of-text`. This creates two classes of problems:

**Collisions**: Two items in the same file on the same line whose text shares the first 80 characters will silently share identity. Resolving one resolves both. This is especially likely for discovered threads, where every item gets `line: 1` hardcoded — so any two items from the same source/location pair will collide if their first 80 chars match.

**Drift**: When you edit MEMORY.md (changing an item's text), or when the discovery agent paraphrases a thread differently across runs, the key changes — and the item loses its connection to its check history, postpones, and linked chats. It appears as a "new" thread even though it's the same work item.

**How production systems handle this**: PagerDuty uses an explicit `dedup_key` supplied by the integration — a stable identifier that's independent of content. Gmail threads by conversation ID. Neither uses content-based keying for *identity*. Content is used for *grouping* and *display*, never for identity.

The discovered-threads cache partially solves this by preserving the canonical text from the first observation. But this only helps when the same synthetic file path is reused — if the agent generates a different path, a new entry is created.

---

### The dedup is "first wins" — not "best wins"

When `mergeThreads` finds a duplicate (two items sharing a Linear issue ID like `AWS-456`), it keeps whichever one appears first in the input array. Since MEMORY items come first, your hand-written annotations win — which is the right default.

But within discovered items, the order is non-deterministic (`Object.values()` of a JSON object). If the same Linear ticket appears from two discovery runs with different text, whichever entry was inserted first into the cache wins. The "better" description (maybe more recent, maybe with a useful URL in the location field) could be the one that gets dropped.

PagerDuty's approach: when merging, keep the *most recent* alert in the group as the representative, not the first one seen.

---

### There's no feedback loop

Gmail's importance threshold adjusts on every "Mark important / unimportant" click. PagerDuty uses resolution data to identify transient alert patterns. Linear uses every priority assignment as training data for its AI suggestion model.

Your system records every resolve, postpone, and "Work on this" click to disk — but none of those signals feed back into what surfaces next. The discovery agent doesn't know which sources you resolve from most often. The weight function doesn't know which kinds you act on fastest. The suppression rules don't adapt to your behavior patterns.

The EXPERIMENT(weighting) Phase 3-4 plan is designed to address this (observe for 2 weeks, then calibrate coefficients), but the outcome capture isn't happening yet. There's no structured log that says "this item was surfaced on May 5, Pavel resolved it on May 6, from Linear, kind=overdue" — the data needed for calibration.

---

### Filters would help more than ranking

This is the most counterintuitive insight from the research. GitHub manages notifications for ~150M developers and built **no ranking algorithm at all**. They built filters. Saved views like "review-requested, org:awssome, is:pull-request." Within a filter, sort is reverse-chronological. That's the entire algorithm.

The product insight: for a notification stream you actively check (email inbox, work dashboard), the user already has an opinion about what matters — they want a saved view that matches their current focus. For a stream you passively scroll (Instagram, TikTok), the user doesn't know what they want, and ranking is the product.

Your Activity dashboard is closer to the GitHub case. Adding 3-4 filter chips — "Linear only", "KEEP/ESCALATE verdicts", "Pavel mentions", "Last 7 days" — would likely do more for your daily experience than perfecting the weight formula, because it gives you direct control over what's surfaced rather than asking you to trust a score.

---

### The scheduler can't keep up with the item volume

The background auto-check runs every 4 hours and checks 4 items per tick. With discovery potentially surfacing 30 items and MEMORY contributing ~15, you have ~45 active threads. At 4 items per 4 hours, it takes 45+ hours to cycle through all of them. But a verdict's TTL is 6 hours — so by the time the scheduler revisits the first items, their verdicts are already stale.

The batch-check approach (one Claude prompt checking 4 items at once using Haiku) is cost-efficient, but the cadence doesn't match the item volume. The "Scan now" button compensates — it runs on demand with a "force" policy — but the background loop's purpose is to keep verdicts fresh without user action, and it can't do that at the current rate.

---

### Discovery has no budget allocation across sources

The discovery prompt tells the agent "6-15 tool calls TOTAL across all sources." But there's no guidance on how to allocate those calls. If Linear returns 20 items in 3 calls, the agent might spend its entire budget there and never check Notion, PostHog, or claude-output.

PagerDuty's Event Orchestration separates *routing* (which source to check) from *evaluation* (what did we find). The discovery agent conflates both in a single prompt with a single budget. A source-aware budget allocation — or running sources in parallel with per-source sub-tasks — would ensure coverage.

---

## Summary: Where The Algorithm Is vs. Where It Could Be

| Aspect | Today | Better |
|---|---|---|
| **Macro sort** | Kind hierarchy (overdue → today → upcoming → stalled → open) | ✅ Already good — matches production patterns |
| **Suppression** | 4-layer gating (resolved, postponed, CLOSE TTL, snooze fatigue) | ✅ Already strong — aligned with PagerDuty's approach |
| **Intra-kind ranking** | Weight formula dominated by days_overdue; today/upcoming score 0 | Continuous urgency function (e.g., `14 - daysOff` or exponential decay) |
| **Recency** | Not used in ranking at all | Time-since-last-touch as a signal in computeWeight |
| **Identity** | Text-prefix key (`file:line\|text80`) — fragile to edits and paraphrasing | Content-addressed hash or explicit dedup_key for discovered items |
| **Dedup** | First-wins across sources | Most-recent-wins or richest-metadata-wins |
| **Feedback loop** | Actions recorded but don't influence future surfacing | Outcome capture → coefficient calibration (planned but not started) |
| **User control** | Single flat list, no filters | Filter chips for source, verdict, recency (GitHub model) |
| **Scheduler throughput** | 4 items / 4h — 45h to cycle through 45 items | Higher throughput or shorter interval |
| **Discovery budget** | Single prompt, single budget, source-blind | Per-source allocation or parallel sub-tasks |
