Deterministic first
The model is used only for semantic judgement: relevance, vertical, strength, and evidence. Dates, deduplication, clustering, and ranking are plain code, so they are fast, testable, and identical on every run.
Deal Origination Framework (DOF)
A weekly business-signal pipeline for 537 enterprise accounts: collect from newsrooms and news, deduplicate syndicated copies, classify each event once, and rank the top two per vertical.
- Role
- Built
- Where
- Zinnov (Draup)
- When
- 2025 to 2026
- Access
- Work · private repo
Problem
Every week a deal-origination team has to spot business signals (leadership changes, partnerships, acquisitions, investments) across hundreds of enterprise accounts, then pick the strongest two per industry vertical for a dashboard and a deck. By hand that means visiting every newsroom, searching the news, throwing away syndicated copies of the same story, and judging relevance one article at a time.
01
A fixed weekly window
Each run loads the account list and the signal taxonomy and computes the previous Monday to Sunday in one timezone. Discovery looks back 14 days so late-listed releases are still caught; a late item is tagged, never dropped.
02
Fan out per account
Every account gets a newsroom-discovery task and a news-search task in a persistent SQLite queue that workers claim atomically. Newsroom discovery diffs links against a cumulative URL inventory, so only new URLs move on; news-search links are decoded to the real publisher URL.
03
Fetch once, date carefully
All fetches share a pool of at most four long-lived browsers, and HTML or PDF becomes Markdown. The publication date comes from a nine-step ladder, from structured data down to an LLM as the last resort, and every date records its source and confidence.
04
Validate without deleting
Each document gets a window status (in window, outside, unknown date, late arrival) and a company-match check. Rejections are kept as statuses rather than deleted, and a barrier waits for every collect task before moving on.
05
Syndicated copies become one event
Four dedup layers run per account: canonical URL, content hash, title similarity, and near-duplicate content. Union-find then merges syndicated copies into a single event, with the official release as the canonical document.
06
One LLM call per event
The model is called once per event, not per article: ten copies of a story cost one call. A single response carries relevance, vertical, signal strength, a summary, verbatim evidence, and confidence, validated against the taxonomy with one repair turn and one fresh retry.
07
Uncertain events go to an analyst
Low confidence, a vertical mismatch, or a failed validation sends the event to review. Analysts work in an exported spreadsheet with taxonomy dropdowns, and importing their decisions rebuilds the ranking.
08
Rank, report, and prove it
A deterministic sort picks the top two per vertical (strength, confidence, source count, official source, recency), at most one per account. Reports are written atomically, and a run is only marked complete when report row counts match the database.
Measured
- Enterprise accounts monitored weekly
- 537
- Industry verticals ranked
- 8
- Companies processed across 11 production runs
- 760
- Tests in the pipeline
- 97
- LLM call per event, however many copies
- 1
How it's verified
- Harness
- Persistent SQLite task queue with atomic claims, barriers between phases, immediate persistence, and pause and resume.
- Evals
- A one-week real-data run over 20 accounts (591 tasks, 383 documents, 27 events) passed reconciliation; the classifier was stubbed in that run.
- Guardrails
- LLM output is validated against the signal taxonomy with one repair turn and one retry; low confidence or a vertical mismatch goes to an analyst; a run cannot be marked complete unless report counts match the database. In the weekly watcher, Gemini verdicts are re-checked against a three-condition rule and the nine-category list, and unreadable answers stay pending rather than guessed.
- Testing
- 97 tests.
In depth
No silent data loss, no false done
Phases are separated by barriers, every result is persisted as soon as it exists, and runs can pause and resume.
- Rejected documents become statuses, never deletions
- A reconciliation mismatch marks the run FAILED instead of complete
A real-data validation run
A one-week run over the first 20 accounts produced 591 tasks (569 succeeded, 22 failed), 383 documents, and 27 events, and reconciliation passed. That run stubbed the classifier, so it validates collection through clustering, not classification quality.
Browsers as a fixed pool
A browser per page was rejected for cost, and multi-tab control would have depended on undocumented driver internals. A pool of at most four long-lived browsers, one per worker, keeps fetching predictable.
Add-on: the weekly press-release watcher
A second path runs every Monday over about 760 companies and covers exactly the previous Monday to Sunday. It reads each company newsroom first, then news search, validates only the articles published in that week, and writes two reports: a valid-only deliverable and a full audit file where every rejected row keeps its reason.
- Newsroom runs before news search, so the company's own copy wins and the media copy becomes the duplicate
- Three duplicate layers on every article: URL already seen, identical content hash, near-identical title to a recent release
- Publish dates come from the URL or the visible newsroom text and are never guessed; an impossible date counts as unknown
Fixing a noisy news search
About 45% of news-search rows were invalid: category pages, stock trackers, cookie pages, other companies. Adding movement keywords to the query made the search ignore the date range (0 of 89 results fell in the week), while a name-only exact-phrase search kept 93 to 98% in range. So category filtering moved to our side.
- Company names are cleaned of legal suffixes, with a short form and bracket acronym searched too
- Proper encoding fixed 26 companies, including names with an ampersand, that had been returning unrelated news
- A headline must name the company, carry a movement word, not be stock chatter, and fall in the week; only the best 5 per company are kept
- Redirects are decoded last, after filtering, because decoding is rate-limited; a 7-company test ran in 42 s instead of 141 s
HTTP first, Chrome only where needed
Chrome costs 5 to 60 seconds a page and some sites hang; plain HTTP takes about a second but can silently return a partial page. HTTP is trusted per site only once it has matched Chrome, and the site falls back to Chrome automatically when it stops matching.
- Newsroom: HTTP must find at least 95% of Chrome's links; dropping below 80% of its own earlier count, or hitting a block page, sends it back to Chrome
- Article: HTTP text must reach 90% of Chrome's on that site, and every page still needs 500+ characters, no block page, and no JavaScript-only shell
- If Chrome fails, an HTTP result is used only when it passes the same checks; otherwise the page stays pending
Validation that re-checks the model
Each article gets a fresh Gemini chat and must come back as JSON with company match, one of nine signal categories (or NONE), a framed headline, the news date, confidence, a reason, and a verbatim evidence sentence. A row is valid only when the company is a direct party, the category is a real signal, and the event is announced, not rumoured or recapped.
- Valid with no company match, or valid with NONE, is turned into invalid
- A category outside the nine becomes NONE; an invalid row never keeps a headline or category
- Unreadable JSON is asked again in a new chat, then left pending for the next run, never guessed
- Reason, evidence, confidence, and the raw answer are stored so a reviewer can confirm a row in seconds
Resumable by re-running the same command
Every scraped URL is saved to state immediately, so a crash, Ctrl+C, or power cut loses nothing: re-running the same command skips finished companies and retries failed ones, and a flag skips discovery when only scraping is left.
- Search failures retry after 20 s and then 60 s, fixing a network burst that had failed 759 companies in one run
- Rate limits pause with growing waits, up to 30 minutes
- Input headers are matched case-insensitively with the BOM skipped, after one account list had loaded zero companies without an error
Decisions
- 01Classify events, not articles: syndicated copies collapse first, so the model sees each story once.
- 02The taxonomy is enforced by a validator, never trusted from the model output.
- 03Analyst review happens in a spreadsheet the team already uses, with dropdowns generated from the taxonomy.
- 04Search news by company name only and filter headlines in code: adding keywords made the search ignore the date window.
- 05Trust plain HTTP per site only after it matches Chrome; slower and complete beats faster and missing news.
- 06Deliver valid signals only, but keep every rejected row with its reason in a separate audit report.
Stack
Python · Botasaurus (Chrome pool) · trafilatura · PyMuPDF · gnews · RapidFuzz · Pydantic · SQLite (WAL) · openpyxl · Gemini · pytest