Spinneret

Overview

Spinneret is two systems that need each other.

The first watches niche public sources for the moment a company commits money to something specific, then ranks those companies by how likely they are to buy right now, with the reasoning attached to every score.

The second keeps that feed honest. Scrapers break quietly, so after every run Spinneret measures what came back, compares it against a contract, and when the data stops matching it writes its own repair prompt from those measurements, sends it through Bright Data Scraper Studio, approves the patch and re-runs to prove the fix worked.

Built for Into the Scrape-Verse, the Bright Data and WeMakeDevs hackathon, in August 2026.

The Problem

Scrapers rarely fail loudly.

A site ships a redesign, a CSS selector stops matching, and the extraction code keeps running perfectly. Same row count, same schedule, same green job. The rows are simply empty.

Nothing in a normal pipeline notices. HTTP returned 200 and no exception was thrown. Three weeks later a sales team reports that leads dried up, and someone starts bisecting a month of data to find when the values turned into nulls.

A lead-signal product that silently goes stale is worse than no product at all, because it converts “I do not know” into “I am confidently wrong”. Freshness is the entire value proposition, so self-healing is not a bonus feature here. It is what makes the core promise true.

Goals of the Project

System Architecture

Four layers, with dependencies pointing in one direction only. Nothing in the domain core imports a database, a network client or a React component, which is what lets the self-healing decision logic be unit-tested with no I/O at all.

src/
├─ app/, components/     interface: dashboard, docs, HTTP routes, SSE feed
├─ services/             sentinel loop, scheduler, background jobs, read model
├─ adapters/, db/        the only modules that spawn the CLI or write SQL
└─ core/                 health, drift, prompts, signals, cooldown  (imports nothing)

The Self-Healing Loop

Seven stages. The first three run on every observation. The last four only run when the diagnosis found evidence worth acting on.

Key Features

The scraper for Remote GTM job board (https://weworkremotely.com/categories/
remote-sales-and-marketing-jobs) is returning incomplete data. Fields never
returned as keys in the output: company_name, job_title, location, job_url.
Re-derive selectors for company_name, job_title, location, job_url from the
current live DOM and return the full contracted schema: company_name,
job_title, location, job_url.

Four clauses, each doing a job: name the target, state the evidence with specific fields, say what to do and where to look, and restate the required schema so a partial repair is not mistaken for success. When a collector is only partially broken, a fifth clause appears automatically, fencing off the fields that still work so a rewrite cannot regress them.

None of it is simulated. The rows are ones those collectors genuinely returned, and the scoring, drift classification and prompt composition are the same functions the sentinel runs in production. It runs entirely in the browser, which is a direct payoff from the architecture: the scoring layer performs no I/O, so it ships to the client and works on the hosted snapshot with zero latency.

A value only counts if it carries information. null, empty strings, whitespace and the literal strings "null", "N/A" and "undefined" all count as missing, as do empty arrays and objects. Broken scrapers emit these constantly, and treating them as real data is exactly how a hollow pipeline passes as healthy.

The unreachable case matters more than it looks. A timeout returns nothing, so every field appears absent, which is indistinguishable from a total site rewrite. Healing in that state spends a cycle and hands the healing AI no live DOM to work from.

When no rule fires, no signal is emitted. Spinneret never invents intent it cannot justify, and there is a unit test asserting exactly that.

Live Collectors

Three collectors, deliberately different in shape so they exercise different parts of the health model: a paginated list, a JavaScript-rendered directory with an array field, and a single-subject changelog with no company column at all.

Every target is public, carries no personal data, sits behind no login or paywall and has no pre-built Bright Data scraper. LinkedIn was deliberately avoided precisely because it already has one, and using it would have meant not really using Scraper Studio.

Tech Stack

Database Design

Two details are worth noting. expected_fields is a contract written independently of the scraper, which is what lets the sentinel notice a gap nobody told it about. And signals.fingerprint is derived from what was observed rather than how it was worded, so re-observing the same job openings is silent while a genuinely new posting still creates a signal.

Application Flow

Challenges and Solutions

Best Practices

Deployment and Data Ethics

The hosted build is a showcase, not a control plane, and the header says SNAPSHOT rather than LIVE. A serverless filesystem is read-only, and a heal runs for five to twenty-five minutes shelling out to the Bright Data CLI, far outside any request lifetime. Rather than ship buttons that fail, the deployed build serves a committed snapshot of real runs and disables those actions with tooltips explaining why. The Contract Lab stays fully interactive, because the scoring layer needs no I/O.

Public sources only. No login-walled or paywalled content, and no personal data. Signals are scored at company level and derived from public job postings, directory listings and published changelogs. A job posting contributes its title and its employer, never a named person.

Conclusion

Most scraping projects treat breakage as an operational chore to be handled later. Spinneret treats it as the central design problem, because a signal product whose data quietly goes stale is worse than one that does not exist.

The result is a system that measures itself after every run, writes its own repair instructions from those measurements, and records the before and after so the claim can be audited rather than believed. One collector went from 20 to 97 that way, and the Contract Lab exists so anyone can reproduce the reasoning instead of taking my word for it.