Open data
Download the index
The index is replaceable. Everything it knows that a reader may know is published as a full export, a dump, that anyone can download and load into an index of their own.
The latest dump
When this page was built, the newest dump was the snapshot taken at . A new one is written every night at 04:30 UTC, and latest.jsonalways names the newest.
- sources.ndjson.gz
2 lines · 446 B
One line per source.
sha256 0bb6045c5ba3c73d6756f3b5ec2e8910b28f147fd80f6403d542554f8879ba8c
- entities.ndjson.gz
24 lines · 1.9 KiB
One line per entity, merged ones included.
sha256 1333fd9c883d4c4679a2288d874586caae364238233b04163aae8479bf72915c
- records.ndjson.gz
9 lines · 1.2 KiB
One line per record, withdrawn ones included.
sha256 cfdfb40fd5cdd941d65adbe559ca491261971ca328316741630d155e179a4600
- groupings.ndjson.gz
15 lines · 501 B
One line per grouping.
sha256 5c562f31473d5f2b8f8551caca0f1cbb2d2f3e35f542af1e2ee6e5f35f3f261e
- merges.ndjson.gz
0 lines · 20 B
One line per merge, reverted ones included.
sha256 59869db34853933b239f1e2219cf7d431da006aa919635478511fabbfc8849d2
This dump’s directory is https://dumps.prodid.org/v1/2026-10-29/, and its manifest is manifest.json, written at 2026-10-29 04:30 UTC. Its licences read index: CC0-1.0 and claims: source.
Where dumps are kept
A dump is a directory named after the UTC date of its snapshot, under the version it is written in. v1/ holds v1 dumps, whose manifest and whose source and entity lines are v1 documents. It holds five data files and a manifest.
https://dumps.prodid.org/v1/<YYYY-MM-DD>/manifest.json
https://dumps.prodid.org/v1/<YYYY-MM-DD>/sources.ndjson.gz
https://dumps.prodid.org/v1/<YYYY-MM-DD>/entities.ndjson.gz
https://dumps.prodid.org/v1/<YYYY-MM-DD>/records.ndjson.gz
https://dumps.prodid.org/v1/<YYYY-MM-DD>/groupings.ndjson.gz
https://dumps.prodid.org/v1/<YYYY-MM-DD>/merges.ndjson.gz
https://dumps.prodid.org/v1/latest.jsonlatest.json is written last. It is a copy of the newest dump’s manifest.json, written after every file of that dump and its manifest are in place. A reader that starts from it never finds a manifest naming a file that is not there yet, and finds the directory from the manifest’s own snapshot_at.
Retention. A daily dump is kept for 7 days. The first dump of each UTC month is kept for good. A dump that is kept is never rewritten: its directory, its files and its manifest stay byte for byte as they were.
v0 dumps. The v0 dumps written before v1 stay under https://dumps.prodid.org/v0/, under the same retention, and v0/latest.json keeps naming the last of them. They validate against the v0 schemas for good.
The manifest. It names each of the five files exactly once, with the SHA-256 and the size of the file as served, compressed, and its number of lines once decompressed. Check each file against its digest before you load it. The schema is https://prodid.org/schemas/v1/org.prodid.dump.json, with vectors in /spec/fixtures/v1/dump.
What each file holds
Every file is NDJSON compressed with gzip, one JSON object per line. All five are read from one consistent snapshot, so they agree with each other: every source a line names has its line in sources.ndjson.gz, and every opid a line names has its line in entities.ndjson.gz.
| File | One line per | The line |
|---|---|---|
| sources.ndjson.gz | source | The source’s status, exactly as /source/{name} answers it. |
| entities.ndjson.gz | entity, merged ones included | The entity document exactly as /id/{opid}?history=1 answers it, withdrawn claims included. |
| records.ndjson.gz | record, withdrawn ones included | source, key, feed, opid, body, first_seen, last_seen, withdrawn_at. |
| groupings.ndjson.gz | grouping | source, kind, key, opid. |
| merges.ndjson.gz | merge, reverted ones included | from, into, decided_at, reverted_at. |
Entities that were merged
An opid that was merged answers 308 at the resolver, so it has no document of its own to copy. Its line is the document it would answer were it not merged — its own identifiers and claims stay on it, which is what makes a merge reversible — with one more member, merged_into, the opid of its survivor. Merges do not chain: the survivor is never itself merged.
A record line
One record as the index keeps it. Every time is UTC, ISO 8601.
| Field | What it is |
|---|---|
| source | The source’s name. |
| key | The record’s key in that source. |
| feed | The URL of the feed it was read from. Absent for a record a dataset import wrote. |
| opid | The item it resolved to. |
| body | The org.prodid.item the publisher sent, unchanged. |
| first_seen | When the index first read it. |
| last_seen | When the index last read it. |
| withdrawn_at | Present exactly when the record is withdrawn. |
A grouping line
Which entity a source’s grouping resolved to. kind is family or variant, and key is what the source’s records are grouped on, written out rather than digested, so that another implementation can match its own grouping to it.
family
{ "group": "<group>" }
for a publisher group; { "key": "<key>" } for a record with no group, which is a family of its own.
variant
{ "family": <the family’s key>, "variant": { <axis>: <value>, … } }
the record’s variant, {} when it has none.
A merge line
One merge decision: from the merged opid, into its survivor, decided_at when it was made, and reverted_at, present exactly when it was reverted.
What a dump never holds
A dump says no more than the resolver already does, plus the records, groupings and merges its answers are built from.
- A publisher’s contact address, which the index keeps only to send the lapse notice.
- The crawl log, the crawler’s error text, or anything else about a crawl that the source’s status does not show.
- Why a merge was made or who decided it.
- Why a media hash was blocked.
- Media bytes. A media claim’s media_url addresses the file, and a blocked hash is not served.
- The index’s internal row identifiers. Every line names a source by its name and an entity by its opid.
- The grants of authority and their evidence. A source’s line says authoritative if it was at snapshot_at, but an index loaded from the dump holds no grant until its own operator makes one.
Licences
Two rules, and the manifest’s licenses states both.
index: CC0-1.0
Everything the index makes
opids, the identifier mappings, the family and variant hierarchy, groupings and merges.claims: source
Every claim and every record
Under the licence its own source declares: the
licenseof that source’s line insources.ndjson.gz. A source that declares none has granted nothing, and prodid grants nothing on its behalf.
An open dataset that publishes no discovery document of its own enters the index as a source named dataset:<slug>, imported by the index’s operator from the dataset’s public export. Such a source is always unverified, and its license is the dataset’s own. The rules are in the specification.
Load it into your own index
prodid-batch load-dump loads a v0 or v1 dump into an empty PostgreSQL 18 database migrated to the same version, in one transaction. It takes a dump’s dated directory, its manifest.json, or a latest.json, local or over HTTPS.
git clone https://github.com/productdevbook/prodid.git && cd prodid
bun install --frozen-lockfile
export DATABASE_URL=postgres://<user>:<password>@<host>:<port>/<database>
bun run --filter '@prodid/db' migrate
PRODID_OPERATOR=<your name> bun apps/batch/src/cli.ts load-dump https://dumps.prodid.org/v1/latest.jsonEvery file is checked against the manifest’s sha256, bytes and lines before anything is written, and a remote dump is downloaded to a temporary directory first. A database that is not empty is refused. PRODID_OPERATOR is recorded as the decider of the loaded merges.
To read a file without loading it, check it against its digest and decompress it. The digest below is the one in the latest manifest when this page was built.
curl -sO https://dumps.prodid.org/v1/2026-10-29/sources.ndjson.gz
sha256sum sources.ndjson.gz
0bb6045c5ba3c73d6756f3b5ec2e8910b28f147fd80f6403d542554f8879ba8c sources.ndjson.gz
gunzip -c sources.ndjson.gz | head -n 1The full rules are in the specification under Dumps.