Specification

prodid spec v1

Status: v1.0.0, 2026-10-28. Stable: it changes only by addition, as Versions says.

Versions

This is version 1 of the specification. What changed from v0, and why, is listed in v1-from-v0.md.

Every document this specification defines says which version it is, and a reader validates it against that version’s schema:

A feed record has no version of its own: the v0 and v1 schemas of org.prodid.item accept exactly the same records, so a feed reads the same under either. The schemas of each version are served at https://prodid.org/schemas/v<N>/<$type>.json, which is also each schema’s $id. Both versions stay served for good, because other documents and validators hold those URLs as names.

What the index accepts. A discovery document may be v1, or v0 until a cut-off: a crawl that starts before 2027-01-26T00:00:00Z, 90 days after v1.0.0, accepts a v0 document, and every crawl that starts at or after that instant refuses one, exactly as it refuses any document that does not validate. That is a definite negative: the source lapses, its publisher.contact is notified, and the 30 days before its claims are withdrawn begin. Moving a document to v1 is one edit, "version": 0 to "version": 1; nothing else in a discovery document, in a feed or in a _prodid record changes.

What the index answers. An index answers in the version it implements, and an index implementing v1 answers v1: every entity, every source status and every dump manifest carries "version": 1, and so does every line of a dump’s sources and entities files. There is no version negotiation. The paths /id, /01 and /source, the problem types and the media URLs carry no version and do not change. A dump that is kept stays in the version it was written in (Dumps).

The _prodid record keeps v=prodid0. The tag is the version of the record, not of this specification, and v1 defines the same record as v0, so a domain that delegates has nothing to change. A record version changes only with a record that a reader of the old one could not parse, and never lightly: a reader that does not know a version ignores the record, and a domain whose only record is ignored falls back to .well-known on the domain, the weaker proof.

Within v1. v1 changes only by addition: a new optional member, a well-known identifier scheme, a problem type, a vector. Every document valid under an earlier v1.x stays valid under a later one and means the same; a document that uses an addition is valid from the v1.x that added it on. The v1 schemas are updated in place under /schemas/v1/, and version stays 1. Anything else, a valid document becoming invalid or a member changing its meaning, is v2: new schemas under /schemas/v2/, "version": 2, and a stated period in which v1 documents are still accepted, as v0’s 90 days are here.

Identity

Every entity in the index has an opid: 16 characters of lowercase Crockford base32, assigned by the index.

^[0-9a-hjkmnp-tv-z]{16}$

An opid is opaque. It encodes no brand, category, country or time, and must never be parsed. It is never reused and never deleted; when two entities are merged, the losing opid keeps resolving and redirects to the surviving one.

Entities have one of three kinds:

A barcode is not a product. GTIN, MPN, ASIN, UDI-DI and every other code are identifiers that resolve to an opid.

Identifiers

An identifier is a { scheme, value } pair. Schemes are lowercase. Well-known schemes:

Unknown schemes are accepted and stored.

mpn, sku and rcn are the scoped schemes: a value of one of them means nothing without the scope it belongs to, so a reader is always given both. The scope a reader sees is the normalized brand for mpn and the source’s current name for sku and rcn. Every other scheme is global and carries no scope at all.

Which GTINs are identifiers

A correct check digit is not enough to make a number a global identifier.

• All zeros is not a GTIN, at any length. Catalogues use it as a “no barcode” placeholder, its check digit is valid, and taking it at face value would merge every product that carries it into one entity. A record whose gtin is all zeros is invalid, like a record with a wrong check digit.
• Restricted circulation numbers are not global. GS1 reserves ranges for numbers whose meaning is local to a company or a region, so the same value means different things at two sources. The index stores them under the scheme rcn, scoped to the source, and never merges two sources on one. /01/{gtin} answers a restricted number with a problem response instead of resolving it.

Classification reads the normalized 14-digit form, never the spelling: the index stores only that form, so the same number has to classify the same way however it was written. A 14-digit form that begins with six zeros is a GTIN-8 — GS1 leaves the GTIN-13 prefixes 0000001–0000099 unused to avoid exactly this collision — so the GTIN-8 rule governs it whether the publisher wrote 8, 13 or 14 digits.

01234565 is a restricted GTIN-8; written out as 0000001234565 or 00000001234565 it is the same restricted number, which a rule keyed on the written length would miss. The GS1 prefix 0000000 needs no row of its own: a 13-digit number beginning with it has eight leading zeros once normalized, so the first row reads it as a GTIN-8 beginning with 0, which is restricted as well. The two readings agree, which is the point.

Verified against the GS1 General Specifications, figures 1.4.2-1, 1.4.3-1 and 1.4.5-1: GS1 Prefix 02 and 20–29 are “used to issue Restricted Circulation Numbers within a geographic region”, 04 and 0000000 “within a company”, and the GS1-8 Prefix ranges 000–099 and 200–299 likewise. 05 is reserved for future use, not restricted, and is treated as an ordinary GTIN.

Vectors: fixtures/gtin/, one file per case as { "value": "...", "verdict": "gtin" | "rcn" | "invalid" }.

Brand normalization

mpn is scoped to the brand, and a brand written two ways is two scopes, so the brand is normalized before it is used as a scope. The normalized form is, in order:

1. Unicode NFKC;
2. lowercase, locale-independent;
3. ı (U+0131) becomes i, and every combining dot above (U+0307) in the run directly following an i is removed — the whole run, not one;
4. runs of whitespace collapse to one space, and the ends are trimmed.

Step 3 exists because locale-independent lowercasing turns İ (U+0130) into i + U+0307: without it ARÇELİK and Arçelik are two brands, and IŞIK, Işık and ışık are three — in this project’s first market.

Normalizing a normalized brand must return it unchanged. That is why step 3 takes the whole run: removing one dot at a time would make İ̇lik land on i̇lik and only then on ilik, and since the normalized brand is the mpn scope, a second pass would quietly move every mpn that brand ever published.

The normalized form is what scope carries in a resolver response; the brand as the publisher wrote it stays in the brand claim. Vectors: fixtures/brand/, as { "input": "...", "normalized": "..." }.

Publishing

A source’s identity is a domain it controls. Hosting is separate from identity: where the documents are served is delegable and can change without changing the identity or any opid. A source is permanent; it lapses, it is never deleted.

Discovery

Discovery runs from scratch on every crawl, DNS included. Nothing about the path to a source is remembered between crawls, so a delegation that is removed is gone at the next crawl.

1. DNS TXT at _prodid.<domain>. If a v=prodid0 record exists it is followed exclusively, even when it names the domain itself. The record names a host; the document is always https://<host>/.well-known/prodid.json.

   _prodid.acme.example. IN TXT "v=prodid0; host=acme.prodid.org"

2. https://<domain>/.well-known/prodid.json, used only when no v=prodid0 record exists at _prodid.<domain>.

DNS comes first because it is the stronger proof: whoever can put a file on a host is not always who owns the domain, and the owner must be able to override that. There is no fallback from step 1 to step 2. A record that exists but is malformed, over-long or duplicated ends discovery; it never sends the crawler to the weaker path.

A step succeeds when all of this holds:

• HTTP 200, after at most the one permitted redirect;
• a JSON content type: application/json, or any application/…+json;
• a body that validates against org.prodid.feed in a version the index accepts;
• the domain rule below.

Anything else is not an answer: a 404, a 200 carrying an HTML fallback page, a single-page app that answers every path, a document served as text/html.

The domain rule. A document reached through a TXT record whose <host> is not the domain itself must carry "domain": "<domain>", equal to the domain being discovered. A delegated document without it is rejected — absence is never taken as consent. domain may be absent only when the domain serves its own document, in step 2 or through a record naming the domain itself; when it is present there it must still equal that domain. A <domain> → www.<domain> redirect is inside step 2 and is not delegation, so it does not make domain required.

A document whose domain names another domain, and to which that domain does not delegate, speaks for nobody. It is ignored, and nothing is published under either domain because of it.

The _prodid TXT record

An RR at _prodid.<domain> is a prodid record when its value, after concatenation, is a tag list whose first tag is v with the value prodid0. Every other TXT RR at that name — verification tokens, v=spf1, a v= version this spec does not define — is ignored. prodid0 is the version of the record, not of the specification, and v1 keeps it.

The character-strings of one RR are concatenated in order, with no separator, before parsing. The concatenated value is at most 512 bytes; a prodid record over that is malformed, not ignored.

record  = *WSP version *( sep tag ) [ sep ] *WSP
version = %s"v" *WSP "=" *WSP %s"prodid0"
sep     = *WSP ";" *WSP
tag     = name *WSP "=" *WSP value
name    = LOWER *( LOWER / DIGIT / "-" )
value   = *VALCHAR
LOWER   = %x61-7A
DIGIT   = %x30-39
VALCHAR = %x21-3A / %x3C-7E   ; printable, no space, no ";"
WSP     = SP / HTAB

The grammar is necessary, not sufficient. In addition:

• v=prodid0 is the first tag. A value whose first tag is anything else is not a prodid record, ignored rather than rejected.
• Tag names are lowercase. A tag name that appears more than once makes the record malformed.
• Unknown tags are ignored; a later version may define more.
• host is required. A prodid record without it is malformed.
• host is normalized before it is read: lowercased, and one trailing dot removed. DNS names are case-insensitive and the absolute form with the trailing dot is what many DNS panels write, and under the lapse rule a capital letter would otherwise cost a publisher its catalogue. ACME.Prodid.org. and acme.prodid.org are the same host.
• The normalized value must then satisfy the same rule as the document’s domain field: at least two labels, at most 253 bytes in total, the last label starting with a letter, and therefore never an IP literal. It is ASCII: punycode A-labels such as acme.xn--p1ai are hostnames like any other, a U-label such as açme.example is not, and a second trailing dot is not.
• The domain field inside the document stays strictly lowercase, with no trailing dot. Software writes it; a DNS panel does not.

Because the record names a host and the path is fixed, there is nothing in it that can express a port, a path, a query, a fragment, userinfo or a scheme; a value carrying any of them is simply not a hostname, so the record is malformed.

Delegations, every one of them to acme.prodid.org:

v=prodid0; host=acme.prodid.org
v=prodid0;host=acme.prodid.org
  v=prodid0 ;  host = acme.prodid.org
v=prodid0; host=acme.prodid.org;
v=prodid0; host=acme.prodid.org; contact=data@acme.example
v=prodid0; host=ACME.prodid.org.

The last one is the normalization at work: the case and the trailing dot are how a DNS panel writes the same name.

Malformed, so discovery fails:

v=prodid0
v=prodid0; host=acme.prodid.org; host=evil.example
v=prodid0; Host=acme.prodid.org
v=prodid0; host=açme.example
v=prodid0; host=acme.prodid.org..
v=prodid0; host=192.0.2.10
v=prodid0; host=acme.prodid.org:8443
v=prodid0; host=https://acme.prodid.org
v=prodid0; host=acme.prodid.org/.well-known/prodid.json

Not a prodid record, so ignored:

host=acme.prodid.org; v=prodid0
v=prodid1; host=acme.prodid.org
v=spf1 include:_spf.example ~all

Machine-readable vectors, one file per case: fixtures/txt/.

Feeds

The discovery document (schema, example, delegated example) lists the feeds, each served from the same host as the document itself. Both hosts are normalized the same way as a delegation’s host before they are compared. A document listing a feed on any other host, or listing more than 100 feeds, is rejected whole: no feed from it is fetched, and discovery ends with no acceptable document. Serving feeds from a separate CDN hostname is not supported in v1.

A feed is NDJSON: one org.prodid.item record per line (one record, a feed). The index validates every line and records each field as a claim made by the source domain.

A line that is over the size cap, unparseable or invalid is skipped and counted. Because a skipped line’s key is unknown, a feed with skipped lines withdraws nothing — but only for seven days. After seven days of crawls that keep skipping lines, absent keys are withdrawn as usual: a broken line may delay withdrawal, it may not disable it. The count is public in the source’s status, so this never happens silently.

A feed that cannot be fetched at all is a different failure and has its own bound: the crawl becomes a transport failure, which lapses the source after seven days of it.

A feed the document stops listing is retired. When a crawl reads an acceptable document that no longer names a feed URL, that feed’s live records are withdrawn — exactly as if every one of its keys had disappeared — and the URL is not fetched again. Listing it again later brings it back as an ordinary feed, read from scratch. A retired feed is never a failed feed: it was not fetched, so it cannot have failed, and it never counts in feeds_failed.

Retiring withdraws at once and the seven-day grace does not apply to it. The grace exists for a feed that is still listed and whose lines the crawler could not read, where the publisher may not know anything is wrong; removing a URL from the document is the publisher saying so on purpose, and the index takes it at its word even if that feed’s last crawl skipped lines.

A 304 on the discovery document retires nothing. The document has not changed, so the list in it has not changed; only a document that was read and parsed can retire a feed.

Three rules make a record invalid that the schemas cannot express, and a record that breaks any of them is skipped like any other:

• Every string must be storable. No U+0000, and no lone surrogate: the string has to be well-formed Unicode. RFC 8785 makes a lone surrogate an error when canonicalizing, and the index’s store rejects both, so a record carrying one cannot be kept at all — and one such line must cost its own line, not the whole ingest.
• A gtin identifier must be a GTIN: 8, 12, 13 or 14 digits, a correct check digit, and not all zeros. Which GTINs are identifiers has the rest, including restricted numbers, which are stored rather than rejected.
• A record must carry at least one identifier the index can store. An identifier value is not blank and is not padded — the schema requires at least one character that is not whitespace, and no whitespace at either end — and an mpn needs the record’s brand to normalize to something, because that is the scope it is unique within. A record with nothing usable left identifies nothing, so matching it would mint a fresh entity on every single crawl; it is skipped instead.

fixtures/ndjson/ is a feed with one of each defect and the line numbers that must be skipped.

Three ways to publish

All three produce the same records.

A prodid record wins over a file on the domain, so a domain that serves its own document must not delegate elsewhere — and, as Security considerations explains, should say so with a record of its own.

A name under the hosting zone is a case of its own: <slug>.prodid.org is always a hosted source and always unverified, however discovery reached it. Pinging a slug runs discovery like any other name, and the document it finds there is served from prodid.org’s domain, so it proves control of a slug and nothing more. Only a domain’s own _prodid record, naming that host, turns the slug into a verified source.

Sources that are not the brand owner — retailers, registries, open datasets, contributors — publish the same way. Their claims are attributed to them and never overwrite anyone else’s.

Dataset sources

An open dataset that publishes no discovery document of its own enters the index as a source of kind dataset: the index’s operator imports its public export.

• Name. A dataset source is named dataset:<slug>. The slug is 3 to 32 lowercase letters, digits and hyphens, begins and ends with a letter or a digit, and has no -- in its third and fourth places — the grammar of a hosted slug. A hostname cannot contain :, so a dataset’s name never collides with a domain or a slug host. /source/dataset:<slug> answers its status like any other source’s.
• Trust. A dataset source is always unverified. An import proves nothing about who is speaking: the operator mapped someone else’s export into records, and nobody at the dataset took part.
• Import. The export is mapped into org.prodid.item records, and those are ingested exactly as a feed’s are: the same record rules, the same claims, identifiers and grouping. A full import withdraws the keys absent from it, as a feed fetched whole does. Each import is reported as the source’s last_crawl, with via set to import; A source’s status says what else it carries.
• Licence. The source’s license is the dataset’s own, and every claim the source makes carries it.
• No discovery. A dataset source is never discovered, so it is never pinged and it never lapses. dataset:<slug> is not a hostname, and a ping naming it is an invalid-request like any other name that is not one.

Trust

Trust is a property of the source, not of a claim.

Only a source whose identity is a domain can be more than unverified. A <slug>.prodid.org host is prodid.org’s domain, not the publisher’s, so serving a document there proves nothing about the publisher: a hosted source stays unverified however well it publishes, until a domain’s _prodid record names that host and the document names the domain back. That is the moment it becomes that domain, and the same moment its trust rises.

authoritative is granted on evidence, per brand, and can be revoked. No crawl can prove who owns a brand, so the index’s operator grants it, and only on evidence: the GS1 company prefix of the GTINs the source publishes under that brand is licensed to the organization that controls the source’s domain, as GS1’s public “Verified by GS1” lookup shows, and the URL of that lookup is kept with the grant as its evidence. A grant names one brand, in its normalized form. A source may hold grants for several brands, and the operator revokes each on its own.

A grant does not outlive the proof under it. A source is authoritative exactly while control of its domain is proven, as verified requires, it is not lapsed, and it holds at least one grant that has not been revoked. A source that loses that proof, or that lapses, drops at once to the trust its remaining proof supports — verified while it is lapsed, lower if that proof is gone too — with no operator action, and it is authoritative again as soon as all three hold again. Trust stays a property of the source: a source that holds a grant for one brand is authoritative in every claim it makes, including its claims about brands it holds no grant for.

Crawling

There is no publishing API key and no account is needed to be indexed:

POST https://prodid.org/ping
Content-Type: application/json

{"domains": ["acme.example", "beta.example"]}

{"domain": "acme.example"} is the single-domain form; a request carries one form or the other, and at most 1000 domains. A ping asks the index to run discovery for those domains now. It grants the caller no say over content, but it does make the index issue DNS queries and HTTP fetches on the caller’s word, so the crawler is built as if every ping were hostile.

That is the whole vocabulary, and 202 is all it says. It does not report whether a domain is known, whether anything was found, what was fetched, what changed, or whether someone else pinged it a minute ago: a domain already inside its own crawl window is folded into the schedule it already has, and the answer stays 202. A request naming an invalid domain is rejected whole rather than per domain, so no answer varies with anything except the caller’s own request. Crawl results reach the public only as the source’s own status.

Fetch rules, for pinged and scheduled crawls alike:

• https only, on the default port;
• public DNS names that resolve to public addresses; never an IP literal, and a name that resolves only to private, loopback, link-local or unique-local addresses is not fetched;
• at most one redirect, and only <domain> → www.<domain> with the same scheme and path; any other redirect is not followed;
• a delegation names a host and the document path is fixed, and feeds are always on the document’s own host, so neither a TXT record nor a document can aim the crawler at an arbitrary URL.

Caps:

Rate limits:

A caller is a client address: an IPv4 /32 or an IPv6 /64.

The per-host budget has two lanes, and both grow with v, the number of sources currently verified at that host:

The verified lane is reserved: discovery for strangers cannot consume it, so pointing thousands of domains at a publisher’s host cannot starve that publisher’s own crawl. Growth is safe because v counts only sources whose discovery has succeeded at that host, which means a document there named them and they agreed — a host serving documents for thousands of domains has asked to be crawled. Domains an attacker points at a host it does not control never verify, so they never raise v, and a host nobody is verified at never gets more than the base 40 and 20. A fetch held back by either lane is deferred and rescheduled. Being deferred is not an outcome: it is not a failure, not a negative, and never a lapse.

A crawl costs one fetch for the discovery document plus one per feed the document lists. First contact is a document fetch in the unverified lane; everything after it, for that source at that host, is in the verified lane.

A platform on its merchants’ own domains. The document for each merchant is served on that merchant’s domain, so every host has its own budget and v = 1; nothing contends with anything. What bounds it is the ping: 100 000 domains at 1000 per request is 100 requests, and at 60 requests per hour one caller is done in 1 hour 40 minutes. This is the topology to prefer where a platform can choose, and it is also the only one where a merchant can leave with its data path intact.

A platform delegating 100 000 merchants to one host. Every first contact is in that host’s unverified lane. It starts at the base 20 per minute and accelerates as merchants verify: 20/min until v reaches 10 000 (500 minutes, 8 hours), then v/500 per minute, which is exponential with a 500-minute time constant and covers the remaining 90 000 in 500 · ln 10 ≈ 1150 minutes (19 hours) — about 27 hours in all, against 83 hours at a flat 20 per minute. After that the host is at v = 100 000, so the verified lane is at its 600-per-minute cap: a full re-crawl is 100 000 document fetches in under 3 hours, or about 5 and a half with one feed each, against 42 hours at a flat 40. The first day is the price of one host; per-merchant hosts do not pay it.

An attacker aiming the crawler at a victim host. Nothing the attacker does raises v at a host it does not control, so the victim’s unverified lane stays at the base 20 fetches per minute: 1200 an hour, at most 20 MiB a minute, no matter how many domains, callers or zones are involved — and the victim’s own crawls keep their verified lane throughout. The attacker’s own ceiling is the same 100 requests per 1 hour 40 minutes, each ~24 KB buying at most 1000 DNS lookups and 1000 document fetches of at most 1 MiB. Spreading the load instead of concentrating it runs into the other two limits: 1000 new names per hour per registrable domain, 60 000 domains per hour per caller, and nothing at all that the domains themselves do not publish.

Every known source is re-crawled on a schedule; a ping only makes the next crawl happen sooner. Conditional requests are not a shortcut past discovery: discovery reruns in full, DNS included, and only once it has re-confirmed the path are the document and the feeds fetched with If-None-Match from the stored ETag and If-Modified-Since from the stored Last-Modified. A 304 then means that file has not changed. Publishers should serve ETag or Last-Modified on discovery documents and feeds.

Lapse

A source whose discovery stops succeeding is lapsed, not deleted.

A crawl succeeds only when everything the document lists was read: an acceptable document, and every feed in it answered 200 or 304.

A definite negative is discovery ending with no acceptable document, whether none was found at all or one was fetched and refused: no prodid record and no acceptable document on the domain; a prodid record that is malformed, over-long or duplicated; or a document that is rejected — not 200, not JSON, not valid, not agreeing about domain, or listing a feed off its host. It sets the lapse at once, with its timestamp, and readers see it.

A transport failure is a crawl that could not be completed: DNS timeout or SERVFAIL, connection or TLS failure, 429, 5xx, a cap or a timeout reached — on the document or on any feed it lists. One changes nothing. Seven consecutive days in which every attempt ends this way set the lapse too, so serving errors forever is not a way to freeze claims where they are.

That benefit of the doubt belongs to a host, not to a source. It applies only at a host this source has already been verified at. At a host it has never been verified at, any failure — a timeout or a 5xx included — is a definite negative and lapses the source at once. Otherwise a verified source could re-point its _prodid record at a host it does not control and read back, from its own outcome and from whether its lapse starts today or in a week, whether that host is up. A publisher moving hosts loses nothing to this: the first crawl that succeeds at the new host clears everything, and even a lapse is recoverable for the 30 days before claims are withdrawn.

A feed that cannot be read is therefore not a free pass. The feeds that did answer are still ingested, and no feed withdraws anything on a crawl that failed — but the crawl counts as a transport failure and feeds_failed says so in the source’s status, so a valid document in front of a permanently broken feed lapses on the same seven-day clock as a domain that stopped answering at all. Nothing can hold a source’s claims still and healthy-looking for ever.

While a source is lapsed it makes no new claims, its existing claims stay readable and stay marked, and its publisher.contact is notified. A crawl of a lapsed source writes nothing until it has succeeded in full; only then is the lapse cleared and the source ingested again. After 30 days of continuous lapse its records and their active claims are withdrawn together; they remain as history. Those 30 days are the window in which a publisher can notice and put the record, the document or the feed back.

The lapse notice. Every successful crawl keeps the publisher.contact of the document it accepted, or keeps no address when that document has none; a crawl that does not succeed leaves what was kept as it was. When a source lapses, the index sends that address one notice. It is one per lapse: a source that is already lapsed is not notified again however many crawls fail, and a lapse that is cleared and later starts again is a new lapse with a new notice. A source whose last accepted document carried no contact is not notified at all.

The notice says no more than the source’s status already does, in this order:

1. the source’s name and when the lapse started, in UTC, ISO 8601;
2. the outcome of the crawl that lapsed it, definite-negative or transport-failure, and what that outcome means;
3. the day its records and active claims will be withdrawn if nothing changes, 30 days after the lapse started;
4. its status page, https://prodid.org/source/<name>;
5. what to do: put the record, the document or the feed back, then ping;
6. why this address received it: it is the publisher.contact of the source’s own discovery document.

It never contains the crawler’s error text, an HTTP status the status does not show, who asked for a crawl or how often, or anything about a host other than the one the source was last verified at.

The kept address is never published. The status has no field for it, and no answer the index serves carries it. Because the address is whatever the document says, a notice can reach someone who never asked for one; that is why there is one per lapse, why only a source that has been discovered successfully can lapse, and why the notice says where the address came from.

Recovery does not depend on anything changing. Once discovery succeeds again, everything the source publishes at that moment is asserted again, whether or not a single byte moved: a publisher that comes back serving the identical document and the identical feed gets its catalogue back. Conditional requests are an optimization the index gives up whenever it has nothing live from a source, never a reason for a returning publisher to stay withdrawn.

Claims re-asserted after such a withdrawal are new assertions, with a new seen_at: for the length of the gap nobody verifiable stood behind them, and saying otherwise would date a fact to a time when it had no source. The withdrawn rows stay, so ?history=1 shows the gap.

Identity continuity

A source is permanent and keeps its identifiers when its name changes. sku is scoped to the source itself, not to its name.

A publisher that starts at <slug>.prodid.org and later adds a _prodid.<domain> record naming that same host becomes <domain> at the first crawl that confirms the delegation: the same source, the same records, the same opids, with trust rising from unverified to verified. The hosted publisher writes domain into the served document only once it has seen the TXT record itself; until then the slug publishes under its own unverified identity, because a document naming a domain that does not delegate to it speaks for nobody.

A domain that changes hands is a new controller of the same identity. prodid does not assert that a source is the same organization over time; the crawl log and the timestamp on every claim are the record of who said what, and when.

Records

Publishers send flat items. group ties the variants of one product together, the way item_group_id does in a Google Merchant feed; the index builds families and variants from it. Publishers never send an opid.

key is the record’s stable name within the feed. Publishing a record with an existing key replaces the previous one. A key that disappears from a feed the crawler fetched whole marks the source’s claims about it as withdrawn; the entity and its opid remain. A feed with skipped lines withdraws nothing for seven days, and after that withdraws absent keys as usual. Dropping a feed from the discovery document withdraws every key it carried, at once and without that grace, because it is the publisher saying the whole file is gone.

Security considerations

Write access to _prodid.<domain> is control of the identity, exactly as it is for DKIM or an ACME DNS-01 challenge. It must not be delegated to a vendor, and a domain whose DNS is shared is an identity that is shared.

Where no record and no document exist yet, whoever first publishes one becomes the source for that domain. That is the trust model, not a flaw in it: the index can only believe what the domain says about itself.

• Publish a record even when you serve your own document. v=prodid0; host=<domain> costs nothing and changes what an attacker with add-only DNS access can do. With no record, a record they add takes the identity over. With your record already there, a second one makes discovery fail with “more than one record”: a lapse, which is visible, notified and recoverable, instead of a silent takeover.
• Lapse is the alarm. The index notifies publisher.contact when a source lapses, and the 30 days before claims are withdrawn are the recovery window. A contact address that nobody reads turns a recoverable event into a silent one.
• A short hijack leaves a long trace. Claims made while an attacker controlled the DNS or the host are attributed to the source and stay until they are withdrawn, and media fetched then keeps its hash. The crawl log and the claim timestamps show exactly when they arrived, which is what makes an hour of control an hour of damage rather than a permanent one.
• A domain that changes hands is a new controller of the same identity. Readers who care about provenance read seen_at and the source’s status, not just the value.
• Trust is about control, not truth. verified says the index proved who serves the domain. It says nothing about whether what the domain publishes is correct, and a reader that shows a claim without its source is showing an unattributed assertion.

Claims

The index never stores a bare field. It stores claims:

(opid, property, value, source, seen_at, evidence)

Two sources may disagree about the same property; both claims are kept, each attributed to its source. Matching decisions, the statement that two records describe the same entity, are claims too, and can be reversed.

Property names

A published record becomes one claim per fact, under these names and no others:

<axis> and <name> are exactly the publisher’s own keys, which org.prodid.item already restricts to ^[a-z][A-Za-z0-9]*$. The two prefixes are not decoration: flattened, a publisher property called name or brand would collide with the built-in claim of the same name, and a variant axis would collide with a properties entry of the same name. There is exactly one dot, and v1 defines no other property names.

markets and media are sets, so each element is its own claim, as the table says: ["TR", "DE"] is two markets claims carrying "TR" and "DE". Reordering the list in the feed then changes nothing, and a market added or dropped withdraws exactly one claim instead of rewriting the set.

lang is carried only by claims whose value is text the publisher wrote: brand, name, variant.*, and those properties.* whose value is a string or a list of strings. category, markets and media never carry it.

A media claim is the one whose value is not the publisher’s raw value. The publisher writes a URL and a license; the claim carries what the index stored:

{
  "property": "media",
  "value": {
    "sha256": "e3943ea26ce08ae81809e62936dc4f934fa04da6cd57bd9ab2a90b79da751c09",
    "url": "https://acme.example/img/tj2041-navy.webp",
    "media_url": "https://media.prodid.org/sha256/e3943ea26ce08ae81809e62936dc4f934fa04da6cd57bd9ab2a90b79da751c09",
    "license": "CC-BY-4.0",
    "kind": "image"
  },
  "source": { "name": "acme.example", "trust": "verified", "lapsed": false },
  "seen_at": "2026-09-14T09:12:03Z"
}

A reader fetches media_url, which is immutable and always answers the same bytes. url is where the source published the file, kept as provenance; it may have changed or gone since.

Reading

The resolver

All three answer application/json when the request accepts it and an HTML page otherwise.

The record is open, so a web page on any origin may read it directly. Every JSON answer of /id, /01, /source, /schemas/v0/ and /schemas/v1/ — an entity, a status, a schema, a problem, the 304 that revalidates one of them and the 308 a merged opid answers — carries Access-Control-Allow-Origin: *. HTML answers do not carry it.

/01/{gtin} accepts GTIN-8, -12, -13 and -14 and answers the item itself rather than redirecting, so a QR scan costs one round trip; the Content-Location header gives the canonical opid URL. v1 resolves on the GTIN alone; further GS1 Digital Link path segments such as /10/{lot} and /21/{serial} are accepted and ignored.

An opid that was merged into another answers 308 Permanent Redirect to /id/{survivor}, and the survivor lists it in merged_from. Merges are reversible, which is why the redirect is per-request and the losing opid keeps resolving forever.

?history=1 adds the withdrawn claims, each carrying withdrawn_at. Without it, only active claims are returned.

An entity

Schema. Item, family.

{
  "$type": "org.prodid.entity",
  "version": 1,
  "opid": "kq7m3v9dn2hb4t5x",
  "kind": "item",
  "parents": { "family": "3n7k2q9vdfhb4mta", "variant": "9p2d4kv7mq3nbhxf" },
  "identifiers": [{ "scheme": "gtin", "value": "09506000134352" }],
  "claims": [
    {
      "property": "name",
      "value": "Trail Jacket",
      "lang": "en",
      "source": { "name": "acme.example", "trust": "verified", "lapsed": false },
      "seen_at": "2026-09-14T09:12:03Z"
    }
  ]
}

The response has no bare fields. There is no top-level name or brand: every value a reader sees is inside a claim that carries its source — the source’s identity, its trust and whether it is lapsed — and the seen_at at which that source began asserting it. Two sources disagreeing produce two claims, and choosing between them is the reader’s decision, made with the source in hand.

seen_at is when this source started asserting this value and has not stopped since, not when the fact was first published anywhere. It survives a re-crawl that finds the same value, which is the point of it; it does not survive a withdrawal. A source lapsed for 30 days has its records and claims withdrawn, and what it publishes afterwards is asserted again from that day, because for the length of the gap nobody verifiable stood behind it. ?history=1 shows the withdrawn rows, so the gap is visible rather than implied.

parents and children are the hierarchy as opids: an item names its variant and its family, a variant names its family and lists its items, a family lists its variants and has no parents. An item has no children. Property names are the namespaced ones above, so a reader can tell name from properties.name without guessing.

An identifier is global or scoped, never ambiguously either. gtin is the normalized 14-digit form, never all zeros, and carries no scope; mpn, sku and rcn carry a non-empty scope — the normalized brand, or the source’s name — and nothing else carries one at all. That is what keeps a restricted number from reading as a global one, which is the whole reason rcn exists.

Three rules hold that the schema cannot state, because each compares two values in the same document:

• no opid appears in its own parents, children or merged_from;
• withdrawn_at is never before seen_at;
• a media claim’s media_url addresses its own sha256.

The conformance harness checks all three.

A source’s status

Schema. Example.

{
  "$type": "org.prodid.source",
  "version": 1,
  "name": "acme.example",
  "kind": "domain",
  "trust": "verified",
  "verified_by": "dns-txt",
  "verified_at": "2026-09-14T09:12:03Z",
  "lapsed": false,
  "license": "CC0-1.0",
  "last_crawl": {
    "at": "2026-09-19T06:00:11Z",
    "via": "dns-txt",
    "document_url": "https://acme.prodid.org/.well-known/prodid.json",
    "outcome": "success",
    "http_status": 200,
    "records_seen": 3,
    "records_changed": 1,
    "lines_skipped": 0,
    "feeds_failed": 0,
    "media_failed": 0
  }
}

This is how a publisher sees that something is wrong: the outcome of the last crawl, how many lines were skipped, how many feeds could not be read, and whether the source is lapsed and since when. trust above unverified exists exactly when the proof does, so verified_by and verified_at are present together or not at all, and only a source whose identity is a domain can be more than unverified. records_changed is never more than records_seen, which is a rule the schema cannot state.

Two things keep this page from becoming a probe:

• A domain becomes a source at its first successful discovery, not before. Until then there is no row, and /source/{name} answers 404 — the same 404 as a name nobody has ever pinged. Pinging a domain therefore tells the caller nothing, because it creates nothing to read.
• http_status and document_url are shown only for the host the source was last verified at. Otherwise anyone could register evil.example, point _prodid at a host they do not control, ping it, and read back through this page how that host answers — and the crawler’s egress is not theirs, so the answer would see past address, network and geography blocks that the host set up on purpose. The assurance is not only that this page says nothing about other callers; it is that it says nothing about any host but the one this source is published from.

It never reports who asked for a crawl, how often anyone asks, or the crawler’s internal error text.

via says how the last crawl reached the records: well-known or dns-txt through discovery, or import for a dataset source, and import for nothing else. An import is not a fetch on anyone’s word — the operator runs it against an export of the operator’s choosing, and nobody can aim it at a host — so its document_url is always shown, as the public URL of the export it read. It has no http_status, and its outcome is always success: an import that does not finish is not reported.

Errors

Every error answers application/problem+json (RFC 9457). The status says what kind of failure it is; the type says why:

{ "type": "https://prodid.org/problem/unknown-opid", "title": "No entity with that opid", "status": 404 }

rate-limited carries Retry-After; nothing else does.

invalid-gtin, restricted-gtin and blocked-media are decided by the request alone — the same path will always answer the same way — so they may be cached. Every other problem depends on the state of the index or on what is routed, and is served Cache-Control: no-store: a 404 today is the answer for an opid or a domain that may exist tomorrow.

Media

A record’s media entries name files the index fetches, stores under their SHA-256, and serves from:

https://media.prodid.org/sha256/{hex}

A media claim exists only once the bytes are stored, and it carries the digest, the original URL, the licence, the source and the fetch time. Files are immutable: the URL is the digest, so it can never come to mean something else. A hash on the takedown list is deleted and cannot be stored again.

Everything below is per entry. A media entry that is refused, too big, or unreachable costs that entry and nothing else: the record is never invalid because of its media, and no media failure lapses a source.

What is ingested

The type is decided by sniffing the bytes. The publisher’s Content-Type is a hint and nothing more — it is the one part of a fetch a hostile or careless origin fully controls, and it is wrong often enough in ordinary catalogues to be useless as a gate.

All six are registered with IANA: image/jpeg and image/gif in RFC 2045 and RFC 2046, image/png by the W3C, image/webp in RFC 9649, image/avif by the Alliance for Open Media, application/pdf in RFC 8118. An APNG is a PNG with extra chunks, so it sniffs as PNG and is stored as one.

The sniffed type and the entry’s kind must agree. A PDF published as kind: image, or a JPEG as kind: document, is skipped: the publisher and the bytes disagree about what the file is, and guessing which is right is how the wrong thing gets served.

Nothing else is ingested. In particular:

• SVG is out. It is XML that can carry <script>, event handlers and external references, and served from media.prodid.org it would be active content on our own origin — the classic stored-XSS carrier. No amount of header hardening makes that a good trade for a product photo, which is raster anyway.
• HTML and XHTML are out, for the same reason and more directly.
• Archives are out — zip, gzip, tar. What is inside is invisible at fetch time, so ingesting one means redistributing something nobody looked at.
• Anything executable is out: binaries, installers, WebAssembly, scripts of every kind.
• Office documents are out in v1. They are macro carriers with a long history of it, and a publisher who wants a spec sheet carried can publish a PDF.

The list is short on purpose. Widening it is a spec change, not a configuration one.

Caps

20 MiB holds any product photograph a publisher has a reason to serve, including a 4000 × 4000 original; 50 MiB holds a long illustrated manual. 20 entries is a gallery plus its documents — Google’s own product feed allows eleven images — and a record listing more is a catalogue pretending to be a product.

Exceeding any of them skips the entry. The twenty-first entry of a record is not fetched; an image that turns out to be 30 MiB is dropped mid-fetch and not stored.

Fetching

Media is fetched under the same guard as everything else the crawler touches: https only, default port, a public DNS name that resolves to a public address, never an IP literal, and at most the one <host> → www.<host> redirect.

The same-host rule does not apply to media. A feed must live on the document’s host; an image must not. Product photography lives on CDNs, and forcing publishers to proxy their own images through the feed host would cost them money and us freshness. A media URL is therefore a fetch of a host the publisher picked, which has two consequences worth stating plainly:

• Budget. Media fetches count against the fetched host’s per-host budget, in the unverified lane unless the source is verified at that host, and are additionally capped at 10 per minute and 200 MiB per minute for one host. Without that second cap a 20 MiB media cap would be a far larger amplifier than the 1 MiB discovery document ever was.
• Oracle. A publisher can learn one bit about a host it named: whether the fetch produced a claim. It never learns a status code, an error, a redirect target or a timing — media_failed in the source status is a count for the whole crawl, not a per-entry report. That bit is the minimum a publisher needs to fix its own catalogue, and it is expensive: it costs a verified domain, it is bounded by that host’s own rate limit, and it says nothing about anyone else’s source.

When it fails, and when it changes

A media entry that cannot be fetched produces no claim. The record is stored and its other claims stand; the entry is counted in media_failed and tried again on the next crawl of that record, which is the ordinary schedule and not a retry loop of its own. Nothing about it lapses the source: lapse is decided by discovery, and media is not discovery.

Because the claim is keyed to the digest, the two ways a file changes both resolve themselves:

• The URL changes, the bytes do not. The digest is the same, so the stored file is the same and media_url does not move. The claim’s value does change, because url is part of it, so the old claim is withdrawn and a new one is asserted — the file a reader loads never flickers, and the record of where it was published stays honest.
• The URL stays, the bytes change. A new digest means a new file, stored beside the old one, and a new claim; the claim on the old bytes is withdrawn because the record no longer asserts it. The old file stays addressable, because something may have linked it.

An unchanged record gives no signal that its bytes moved, so media is re-fetched when the record changes and otherwise no more often than every 30 days, with If-None-Match and If-Modified-Since where the origin offers them.

Serving

Every file is served with:

The CSP and the sandbox are belt and braces for a store that holds no active types anyway; a PDF is downloaded rather than opened in a viewer that would run inside our origin. A year of immutable caching is safe for the same reason the URL is a digest.

A blocked hash answers 410 with …/problem/blocked-media, not 404: the file existed, it was removed deliberately, and it will not come back. Claims pointing at a blocked hash are withdrawn, and a record that re-publishes the file does not bring it back.

Licences that can be ingested

The index stores the file and redistributes it, so a licence that does not allow redistribution cannot be ingested at all. The entry’s license must be exactly one of these SPDX identifiers:

CC0-1.0  CC-BY-4.0  CC-BY-SA-4.0  CC-BY-ND-4.0  CC-BY-3.0  CC-BY-SA-3.0  CC-BY-ND-3.0

All seven are current SPDX identifiers (list 3.29.0) and all seven permit verbatim redistribution by anyone, which is exactly what we do. ND is on the list because we redistribute the file unmodified; every NC variant is off it, because our readers include commercial ones and we cannot pass on a restriction we do not track. Anything else — an unlisted identifier, a misspelling, a bare “all rights reserved” — is not ingested.

It is checked as a string, before the fetch, so a licence we cannot use costs nobody any bandwidth. The claim keeps the identifier, so a reader always knows the terms of the file it is looking at.

Dumps

The index is replaceable. Everything it knows that a reader may know is published as a full export, a dump, that anyone can download and load into an index of their own.

https://dumps.prodid.org/v1/<YYYY-MM-DD>/manifest.json
https://dumps.prodid.org/v1/latest.json

A dump is a directory named after the UTC date of its snapshot_at, under the version it is written in: v1/ holds v1 dumps, whose manifest and whose source and entity lines are v1 documents. The v0 dumps written before v1 stay under v0/, under the same retention, and v0/latest.json keeps naming the last of them. It holds five data files and a manifest (example). latest.json is a copy of the newest dump’s manifest.json, and it is written last, after every file of that dump and its manifest are in place: a reader that starts from latest.json never finds a manifest naming a file that is not there yet, and finds the directory from the manifest’s own snapshot_at.

Files

Every data file is NDJSON compressed with gzip, one JSON object per line. All five are read from one consistent snapshot of the index, taken at snapshot_at, so they agree with each other: every source a line names has its line in sources.ndjson.gz, and every opid a line names has its line in entities.ndjson.gz.

An opid that was merged answers 308 at the resolver, so it has no document of its own to copy. Its line is the document it would answer were it not merged — its own identifiers and claims stay on it, which is what makes a merge reversible — with one more member, merged_into, the opid of its survivor. Without merged_into the line validates as org.prodid.entity. merged_into is never the line’s own opid, and names an entity that is not itself merged, because merges do not chain.

A record line is one record as the index keeps it. source is the source’s name; key is the record’s key in that source; feed is the URL of the feed it was read from, absent for a record a dataset import wrote; opid is the item it resolved to; body is the org.prodid.item the publisher sent, unchanged; first_seen and last_seen are when the index first and last read it; withdrawn_at is present exactly when the record is withdrawn. Every time is UTC, ISO 8601.

A grouping line says which entity a source’s grouping resolved to: source is the source’s name, kind is family or variant, and opid is that entity. key is what the source’s records are grouped on, written out rather than digested, so that another implementation can match its own grouping to it:

A merge line is one merge decision: from the merged opid, into its survivor, decided_at when it was made, and reverted_at, present exactly when it was reverted.

What a dump never holds

A dump says no more than the resolver already does, plus the records, groupings and merges the resolver’s answers are built from. It never holds:

• a publisher’s contact address, which the index keeps only to send the lapse notice;
• the crawl log, the crawler’s error text, or anything else about a crawl that the source’s status does not show — the log is per-host observation, which the status withholds on purpose;
• why a merge was made or who decided it;
• why a media hash was blocked;
• media bytes. A media claim’s media_url addresses the file, and a blocked hash is not served;
• the index’s internal row identifiers. Every line names a source by its name and an entity by its opid;
• the grants of authority and their evidence. A source’s line says authoritative if it was at snapshot_at, but an index loaded from the dump holds no grant until its own operator makes one.

Licences

The manifest’s licenses says what the contents are under, and the rule is the one in Licensing:

• everything the index makes — opids, the identifier mappings, the family and variant hierarchy, groupings and merges — is CC0-1.0;
• every claim and every record is under the licence its own source declares, the license of that source’s line in sources.ndjson.gz. A source that declares none has granted nothing, and prodid grants nothing on its behalf.

Manifest

org.prodid.dump names each of the five files exactly once, with the SHA-256 and the size of the file as served, compressed, and its number of lines once decompressed. A reader checks each file against its digest before loading it. created_at is when the manifest was written and is never before snapshot_at, a rule the schema cannot state. Vectors: fixtures/v1/dump/.

Retention

A daily dump is kept for 7 days. The first dump of each UTC month is kept for good. A dump that is kept is never rewritten: its directory, its files and its manifest stay byte for byte as they were.

Conformance

bun run check in this directory validates every example and fixture against the schemas of its version, and runs the vector sets through reference implementations of the rules above.

The TXT, GTIN, brand, NDJSON and Merchant vectors hold for v0 and v1 alike, because v1 changed none of those rules; the harness runs the NDJSON and Merchant sets under the item schemas of both versions. v0 is kept beside v1, unchanged: its schemas are the files directly in schemas/, its examples are examples/, and its vectors are fixtures/valid/, fixtures/invalid/ and fixtures/dump/, which a reader of v0 documents, a kept v0 dump among them, still checks against.

Every file in fixtures/v1/invalid/ has an entry in fixtures/v1/expected.json, and every file in fixtures/v1/dump/invalid/ one in fixtures/v1/dump/expected.json, as their v0 counterparts in fixtures/expected.json and fixtures/dump/expected.json do, naming the instance path, the schema keyword and, where it disambiguates, the error parameters that must reject it. A fixture that fails for a new and unrelated reason is a failure, not a pass. Six of the keywords name rules that live on top of the schemas rather than inside them, because JSON Schema cannot express them: storable for a string that cannot be stored, gtin for a value that is not a GTIN, identifiable for a record with no identifier the index can store, self-reference for an entity that lists its own opid, order for a withdrawn_at before its seen_at or a dump’s snapshot_at after its created_at, and digest for a media_url that does not address its claim’s sha256.

A TXT vector is { "value": [...], "outcome": "delegation" | "ignored" | "malformed", "host": "..." }, where value is the RR’s character-strings in order and host is the normalized host, present exactly when the outcome is delegation. The three outcomes are behavioural, not cosmetic: ignored means the RR is not a prodid record and discovery carries on without it, malformed means discovery fails.

Two independent implementations — the one in validate.ts and the crawler’s — agreeing on these vectors is the point of all of them.

The Merchant fixture is derived, not asserted by hand: bun run validate.ts --emit-merchant rewrites the expected NDJSON from the reference mapping in the harness, and the check fails whenever the file on disk says something the mapping does not.

Writing a conformance runner

The vectors are data, so an implementation in any language can run them without validate.ts. It needs a JSON parser, a JSON Schema 2020-12 validator and the rules of this text; every vector is a file under schemas/, examples/ or fixtures/, and nothing else in this directory is needed. Each schema’s $id is https://prodid.org/schemas/v<N>/<file name>, and the harness fails when one is not, so a runner may load them by $id. Each paragraph below is one vector format and what passing it means, exactly as the harness decides it.

Documents and records. A file in fixtures/v1/examples/, fixtures/v1/valid/, fixtures/v1/invalid/, fixtures/v1/dump/valid/ or fixtures/v1/dump/invalid/ is one JSON document, except fixtures/v1/examples/items.ndjson, a feed whose every line is a record. To check one:

1. Parse it. A string escape for a lone surrogate, "\ud800", must come through as that code unit, not as U+FFFD and not as a parse error: the storable rule below has to see it.
2. Pick the schema by the document’s $type, from the version of the directory the file is in — schemas/v1/ for everything under fixtures/v1/ — and never from the document’s own version, which several invalid fixtures get wrong on purpose. A document with no $type, or with one that version has no schema for, fails, in whichever directory it is.
3. Validate it against that schema, collecting every error rather than stopping at the first, and with format asserted rather than only annotated: the schemas use date-time (RFC 3339, with an offset), uri (absolute, RFC 3986) and email.
4. Apply the rules the schemas cannot state, whatever the schema reported. Each yields a problem with an instance path and a keyword, and no parameters:

A document is valid when neither the schema nor any rule reports a problem. Every example and every file in a valid/ directory must be valid; in items.ndjson every line must be a valid record, and the only empty line allowed is the one after the final newline.

expected.json. fixtures/v1/expected.json covers fixtures/v1/invalid/, and fixtures/v1/dump/expected.json covers fixtures/v1/dump/invalid/. Each is one JSON object; every member name is the name of a file in that invalid/ directory, and every value names one defect:

{ "instancePath": "/claims/0", "keyword": "required", "params": { "missingProperty": "source" } }

An invalid fixture passes when it parses, its $type has a schema in its version, it yields at least one problem, and at least one of its problems has exactly that instancePath, exactly that keyword, and, for each member of params when there is one, a parameter of the same name with an equal value; parameters params does not name are not compared. Other problems may come with the expected one, but a fixture rejected only for other reasons fails. A fixture with no entry fails, and so does an entry with no fixture.

instancePath is a JSON Pointer (RFC 6901) to the value the problem is about, "" for the document itself, with ~ and / in member names written ~0 and ~1. A schema problem is named as the harness’s validator, Ajv, names it; any other validator’s output maps onto it like this:

• keyword is the name of the schema keyword that failed, spelled as JSON Schema spells it; the vectors use required, additionalProperties, type, enum, const, pattern, format, minimum, minItems, maxItems, contains, oneOf and not;
• a value that meets a boolean false schema — a member a conditional forbids, as { "properties": { "parents": false } } does — is the keyword false schema, at that value;
• required is at the object that lacks the member, with params.missingProperty its name, and additionalProperties is at the object that has the extra member, not at the member;
• every other keyword is at the value it applies to: oneOf at the value that matched no branch or more than one, contains, minItems and maxItems at the array.

NDJSON. fixtures/ndjson/expected.json names each .ndjson file beside it with the line numbers, counted from 1 and ascending, that a reader must skip. Split the file on LF. A line that is empty or only whitespace is not a record and is not skipped; every other line is skipped when it is not JSON, or when it is not a valid org.prodid.item by the rules for documents above. The skipped lines must equal the expected list exactly, under the v1 item schema and under the v0 one alike.

Merchant. fixtures/merchant/expected.json names each .csv file beside it with ndjson, the file of records it must produce; declared, the lang, markets and license the import declares for the whole file; and skipped_rows. The CSV has a header row, separates cells with commas, quotes a cell with " and writes a quote inside one as "", and has no line break inside a cell. To check it: the declared license must be one of the licences that can be ingested; map every row after the header as the appendix says, check each result as an org.prodid.item by the rules for documents above, and keep the valid ones in row order. The rows whose record is invalid, numbered with the header as row 1, must equal skipped_rows; there must be as many kept records as the NDJSON file has lines, each equal to its line; and every line of that file must itself be a valid record. All of it holds under the v1 item schema and under the v0 one. The harness compares each line as text, because it wrote the file with its own serializer; a runner in another language compares each line as a JSON value — the same members with equal values, arrays in order — since member order is a serializer’s choice.

Wikidata. fixtures/datasets/wikidata/entities.json is a JSON array of entities as a Wikidata JSON dump writes them, and expected.json names every entity’s id with what an import into the dataset source dataset:wikidata makes of it: { "ok": true, "record": { … }, "dropped": [ … ] }, or { "ok": false, "skip": "<reason>" }. This is one importer’s mapping rather than a rule of the protocol, and an implementation that does not import Wikidata has no need to run it. A brand’s English label is read off the brand’s own entity in the same file, its labels.en.value trimmed; a brand whose entity is not in the file has none. Each entity is then mapped in this order, and the first step that fails is its skip:

1. It is an item: type is item and id matches ^Q[1-9][0-9]*$. Otherwise not-an-item.
2. Its GTINs are the string values of its P3962 statements at best rank: the preferred statements when there are any, else the normal ones, never the deprecated; a statement whose mainsnak.snaktype is not value carries none. Each value that, trimmed, is a GTIN — digits, of length 8, 12, 13 or 14, not all zeros, with a correct check digit; a restricted number is kept — becomes { "scheme": "gtin", "value": <the trimmed value> }, once per normalized 14-digit form, in statement order. Each one that is not adds P3962: <the value as written> (<problem>) to dropped, the problem being format, all-zero or check-digit, tested in that order. No GTIN left: no-gtin.
3. Its brands are the ids, matching ^Q[1-9][0-9]*$, of its P1716 statement values at best rank, each once, in statement order. The first with a non-blank English label becomes brand; every other one adds P1716: <id> to dropped. None with a label: no-brand.
4. Its own English label, trimmed, is name. None, or blank: no-label.
5. { "$type": "org.prodid.item", "key": <id>, "brand": …, "name": …, "lang": "en", "identifiers": [ … ] }, with those members only, is checked as an org.prodid.item. Invalid: invalid.

A mapped entity passes when record equals what it maps to, as a JSON value, and dropped equals its list in order; a skipped one passes when its reason equals skip. The ids in the file and the names in expected.json must be the same set. These vectors are run by the reference implementation’s own tests rather than by validate.ts.

TXT, GTIN and brand. Each file is one JSON object with exactly the members below and no other; a vector with any other member is itself broken.

• fixtures/txt/: value, outcome and, exactly when outcome is delegation, host, as described above. Every string in value is at most 255 bytes, and host is a hostname by the discovery document’s domain rule. Join the strings in order with nothing between them, read the result as The _prodid TXT record says, and compare the outcome and, for a delegation, the normalized host.
• fixtures/gtin/: value and verdict. Classifying value as Which GTINs are identifiers says gives verdict: gtin, rcn for a restricted number, or invalid.
• fixtures/brand/: input and normalized. Normalizing input gives normalized, and normalizing normalized gives it back unchanged.

v0. The v0 vectors run exactly as the v1 ones do, against the v0 schemas, the files directly in schemas/: examples/, fixtures/valid/, fixtures/invalid/ with fixtures/expected.json, and fixtures/dump/valid/ and fixtures/dump/invalid/ with fixtures/dump/expected.json. The rules, the keywords and the meaning of expected.json are the same. An implementation runs them when it reads v0 documents: a v0 discovery document before the cut-off in Versions, or a kept v0 dump.

Appendix: importing a Merchant feed

The first publishers arrive with a spreadsheet, not a web server. A Google Merchant product file is the format they already keep, so the hosted publisher imports one and writes org.prodid.item lines out of it. This appendix is the mapping, against Google’s current product data specification.

A Merchant file describes offers: what a seller sells, at what price, where, today. prodid records what a product is. The mapping is mostly a matter of keeping the second and dropping the first.

Why the offer is dropped. Price, availability, shipping and condition belong to a seller on a day, not to the product: they differ between sellers, they change hourly, and a claim that stale would be worse than no claim. link goes the same way — a storefront URL is where a seller sells it, not a property of the thing.

Variant axes versus qualifiers. Google lists six attributes as the ones that distinguish products inside an item_group_id — color, pattern, material, age_group, gender and size — plus variant_option for anything else. Four of them become axes here: color, size, material and pattern, the ones a buyer actually picks between. gender and age_group stay properties, and so do size_type and size_system, which Google does not list as variant attributes at all: they describe the whole group, and as axes they would split one product into several the moment a publisher spelled one of them two ways.

Repeated fields. Google’s data sources write a repeated attribute in one of two ways, and the import accepts both: every value in one cell separated by a comma with no space, or the attribute’s name repeated as the header of several columns, one value each. A cell is split on commas only when its header appears once in the whole file; when a header is repeated, every cell under it is taken whole. That is the rule, and it is what makes the second form the way to carry a value with a comma in it — in the first form a comma always separates, which is why Google’s instruction for a URL is to percent-encode it as %2C. The two forms are therefore chosen per file, not per cell: a file that repeats a header has said how that attribute is written everywhere in it.

The attributes read as repeated are additional_image_link, product_type, product_highlight, product_detail and certification; every other value is taken whole, so a description with commas in it stays one string. A repeated attribute always produces a list, even when the file gives it one value, so a consumer never has to handle two shapes.

Product details. Each product_detail value is three colon-separated sub-attributes: a section name, an attribute name and a value. The attribute name becomes a properties key by lowercasing it, dropping spaces, underscores and hyphens, and capitalizing each word after the first — Machine wash becomes machineWash. A detail that does not split into three parts, or whose name does not survive as ^[a-z][A-Za-z0-9]*$, is dropped and reported.

The section name is not part of the key, so two details can want the same one. A detail never overwrites anything: the first value for a key wins, and any later detail with that key is dropped and reported with the row’s other dropped values. The same holds against the properties the mapping itself produces — a detail that derives productType, description or any other mapped key is dropped, not merged — so a product_detail column can never quietly redefine what the rest of the row said.

Categories. category in a prodid record is a GS1 GPC brick code, and Google’s taxonomy is a different taxonomy. Translating one into the other automatically would be guessing, so the import does not: both Google values are carried verbatim as properties, and category is set only if the publisher supplies a GPC code itself.

Language and markets. The product data specification has no language or country attribute: they are settings of the data source, so one file is one language and one set of markets. The import declares them and they become lang and markets on every record of the file.

Media licences. The Merchant format has no licence column, and media is not ingested without one. The import therefore declares a single SPDX identifier that covers every image and document in the file, and it must be one of the ingestible ones; the declaration is recorded on every media entry the file produces. If the file carries a license column of its own, a row’s value wins over the declaration. A file whose declared licence is not ingestible produces records without media, not records that fail.

Rows that cannot become a record. A row with no usable identifier, a GTIN that fails its check digit, or anything else the record rules reject is skipped, and the import reports it by row number, counting the header as row 1 so the number matches what the publisher sees in the spreadsheet. One bad row costs its own row.

fixtures/merchant/ is a worked example: feed.csv, the feed.ndjson it must produce, and expected.json naming the declared language, markets and licence, and the row that must be skipped — row 6, whose GTIN check digit is wrong. Row 2 carries the comma form under a single additional_image_link header and the column form under two product_type headers, one of which holds a value with a comma in it; row 3 carries three details that collide, two of them dropped. The check derives the records from the CSV itself and compares them line by line, so the file and this table cannot drift apart.

Licensing

This specification and the schemas are CC0. opids and identifier mappings published by prodid.org are CC0. Every other claim carries the license of its source.