Before you can improve how product data reaches your catalogue, you need an honest inventory of where it comes from today. In most distribution businesses there are seven sources. They are not equally difficult, and the difference between them is where the cost sits.
The seven sources
- 01The supplier's public website. The most common source, and the most complete. Everything you need is usually there — description, specification, images, datasheets — in a format designed for humans rather than systems.
- 02A supplier PDF catalogue. Looks structured, is not. A PDF is a description of where ink goes on a page; the table you can see is not a table to a machine.
- 03A spreadsheet from the supplier. The best case when it exists, which is rarely. Column names vary between suppliers and often between versions from the same supplier.
- 04A supplier portal or dealer extranet. Often the richest source, behind a login, with per-dealer pricing. Frequently no export.
- 05Email attachments. New product announcements, price-list updates, discontinuation notices. Unstructured, unarchived, and usually the only record of a change.
- 06A manufacturer content syndication feed. Excellent coverage for global IT and consumer electronics. Poor to non-existent for industrial, electrical and technical brands.
- 07Your own institutional knowledge. Cross-references, compatibility, what customers actually ask for. This exists only in people.
Ranking by cost to process
Ranked by the effort required to turn a source into a usable product record:
| Source | Effort | Why |
|---|---|---|
| Syndication feed | Lowest | Already structured; map fields once |
| Supplier spreadsheet | Low | Structured, but mapping differs per supplier |
| Supplier website | Medium | Complete but unstructured; automatable per site |
| Dealer portal | Medium–high | Rich, but usually no export and login-gated |
| PDF catalogue | High | Extraction is unreliable; tables break |
| Email attachments | High | No structure, no history, easily missed |
| Institutional knowledge | Highest | Exists only in a person; must be interviewed out |
The counter-intuitive finding is that the supplier website — which feels like the most manual source — is the most automatable. It is consistent within itself. Once you have worked out how one manufacturer structures their product pages, that knowledge applies to every product they sell, and it keeps applying until they redesign.
Which sources automate well
A source automates well when it is consistent, addressable and complete.
- Consistent — the same information appears in the same place on every product page.
- Addressable — there is a stable URL or identifier per product.
- Complete — the information you need is actually present, rather than implied.
Supplier websites usually satisfy all three, which is why per-vendor configuration is worth building. Configure a manufacturer's site once and the four hundredth product from that manufacturer costs almost nothing to load.
PDFs satisfy the first two and fail the third in a specific way: the information is present visually but not structurally. Extracting a specification table from a PDF works until a footnote, a merged cell or a unit in the header breaks it, and you cannot tell from the output that it broke.
Which will always need a person
Two sources will never automate, and you should stop trying.
Email attachments are unstructured by nature. The right response is not extraction but process: a single mailbox that supplier communications go to, and a rule that price and specification changes get logged when they arrive rather than when somebody notices.
Institutional knowledge — which gland fits which cable, which part supersedes which, what customers actually ask for — cannot be scraped from anywhere because it exists nowhere. It has to be interviewed out of people and written into your own data as compatibility relationships and cross-references. That is a genuine project, and it is the one that most differentiates a good catalogue from a complete one.
Do the inventory before you do anything else. Most businesses find they are treating an automatable source as manual work, and a genuinely manual source as something they keep meaning to get to.
Common questions
- Where do distributors get their product data from?
- Seven sources, in roughly descending order of frequency: the supplier's public website, supplier PDF catalogues, spreadsheets sent by the supplier, dealer portals and extranets, email attachments, manufacturer syndication feeds, and the institutional knowledge held by staff.
- Which product data source is cheapest to work with?
- A manufacturer syndication feed, where one exists — the data is already structured and you map the fields once. Coverage is good for global IT and consumer electronics and poor to non-existent for industrial, electrical and technical brands, which is why most distributors never get to use one.
- Can product data be extracted from supplier PDFs reliably?
- Not reliably. A PDF describes where ink goes on a page, so the table you can see is not a table to a machine. Extraction works until a footnote, a merged cell or a unit in a header breaks it, and the output gives you no signal that it broke.
Where to next
Who wrote this
The Smart Station team
Written by the engineering and delivery team at Smart Station (Pty) Ltd, the South African software company that builds GetShopSync. The material here comes from building catalogue systems for distributors — it is what we have measured and what we have got wrong, not a survey of the literature.