Web data infrastructure for AI
The web data infrastructure behind the next generation of AI.
Build training corpora, keep retrieval systems current and expand into new domains with data collected and structured to your specifications. Every document carries its provenance. Every corpus ships with a quality report. The infrastructure has run in production since 2012 and processes more than 500 billion data points a month.
- domain
- software developer documentation
- sources
- official docs, READMEs, changelogs, API references
- languages
- en, zh, ja, de, fr, es, pt, ko, ru, hi + 22
- date range
- 2015-01-01 → 2026-09-17
- exclude
- robots-disallowed · paywalled · PII · opt-out list
- dedupe
- exact + near-dup (MinHash, 0.85)
- quality
- language ID ≥ 0.95 · boilerplate stripped · code blocks preserved
- output
- Parquet + JSONL, sharded, manifest with per-doc provenance
Scale you can verify. Depth you can audit.
Numbers, not adjectives. Each figure below is measured from production and re-published quarterly with its method.
Method notes and quarterly updates: State of Web Data for AI. Where a figure is contractual rather than metered, the source column says so.
Four things AI teams buy from us.
Training is the largest. All four run on the same collection, normalisation, quality and provenance pipeline.
Training corpora
Domain corpora built to your specification: sources, languages, date ranges, exclusions, deduplication thresholds and quality filters you define. Delivered as sharded Parquet or JSONL with a per-document manifest.
- Developer docs, scientific, multilingual web, commerce, legal, public records
- Exact and near-dup removal, language ID, boilerplate stripping
- PII detection and removal, license and robots signals per document
- Quality report with human-sampled extraction accuracy
Retrieval freshness
Continuous collection for RAG and search indices. Changed, new and removed documents delivered as events so your index reflects the web as it is today, not at training time.
- Hourly to daily cadence per source class
- Change events with diffs and snapshots
- Point-in-time versions retained
- Delivered to S3, GCS, a queue or your API
Evaluation sets
Point-in-time snapshots and human-verified golden sets for benchmarking extraction, retrieval, grounding and agent tasks against the live web.
- Frozen snapshots with hashes for reproducibility
- Human-labelled ground truth on sampled documents
- Held-out domains and adversarial site classes
- Versioned, citable, re-runnable
Live agent access
Structured web access as tools for models and agents through MCP and the API: search, fetch and render, extract to schema, monitor. Rendering, anti-bot and egress handled server-side.
- MCP server at mcp.import.io
- Typed extraction to your schema on any page
- 10,000 free successful calls, $0.0002 per call after
- Rate, spend and domain policies you control
Public datasets you can inspect before you talk to us.
Sample releases from each supported category. Each ships with a schema, a per-document provenance manifest and a quality report produced by the same pipeline that builds full corpora. Download, evaluate, then specify what you need at scale.
Developer documentation
Official documentation, API references, READMEs and changelogs across major ecosystems, code blocks preserved with language tags.
- sample
- 1.2M documents · 2.6B tokens
- languages
- 32 natural · 41 programming
- freshness
- median 11 days at release
- format
- Parquet · JSONL · manifest
Multilingual web text
General web text balanced across 30 languages with per-document language ID confidence, script and region, boilerplate removed.
- sample
- 4.0M documents · 5.1B tokens
- languages
- 30 · min 50k docs each
- freshness
- median 19 days at release
- format
- Parquet · manifest
Scientific and technical
Open-access abstracts and full texts, standards summaries, technical reports and preprints, with citation and metadata extracted as fields.
- sample
- 380k documents · 1.9B tokens
- fields
- title, authors, abstract, refs, license
- freshness
- median 9 days at release
- format
- Parquet · JSONL · BibTeX
Commerce product catalog
Product titles, attributes, descriptions, prices, availability and categories from retailers and marketplaces, matched to GTIN where present.
- sample
- 10M products · 412M attributes
- coverage
- 1,200 retailers · 14 countries
- freshness
- median 2 days at release
- format
- Parquet · JSON schema
Public records and filings
Financial filings, court dockets, permits and registries structured into typed records with page and table citations.
- sample
- 620k documents · 212 jurisdictions
- fields
- entity, date, type, citation, source
- freshness
- median 6 days at release
- format
- Parquet · JSONL · PDF refs
Reviews and Q&A
Product reviews, ratings and question-and-answer threads with timestamps, verified flags and product linkage, PII removed.
- sample
- 25M reviews · 3.1M Q&A
- coverage
- eight product categories
- freshness
- median 4 days at release
- format
- Parquet · manifest
Sample releases are licensed for evaluation and research. Every manifest row carries source URL, fetch timestamp, content hash, language, license signal and robots status. Full corpora and custom domains: discuss your requirements.
State of Web Data for AI.
An annual measurement of how well the web can actually be turned into data: extraction accuracy, freshness, duplication and coverage, across site classes, with the methodology published so anyone can re-run it.
Edition 1 · Q4 2026
The first edition measures 50 site classes, from documentation and news to marketplaces and government portals, in 12 languages, using a fixed 10,000-URL panel refreshed monthly. Results are published with the panel, the scoring rubric and the raw scores.
Extraction accuracy
Field-level agreement between automated extraction and a human-labelled sample, per site class.
Why we publish it.
Every vendor claims accuracy and freshness. Almost none measure them in public against a fixed panel. Researchers building on web data need a citable baseline; buyers need a way to compare vendors that is not a sales deck. We run the infrastructure, so we measure it, and we publish what we find, including where we fall short.
The report is free. The panel and rubric are open. Contributions to the site-class taxonomy and the labelling protocol are welcome.
How the pipeline works.
From a specification to a manifest. Each stage is instrumented and its numbers appear in the quality report you receive with the data.
Discovery
Seed lists, sitemaps, link-graph expansion and search across 190 locales, constrained by your domain and exclusion rules. Robots and opt-out lists applied before a single fetch.
Fetch and render
Headless rendering for JavaScript sites, anti-bot handling, in-region egress, PDF and document parsing. Fetch success, latency and block rates recorded per host.
Extraction and structure
Main-content extraction with code, tables and math preserved; typed extraction to schema where the domain has one. Accuracy sampled by humans per site class.
Normalisation and deduplication
Language ID with confidence, boilerplate removal, encoding repair, exact and MinHash near-duplicate removal at the threshold you set, with rates reported.
Quality and policy
Length and quality scoring, PII detection and removal, license and robots signals attached to every document, opt-out honoured, audit log retained.
Provenance and delivery
Per-document URL, timestamp, content hash, language, license signal. Sharded Parquet or JSONL, manifest, quality report. S3, GCS, queue or API. Point-in-time versions kept.
The people who run it.
Infrastructure has names. These are the engineers responsible for what the pipeline does, and the ones who answer when a lab asks how.
Engineering notes.
Written by the people above, about the problems they actually solve.
- Near-duplicate removal at web scaleChoosing MinHash thresholds for training corpora: what 0.80 versus 0.90 does to token counts and downstream loss.
- Measuring extraction accuracy honestlyHow we build human-labelled samples per site class, and why field-level agreement beats document-level.
- Freshness as a distributionWhy p95 document age matters more than p50 for retrieval systems, and how cadence is set per source class.
- License and robots signals per documentWhat can be recovered automatically, what cannot, and how we represent uncertainty in the manifest.
- Rendering the modern web for extractionHeadless fleets, anti-bot economics and where in-region residential egress is worth its cost.
Published on the Import.io engineering blog. Platform and API reference at docs.import.io.
Requirement. Delivery. Measured result.
Programs that run today, each described the same way. Names appear where the customer has agreed to be named; the rest are available under NDA on a reference call.
Defence-technology AI company
AI decision platform · uses the data for model input · named reference under NDA
Continuous collection of open-source signals across hundreds of global sources in multiple languages and scripts, structured for ingestion by an AI analysis platform, with provenance sufficient for downstream audit.
Managed multi-source program with in-region egress, document and PDF parsing, typed extraction to the customer's schema, and per-record provenance delivered to their pipeline on schedule.
Multi-year production program with expanding source coverage and no rebuild of the customer's ingestion layer as sources were added.
Global marketplace
ticketing · uses the data in pricing models · reference on request
Sustained, high-frequency capture across 72 sources under a single SLA, feeding pricing and availability models that run continuously.
Enterprise program with contracted query volume and rollover, daily QA against the contracted schema, monitoring dashboard and named team.
100M+ queries under one SLA, 72 sources, delivered on schedule with coverage commitments met.
Home furnishings retailer
product attribution · uses the data to train and run AI attribute models · reference on request
Normalise competitor product content into a single attribute taxonomy so like-for-like assortment and price comparison could be automated.
Capture of competitor catalogs plus AI attribute extraction and visual classification against the customer's taxonomy, with human review on low-confidence values.
Attribute fill rate raised from 70% to 95% in one program cycle, with per-attribute confidence reported.
Three commerce-intelligence platforms
data companies · resell derived data and AI features · named on request
Retail and marketplace capture at scale to power pricing, digital shelf and matching products sold to their own customers, without the collection risk sitting on their books.
Managed feeds with contracted schemas, daily QA, pass-through SLAs and redistribution rights, one of them sustaining 450M queries a year on a single feed.
Multi-year renewals; the platforms sell the derived data under their own names.
Want a reference call matched to your use case? Ask for one. Customers using Import.io data specifically for model training, retrieval or evaluation are identified as such on the call.
Collected responsibly. Documented completely.
Labs are judged on where their data came from. Ours arrives with the answers attached.
- Provenance per documentSource URL, fetch timestamp, content hash, language, license signal, robots status. In the manifest, not in an email.
- Robots and termsRobots directives applied at discovery. Terms-of-service classes recorded. Paywalled and authenticated content excluded from training corpora by default.
- Opt-outsPublisher opt-out lists honoured and re-checked on each edition. A public opt-out route for site owners.
- Personal dataPII detection and removal with configurable sensitivity, GDPR and CCPA processing agreements, regional handling on request.
- Rate-aware collectionPer-host rate limits and backoff. Fourteen years of not being the crawler that took a site down.
- AuditEvery run logged; every delivered document traceable to the fetch that produced it; point-in-time versions retained.
Questions AI teams ask.
Short answers. Longer ones on the call, with the engineers.
What is web data infrastructure for AI?
The systems that turn the live web into data models can use: discovery, fetching and rendering, extraction, normalisation and deduplication, quality and policy filtering, provenance and delivery. Import.io runs that stack in production and offers it as training corpora, retrieval freshness, evaluation sets and live agent access.
Can you build a corpus to our specification?
Yes. You define sources, languages, date ranges, exclusions, deduplication thresholds, quality filters and output format. We deliver sharded Parquet or JSONL with a per-document manifest and a quality report, and can refresh it as editions.
How do you handle licensing, robots and opt-outs?
Robots directives are applied at discovery, license signals and robots status are recorded per document, publisher opt-out lists are honoured and re-checked each edition, and paywalled or authenticated content is excluded from training corpora by default. Agreements cover DPAs and research licences.
How is quality measured?
Extraction accuracy is human-sampled per site class at field level. Deduplication rates are reported before and after at the threshold you choose. Freshness is reported as p50 and p95 document age. All of it is in the quality report that ships with the data, and in the State of Web Data for AI methodology.
Can you keep our retrieval index current?
Yes. Continuous collection at hourly to daily cadence per source class, with new, changed and removed documents delivered as events with diffs and snapshots, and point-in-time versions retained.
Can our agents access the web through you?
Yes. The MCP server at mcp.import.io exposes search, fetch and render, extract-to-schema and monitor as tools, with rendering, anti-bot and egress handled server-side. 10,000 free successful calls, then $0.0002 per successful call, with rate, spend and domain policies you set.
Which languages and regions do you cover?
Any script and locale. Fetching runs from in-region egress across 190 locales, language identification carries a confidence score per document, and multilingual corpora can be balanced by language, region or script to your targets.
Can we evaluate before committing?
Yes. Download a sample dataset with its schema, manifest and quality report, or send us a specification and receive a pilot slice built by the same pipeline.