Original Data Pipeline: How AI Collects Proprietary Evidence for Every Article

What this article answers

• How does the pipeline collect original data for articles automatically?
• What source types does the Artifact Generation System pull from?
• How does the Original Data Bridge connect Mentions and Reddit scans to content?
• What prevents the system from fabricating data or hallucinating sources?
• Why is original data the most important factor for AI citations?

Key takeaways

• Original data is the single biggest differentiator for AI citations - generic industry stats have zero competitive value because AI already has them from thousands of sources.
• The pipeline collects from six source types - NewsAPI, Reddit, YouTube, podcasts, Substack, and Medium - plus an optional Trade Sonar pass for B2B-industrial topics. Each item is condensed into structured extractions.
• The Original Data Bridge automatically converts Mentions and Reddit scan results into knowledge items the article pipeline can consume - no manual data entry required.
• Deduplication uses URL matching plus title fuzzy matching, and every claim must reference a verifiable source URL - preventing fabrication at the system level.
• Articles built with artifact-sourced data contain current statistics, named expert contradictions, and time-horizon predictions that no competitor can replicate.
• The full pipeline runs automatically during article creation: Wave 1 + Wave 2 collect sources, the synthesizer assembles a 13-section artifact in runtime state, and B-04/B-05/B-07/B-24 weave the data into prose.

Why Is Original Data the Deciding Factor for AI Citations?

AI engines skip pages that restate what they already know, citing only content with information unavailable elsewhere, which makes original data the single deciding citation factor.

AI engines have been trained on the entire public internet. When ChatGPT or Claude encounters a page that restates what they already know from thousands of other sources, there is no reason to cite that specific page. The information is redundant. The page adds nothing.

Original data changes the equation. When your article contains a statistic from a government database published last month, a contradiction between two named industry experts sourced from different publications, or a trend analysis synthesized from 15 Reddit threads - the AI engine encounters information it cannot get elsewhere. That is what triggers a citation.

Here is the test we apply to every paragraph: remove the brand name and ask whether this content could appear on a competitor’s site. If a home health care company publishes “the industry is growing rapidly and technology is improving outcomes” - that sentence could appear on any of 500 competing sites. It is not original. But if the same company publishes “CMS reduced the home health payment rate by 1.7% for 2026, while agencies using remote patient monitoring report 23% fewer 30-day readmissions according to JAMA Network” - that specific synthesis of current data points, with sources, is original analytical work.

The problem is that producing original data manually is expensive. Researching current statistics, finding expert contradictions, tracking sentiment shifts across Reddit and industry forums - this takes hours per article. Most content teams skip it and default to whatever the LLM generates from training data. The Original Data Pipeline automates the entire process, making every article rich with proprietary evidence by default.

How Does the Artifact Generation System Work?

The Artifact Generation System runs before any writing block, collecting from six source types across two waves, condensing each piece, then storing structured corpus for downstream use.

The Artifact Generation System runs automatically during article creation - before any writing blocks execute. It collects live data from six source types in two waves, condenses each piece into structured extractions, and stores the resulting corpus in runtime state so every downstream block can read it.

Collection runs in two waves. Wave 1 hits free API sources that always run: NewsAPI returns the most relevant recent articles for the topic queries (up to 5 items, deduped). Wave 2 uses ScrapingDog SERP to discover content on Reddit (up to 10 threads with comments), YouTube (up to 3 videos with transcripts), podcasts (up to 3 with transcripts where available), Substack (up to 3 articles), and Medium (up to 3 articles). B2B-industrial topics trigger an additional Trade Sonar pass in Wave 2 to surface industry-specific coverage.

Every scraped piece goes through LLM condensation using a Claude Code CLI subprocess (claude -p with a logged-in session) rather than the Anthropic SDK - the same Claude session you use interactively, called automatically. The condensation prompt extracts every meaningful fact, data point, statistic, expert opinion, prediction, entity mention, disagreement, and trend indicator. The output is structured into six categories: key facts with source attribution, opinions and predictions, entities with roles, notable quotes with attribution, trends and signals, and methodology references. This is comprehensive fact extraction - not summarization.

After condensation, deduplication removes redundant results using URL matching and title fuzzy matching. The deduplicated corpus is stored in the runtime web_corpus state so every writing block can read it during generation. A synthesizer pass produces a 13-section structured artifact with themes, entity maps, predictions at three time horizons, contradictions between sources, trend analysis, interesting standalone facts, and contrarian takes - all grounded in the collected sources.

What Does the Original Data Bridge Connect?

The Original Data Bridge unifies knowledge items, client insights, and mentions and Reddit scan data through one knowledge table, feeding every article pipeline automatically.

Before the Original Data Bridge, three data systems existed in isolation. Knowledge items lived in their own table and only the Reddit drafter consumed them. Client insights were manually entered and fed into B-46 and writing blocks. Mentions and Reddit scan data powered the visibility UI but never reached the article pipeline. Three pools of proprietary data, none talking to each other.

The bridge unifies all three through the knowledge items table as the universal original data layer. Mentions scans and Reddit scans now automatically generate knowledge items after every run. When you scan a domain’s brand mentions across the web, the system creates data point items with current citation counts, sentiment trends, and competitor comparison metrics. When a Reddit scan finds threads discussing the client’s industry, it creates case study and expert items from the highest-signal discussions.

These knowledge items then flow into the article pipeline through the same channel as manually entered client insights. They are loaded into runtime state alongside the collected web corpus, and every writing block reads from that state during generation. B-04 (the Outline Generator) uses the executive summary and top themes to choose sections. B-05 (the Section Brief Writer) attaches specific knowledge items to each section brief so individual paragraphs cite proprietary data. B-07 (the Section Writer) weaves the cited claims into prose. A data point knowledge item becomes a UNIQUE_CLAIM with type proprietary_data. A case study item becomes an OWNED_INSIGHT. An expert item gets attributed by name.

The practical result: a client who has run a Mentions scan and a Reddit scan before creating their first article will see that article automatically incorporate real brand mention data, actual Reddit sentiment, and competitive positioning - all without anyone manually typing a single insight. The pipeline fetches up to 30 knowledge items per domain, grouped by type, and injects them with clear instructions to weave them naturally and never fabricate beyond what the items actually state.

How Do Deduplication and Source Validation Prevent Fabrication?

Every claim must trace to a source URL, with condensation prompts attaching attribution and deduplication running on both URL and semantic dimensions before synthesis.

Fabrication prevention is built into every stage of the pipeline, not bolted on as a final check. The system operates on a fundamental constraint: every claim must trace back to a verifiable source URL.

At the collection stage, every scraped piece retains its source URL, publish date, and domain. The condensation prompt explicitly instructs the model to extract facts with source attribution - if a fact cannot be attributed to the specific source being condensed, it does not enter the corpus. This prevents the LLM from injecting its own training data during the extraction phase.

Deduplication runs after condensation and before synthesis. It operates on two dimensions. URL-level deduplication catches the same page appearing across multiple search queries or source types. Title-level fuzzy matching catches the same story reported by different outlets - keeping the most detailed version and discarding near-duplicates. This prevents the synthesis stage from over-weighting a single data point that appeared in multiple sources.

During synthesis, the 13-section artifact structure enforces source grounding at every level. Each theme must list source URLs as key evidence. Each prediction must include a reasoning chain referencing specific sources. Each contradiction must name both sides with their respective source URLs. The interesting facts section requires a source URL and attribution for every entry. If the model cannot ground a claim in the collected corpus, the claim does not make it into the artifact.

The writing blocks that consume the artifact inherit this provenance chain. When B-07 (the Section Writer) writes a paragraph citing a statistic, that statistic traces back through the artifact to the condensed extraction to the original scraped source. The system does not generate claims and then look for sources to support them. It collects sources first, extracts their claims, and then synthesizes those specific claims into article content.

What Does the Full Pipeline Look Like End to End?

Five stages run automatically: plan sources, collect and condense, deduplicate, synthesize, then store in runtime state so writers cite real evidence-library URLs.

The pipeline executes five stages during every article creation, running automatically without manual intervention.

Stage 1 - Plan Sources. When article creation begins, an AI model receives the article topic, client industry, and client region, then chooses query terms for each source type. NewsAPI (Wave 1) always runs. SERP-based discovery for Reddit, YouTube, podcasts, Substack, and Medium runs in Wave 2. B2B-industrial topics trigger an additional Trade Sonar pass.

Stage 2 - Collect and Condense. The system fires parallel requests using ScrapingDog for SERP queries and direct fetches for content. Rate limiting uses a token bucket pattern shared with the Mentions system. Each scraped result goes through Claude Code CLI condensation (one claude -p subprocess per item), producing structured extractions in parallel.

Stage 3 - Deduplicate and Validate. URL-level and title-level fuzzy matching removes redundant results. Source URLs are validated as resolvable. The deduplicated corpus is stored in the runtime web_corpus state so it persists with the article and can be inspected later.

Stage 4 - Synthesize Artifact. A synthesizer pass reads the deduplicated corpus alongside any existing knowledge items from the Original Data Bridge. It produces a 13-section artifact: executive summary, themes with source counts and emerging/sustained/fading status, sentiment analysis by source type and entity, entity map with roles and positions, predictions at short/medium/long time horizons with confidence levels, defensible theses with counter-arguments, temporal trends, contradictions between sources, statistical summary with chart data, interesting standalone facts, image generation prompts, contrarian takes, and identified knowledge gaps.

Stage 5 - Inject into Blocks. The artifact and knowledge items flow into every writing block via runtime state. B-04 (Outline Generator) consumes the executive summary and top themes. B-05 (Section Brief Writer) attaches specific facts, contradictions, and entity context to each section brief. B-07 (Section Writer) weaves those source-grounded claims into prose. B-24 (FAQ Generator) draws from interesting facts and contrarian takes. The result is an article where every substantive claim traces back to a live source collected during that specific article’s creation.

The original data pipeline runs five stages that each contribute a different intelligence layer to the final article.

Where Can You Learn More About Original Data for AI Visibility?

• Schema.org Article Type Reference -https://schema.org/Article
• Schema.org CreativeWork isBasedOn Property -https://schema.org/isBasedOn
• Google Article Structured Data -https://developers.google.com/search/docs/appearance/structured-data/article
• Anthropic Claude Models Overview -https://docs.anthropic.com/en/docs/about-claude/models
• OpenAI ChatGPT Web Search -https://platform.openai.com/docs/guides/tools-web-search

Related topics

• original-data
• brand-mention-monitoring
• reddit-ai-discovery
• content-cluster-strategy