AI News Pipeline Documentation

Process:

1. Fetch from 2 distinct systems:
   • RSS/feed collection: 210 curated feeds spanning 204 publishers and 35 categories (wire services, tier-1 tech press, research labs, regulator blogs, and more)
   • News APIs: NewsAPI & NewsData.io queries across 9 industry keyword sets
2. Initial Filtering:
   • Block known low-quality aggregators
   • Remove duplicate URLs at insert time
3. Source Name Completion:
   • Query the source registry for the complete publisher name
   • If not found, call AI to expand abbreviated names
   • Examples: "techcrunch" → "TechCrunch", "smh" → "The Sydney Morning Herald"
4. Deduplication:
   • URL uniqueness constraint plus published-date matching
   • Measured duplicate rate across the database: 0.00%

Process:

1. Canonical taxonomy:
   • 76 validated industries with fixed category mappings
   • Labels are validated against the taxonomy before they are written — no free-text variants
2. AI Classification:
   • Routine labelling runs on fast, cost-efficient models (Gemini 2.0 Flash class)
   • Premium models are reserved for complex or ambiguous articles and quality assurance
   • Prompt includes title, summary, and content preview
3. Classification Output:

   {
     "category": "AI in Healthcare",
     "industry": "Healthcare",
     "impact_level": "HIGH",
     "confidence": 95,
     "reasoning": "Article covers FDA-approved AI diagnostic tool"
   }

4. Fair rotation:
   • Intake is balanced across industries so high-volume sectors cannot starve quieter ones
   • Every one of the 76 industries receives coverage over each collection window

Process:

1. Source-specific extraction strategies:
   • Harvard Business Review: extract from article body
   • arXiv: parse abstract and paper sections
   • Medium/Substack: handle paywall content
   • VentureBeat/TechCrunch: extract from article containers
   • Generic: strip HTML, decode entities, extract text
2. Validation checks:
   • Minimum content length
   • Valid URL structure
   • Not all-caps spam
   • Contains readable text
3. Park-and-retry, never delete:
   • Failed scrapes are parked and retried on later runs with fresh strategies
   • Articles are never deleted for scrape failures — a temporarily unreachable article stays in the queue

Process:

1. AI Content Extraction:
   • Objective: extract ONLY the article body — no navigation, ads, or footers
   • Articles are retried until cleaned; nothing is discarded
2. AI Prompt Instructions:
   • Remove: navigation, cookie notices, ads, "share this", related articles, breadcrumbs
   • Keep: full article text, quotes (exact wording), statistics, author names in content
   • Preserve: paragraph structure, subheadings, bullet points
   • Detect incomplete articles: flag with "INCOMPLETE_CONTENT: [reason]" and retry later
3. Validation:
   • Word count recalculated from the cleaned text
   • Cleaned content stored alongside the raw scrape for auditability

Process:

1. Structural Quality Assessment (0-100 scale):
   • HTML Tag Density: penalty for excessive markup residue
   • Line Break Issues: penalty for 3+ consecutive newlines
   • Special Character Noise: penalty for repeated symbols
   • Encoding Problems: penalty for unresolved HTML entities
   • Short Paragraphs: penalty for stub paragraphs
   • Repetitive Text: penalty for duplicate sentences
2. Citation/Attribution Analysis (0-100 scale):
   • Named source quotes
   • External links to references
   • Data points: percentages, dollar figures, user counts
   • Company mentions: OpenAI, Google, Microsoft, etc.
3. Scoring outcome:
   • Score < 70: marked for re-cleaning
   • Score ≥ 70: approved for analysis use

6A. Source Credibility Scoring (0-100)

• Tier 1 Sources (85-95): Bloomberg, Reuters, OpenAI Blog, Nature, Science
• Tier 2 Sources (70-84): TechCrunch, VentureBeat, Forbes, HBR, Wired
• Tier 3 Sources (55-69): Medium, Substack, company engineering blogs
• Unknown Sources (50-64): Deterministic hash-based scoring (no randomness)

6B. Peer Review Detection

Check for: DOI, PubMed ID, arXiv ID, "peer-reviewed" mentions

Bonus points, NOT a requirement (real AI news is rarely peer-reviewed)

6C. Content Type Classification

• Long-form: 1200+ words
• Standard: 400-1199 words
• Brief: 150-399 words
• Breaking News: <150 words + <48 hours old
• Press Release: detected by keywords ("announces", "launches")

6D. Content Completeness Score (0-100)

Adjusted by content type:

• Title quality: 15 points (length, clarity)
• Summary quality: 15 points (length, informativeness)
• Content depth: 35 points (word count, adjusted by type)
• Structure: 10 points (headings, paragraphs, lists)
• Metadata: 25 points (URL, image, date, author, tags)

6E. Citation/Attribution Score (0-100)

Real-world journalism standards:

• External links: 20 points (up to 4 unique domains)
• Attributed quotes: 25 points (named person + quote)
• Data-driven references: 25 points (%, $, user counts)
• Named companies: 15 points (OpenAI, Google, etc.)
• Research references: 15 points ("study shows", "report finds")

6F. Red Flag Detection

• Critical: no content (<30 words), all-caps title
• Warning: sensational language, clickbait patterns, promotional content
• Minor: no author, no publish date, short title

6G. AI Deep Quality Analysis

Premium model rubric (0-100 scale):

• Reporting Substance: 30 points (facts, sources, context)
• Factual Specificity: 25 points (numbers, attribution)
• Professional Quality: 20 points (writing clarity)
• Business Relevance: 15 points (strategic value)
• Information Value: 10 points (newsworthiness)

6H. Composite Quality Rating (0-5 stars)

quality_rating = (
  source_credibility * 0.25 +
  content_completeness * 0.25 +
  citation_score * 0.20 +
  ai_deep_score * 0.25 +
  peer_review_bonus * 0.05
) / 20 - red_flag_penalty

6I. Quality Badge Assignment

• Verified: Source ≥ 85 + Quality ≥ 4.0 + Peer-reviewed
• Premium: Source ≥ 75 + Quality ≥ 3.5
• Standard: Quality ≥ 2.5
• Unverified: Quality < 2.5

How work moves through the pipeline:

• Bounded batches: every stage processes a capped batch per run, so no single run can time out or monopolize resources
• Leases, not locks: work items are claimed with short leases; a crashed run releases its work automatically for the next run
• Park-and-retry: items that fail repeatedly are parked with their failure history and retried on later runs — never silently deleted
• Fair industry rotation: intake and processing rotate across all 76 industries so every sector is covered in every window
• Nothing deleted: articles, content, and history are retained; the database grows as a permanent, auditable record

Weekly AI Trends Report

Generated once a week on Sunday morning.

• Analyzes the week's qualified articles against a year-to-date and rolling 90-day context window
• Includes a prior-week comparison so movement week-over-week is explicit
• Multi-phase generation with a single-flight lease — only one report run can be active at a time, and a stalled run is detected and recovered automatically

On-Demand AI News Reports

Separate credit-based reports that subscribers can generate on demand from the same qualified-article pool.

Article eligibility for analysis

• Cleaned content available
• Quality rating at or above the analysis threshold
• Industry classified against the canonical taxonomy
• No critical fact-check flags