Product Data Enrichment Dashboard

Overview

A queue-driven enrichment dashboard that ingests CSV/Excel product files, looks them up against an internal nine-domain classification scheme, searches the brand web, scrapes structured specs, and uses an LLM to propose SEO-ready titles, descriptions, and HTML blocks. Every proposal lands in a review queue with a confidence score and a source URL. The system never silently overwrites master data.

Problem

Mike Sport's multi-brand catalog had thousands of products with sparse, inconsistent attributes spanning Adidas, Asics, Nike, Puma, and dozens of other brands. Pure-LLM enrichment was tempting but dangerous: a single hallucination could pollute master data and propagate to every downstream system. Manual entry was untenable. The team needed AI scale with human governance.

Approach

Treat the LLM as a research assistant, not a writer. Every enrichment becomes a proposal with provenance. The reviewer sees the source URL, confidence, and the LLM's reasoning, and approves or rejects per field. Provider-agnostic abstraction lets the model be swapped (Anthropic by default, OpenAI optional) without any change to callers.

Architecture

• Frontend: Next.js 14 + Tailwind. Excel/CSV upload, column auto-detection, inline editing, bulk approvals, audit trail.
• Backend: Express + TypeScript with Prisma ORM and PostgreSQL.
• Queue: BullMQ on Redis (optional, with a fast in-memory queue as fallback). Each enrichment task is a job with status, retries, and confidence-based routing. Parallel batch size is configurable (default 3), rate-limited to 50 RPM.
• AI provider: Pluggable. A single interface fronts the model; concrete classes implement Anthropic (default) and Google Gemini backends, routable via an AI_PROVIDER env var.
• Search and scrape: SerpAPI / Serper for brand web search, ScrapingBee for structured spec extraction. Both pluggable.
• Reference repository: A 9-domain classification index (Division, Category, Product Group, Family, Brand, Gender, Season, Country of Origin, HS Code) constrains the search space and gives every product a canonical place.

Tech stack

• Frontend: Next.js 14, React 18, TypeScript, Tailwind CSS
• Backend: Express 4.21, TypeScript, Prisma, PostgreSQL 16
• Queue: BullMQ, Redis (optional; in-memory fallback)
• AI: @anthropic-ai/sdk (Claude Haiku 4.5 default), Google Generative AI SDK (Gemini 2.0 Flash fallback)
• External: SerpAPI / Serper, ScrapingBee, custom web scraper
• Deploy: Docker Compose + Nginx reverse proxy at enrichment.mikesport.tech (Render blueprint also provided)

AI work

• Two-stage AI pipeline: stage one runs Claude Haiku with up to 4 web searches per product to discover data, stage two runs a second Haiku (or optional Sonnet) judge pass that verifies and refines the draft and rejects product substitutions. Cost lands around $40 per 1,000 rows with Haiku throughout, or about $60 for near-95% accuracy with a Sonnet judge. Synthesis is toggleable.
• Structured-output enrichment service with field-level confidence scores.
• Provider-agnostic adapter so models are swappable without changing callers.
• Prompt caching (write 1.25x, read 0.1x) factored into a real-time per-row, per-job, and cumulative cost model shown in the UI.
• Blacklisted domains (mikesport.com and regional variants) are never scraped or cited, preventing a self-referential data loop.
• Audit trail linking every enriched value to a source URL the reviewer can click.
• Governance-first design: AI never overwrites verified master data. It always proposes; humans confirm.
• Prompts versioned in the repo and tested against a sample fixture set.

Engineering highlights

• Column auto-detection: drop in a vendor's spreadsheet and the system maps their column names to the canonical schema using fuzzy matching plus an LLM tiebreaker.
• Confidence-gated review: high-confidence enrichments stream into a fast-approve view; low-confidence go to a deeper review queue.
• Bulk operations: approve, reject, or override hundreds of proposals at once.
• Export pipeline: enriched workbook + audit tabs, ready for downstream catalog systems.
• Retry-aware queue: BullMQ handles rate limits gracefully across SerpAPI, ScrapingBee, and the LLM.

Outcome

Live in production at enrichment.mikesport.tech on a Docker plus PostgreSQL stack behind Nginx. Confidence-gated review reduces manual effort dramatically while keeping the brand's data team in control. The "propose, never overwrite" pattern has become the team's default for any AI-assisted operation on master data.

Lessons

• The most valuable thing an LLM can produce in a data pipeline is a proposal with provenance, not an answer.
• BullMQ + Redis is overkill for a hundred items but exactly right for a thousand. The queue is the contract between the AI and the reviewers.
• The hardest part wasn't the model. It was building a review UX that a non-engineer can trust at speed.