NoCodeAPI

Try For Free
Login
View all articles

PDF Processing Pipeline – The Complete Technical Guide for Building AI-Ready Material Catalogs

Modern AI applications rely on clean, structured, and deeply connected data. But PDFs—especially complex material catalogs filled with images, specifications, and tables—remain one of the hardest data sources to process reliably.

At NoCodeAPI, we built a 14-stage intelligent PDF-to-AI pipeline that transforms raw supplier catalogs into high-quality, searchable knowledge with semantic embeddings, product metadata, image intelligence, and advanced relationships.

This post breaks down the entire pipeline end-to-end, covers the architectural decisions, and explains why each stage matters for AI-powered search, RAG systems, and product intelligence.

🚀 Pipeline Overview

A progressive “knowledge refinement system” from PDF ingestion to fully structured AI-ready data.

STAGE 0A → Product Discovery
STAGE 0B → Document Entity Discovery
STAGE 1 → Focused Extraction
STAGE 2 → Text Extraction
STAGE 3 → Semantic Chunking
STAGE 4 → Text Embeddings
STAGE 5 → Image Extraction + CLIP Embeddings
STAGE 6 → Image Analysis (Async)
STAGE 7 → Product Creation
STAGE 8 → Entity Linking
STAGE 9 → Completion

Each stage builds on the previous one with full checkpoint recovery and modular services.

🧠 Stage 0A — Product Discovery (0–10%)

Goal: Identify every product with its complete metadata in a single inseparable extraction pass.

Models: Claude Sonnet 4.5 or GPT‑4o

Products are extracted atomically: name, factory, specs, variations, metadata, and page ranges.

Example:

{
  "products": [
    {
      "name": "NOVA",
      "description": "Modern ceramic tile collection",
      "metadata": {
        "designer": "SG NY",
        "factory": "Castellón Factory",
        "dimensions": ["15×38", "20×40"],
        "slip_resistance": "R11",
        "material": "ceramic",
        "finish": "matte"
      },
      "page_range": [12, 14],
      "confidence": 0.95
    }
  ]
}

Stored in: products (JSONB)

🏛️ Stage 0B — Document Entity Discovery

Extracts non-product entities such as:

  • certifications
  • installation guides
  • factory documents
  • logos

Example:

{
  "certificates": [
    {
      "name": "ISO 9001:2015",
      "issuer": "TÜV SÜD",
      "factory_name": "Castellón Factory",
      "page_range": [45, 46]
    }
  ]
}

Stored in: document_entities

📄 Stage 1 — Focused Extraction (15–30%)

We use product page ranges to reconstruct a clean, product‑only PDF.

Benefits:

  • 40–60% speed boost
  • Cleaner chunking
  • Less noisy embeddings

🔠 Stage 2 — Text Extraction (30–40%)

Tool: PyMuPDF4LLM

Outputs structured markdown preserving hierarchy.

Example:

# NOVA
## Specifications
- Material: Ceramic
- Slip Resistance: R11
- Thickness: 8mm

## Description
Modern ceramic tile collection engineered for indoor/outdoor use.

🧩 Stage 3 — Enhanced Semantic Chunking (40–50%)

A 5-layer, product-aware chunking engine:

  1. Product boundary detection
  2. Semantic chunk segmentation
  3. Context enrichment (product_id, product_name)
  4. Metadata-first architecture
  5. Chunk‑to‑chunk relationships

Example:

{
  "id": "chunk_1",
  "content": "NOVA tiles provide...",
  "metadata": { "product_id": "prod_1" },
  "quality_score": 0.92
}

🧬 Stage 4 — Text Embeddings (50–60%)

Model: OpenAI text-embedding-3-small

  • 1536‑D vectors
  • Stored in pgvector
  • Indexed for fast RAG

🖼️ Stage 5 — Image Extraction + CLIP Embeddings (60–80%)

Extracts each image and generates 5 independent CLIP embeddings using Google SigLIP ViT‑SO400M:

  • visual
  • color
  • texture
  • application
  • material

Memory‑Safe Refinement:
10–15MB constant memory, even with 900+ images.

Example:

{
  "images_saved": 900,
  "clip_embeddings_generated": 4500,
  "memory_usage": "10-15MB",
  "processing_time": "3-5 seconds per image"
}

🔍 Stage 6 — Image Analysis (Async) (80–85%)

Model: Llama 4 Scout 17B Vision

Extracts:

  • OCR
  • materials
  • visible specs
  • quality score
{
  "ocr_text": "Material: Wool, 100%",
  "materials": ["Wool"],
  "quality_score": 0.87
}

🏗️ Stage 7 — Product Creation (85–92%)

Two‑step reasoning:

  • Haiku → Fast candidate detection
  • Sonnet → Deep product validation & metadata enrichment
{
  "product_id": "prod_1",
  "name": "NOVA",
  "metadata": { "material": "ceramic" },
  "confidence_score": 0.95
}

🔗 Stage 8 — Entity Linking (92–97%)

Connects all objects via weighted relevance:

  • 40% page overlap
  • 40% visual similarity
  • 20% confidence

Creates a graph of:

  • Product → Image
  • Chunk → Product
  • Chunk → Image

🎉 Stage 9 — Completion (97–100%)

Finalization:

  • validation
  • summary reports
  • job triggers

Document becomes fully AI‑ready.

🔄 Checkpoint Recovery System

Supports 9 checkpoints:

  • INITIALIZED
  • PDF_EXTRACTED
  • CHUNKS_CREATED
  • TEXT_EMBEDDINGS_GENERATED
  • IMAGES_EXTRACTED
  • PRODUCTS_DETECTED
  • PRODUCTS_CREATED
  • COMPLETED

Crash-safe and auto‑resumable.

📊 Performance (Real Example: Harmony PDF)

  • 71 pages
  • 11–14 products
  • 900+ images
  • 107 chunks
  • 4500 CLIP embeddings
  • 45–75 minutes
  • 10–15MB memory
  • 95%+ product detection accuracy

🏗️ Modular Architecture

Refactored from a 2900‑line script into:

Service Layer

  • ImageProcessingService
  • ChunkingService
  • RelevancyService

Internal API Endpoints

  • POST /api/internal/classify-images/{job_id}
  • POST /api/internal/upload-images/{job_id}
  • POST /api/internal/save-images-db/{job_id}
  • POST /api/internal/create-chunks/{job_id}
  • POST /api/internal/create-relationships/{job_id}

Main Orchestrator

POST /api/rag/documents/upload → returns job_id and starts the pipeline.

📌 Final Notes

  • Pipeline Version: 9‑Stage Optimized
  • Status: Production
  • Updated: Nov 18, 2025

Major improvements:

  • Combined image extraction + CLIP
  • Memory‑safe embedding generation
  • Full checkpoint recovery
  • Modular, testable architecture

🙌 Stay tuned @nocodeapi