Tokigen

A powerful e-commerce product scraping toolkit for AI models, implementing the Model Context Protocol (MCP).

Installation

npm install @modelcontextprotocol/tokigen

MCP Integration

Tokigen implements the Model Context Protocol (MCP) v1.0, providing a standardized interface for AI models to:

• Extract product data from e-commerce websites
• Stream results for efficient processing of large datasets
• Validate and transform data according to the MCP schema

Using with AI Models

import { mcp } from "@modelcontextprotocol/tokigen";

// Single batch product retrieval
const stream = await mcp.context.getProducts(
  "https://example-store.com/products"
);
for await (const product of stream) {
  // Process each product
}

// Streaming with custom options
const options = {
  batchSize: 20,
  maxConcurrent: 2,
  timeout: 60000,
};

for await (const product of mcp.context.streamProducts(
  "https://example-store.com/products",
  options
)) {
  // Process each product in real-time
}

Product Schema

Products follow this standardized schema:

interface Product {
  brand: string;
  name: string;
  category: string;
  category_en?: string;
  current_price: number;
  original_price?: number;
  currency: string;
  discount: boolean;
  discount_amount?: number;
}

Features

• 🤖 Full Model Context Protocol (MCP) v1.0 implementation
• 🌐 Headless browser scraping with Puppeteer
• ✨ TypeScript support with complete type definitions
• 🔄 Streaming support for large datasets
• ⚡ Rate limiting and retry mechanisms
• 🛡️ Zod-based schema validation
• 🔍 Flexible product data extraction
• 📊 Batch processing capabilities

Development

# Install dependencies
npm install

# Build the project
npm run build

# Run tests
npm test

# Validate MCP implementation
npm run mcp:validate

# Start development server
npm run dev

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

License

MIT License - see LICENSE file for details