Odds Portal Scraper CLI

A command line tool for scraping soccer odds data from Odds Portal. The tool supports both historical odds data and upcoming matches across multiple leagues and formats.

Features

• Scrape historical odds data for specified date ranges
• Get odds for upcoming matches
• Support for multiple leagues and odds formats
• Export data to local files or S3 buckets
• Built with Puppeteer for reliable scraping
• Configurable logging and error handling

Installation

Prerequisites

• Node.js (v18 or higher)
• npm or yarn
• AWS CLI configured (for S3 exports)

Install via npm

npm install -g odds-portal-scraper

Install via yarn

yarn global add odds-portal-scraper

Usage

Commands Overview

odds-portal [command] [options]

Available Commands

| Command | Description | |-----------------|--------------------------------------------------| | historic | Scrape historical odds data | | next-matches | Scrape odds for upcoming matches | | soccer-leagues | List available leagues | | odds-format | List supported odds formats |

Command Details

Historic Odds

Scrape historical odds data for a specified league and date range.

odds-portal historic <leagueName> <startYear> <endYear> [options]

Parameters:

• leagueName: Name of the league (use soccer-leagues command to see available options)
• startYear: Starting year for historical data (e.g., 2020)
• endYear: Ending year for historical data (e.g., 2023)

Options:

| Option | Description | |-----------------|--------------------------------------------------| | --odds-format | Odds format (default: EU) | | --local | Local directory to save JSON files | | --s3 | S3 bucket name to upload data |

Example:

odds-portal historic premier-league 2020 2023 --local ./data --odds-format eu

next-matches

Scrape odds for upcoming matches in a specified league.

Scrape next matches odds data for a given soccer league.

Parameters:

• leagueName: Name of the league (use soccer-leagues command to see available options)

Options:

| Option | Description | |-----------------|--------------------------------------------------| | --odds-format | Odds format (default: EU) | | --local | Local directory to save JSON files | | --s3 | S3 bucket name to upload data |

odds-portal next-matches premier-league --local ./data --odds-format eu

This will scrape next matches odds data for the English Premier League.

Soccer Leagues

List all available leagues for scraping.

odds-portal soccer-leagues

Odds Formats

List all supported odds formats.

odds-portal odds-format

Configuration

Environment Variables

| Variable | Description | |------------------------|--------------------------------------------------| | ODDS_PORTAL_PROXY_URL | Proxy URL for scraping | | AWS_ACCESS_KEY_ID | AWS access key for S3 exports | | AWS_SECRET_ACCESS_KEY | AWS secret key for S3 exports |

Data Structure

Scraped data is saved in JSON format with the following structure:

{
  "scrapedAt": "2023-10-15 14:30:00",
  "day": "Sunday",
  "date": "2023-10-15",
  "time": "14:00",
  "homeTeam": "Manchester United",
  "awayTeam": "Liverpool",
  "mlFirstHalf": [
    {
      "bookMakerName": "Bet365",
      "hw": "2.10",
      "d": "3.25",
      "aw": "3.50"
    }
  ],
  "mlSecondHalf": [...],
  "mlFullTime": [...],
  "leagueName": "premier-league",
  "underOver25": [
    {
      "bookmakerName": "Bet365",
      "oddsOver": "1.80",
      "oddsUnder": "2.00"
    }
  ],
  "underOver15": [...],
  "underOver35": [...]
}

Error Handling

The tool includes comprehensive error handling with the following features:

• Automatic retries for failed requests
• Detailed error logging
• Graceful shutdown on critical errors
• Validation of input parameters

Contributing

Contributions are welcome! Please follow these steps:

1. Fork the repository
2. Create a new branch (git checkout -b feature/your-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin feature/your-feature)
5. Create a new Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For support or feature requests, please open an issue on the GitHub repository.