Uniqhtt

Welcome to Uniqhtt, a powerful and versatile Node.js library designed to handle advanced HTTP requests, response processing, and web automation. This library stands out with its comprehensive support for cookie management, HTML parsing, CSS transformation, and Wget-like recursive website cloning features. Uniqhtt provides a solid solution for web scraping, data extraction, and automation tasks, making it a valuable tool for developers working in diverse environments ranging from Node.js to serverless platforms and browsers.

Table of Contents

• Key Features
• Installation
• Basic Usage
• HTTP Methods
• Request Options
• HTTPS Options
• Cookie Management
• Wget-like Options
• Storage Options
• Advanced Features
• Error Handling
• TypeScript Support
• Contributing

Key Features

Uniqhtt brings a suite of robust features to empower your HTTP requests and web automation workflows:

1. Cross-Platform Compatibility: Runs seamlessly in Node.js environments, browsers, and edge computing platforms.
2. Advanced Cookie Management: Utilizes tough-cookies for sophisticated session handling, making it easy to manage and persist cookies.
3. HTML Parsing & Manipulation: Powered by jsdom, enabling developers to parse and manipulate HTML structures with ease.
4. CSS Processing: Supports CSS transformations and processing via postcss.
5. Recursive Website Cloning: Emulate Wget-like behavior for cloning entire websites with options for recursion, rate limiting, and user-agent customization.
6. Flexible Storage Options: Store data locally, on S3-compatible storage (including Cloudflare R2), or via FTP.
7. Wget-like Functionality: Clone websites, respect robots.txt, and support recursive downloads with granular control.
8. Customizable User Agents: Define custom user agents to mimic browser requests or other clients.
9. Rate Limiting: Prevent overloading servers with configurable rate limits.
10. Parallel Downloads: Handle multiple downloads concurrently to increase performance.
11. Error Handling: Advanced retry mechanisms, detailed error reporting, and customizable error handling.
12. TypeScript Support: Full support for TypeScript, offering type safety and enhanced development experience.

Installation

To install Uniqhtt, use npm or yarn:

npm install uniqhtt

Or, using yarn:

yarn add uniqhtt

This will install Uniqhtt along with its dependencies, including jsdom, fetch-cookie, and postcss.

Basic Usage

Uniqhtt provides a simplified API for making HTTP requests similar to popular libraries like Axios, while offering advanced features for handling cookies, scraping, and cloning websites.

Example: Simple GET Request

const uniqhtt = require('uniqhtt');

uniqhtt.get('https://example.com')
  .then(response => {
    console.log(response.data); // HTML or JSON response
  })
  .catch(error => {
    console.error('Error occurred:', error);
  });

Example: POST Request with Cookies

uniqhtt.post('https://example.com/login', {
  data: {
    username: 'user',
    password: 'pass',
  },
  cookies: true
})
  .then(response => {
    console.log('Login successful:', response.data);
  })
  .catch(error => {
    console.error('Login failed:', error);
  });

For a comprehensive guide on using Uniqhtt, keep reading the detailed documentation below.

HTTP Methods

Uniqhtt supports all major HTTP methods, enabling it to cover a wide range of use cases for interacting with RESTful APIs or handling custom HTTP interactions.

• GET: Retrieve data from a server.
• POST: Submit data to be processed by a server.
• PUT: Update existing data on a server.
• DELETE: Remove data from a server.
• PATCH: Apply partial modifications to data.
• HEAD: Retrieve metadata about a resource.
• OPTIONS: Get supported HTTP methods for a given URL.

Request Options

Uniqhtt's request options mirror the popular Axios-like structure, making it easy for developers familiar with other libraries to integrate.

Common Options:

• url: The URL to which the request is sent (string or URL object).
• method: The HTTP method (default: GET).
• headers: An object representing custom HTTP headers to send along with the request.
• body: The body of the request, used in methods like POST, PUT, and PATCH.
• params: URL parameters to append to the request URL (object).
• timeout: Request timeout in milliseconds.
• followRedirect: Automatically follow HTTP 3xx responses (boolean).
• maxRedirects: The maximum number of redirects to follow.
• compress: Request a compressed response if supported by the server (boolean).
• agent: Custom HTTP or HTTPS agent for more advanced network handling.
• auth: Basic authentication credentials (object with username and password).
• responseType: Expected type of response (json, text, arraybuffer, etc.).
• validateStatus: Function to determine if the HTTP response status code should be considered successful.
• transformRequest: Function to modify request data before sending it to the server.
• transformResponse: Function to process response data before returning it to the caller.

Example: Custom Headers

uniqhtt.get('https://api.example.com/data', {
  headers: {
    'Authorization': 'Bearer token',
    'Content-Type': 'application/json'
  }
})
  .then(response => console.log(response.data));

HTTPS Options

For secure communication, Uniqhtt offers flexible HTTPS options to handle advanced scenarios, such as mutual TLS.

• rejectUnauthorized: Disables SSL certificate verification (boolean).
• cert: Client certificate for mutual TLS authentication.
• key: Client key for mutual TLS.
• ca: Custom Certificate Authority (CA) bundle.
• pfx: PKCS#12 formatted key and certificate chain.
• passphrase: Passphrase for encrypted private key or PFX file.

Cookie Management

Uniqhtt integrates tough-cookies for advanced cookie handling, making it a great tool for web scraping or interacting with stateful HTTP sessions.

Key Cookie Management Methods

• setCookies(cookies, startNew): Set cookies for subsequent requests.
• getCookies(): Retrieve current cookies in use.
• clearCookies(): Clear all cookies.
• parseCookies(cookieString, url): Parse a raw cookie string into individual cookies based on the provided URL.

Example: Persistent Session with Cookies

uniqhtt.setCookies([{ name: 'session', value: 'abc123', domain: 'example.com' }]);

uniqhtt.get('https://example.com/dashboard')
  .then(response => {
    console.log('Dashboard data:', response.data);
  });

Wget-like Options

Uniqhtt can emulate Wget functionality, enabling recursive website downloading, link rewriting, and local storage of complete websites.

Key Options:

• recursive: Download recursively (boolean).
• level: Set the depth of recursion.
• waitBetweenRequests: Pause between requests (milliseconds).
• limitRate: Throttle the download speed (bytes per second).
• userAgent: Specify a custom user agent string.
• convertLinks: Convert links from relative to absolute.
• retryAttempts: Set the number of retries for failed requests.
• cookies: Enable or disable cookie support during scraping.

Example: Cloning a Website

uniqhtt.clone('https://example.com', {
  recursive: true,
  level: 3,
  convertLinks: true,
  outputDirectory: './website_clone'
})
  .then(() => console.log('Website cloned successfully!'))
  .catch(error => console.error('Error while cloning website:', error));

Storage Options

Uniqhtt supports various storage options to fit different needs for persisting downloaded content.

• Local Storage: Save data directly to the local file system.
• S3-Compatible Storage: Compatible with AWS S3 and services like Cloudflare R2.
  • s3Config: Configuration object with bucket, region, accessKeyId, and secretAccessKey.
• FTP: Save content via FTP.
  • ftpConfig: Configuration object with host, port, user, and password.

Advanced Features

Uniqhtt offers several advanced features to enhance its capabilities:

• Parallel Downloads: Support for downloading multiple files simultaneously.
• Rate Limiting: Prevent overwhelming servers with too many requests too quickly.
• OnProgress & OnFinished Callbacks: Track progress and completion of long-running tasks.
• Uniqhtt Instance for Cloudflare Workers: Use the library with Cloudflare R2 by passing an R2 bucket to the createUniqhtt() function.

Error Handling

Uniqhtt provides comprehensive error handling, ensuring developers have access to all necessary details when something goes wrong:

• Detailed Error Information: Errors include status codes, request details, and response data.
• Custom Error Classes: Handle specific error scenarios more effectively by extending built-in JavaScript error types to provide better context. For example, a RequestError might include details about the failed HTTP request, while a ResponseError could provide insight into the server response that caused the failure.
• Retry Mechanisms: Automatically retry failed requests based on customizable retry logic. You can specify the number of retry attempts and the delay between retries, making Uniqhtt robust for unstable network environments or transient server issues.

Example: Custom Error Handling with Retries

uniqhtt.get('https://example.com/data', {
  retryAttempts: 3,  // Retry up to 3 times
  retryDelay: 1000   // Wait 1 second between retries
})
  .then(response => {
    console.log('Data received:', response.data);
  })
  .catch(error => {
    if (error.isRetryError) {
      console.error('Request failed after multiple retries:', error);
    } else {
      console.error('An unexpected error occurred:', error);
    }
  });

TypeScript Support

Uniqhtt is fully compatible with TypeScript, offering a strong typing system that enhances developer experience by catching errors at compile time and providing better IntelliSense support. TypeScript definitions are bundled with the package, ensuring you don’t need to install additional typings.

Example: TypeScript Usage

import uniqhtt from 'uniqhtt';

uniqhtt.get('https://example.com')
  .then((response: uniqhtt.UniqhttResponse<string>) => {
    console.log(response.data);  // Data typed as string
  })
  .catch((error: uniqhtt.UniqhttError) => {
    console.error('Error occurred:', error.message);
  });

TypeScript Types Overview

• UniqhttRequestConfig: Interface defining the options for making HTTP requests.
• UniqhttResponse: Interface for handling the structure of HTTP responses, with generic typing to represent the expected response data format.
• UniqhttError: Custom error type to encapsulate errors encountered during requests.
• UniqhttCookieOptions: Interface defining cookie-related options for requests.

By utilizing TypeScript, Uniqhtt ensures that developers can take full advantage of type safety, making it easier to spot errors early in the development process and ensuring higher quality code.

Contributing

We welcome contributions to Uniqhtt! Whether you're fixing bugs, improving documentation, or adding new features, your input is valuable.

Steps for Contributing

1. Fork the repository on GitHub.
2. Create a new branch for your feature or bugfix.
3. Write clear commit messages and document any changes made.
4. Ensure all new features include appropriate unit tests and documentation updates.
5. Open a pull request, clearly describing the changes and providing context if necessary.

Before submitting, make sure your code passes all tests and linting checks to ensure consistency and quality.

Conclusion

Uniqhtt is a comprehensive, advanced HTTP client for Node.js, offering a range of features designed to handle everything from basic HTTP requests to complex web scraping and automation tasks. Its support for cookie management, recursive website cloning, parallel downloads, flexible storage, and advanced error handling makes it a powerful tool for developers working on a wide variety of projects.

Whether you're building a scraping engine, an API client, or an automated testing tool, Uniqhtt provides the tools you need in a simple, highly customizable package. With features like TypeScript support, retry mechanisms, and seamless cookie handling, Uniqhtt is designed to handle modern web interactions efficiently and securely.

For more detailed examples, head over to the [official documentation] and start exploring all that Uniqhtt has to offer!

Happy coding! 🎉

License

Uniqhtt is released under the MIT License. See the LICENSE file for more details.