References AI CLI

This CLI tool helps you to list all of document urls, and export all of your source code to feed ChatGPT.

Installation

You can install the CLI tool using npm:

npm install -g  @gpuless/references-ai

Usage

The CLI tool provides the following commands:

refer install: Crawls and downloads webpages based on a list of URLs defined in the package.json file under the references key. refer export: Generates a CSV file containing the contents of the downloaded webpages.

Crawling and Downloading Webpages

To crawl and download webpages, you need to define a list of URLs in your package.json file. Add a references key with an array of URLs as its value, like this:

{
  "references": [
    "https://example.com",
    "https://example.org"
  ]
}

Then, run the following command:

refer install

This command will crawl the webpages and download their contents to the dist/url_list directory. It will include URLs like https://example.com/sub/a.html and https://example.com/sub/sub/a.html but will not include https://example.com/sub2.

Generating a CSV File

To generate a CSV file containing the contents of the downloaded webpages, run the following command:

refer export

This command will generate a CSV file named all_code.csv in the dist directory. The CSV file will have the following columns: Content, Title, Url, and Page.

Files listed in the .gitignore file in the current directory will be ignored. Binary files, which contain a 1-byte null, will also be ignored.

Development

To set up the development environment, clone the repository and install the dependencies using npm:

git clone https://github.com/RAKUDEJI/References-AI
cd References-AI
npm install

To build the project, run:

npm run build

To enable live-reloading during development, run:

npm run dev

This command will watch for changes in the src directory and rebuild the project whenever a change is detected. It also installs the CLI tool globally, so you can use it by running the refer command.