github-resource-converter

v1.1.6

Published

3 years ago

Convert and export GitHub resources--Issues and Pull Requests--to CSV and JSON.

Downloads

`github-resource-converter`

Convert GitHub Issues and Pull Requests to JSON and CSV from a Terminal or Node.js app.

1. Overview

github-resource-converter (alias grc) exports your GitHub and GitHub Enterprise repositories' Issues and Pull Requests to CSV and JSON file formats. It's helpful whenever you need to:

Use spreadsheets to analyze, modify, print, or summarize large amounts of data with Pivot Tables and other important financial or statistical operations

Example:

# Convert all GitHub issues to CSV:
$ github-resource-converter \
  --owner foo \
  --repo  bar
# => Saved "foo-bar-issues-export.2018-03-20T02_11_04_356Z.csv".

Share data with other tools like GitLab and JIRA.

Example:

# Convert and save all GitHub Enterprise
# Pull Requests as JSON (using the grc alias):
$ grc \
  --owner gregswindle \
  --repo github-resource-converter \
  --resource-type pr \
  --dest './docs/reports/export.json'
# Saved "docs/reports/gregswindle-github-resource-converter-pr-export.2018-03-20T02_18_33_682Z.json".

2. Installation

Required: github-resource-converter is written in JavaScript (CommonJS) for Node.js version

which must be installed prior to use. Node.js requires npm, which is used for installing dependencies. (npm installs with Node.js.)

Recommended: To avoid rate-limiting, you should create a personal access token External link and save your personal access token.

MacOS and Unix

$ mkdir -p /usr/local/etc/github-resource-center/envvars/
$ touch /usr/local/etc/github-resource-center/envvars/.env
$ echo "GITHUB_ACCESS_TOKEN="{your-personal-access-token-value}" > \
  /usr/local/etc/github-resource-center/envvars/.env

Windows

> md -p C:\usr\local\etc\github-resource-center\envvars\
> touch C:\usr\local\etc\github-resource-center\envvars\.env
> echo "GITHUB_ACCESS_TOKEN="{your-personal-access-token-value}" >>
  C:\usr\local\etc\github-resource-center\envvars\.env

2.1. For Terminal/command-line usage

# Install globally to execute from a Terminal/command-line
$ npm i -g github-resource-converter

2.2. As a application dependency

# Install as a dependency within a Node.js app
$ npm i --save github-resource-converter

3. Usage

The following examples assume that github-resource-converter is installed globally and invoked from a Terminal (command-line interface)

3.1. Formatting

--dest export.[csv|json]

You can convert GitHub (Enterprise) Issues and Pull Requests into two file formats: CSV and JSON.

CSV is the default format.
JSON formatting requires a --dest value with a .json file extension.

3.2. Resource types

--resource-type, -t

Convert and export GitHub (Enterprise) by --resource-type: issues, pull_requests, or both (all).

issues is the default resource-type.
prs require a --resource-type or -t value of
- pr
- prs
- pull_request
- pull_requests
--resource-type all will export all issues and prs into a single file.

3.3. Filtering

Filtering is currently unavailable.
If you're interested in CONTRIBUTING to features like filters--e.g., only select "open" issues--we're happily accepting pull requests!

3.4. Exporting

3.4.1. Issues

CSV

GitHub:

# GET https://api.github.com/rrepos/:owner/:repo/issues
$ github-resource-converter \
  --owner gregswindle \
  --repo  github-resource-converter \
  --dest  './docs/reports/export.csv'

GitHub Enterprise:

# GET https://api.ecorp.com/api/v3/repos/:owner/:repo/issues
$ grc --base-url https://api.ecorp.com/api/v3 \
  --owner evilcorp \
  --repo ecoin

JSON

GitHub:

# GET https://api.github.com/repos/:owner/:repo/issues
$ github-resource-converter \
  --owner gregswindle \
  --repo  github-resource-converter \
  --dest  './docs/reports/export.json'

GitHub Enterprise:

# GET https://api.ecorp.com/api/v3/repos/:owner/:repo/issues
$ grc --base-url https://api.ecorp.com/api/v3 \
  --owner evilcorp \
  --repo ecoin
  --dest ./export.json

3.4.2. Pull Requests

CSV

GitHub:

# GET https://api.github.com/repos/:owner/:repo/pulls
$ github-resource-converter \
  --owner gregswindle \
  --repo  github-resource-converter \
  --resource-type pr

GitHub Enterprise:

# GET https://api.ecorp.com/api/v3/repos/:owner/:repo/pulls
$ grc --base-url https://api.ecorp.com/api/v3 \
  --owner evilcorp \
  --repo ecoin \
  --resource-type pr

JSON

GitHub:

# GET https://api.github.com/repos/:owner/:repo/pulls
$ github-resource-converter \
  --owner gregswindle \
  --repo  github-resource-converter \
  --resource-type pr
  --dest './export.json'

GitHub Enterprise:

# GET https://api.ecorp.com/api/v3/repos/:owner/:repo/pulls
$ grc --base-url https://api.ecorp.com/api/v3 \
  --owner evilcorp \
  --repo ecoin \
  --resource-type pr
  --dest './export.json'

3.4.3. All (open and closed issues and pull requests)

CSV

GitHub:

# GET https://api.github.com/repos/:owner/:repo/pulls
$ github-resource-converter \
  --owner gregswindle \
  --repo  github-resource-converter \
  --resource-type all

GitHub Enterprise:

# GET https://api.ecorp.com/api/v3/repos/:owner/:repo/pulls
$ grc --base-url https://api.ecorp.com/api/v3 \
  --owner evilcorp \
  --repo ecoin \
  --resource-type all

JSON

GitHub:

# GET https://api.github.com/repos/:owner/:repo/pulls
$ github-resource-converter \
  --owner gregswindle \
  --repo  github-resource-converter \
  --resource-type all
  --dest './export.json'

GitHub Enterprise:

# GET https://api.ecorp.com/api/v3/repos/:owner/:repo/pulls
$ grc --base-url https://api.ecorp.com/api/v3 \
  --owner evilcorp \
  --repo ecoin \
  --resource-type all
  --dest './export.json'

3.5. Command-line flags

3.6. Errors

Errors are written to the console (stdout) as JSON:

# Attempt to fetch issues from a repository that doesn't exist
$ grc --owner example --repo error
[2018-03-20T02:31:24.737Z] ERROR: github-resource-converter/48219 on User.router.home: {"message":"Not Found","documentation_url":"https://developer.github.com/v3/issues/#list-issues-for-a-repository"}
  HttpError: {"message":"Not Found","documentation_url":"https://developer.github.com/v3/issues/#list-issues-for-a-repository"}
      at response.text.then.message (/p/a/t/h/github-resource-converter/node_modules/@octokit/rest/lib/request/request.js:56:19)
      at <anonymous>
      at process._tickCallback (internal/process/next_tick.js:188:7)

3.7. Info

The --help flag displays all options:

$ grc --help

  Convert and export GitHub resources--Issues and Pull Requests--to CSV and JSON.

  Usage

    $ grc [options] [info]
    $ github-resource-converter [options] [info]

  Options

    --base-url           The GitHub REST API v3 URL origin, or a
                         GitHub Enterprise URL origin and path-prefix.
                         [Default: 'https://api.github.com']
    --dest,          -d  The CSV's destination path and file name.
                         [Default: './resources.csv']
    --no-auto-filename   Disable automatic file naming.
                         [Default: false]
    --owner,         -o  The GitHub account name or organization name.
    --repo,          -r  The name of the GitHub (or GitHub enterprise)
                         repository.
    --resource-type, -t  "issues", "prs", or "all".
                         [Default: 'issues']

  Info

    --help     Show this dialog.
    --version  Display the installed semantic version.

  Examples

    $ grc --owner github --repo hub
      // => Exported CSV to /path/of/cwd/issues.csv.

    $ grc --owner github --repo hub -dest './reports/issues/YYYY-MM-DD.csv'
      // => Exported CSV to /path/to/reports/issues/YYYY-MM-DD.csv.

    $ grc --owner example --repo error
      // =>
      [2018-03-19T08:04:06.596Z] ERROR: github-resource-converter/00000 on localhost: Cannot destructure property `data` of 'undefined' or 'null'.
        TypeError: Cannot destructure property `data` of 'undefined' or 'null'.
            at paginate (/p/a/t/h/github-resource-converter/lib/base-resource-converter.js:39:16)
            at <anonymous>
            at process._tickCallback (internal/process/next_tick.js:188:7

Use the --version flag to see which version you have installed:

$ github-resource-converter --version
# => 1.0.1

4. API

Test the github-resource-converter (grc) API in your Web browser .

4.1. `grc.authenticate({token, type, key}): void`

Most GitHub API calls don't require authentication. Rules of thumb:
If you can see the information by visiting the site without being logged in, you don't have to be authenticated to retrieve the same information through the API.
If you want to change data, you have to be authenticated.
octokit/rest.js. (2018). GitHub. Retrieved 21 March 2018, from https://github.com/octokit/rest.js#authentication

4.1.1. Parameters

4.1.2. Example

// Token (https://github.com/settings/tokens)
grc.authenticate({
  token: 'secrettoken123',
  type: 'token'
})

4.2. `grc.getAll({owner, repo}): Promise`

Retrieve all open and closed Issues and Pull Requests from a GitHub project.

GET

/repos/:owner/:repo/issues
/repos/:owner/:repo/pulls

grc.getAll combines the results of grc.issues.getForRepo and grc.pullRequests.getForRepo into a single Array<object>.
For more information, see
grc.issues.getForRepo and
grc.pullRequests.getForRepo below.

4.3. `grc.issues.getForRepo({owner, repo}): Promise`

Retrieve all open and closed issues for a repository with this proxy method for octokit.issues.getForRepo .

GET

/repos/:owner/:repo/issues

4.3.1. Parameters

4.3.2. Examples

async/await:

const result = await grc.issues.getForRepo({
  owner: 'gregswindle',
  repo: 'eslint-plugin-crc'
})

Promise:

grc.issues
  .getForRepo({
    owner: 'gregswindle',
    repo: 'eslint-plugin-crc'
  })
  .then(result => {})
  .catch(err => {})

4.4. `grc.logger`

A trentm/node-bunyan logger instance that uses the LONG thlorenz/bunyan-format writeable stream for output.

bunyan.INFO is the default log LEVEL.

Logger {
  domain: null,
  _events: {},
  _eventsCount: 0,
  _maxListeners: undefined,
  _level: 30,
  streams:
    [{
      type: 'stream',
      stream: BunyanFormatWritable {
      },
      closeOnExit: false,
      level: 30,
      raw: false
    }],
  serializers: null,
  src: false,
  fields: {
    name: 'github-resource-converter',
     hostname: 'Gregorys-MBP-2.fios-router.home',
    pid: 76698
  },

  debug(): void,
  error(): void,
  fatal(): void,
  info(): void,
  trace(): void,
  warn(): void
}

4.5. `grc.options`

Contains default values for api, cli, and metadata.

{
  "api": {
    "baseUrl": "https://api.github.com",
    "headers": {
      "Accept":
        "application/vnd.github.v3+json, application/vnd.github.symmetra-preview+json",
      "user-agent": "gregswindle/github-resource-converter v1.0.1"
    },
    "owner": null,
    "repo": null,
    "requestMedia": "application/vnd.github.v3+json",
    "timeout": 0
  },
  "cli": {
    "flags": {
      "baseUrl": {
        "default": "https://api.github.com",
        "type": "string"
      },
      "dest": {
        "alias": "d",
        "default": "./export.csv",
        "type": "string"
      },
      "no-auto-filename": {
        "default": false,
        "type": "boolean"
      },
      "owner": {
        "alias": "o",
        "type": "string"
      },
      "repo": {
        "alias": "r",
        "type": "string"
      },
      "resource-type": {
        "alias": "t",
        "default": "issues",
        "type": "string"
      }
    }
  },
  "meta": {
    "input": [],
    "flags": {
      "noAutoFilename": false,
      "baseUrl": "https://api.github.com",
      "dest": "./export.csv",
      "d": "./export.csv",
      "resourceType": "issues",
      "t": "issues"
    },
    "pkg": {
      "name": "github-resource-converter",
      "description":
        "Convert and export GitHub resources--Issues and Pull Requests--to CSV and JSON.",
      "version": "1.0.1",
      "author": {
        "name": "Greg Swindle",
        "email": "[email protected]",
        "url": "https://github.com/gregswindle"
      },
      "bin": {
        "grc": "lib/cli.js"
      },
      "bugs": {
        "url": "https://github.com/gregswindle/github-resource-converter/issues"
      },
      "commitplease": {
        "style": "angular",
        "types": [
          "build",
          "chore",
          "ci",
          "docs",
          "feat",
          "fix",
          "perf",
          "refactor",
          "revert",
          "style",
          "test"
        ],
        "scope": "\\S+.*"
      },
      "contributors": [],
      "dependencies": {
        "@octokit/rest": "15.2.6",
        "bunyan": "1.8.12",
        "bunyan-format": "0.2.1",
        "dotenv": "5.0.1",
        "fs-extra": "5.0.0",
        "insight": "0.10.1",
        "jsonexport": "2.0.11",
        "lodash": "4.17.5",
        "meow": "4.0.0"
      },
      "devDependencies": {
        "@semantic-release/changelog": "2.0.1",
        "@semantic-release/git": "4.0.1",
        "@semantic-release/npm": "3.2.4",
        "ajv": "6.4.0",
        "ajv-keywords": "3.1.0",
        "codacy-coverage": "2.1.1",
        "commitplease": "3.2.0",
        "coveralls": "3.0.0",
        "eslint": "4.19.1",
        "eslint-config-prettier": "^2.4.0",
        "eslint-config-scanjs": "1.0.0-beta4",
        "eslint-config-standard": "11.0.0",
        "eslint-config-xo": "0.20.1",
        "eslint-plugin-import": "2.10.0",
        "eslint-plugin-jsdoc": "3.5.0",
        "eslint-plugin-json": "1.2.0",
        "eslint-plugin-no-unsafe-innerhtml": "1.0.16",
        "eslint-plugin-node": "6.0.1",
        "eslint-plugin-prettier": "^2.2.0",
        "eslint-plugin-promise": "3.7.0",
        "eslint-plugin-security": "1.4.0",
        "eslint-plugin-standard": "3.0.1",
        "eslint-plugin-unicorn": "4.0.2",
        "eslint-plugin-xss": "0.1.9",
        "fixpack": "2.3.1",
        "husky": "^0.14.3",
        "jest": "22.4.3",
        "jest-runner-eslint": "0.4.0",
        "lec": "^1.0.1",
        "lint-staged": "7.0.3",
        "markdown-magic": "0.1.21",
        "markdown-magic-dependency-table": "1.3.2",
        "markdown-magic-install-command": "1.3.1",
        "markdown-magic-package-scripts": "1.2.1",
        "nsp": "^3.2.1",
        "prettier": "1.11.1",
        "semantic-release": "15.1.5"
      },
      "engines": {
        "node": ">=8.0.0"
      },
      "eslintIgnore": ["lib/__tests__/coverage/**"],
      "files": ["lib"],
      "homepage":
        "https://github.com/gregswindle/github-resource-converter/#readme",
      "jest": {
        "automock": false,
        "collectCoverage": true,
        "coverageDirectory": "lib/__tests__/coverage",
        "coverageThreshold": {
          "global": {
            "branches": 100,
            "functions": 100,
            "lines": 100,
            "statements": 100
          }
        },
        "moduleDirectories": ["node_modules", "lib"],
        "testMatch": [
          "<rootDir>/lib/__tests__/**/*.test.js",
          "<rootDir>/lib/__tests__/*.test.js"
        ],
        "watchman": false
      },
      "jest-runner-eslint": {
        "cliOptions": {
          "config": ".github/config/jest-runner-eslint.config.json",
          "fix": true
        }
      },
      "keywords": [
        "QA",
        "convert",
        "converter",
        "converter",
        "csv",
        "export",
        "github",
        "google sheets",
        "issue",
        "json",
        "pr",
        "pull request",
        "pull-request",
        "quality assurance",
        "quality-assurance",
        "reporting",
        "reports",
        "save",
        "sheets",
        "spreadsheets",
        "testing"
      ],
      "license": "MIT",
      "lint-staged": {
        "*.js": ["npm run lint:js", "git add"],
        "*.json": ["npm run lint:json", "git add"],
        "*.md": ["npm run lint:md", "npm run docs", "git add"]
      },
      "main": "lib/index.js",
      "prettier": {
        "semi": false,
        "singleQuote": true
      },
      "release": {
        "generateNotes": {
          "preset": "angular"
        },
        "prepare": [
          "@semantic-release/changelog",
          "@semantic-release/npm",
          "@semantic-release/git"
        ],
        "verifyConditions": ["@semantic-release/npm", "@semantic-release/git"]
      },
      "repository": {
        "type": "git",
        "url":
          "git+https://github.com/gregswindle/github-resource-converter.git"
      },
      "scripts": {
        "docs": "npm run docs:toc",
        "docs:toc":
          "md-magic --config '.github/config/markdown.config.js' --path '**/*.md' --ignore 'node_modules'",
        "lint":
          "npm run lint:js && npm run lint:json && npm run lint:manifest && npm run lint:md",
        "lint:js":
          "node_modules/.bin/eslint -c .github/config/.eslintrc.yml --ext .js . --fix",
        "lint:json":
          "prettier ./**/*.json --ignore-path '.github/config/.prettierignore' --write",
        "lint:manifest": "fixpack",
        "lint:md": "prettier ./**/*.md -prose-wrap always --write",
        "posttest:ci:coverage:codacy":
          "cat ./lib/__tests__/coverage/lcov.info | codacy-coverage",
        "precommit": "lint-staged",
        "prepare": "npm run lint && npm run security",
        "prepublish":
          "lec lib/cli.js -c LF && npm run security && semantic-release -d",
        "prepublishOnly": "npm run prepare",
        "pretest": "npm run lint",
        "security": "npm run security:nsp:scan",
        "security:nsp:scan": "nsp check",
        "test": "jest",
        "test:config": "jest --showConfig",
        "test:watch": "jest ./lib/__tests__/*.test.js --watch",
        "test:watch:all": "jest ./lib/__tests__/*.test.js --watchAll"
      },
      "readme": "ERROR: No README data found!",
      "_id": "[email protected]"
    },
    "help":
      "\n  Convert and export GitHub resources--Issues and Pull Requests--to CSV and JSON.\n\n  Usage\n\n    $ grc [options] [info]\n    $ github-resource-converter [options] [info]\n\n  Options\n    --base-url           The GitHub REST API v3 URL origin, or a\n                         GitHub Enterprise URL origin and path-prefix.\n                         [Default: 'https://api.github.com']\n    --dest,          -d  The CSV's destination path and file name.\n                         [Default: './resources.csv']\n    --no-auto-filename       Don't append an ISO 8601-like timestamp to the\n         output file.\n                         [Default: false]\n    --owner,         -o  The GitHub account name or organization name.\n    --repo,          -r  The name of the GitHub (or GitHub enterprise)\n                         repository.\n    --resource-type, -t  \"issues\", \"prs\", or \"all\".\n                         [Default: 'issues']\n\n  Info\n\n    --help     Show this dialog.\n    --version  Display the installed semantic version.\n\n  Examples\n\n    $ grc --owner github --repo hub\n      // => Exported CSV to /path/of/cwd/issues.csv.\n\n    $ grc --owner github --repo hub -dest './reports/issues/YYYY-MM-DD.csv'\n      // => Exported CSV to /path/to/reports/issues/YYYY-MM-DD.csv.\n\n    $ grc --owner example --repo error\n      // =>\n      [2018-03-19T08:04:06.596Z] ERROR: github-resource-converter/00000 on localhost: Cannot destructure property `data` of 'undefined' or 'null'.\n        TypeError: Cannot destructure property `data` of 'undefined' or 'null'.\n            at paginate (/p/a/t/h/github-resource-converter/lib/base-resource-converter.js:39:16)\n            at <anonymous>\n            at process._tickCallback (internal/process/next_tick.js:188:7)\n"
  }
}

4.6. `grc.pullRequests.getForRepo({owner, repo}): Promise`

Retrieve an array of all open and closed pull requests for a GitHub or GitHub Enterprise repository with this proxy method for octokit.pullRequests.getAll .

GET

/repos/:owner/:repo/pulls

4.6.1. Parameters

4.6.2. Examples

async/await:

const grc = require('github-resource-coverter')

const getAllPullRequests = async (params) = {
  try {
    const WHITESPACE = 2
    const prs = await grc.pullRequests.getForRepo(params)
    grc.logger.info(JSON.stringify(prs, null, WHITESPACE))
  } catch (err) {
    grc.logger.error(err)
  }
}

await getAllPullRequests({
  owner: 'democracy-ia',
  repo: 'govinfo-link-js'
})

Promises:

const grc = require('github-resource-coverter')

grc.pullRequests
  .getForRepo({
    owner: 'democracy-ia',
    repo: 'govinfo-link-js'
  })
  .then(prs => {
    const WHITESPACE = 2
    grc.logger.info(JSON.stringify(prs, null, WHITESPACE))
  })
  .catch(err => {
    grc.logger.error(err)
  })

4.7. `grc.save({data, dest}): Promise`

Export a collection of Issues or Pull Requests to your local filesystem.

4.7.1. Parameters

4.7.2. Examples

async/await:

// Save as JSON
const result = await grc.save({
  data,
  dest: './export.json'
})

Promise:

// Save as CSV
grc
  .save({
    data,
    dest: './export.csv'
  })
  .then(result => {})
  .catch((err = {}))

4.8. `grc.toCsv({data=[]}): Promise`

Converts (deeply) nested JSON into CSV format, returning a Promise<string>.

4.8.1. Parameters

4.8.2. Examples

async/await:
```
const result = await grc.toCsv(data)
```

Promise:

grc
  .toCsv(data)
  .then(result => {})
  .catch((err = {}))

5. Version

6. Contributing

We welcome contributions with GitHub issues and pull requests.

Before embarking on a significant change, please adhere to the following guidelines:

Create an issue—e.g., a defect ("bug") report or a feature request—to propose changes.
Exceptions:
If you're working on documentation and fixing something simple like a typo or an easy bug, go ahead and make a pull request.
Follow the CONTRIBUTING guidelines.
Why:
Standards and guidelines make communication easier. If you're willing and able to program—or want to learn how— following the guidelines will increase the likelihood of adding your changes to the software product.
Read the Code of Conduct.
Why:
It's more fun when everybody's friendly and respectful.
Make a pull request when you're ready for other to review your changes (or you get stuck somewhere).
PR novices:
🙋 Never created a pull request? No problem. 🆓 Take this free online training . (It even covers most of the conventions in the CONTRIBUTING guidelines!)

7. License

Read the NOTICE External link for all third-party software that github-resource-converter uses.

Published

Vulnerabilities

Links

Maintainers

Keywords

Readme

github-resource-converter

Table of contents

1. Overview

2. Installation

2.1. For Terminal/command-line usage

2.2. As a application dependency

3. Usage

3.1. Formatting

3.2. Resource types

3.3. Filtering

3.4. Exporting

3.4.1. Issues

3.4.2. Pull Requests

3.4.3. All (open and closed issues and pull requests)

3.5. Command-line flags

3.6. Errors

3.7. Info

4. API

4.1. grc.authenticate({token, type, key}): void

4.1.1. Parameters

4.1.2. Example

4.2. grc.getAll({owner, repo}): Promise

4.3. grc.issues.getForRepo({owner, repo}): Promise

4.3.1. Parameters

4.3.2. Examples

4.4. grc.logger

4.5. grc.options

4.6. grc.pullRequests.getForRepo({owner, repo}): Promise

4.6.1. Parameters

4.6.2. Examples

4.7. grc.save({data, dest}): Promise

4.7.1. Parameters

4.7.2. Examples

4.8. grc.toCsv({data=[]}): Promise

4.8.1. Parameters

4.8.2. Examples

5. Version

6. Contributing

7. License

`github-resource-converter`

4.1. `grc.authenticate({token, type, key}): void`

4.2. `grc.getAll({owner, repo}): Promise`

4.3. `grc.issues.getForRepo({owner, repo}): Promise`

4.4. `grc.logger`

4.5. `grc.options`

4.6. `grc.pullRequests.getForRepo({owner, repo}): Promise`

4.7. `grc.save({data, dest}): Promise`

4.8. `grc.toCsv({data=[]}): Promise`