npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

dt-python-parser

v0.9.0

Published

There are some python parsers built with antlr4, and it's mainly for the **BigData** domain.

Downloads

1,524

Readme

dt-python-parser

NPM version

English | 简体中文

dt-python-parser is a Python Parser project based on ANTLR4 for the big data field. Through ANTLR4 the default generated Parser, Visitor and Listener objects, we can easily achieve the Syntax Validation (Syntax Validation), ** of Python statements Lexical analysis** (Tokenizer), traverse AST nodes and other functions. In addition, several auxiliary methods are provided, for example, to filter comments of type # and """ in Python statements.

Supported Python syntax version:

  • Python2
  • Python3

Tip: The current Parser is the Javascript language version, if necessary, you can try to compile the Grammar file to other target languages

Installation

// use npm
npm i dt-python-parser --save

// use yarn
yarn add dt-python-parser

Use

Syntax Validation

First, you need to declare the corresponding Parser object. Different Python syntax versions need to introduce different Parser object processing. For example, if it is for Python2, you need to introduce Python2 Parser separately, here we use Python3 As an example:

import {Python3Parser} from'dt-python-parser';

const parser = new Python3Parser();

const correctPython = `print('abc')\nprint('abc')\n`;
const errors = parser.validate(correctPython);
console.log(errors);

Output:

/*
[]
*/

Example of verification failure:

const incorrectPython =
    '! a = 10\nif a> 5:\n print("a bigger than 5")\nelse:\n print("a smaller than 5")';
const errors = parser.validate(incorrectPython);
console.log(errors);

Output:

/*
    [
      {
        startLine: 1,
        endLine: 1,
        startCol: 0,
        endCol: 1,
        message: "extraneous input'!' expecting {<EOF>,'def','return','raise','from','import','import','global','nonlocal','assert', 'if','while','for','try','with','lambda','not','None','True','False','class','yield','yield ','del','pass','continue','break','break', NEWLINE, NAME, STRING_LITERAL, BYTES_LITERAL, DECIMAL_INTEGER, OCT_INTEGER, HEX_INTEGER, BIN_INTEGER, FLOAT_NUMBER, IMAG_NUMBER, DECIMAL_INTEGER, HCTEXGER, HCTEXGER, FLOAT_NUMBER, IMAG_NUMBER,'...','*','(','[','+','-','~','{','@'}"
      }
    ]
*/

First instantiate the Parser object, and then use the validate method to verify the Python statement. If the verification fails, an array containing the error information is returned.

Lexical Analysis (Tokenizer)

In necessary scenarios, you can perform lexical analysis on Python sentences separately to obtain all Tokens objects:

import {Python3Parser} from'dt-python-parser';

const parser = new Python3Parser();
const python ='for i in range(5):\n print(i)';
const tokens = parser.getAllTokens(python);
console.log(tokens);

/*
[
      CommonToken {
        source: [[Python3Lexer], [InputStream] ],
        type: 14,
        channel: 0,
        start: 0,
        stop: 2,
        tokenIndex: -1,
        line: 1,
        column: 0,
        _text: null
      },
    ...
]
*/

Visitor mode (Visitor)

Use Visitor mode to visit the specified node in the AST

import {Python3Parser, Python3Visitor} from'dt-python-parser';

const parser = new Python3Parser();
const python = `import sys\nfor i in sys.argv:\n print(i)`;
// parseTree
const tree = parser.parse(python);
class MyVisitor extends Python3Visitor {
    // Override visitImport_name method
    visitImport_name(ctx): void {
        let importName = ctx
            .getText()
            .toLowerCase()
            .match(/(?<=import).+/)?.[0];
        console.log('ImportName', importName);
    }
}
const visitor = new MyVisitor();
visitor.visit(tree);

/*
ImportName sys
*/

Tip: When using Visitor mode, the method name of the node can be found in the Visitor file in the corresponding Python directory

Listener

In Listener mode, use the ParseTreeWalker object provided by ANTLR4 to traverse the AST and call the corresponding method when entering each node.

import {Python3Parser, Python3Listener} from'dt-python-parser';

const parser = new Python3Parser();
const python ='import sys\nfor i in sys.argv:\n print(i)';
// parseTree
const tree = parser.parse(python);
class MyListener extends Python3Listener {
    enterImport_name(ctx): void {
        let importName = ctx
            .getText()
            .toLowerCase()
            .match(/(?<=import).+/)?.[0];
        console.log('ImportName', importName);
    }
}
const listenTableName = new MyListener();
parser.listen(listenTableName, tree);

/*
ImportName sys
*/

Tip: When using Listener mode, the method name of the node can be found in the Listener file in the corresponding Python directory

Clean up comment content

Clear comments and leading and trailing spaces

import {cleanPython} from'dt-python-parser';

const python = `#it is for test\nfor i in range(5):\n print(i)`;
const cleanedPython = cleanPython(python);
console.log(cleanedPython);

/*
    for i in range(5):
        print(i)
*/

Get comment content

Get comments of type # or """

import { lexer } from 'dt-python-parser';

const python = `"""it is for test"""\nvar1 = "Hello World!"\nfor i in range(5):\n    print(i)`;
const commentTokens = lexer(python);
console.log(commentTokens);

/*
    [
      {
        type: 'Comment',
        value: '"""it is for test"""',
        start: 0,
        lineNumber: 1,
        end: 20
      }
    ]
*/

Other API

  • parserTreeToString (input: string): Parse Python into List-like style tree string, generally used for testing

Roadmap

  • Auto-complete
  • Code formatting
  • Grammar structure optimization
  • Execution efficiency optimization

License

MIT