norl - one-liner's node.js like perl / ruby (CLI tool)

one-liners node.js, helps to write one line stdin filter program by node.js Javascript like perl/ruby. + JSON/CSV/Promise/Async/MultiStream feature(CLI tool/module)

• Documentation(npmjs)
• Bug Report(GitHub)
• Home Page

Example

$ cat test.txt
Hello World
Goodnight World

$ cat test.txt | norl -pe '$_=$_.replace(/World/,"Norl")'
Hello Norl
Goodnight Norl

# -p: execute -e <program> line by line. $_: input/output line from/to stdin/out

$ cat test2.txt
Apple,12
Google,3

$ cat test2.txt | norl -B 'total=0' -ane 'total+=Number($F[1])' -PE '$_=`total:${total}`'
total:15

# -n: same as p but doesn't print at -e <program> line by line
# -a: $F=$_.split(',') before -e <program>
# -B <program> / -E <program>: execute <program> before(-B) / after(-E) stdin processing of -e <program>
# -P print $_ at end of stream

Install

npm install -g norl

Demo

wc -l (counting lines)

$ cat README.md | wc -l | sed 's/^ *//'
#wc -l prints unnecessary white space

$ cat README.md | norl -aPe '=>$F.length'
#norl version

JSON Pretty Print

$ cat test.json
{"a":1,"b":2}

$ cat test.json|norl -jJ
{
	"a": 1,
	"b": 2
}

Embed version string to muliple files(like sed + bash for)

$ norl -Pe '$_=$_.replace(/_VERSION_/g,"1.2.0")' -O destDir/  package.json README.md LICENSE.txt
destDir/package.json
destDir/README.json
destDir/LICENSE.json  (All files "_VERSION_" strings were replaced by 1.2.0)

unix join (sort is not necessary :)

$ cat address.csv
norl,moon
partpipe,mars

$ cat tel.csv
norl,010-342-234
partpipe,010-122-444

$ norl address.csv tell.csv a.csv tel.csv -B 'res={};' -ane '_.set(res,[$F[0],$S],$F[1]);' -E '_.forIn(res,(v,r)=>{$P([r,v[0],v[1]].join(","))})'
norl,010-342-234,moon
partpipe,010-122-444,mars

Adds users to a Twitter list from CSV (with twitter-tool)

$ npm i -g twitter-tool; twitter -i
$ cat list.csv
foo,12345
bar,13432

$ TWITTER_USER=kssfilo LST=list2 CSVFLD=0 norl -anxe $'=>`twitter lists/members/create -o s:${$e("LST")},s_n:${$F[$e("CSVFLD")]},o_s_n:${$e("TWITTER_USER")} `' list.csv
# user @foo and @bar has been added to list2 of @kssfilo

Command line

norl <options> -e '<program>' [-B '<program'>] [-E '<program>'] [files...]

Copyright(c) 2019,kssfilo(https://kanasys.com/gtech/)
one-liners node.js, helps to write one line stdin filter program by node.js Javascript like perl/ruby.+JSON/CSV/Promise/Async/MultiStream feature(CLI tool/module)

Options

- h            help
- ?            
- d            debug mode
- e <program>  one line program (without -n -p option, $_ contains whole data from stdin)
- n            call -e program line by line. $_ contains received line from stdin.(like perl/ruby -ne)
- p            assume loop like -n but console.log($_) each line after -e <program> (like perl/ruby -pe) you can delete current line by $_=null
- a            autosplit mode (splits $_ into $F) default split() pattern is ','(with -n -p) or \n(without -n -p)
- F </regexp/> split() pattern for -a switch (you can use string instead of regex.dont need -a when -F option is specified)
- B <program>  (Begin) additional program which runs BEFORE -e program.for initializing(works with -n -p).
- E <program>  (End) additional program which runs AFTER -e program.for finalizing(works with -n -p).
- j            JSON.parse stdin then stores into $_ (can't use with -n -p)
- J            JSON.stringfy($_,null,"\t") and print it at end of stream after -E program (you can also print Promise/Async.js result.see example)
- P            console.log($_) at end of stream after -E program (you can also print Promise/Async.js callback result.see example)
- C [<sep>]    CSV like output. works with -p. $_=$F.join(<sep>) before console.log($_). use with -a to manipulate CSV like files
- c            same as -C but use default ',' separator.useful for joining options like -cape <program>
- X            execute $_ as shell command after -e <program> then print result line by line. works with -p. like xargs.if you store null into $_. do nothing for this line
- x            same as X but doesn't print the shell command's result. pass through input line to stdout. stops process if shell command returns non zero error. 
- L [<number>] by default, shell commands will be executed sequencial. with -L option, commands will run parallel. same effect for async.js style function but Promise().
- m <modules>  preload module list for example, -m 'fs request'
- M            suppress preloading by NORL_MODULES environment variable.default you can preload modules by NORL_MODULES(see example)
- S            search modules according to node.js manner i.e. <currentdir>/node_modules,<parentdir>/node_modules.. then NODE_PATH(default:NODE_PATH only)
- r            Just run -e <program>. stdin and files will be ignored.
- O <dir>      output directory.if you specify this option,and mutiple files are in arguments. writes output to <dir> with same filenames.see Multi-Output mode section.
	

Program and Namespace

you must enclose your program by single quote '. if you want to use single quote inside, use bash single quote escape mode like this ( norl -re $'console.log("'")' )

• $_: input line or object (-j) or array(multi-input mode w/o -n -p). and output for auto print option (-p / -P / -J)
• $F: splitted array when you specify auto split option (-a / -F).
• $S: stream number when you specify multiple files(multi-input mode) with -n -p option.

other variables which started '$' are preserved.don't use in your program.

by default, only lodash module is preloaded into '_'. you can add other modules by NORL_MODULES or -m option.

Examples

1. Perl/Ruby like stdin processing(-e / -ne / -pe / -a)

$ cat test.txt
Hello World
Goodnight World

$ cat test.txt | norl -pe '$_=$_.replace(/World/,"Norl")'
Hello Norl 
Goodnight Norl

-e : program code. without (-n/-p), -e program is called only one time. $_ contains whole stdin data.

$ cat test.txt|norl -pe 'm=$_.match(/^Hello/);$_=m?$_:"---"'
Hello World
---

-pe : execute program line by line.then print $_ after each -e by console.log($_)

$ cat test2.txt
Apple,12
Google,3

$ cat test2.txt | norl -B 'total=0' -ane 'total+=parseInt($F[1])' -PE '$_=`total:${total}`'
total:15

-ne : same as -pe but doesn't print $_ each line.

-B /-E : executs at begining(-B) / end(-E) of stream. works with -n/-p option.

$ cat test2.txt|norl -a -pe '$_=$F[1]'
12
30

-a option: automatic split. $F=$_.split(',') before -e , you can change separator by -F option

2. Automatic JSON.parse

$ cat test3.json
{
	"s":"Hello World"
}

$ cat test3.json|norl -j -e 'console.log($_.s)'
Hello World

-j option: assume stdin is JSON. $_=JSON.parse(stdin) before -e . only works without (-n / -p)

3. Automatic Print (Text/JSON/CSV) (-P / -J / -c)

$ cat test3.json|norl -je '$_=$_.s'
Hello World (shorthand of above example)

-P option: you can omit console.log($_) by -P option. just assign result into $_ at -e program (without -n) or -E program(with -n)

tips: every $_ of -ne/-pe program result is stored into an Array then pass it to -E via $_. you can check -ne results by just add -P option. redirecting to stderr (-E 'console.error($_') is useful for debugging -ne program.

$ cat test2.txt
Apple,12
Google,3

$ cat test2.txt|norl  -B 'count=0' -ane 'count+=Number($F[1])' -JE '$_={total:count}'
{"total":15}

-J option: same as -P but prints JSON. you must assign any object to $_ in -e(without -n) or -E(with -n)

$ cat test.txt
Hello World
Goodnight World

$ cat test.txt|norl -cpe '$F[0]=$_.length;$F[1]=$_'
11 chars,Hello World
15 chars,Goodnight World

-c option: CSV like output. Joins $F array by $_=$F.join(',') after each -pe . works only with -p. you can change separator by -C

4. Super Short JSON Handling (combine -J + -j)

$ cat test.json
{ "apple": 12, "google": 3 }

$ cat test.json | norl -jJe '$_.google+=1'
{ "apple": 12, "google": 4 }

combining -j +J option: easy to modify JSON file.

$ cat test2.json
{ "status":{ "apple": 12, "google": 3 }}

$ cat test2.json | norl -jJe '$_=$_.status'
{ "apple": 12, "google": 3 }

you can assign a part of input JSON to $_. for extracting some properties.

builtin lodash (_) helps to manipulate objects easier. you can combine multiple JSON files by muti input mode(see another section)

5. Super Short CSV Handling (combine -a + c)

$ cat test.txt
Apple,12
Google,3

$ cat test.txt | norl -cape '$F[1]=Number($F[1])+1'
Apple,13
Google,4

combining -c +a option: you can modify CSV columns by just rewrite $F[n] fields

$ cat test2.txt
partpipe,mars,010-1234-5678,2015
norl,moon,010-9876-5432,2019

cat test2.txt | norl -cape '$F=[$F[0],$F[2]]'
partpipe,010-1234-5678
norl,010-9876,5432

-c + -a option: you can reassign new array into $F, useful to filter columns like this example.

6. Modules

$ export NORL_MODULES="mathjs fs"
$ echo -e "1+2\n3*4"|norl -pe '$_=mathjs.evaluate($_)' 
3
12

you can preload modules by NORL_MODULES environment variable. or -m option.(separated by space with in quote' ')

$ echo -e "1+2\n3*4"|norl -m 'mathjs fs' -pe '$_=mathjs.evaluate($_)' 
3
12

module are searched in NODE_PATH if you want to use global (npm install -g) module. i.e. $ export NODE_PATH=$(npm root -g))

node_modules dirs of current/parent dirs(same manner as node.js) are also used if -S specified(high priority than NODE_PATH)

variable name is basically same as module name but '-' and '.' will be '_'. and @private/ prefix is not used. i.e.

request_promise=require("request-promise"); getopt=require("@kssfilo/getopt");

by default. lodash module is pre-loaded into '_'. and 'path' / 'fs' is available.

7. Promise

$ export NORL_MODULES="request-promise"
$ echo -e "https://www.google.com/robots.txt" |norl -Pe 'return request_promise($_)'
User-agent: ..... 

you can return promise object from -B / -e / -E. norl waits result and print it if -P or -J is specified. result of -B function is always dropped.

$ cat urls.txt
https://www.google.com/robots.txt
nhttps://www.yahoo.com/robots.txt

$ export NORL_MODULES="request-promise fs"
$ cat urls.txt | norl -ne 'return request_promise($_)'  -E 'for(i in $_){fs.writeFileSync(`robots-${i}.txt`,$_[i]) }'
robots-0.txt:contains google.com's robots.txt
robots-1.txt:contains yahoo.com's robots.txt

if Promise is returned by each -pe / -ne program, norl prints(-pe) or collects(-ne) it and Promise.all() to wait before -E program. if -ne, then pass the result array into -E program via $_

8. async.js

$ cat waits.txt
A,5
B,1
C,3

$ cat waits.txt | norl -ane 'return ((name,timeout,cb)=>{console.log(`${name}:${timeout}secs`);setTimeout(()=>{cb(null,name+":OK");},timeout*1000)}).bind(null,$F[0],Number($F[1]));'
A:5secs
B:1secs (<-after 5secs from 1st line)
C:3secs (<-after 1secs from 2nd line)

returned function from each -ne / -pe program will be queued and waits for all callbacks then prints(-pe) or collects(-ne) before running -E program.if -ne, then pass the result array into -E via $_

the function must be async.js style like '(cb)=>cb(null,"OK")' , if you specify -c or -C ,you must return array like '(cb)=>cb(null,[1,2,3])'

you can pass parameters via .bind() like this example. by default, execution is sequential. you can control it by -L [] option. try to append -L 2 to the example above to check behavior. 2 is a number of executables in parallel. if you omit , 16 will be used.

-B / -E function also supports async callback. result of -E will be printed if -P or -J is specified. result of -B function is alwasys dropped.

9. Shell Execution

$ cat test.txt
Hello,World
Goodnight,World"

$ cat test.txt | norl -axpe '$_=`echo ${$F[0]}|tr "o" "O"`'
HellO
GOOdnight 

-x option: execute $_ as shell command after each -pe then print stdout of the command, works only with -p. you can use norl like xargs

note that don't forget to assign $_. -x option executes $_ string not your -pe program itself.

tips: process stops at error condition ($?!=0) at LAST command. you can ignore error code by appending '|cat' at end of shell command like $_='wc -l noexists | cat'

$ echo -e "README.md\nnotexists.txt\npackage.json"|norl -Xpe '$_=`test -e ${$_}`'
README.md
package.json

-X option: same as x but pass-through input line instead of printing stdout of shell command. checks $? result code each execution then print input line if $?==0. unlike -x, -X DOESNT stop execution when $!=0

you can easy to create filter program with 'test' or 'grep'.

tips: all data(code/stdin/stdout/cmd) from each shell command will be collected and passed to -E via $_. appending -E "console.error(JSON.stringify($_,null,2))" is useful for debugging.

10. Result Code

$ if cat README.md|norl -ae 'return $F.length<15?false:true';then echo "README.md is too short";fi
README.md is too short  (if number of lines of README.md are under 15)

if you return a boolean(true/false) at the final (-e or -E) program. the number will be norl's shell result code ($?),true=0/false=1. You can use norl like 'test' command inside bash if.

11. Multi-Input Mode

$ norl -jJe '$_=_.merge($_[0],$_[1])' test1.json test2.json
{ "a": 1, "b": 4, "c": 1 }
# merging 2 json files (using buildin lodash(_))

specifying file instead of stdin is ok. if the number of file is 1. norl treats this file as same as stdin. but you specify 2 or more files on command line, norl will be 'multi-input mode'

without (-p / e), each files are stored in array which passed via $_. i.e. $_=[file1's contents,file2's contents,....]

auto parsing (-J / -a) are also working on multi-input mode. with -J, $_=[parsed1stjson,parsed2ndjson...]. with -a,$F=[splited1stfile,splited2ndfile...]

$ norl -pe '$_=`stream:${$S} ${$_}`' file1.txt file2.txt
stream:0 file1's 1st line
stream:0 file1's 2nd line
stream:1 file2's 1nd line
stream:2 file2's 2nd line

with (-p / -n), same -e program will be called with every file's line. -e program is able to know the file number (0,1,..) by special value $S.

12. Multi-Input-Multi-Out(MIMO) mode

$ norl -Pe '$\_=$\_.replace(/\_VERSION\_/g,"1.2.0")' -O destDir/  package.json README.md LICENSE.txt
destDir/package.json
destDir/README.json
destDir/LICENSE.json  (All files "\_VERSION\_" strings were replaced by 1.2.0)

Similar to Multi-Input Mode, but MIMO mode is very simpler than Multi-Input (single out) mode.

if you specify destination dir by -O option with multiple input files. program will be execute file by file. MIMO mode is just a short hand of

$ for f in file1 file2;do cat $f | norl -Pe 'program' >destdir/$f; done. 

unlike Multi-Input(single out) mode, you can't join each file. but MIMO mode is very useful when embed something to template file like example above.

13. => function abbriviation

if you return a string from the program. it will be copied to $_ automatically. special abbriviation function '=>..' is short hand for '($_,$F,$S)=>..' . you can make the program shorter.

$ echo "Hello" | norl -pe 'return $\_+"World"'
#can be
$ echo "Hello" | norl -pe  '=>$_+" World"'
Hello World

14. special variables which starts from $..

all variable names which start from $.. (except $_ / $F / $S) and _(for lodash) are preserved.but you can use some of preserved functions.

$P and $E are predefined to console.log and console.error this means you can rewrite console.log/console.error by $P / $E e.g.

$ echo "Hello" | norl -e 'console.log($_);console.error("warning")'
#can be
$ echo "Hello" | norl -e '$P($_);$E("warning")'

additinally, $s / $m (lower case) are useful to write String.replace()/String.match() shorter.(only works in -e not for -B -E)

$ echo "Hello World" | norl -pe '=>$_.replace(/world/i,"Norl")'
#can be
$ echo "Hello World" | norl -pe '=>$s(/world/i,"Norl")'
Hello Norl  #definition:$s=(regex,replacement)=>$\_.replace(regex,replacement)

$ echo "Hello World" | norl -pe '=>($_.match(/Hello (W.+)/) || [])[1]'
#can be
$ echo "Hello World" | norl -pe '=>$m(/Hello (W.+1)/)[1]'
World       #definition:$m=(regex)=>($\_.match(regex)||[])

$e is shorthand for process.env. definition is $e=(name)=>process.env[env]

Use as Module

norl provides module interface. you can write your own CLI filter program easier.

npm install norl

echo '{"s":"Hello World"}'| node -e 'require("norl").e(($G,$_)=>{console.log(JSON.parse($_).s);})'
# Hello World

echo -e "Hello\nWorld"| node -e 'require("norl").ne(($G,$_)=>{console.log($_.replace(/Hello/,"Hi!"))})'
# Hi!
# World

echo -e "Hello\nWorld"| node -e 'require("norl").ne(($G,$_)=>{$G.count+=$_.length},($G)=>{$G.count=0},($G)=>{console.log(`Chars:${$G.count}`)})'
# Chars:10

# require.("norl").r(<function(-e)>)

# require.("norl").e(<function(-e)>)
# require.("norl").e(<RegExp|String>,<function(-e)>)

# require.("norl").ne(<function(-e)>,[<function(-B)>,<Function(-E)>)
# require.("norl").ne(<RegExp|String>,<function(-e)>,[<function(-B)>,<Function(-E)>])

# <RegExp|String> is seperator for .split() 
# function(-e):  function($G,$_,$F){...} 
# function(-B):  function($G){...} 
# function(-E):  function($G){...} 
# $G is global object for communicating each functions.

Change Log

• 3.1.x: adds $e(name) (shorthand for process.env[name]) / able to use => with -xpe -Xpe like =>'echo hello'
• 3.0.x: adds $s / $m shorthands for $.replace()/ $.match()
• 3.0.x: -pe 'return "FOO"' prints "FOO" instead of $_. adds special abbriviation '=>...'
• 3.0.x: Breaking change: returning Number will print out the number instead of return code. if you return code like test command, use Boolean.
• 2.4.x: -m './foo/bar' style module path support/-S search node_modules before NODE_PATH/async func at -B suppport/automatic => detection/automatic print for async func on -pe
• 2.3.x: added Multi-Input-Multi-Out mode (-O)
• 2.2.x: supports file input and multi-input mode.
• 2.1.x: controling process.exit(n) code by returning number at final function.(-P -J will be cancelled)
• 2.0.x: -x/-X option, async.js style callback support. -L option
• 1.1.x: adds -c option/able to omit -a when -F is specified
• 1.0.x: first release