pastash
v1.0.47
Published
Spaghetti I/O Processing, Interpolation, Correlation and beyond - nodejs logstash + beat
Downloads
144
Maintainers
Readme
paStash
"When logs give you spaghetti, make pasta"
What is paStash ?
PaStasH (pastaʃ'ʃ-utta) is a NodeJS multi I/O processor supporting ingestion, decoding, interpolation and correlation of data - be it logs, packets, events and beyond. PaStash supports the Logstash configuration format and delivers cross-functionality comparable to "Beats" with custom modules, providing a flexible and agnostic data pipelining tool.
What can I do with paStash ?
paStash is designed manage spaghetti I/O with input, processors and output modules for all seasons, and can be useful in many scenarios, such as parsing logs to objects, distributing data to multiple formats, inter-exchanging and correlating protocols and streams, while interpolating and manipulating data in transit. paStash is developed using NodeJS, which is an ideal language for applications with many IO and offers:
- lower memory footprint
- lower CPU footprint
- lower start-up delay
- ease of extension
paStash configuration is compatible with logstash. You can easily replace a logstash node by a paStash one in most cases. The data are formatted in the same way to be compatible with logstash UIs.
How does it work?
The architecture is identical to logstash architecture. You have to instantiates plugins with the paStash core. There are three type of modules:
- inputs plugins: where datas come into paStash. Examples: file, zeromq transport layer
- filter plugins: extract and manipulate fields from logs, like timestamps. Example: regex plugin
- outputs plugins: where datas leave from paStash: Examples: ElasticSearch , zeromq transport layer.
A typical paStash deployment contains agents to crawl logs and a log server.
On agent, paStash is configured with input plugins to get logs from your software stack, and one output plugin to send logs to a log server (eg. zeromq output plugin).
On the log server, logs come through a zeromq input plugin, are processed (fields and timestamps extraction), and sent to ElasticSearch.
How to use it ?
Installation
NPM install
sudo npm install -g pastash
Manual install
- Install NodeJS, version >= 8.xx
- Install build tools
- Debian based system:
apt-get install build-essential
- Centos system:
yum install gcc gcc-c++ make
- Debian based system:
- Install zmq dev libraries: This is required to build the node zeromq module.
- Debian based system:
apt-get install libzmq1
. Under recent releases, this package is present in default repositories. On ubuntu lucid, use this ppa. On debian squeeze, use backports. - Centos 6: First, add the zeromq repo :
curl https://download.opensuse.org/repositories/network:/messaging:/zeromq:/release-stable/CentOS_6/network:messaging:zeromq:release-stable.repo > /etc/yum.repos.d/zeromq-centos6.repo
. Thenyum install zeromq zeromq-devel
. - CentOS 7: First, add the zeromq repo :
curl https://download.opensuse.org/repositories/network:/messaging:/zeromq:/release-stable/CentOS_7/network:messaging:zeromq:release-stable.repo > /etc/yum.repos.d/zeromq-centos7.repo
. Thenyum install zeromq zeromq-devel
.
- Debian based system:
- Clone repository:
git clone git://github.com/sipcapture/pastash.git && cd pastash
- Install dependencies:
npm install
.
The executable is bin/pastash
Configuration formats
There are two format for configuration. The legacy format uses urls. The new one is identical to the logstash config format.
Note : multiple configuration files can be used in parallel with the --config_dir
switch.
Configuration by logstash config files
Example for an input file
input {
file {
path => '/tmp/toto.log'
}
}
You can use if
to have an event dependent configuration. See here for details.
As for urls, a config can be specified
- directly on the command line
- in a file (use the
--config_file
switch) - in all files in a directory (use the
--config_dir
switch)
Note : the implementation is young, all bug reports are welcome. Note : both formats can be mixed.
Command lines params
--log_level
to change the log level (emergency, alert, critical, error, warning, notice, info, debug)--log_file
to redirect log to a log file.--patterns_directories
to add some directories (comma-separated ,), for loading config, for regex plugin and grok plugins. Grok pattern files must be located under agrok
subdirectory for each specified directory.--db_file
to specify the file to use as database for file inputs (see below)--http_max_sockets
to specify the maximum amount of sockets of http.globalAgent.maxSockets. Default to 100.--alarm_file
to specify a file which will be created if paStash goes into alarm mode.
Examples
Config file for an agent:
input {
file {
path => "/var/log/nginx/access.log"
}
}
output {
zeromq {
address => ["tcp://log_server:5555"]
}
}
Config file for log server:
input {
zeromq {
address => ["tcp://0.0.0.0:5555"]
}
}
filter {
regex {
pattern => http_combined
}
}
output {
elasticsearch {
host => localhost
port => 9200
}
}
See our wiki for many more examples
Plugins list
Inputs
- File
- Syslog
- ZeroMQ
- Redis
- HTTP
- Websocket
- TCP / TLS
- Google app engine
- AMQP
- MQTT
- SQS
- NetFlow
- sFlow
- Bencode
- Freeswitch ESL
- Asterisk AMI
Common concepts / parameters :
Filters
- Regex
- Grok
- Mutate Replace
- Grep
- Reverse DNS
- Compute field
- Compute hash
- Compute date field
- Split
- Truncate
- Rename
- Multiline
- Json fields
- Geoip
- Eval
- Bunyan
- IPProto
- HTTP Status Classifier
- Remove field when equal
- Mustache
- Omit
- LRU
Common concepts / parameters :
Apps with embedded parsers :
Outputs
- ZeroMQ
- ElasticSearch
- Splunk
- Kafka
- Statsd
- InfluxDb
- Gelf
- File
- HTTP Post
- Websocket
- Redis
- Logio
- TCP / TLS
- AMQP
- NSQ
- SQS
- HEP
Common concepts / parameters :
Adding your plugins
You can add easily add your plugins :
Manually :
- create a directory layout in the path of your choice :
/var/my_plugins/inputs
,/var/my_plugins/outputs
,/var/my_plugins/filters
- set the NODE_PATH variable to
NODE_PATH=/var/my_plugins:/node_logstash_path/lib
- add your plugins in
inputs
,outputs
orfilters
directory. In the plugin code, you can reference base plugins withvar base_filter = require('lib/base_filter');
- reference your plugin as usual.
With native packaging
The plugins must be deployed in /var/db/pastash/custom_plugins
. All subdirectories already exist. The NODE_PATH is already set.
Signals
- USR1: stopping or starting all input plugins. Can be used to close input when output targets are failing
- USR2: see below file output plugin
Misc
License
paStash Copyright 2016 - 2018 QXIP BV
node-logstash Copyright 2012 - 2014 Bertrand Paquet
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Made by Humans
This Open-Source project is made possible by actual Humans without corporate sponsors, angels or patrons. If you use this software in production, please consider supporting its development with contributions or donations