Cloud Log Collector

This is a simple log collector system.

Checkout www.cloud-log-collector.com for more details

Here for the french version

The primary goal of this project is to have a simple log collector system, assisted by a configuration wizard, and able to be deployed on several thousand of servers.

A simple and secure log collector.

1. How to install NodeJS (Ubuntu) :

sudo apt --yes install nodejs
sudo apt --yes install nodejs-legacy
sudo apt --yes install npm
sudo npm install npm@latest -g
sudo npm install -g n
sudo n lts

2. How to install Cloud Log Collector :

sudo npm install -g cloud-log-collector

3. How to configure Cloud Log Collector :

sudo cloud-log-collector-config-wizard

Main features are :

• Easy deployment and maintenance
• Automation of addition of client servers
• Automatic and mandatory security
• Irreversibility, ie the impossibility to modify data already transmitted

Why develop a new system ?

There are already many log management tools on the open source market. However, we find them relatively complex to implement, they require regular maintenance, and expertise.
The idea of Cloud Log Collector is not to have advanced functionality like LogsTash to interpret logs, ElasticSearch to serve as a search engine, or Kibana to have a nice graphical interface.

The idea of Cloud Log Collector is to be simple, with no complex functionality, and with security by default.

If you are looking to interpret the logs, you can still use LogsTash or any other tool directly on the logs retrieved by Cloud Log Collector!

Installation process

To install all necessary tools, in root :

npm i -g cloud-log-collector

Use the wizard to create configuration files, in root :

cloud-log-collector-config-wizard

To install server part, in root :

cloud-log-collector-install-server my_conf_file.conf

To install client part, in root :

cloud-log-collector-install-client my_conf_file.conf

If you use the magic token, to auto configure the client part, in root :

cloud-log-collector-install-client-get-token

Wizard will tell you how to install and configure the servers / clients

Wizard example

---------------------------------------------------------------
---------------------------------------------------------------
Wizard for Cloud Log Collector configuration file
---------------------------------------------------------------
Server hostname :cloud-log-collector.myserver.com
Server port (1095) :
SSL Certificate file on the server :/var/mycert.cert
SSL Certificate Key file on the server :/var/mycert.key
Folder used to store all log files on the server (/var/clc/data) :
Folder used to store client credential data on the server (/var/clc/tokens) :
Want to use a magic token ? (automatically generates token from client side at first connection) (y/n) :y
Enter magic token (empty = auto generated) :
Magic token : 68l7drd32eaazmhzr883j6ma6po9yysrdp3ltatxochc3xkgmg179fm960opti6uqx0ux1pc47sonyt6wmqyb
---------------------------------------------------------------
In order to help community, send anonymous data on system usage ? (To help future dev) (y/n) :y
---------------------------------------------------------------
Which log do you want to watch on the client ? (empty to continue) :/var/log/auth.log
Which log do you want to watch on the client ? (empty to continue) :/var/log/syslog
Which log do you want to watch on the client ? (empty to continue) :
---------------------------------------------------------------
Folder to save config file(s) (./conf) :
Once everything is configured, you should delete this folder for security reasons
---------------------------------------------------------------
---------------------------------------------------------------
Process to install Cloud Log Collector on server :
- copy './conf/server.conf' to the server in file '/as/you/want/server.conf'
- run these commands in root :
	npm i -g cloud-log-collector
	cloud-log-collector-install-server /as/you/want/server.conf
delete file '/as/you/want/server.conf'
---------------------------------------------------------------
Do you want me to install server side on this server now ? (y/n) :y

Done !
---------------------------------------------------------------
Process to install Cloud Log Collector on client without magic token :
- copy './conf/client.conf' to the server in file '/as/you/want/client.conf'
	Replace XXXSecretClientKeyXXX by your secret key in '/as/you/want/client.conf'
- run these commands :
	npm i -g cloud-log-collector
	cloud-log-collector-install-client /as/you/want/client.conf
delete file '/as/you/want/client.conf'
To get a secret client key, use the command 'cloud-log-collector-client-key-new XXXClientNameXXX' on the server after installation
To revoke a secret client key, use the command 'cloud-log-collector-client-key-revoke XXXClientNameXXX' on the server after installation
'XXXClientNameXXX' must only contain A-Za-z0-9.-_
---------------------------------------------------------------
Process to install Cloud Log Collector on client with magic token :
- copy './conf/client.conf' to the server in file '/as/you/want/client.conf'
- run these commands :
	npm i -g cloud-log-collector
	cloud-log-collector-install-client /as/you/want/client.conf
delete file '/as/you/want/client.conf'
	CLCMagicToken="My Magic Token" CLCHostname="myhostname.com" cloud-log-collector-install-client-get-token
---------------------------------------------------------------
Do you want me to install client side on this server now ? (y/n) :y
Server hostname in cloud-log-collector (could be different from hostname used to connect client and server) (a-zA-Z0-9.-_ only) :cloud-log-collector-server

Token saved in config file
Done !
Wizard end !

How it works ?

Cloud Log Collector is divided into 2 parts, a thin client running on different servers from which we want to recover the logs, and a server that centralizes all received logs.

A simple solution !

A wizard allows you to simply create configuration files for clients and servers. The installation of the system on the clients is simplified by the possible use of a magic token which allows automatic configuration of security token on client side.

We use this log system in dynamic clusters with automatic creation and destruction of servers. It was therefore essential that the installation procedures be automated.

The most paranoid users can disable the magic token feature to generate their own tokens.

What usefulness ?

In the context of cybersecurity, and in particular new regulatory obligations (GDPR, …), server logs must be saved, in order to be used in a survey, for technical, regulatory or legal reasons (forensic).

This can also be used to easily provide access to numerous logs from numerous servers !

For which platform ?

The main target platform is an heterogeneous set of servers running Linux. We developed the system using systemctl for service management.

We also operate several other types of servers (Windows, …), however, the open source version of Cloud Log Collector currently only supports natively Linux.

Interested by this type of feature? contact us !

How fast ?

The system has been designed to be deployed on thousands of servers for a single collector. In case of temporary congestion, unavailability of the log server, or network connection breakdown, the logs are stored in client-side RAM for later delivery.

What security ?

The entire system has been designed to minimize the risk of log corruption in case of corruption of one of the client servers. Even in the case of complete corruption, only logs sent from the moment the corruption is made can be corrupted. The client cannot access the data already sent.

The system requires the use of a TLS security certificate for communications between servers. We use Let’s Encrypt to generate the certificates, but you can use any certificate. We also use auto certificates generated with master certificate that we add to the certificate group recognized by the server.

Impossible to disable security ! This is part of the principles of :

• Security by design
• Security by default
• Privacy by design
• Privacy by default

What dependencies ?

Cloud Log Collector uses only 3 dependencies :

• readline-sync, for configuration tools that require human interaction (CLI)
• websocket, for communications between components (TLS / SSL certificate required)
• tail-file, to detect new lines in client’s logs

Missing features for your needs ?

Contact us through our website, we can help you !