Seham - Store Express HTTP Activity to MongoDB

Motivation

The following node package was created for the scope of my Bachelor Thesis.

Its purpose is not olny to provide a simple HTTP Logging function for Express APIs but also to give the web developer the ability to blacklist malicious/unwanted IPs. HTTP traffic as well as Client IPs are stored to different collections in a Mongo Database. Furthermore, this node package can be used in conjunction with a Python Flask API that leverages Machine Learning techniques to classify each client's behavior (as normal or abnormal) based on their logged HTTP activity/sessions. It may also be used to automaticaly blacklist clients if their activity/behavior is commonly classified as abnormal.

More specifically, this Node JS Middleware function can be used to:

• Log Client IPs (blacklistIP option available).
• Log HTTP Metadata into MongoDB.
• Group HTTP Metadata into Sessions based on timestamps.

If used in conjuction with the provided Session Behavior Classifier API, it can also be used to:

• Extract Features from HTTP Metadata.
• Use Pretrained Python Classifiers to Classify Web Sessions and Blacklist Malicious IP Addresses.

Prerequisites

• Node JS - as a Backend Service
• Express JS - REST API (express - node package)
• MongoDB - Database (mongoose - node package)

Dependencies - Node Packages

• mime: ^2.4.6
• mongoose: ^5.10.3

Usage

Installation

Install the node package:

npm install seham

Require the module to your app:

const seham = require("seham");

Insert the following line-of-code in your main file (app.js/index.js):

app.use(seham(options)); 	// options argument is optional

between the lines:

app.use(express.json());
... // code
/* INSERT THE LINE-OF-CODE HERE */
... // code
app.use(express.static(path.join(__dirname, "public")));

where options is a JavaScript Object with the properties listed below.

Additional properties (Optional):

The various optional properties are listed below:

| Name | Default | Description | | ---------------- | ---------------- | ---------------- | | blockDuration | 60 | Duration (in seconds) that an IP Address gets blocked.| | groupToSessions| true | Group HTTP to sessions based on time.| | custGroupToSess| timestamp based | Use a custom function to group HTTP into sessions.| | sessTimeThres | 15 | Maximum Time Difference (in seconds) between two consecutive requests of the same session. If Time Def greater than this value, requests are split into separate sessions.| | maxReqsPerSess | 180 | Maximum Number of Requests in a session.| | pythonAPIOpt | - | Session-Behavior-Classifier-API Connection Options.| | customIPHeader | - | The name of request-header to get the client IP from. If not provided common headers/request-properties are used instead.|

The default pythonAPIOpt object is shown below:

pythonAPIOpt : {
  useAPI: false 		// Whether to use the Flask API (if available) or not.
  host: "localhost" 	// The Hostname of the Flask API.
  port: 4000 			// The Port number of the Flask API.
}

When the parameter customIPHeader is not provided then the headers and request properties listed above are used in the following order:

Request Headers:

X-Client-IP
X-Forwarded-For
CF-Connecting-IP (Cloudflare)
Fastly-Client-Ip (Fastly CDN and Firebase hosting header when forwared to a cloud function)
True-Client-Ip (Akamai and Cloudflare)
X-Real-IP (Nginx proxy/FastCGI)
X-Cluster-Client-IP (Rackspace LB, Riverbed Stingray)
X-Forwarded
Forwarded-For

Request Properties:

req.connection.remoteAddress
req.socket.remoteAddress
req.connection.socket.remoteAddress
req.info.remoteAddress

An example of the options object could be:

const options = {
        blockDuration : 24 * 60 * 60,     // 24 hours
        groupToSessions : true,
        sessTimeThres : 30,               // 30 seconds
        maxReqsPerSess : 200,             // 200 requests
        pythonAPIOpt : {useAPI: true, host: "localhost", port: 4000},
        customIPHeader : "Forwarded-IP"
}