Redis Data Lake

Use Redis as a document database with extensive search capabilities

Installation

USING NPM

$ npm install datalake --save

In node.js

const datalake = require('datalake');
const dl = new datalake();

Implementation

Code:

const dl = require('datalake');
const datalake = new datalake();

datalake.MethodName(InputParamater).
    then((response) => {
        // do your logics here
    }).
    catch((err) => {
        console.log(err);
    });

Available Methods

| S.No | API Name | Method | Required Fields | Optional Fields | Sample PostData | Notes | | ---- | -------- | ------ | --------------- | --------------- | --------------- | ----- | | 2 | /CloseConnection | GET | ------------ | ------------ | ------------ | ------------ | | 2 | /ConfigureSearchIndex | POST | ShortCodes(sc, type, csv), Schema, Keyword | ------------ |/ConfigureSearchIndex | ------------ | | 2 | /CreateBackupData | POST | Schema, Guid | ------------ | /CreateBackupData | ------------ | | 2 | /CreateConnection | POST | host,dbname | port,password,maxConnection | /CreateConnection | ------------ | | 2 | /Fetch | POST | | 2 | /GetAllSchemaData | POST | Schema| Keyword, Guid | /GetAllSchemaData | This API cannot used in big data (Server will crash) | | 2 | /GetCacheData | POST | Schema,Key | ------------ | /GetCacheData | ------------ | | 2 | /getDistinctRecords | POST | | 2 | /InsertData | POST | Schema, Keyword, MetaData | ------------ | /InsertData | ------------ | | 2 | /ListSchemas | GET | ------------ | ------------ | ------------ | ------------ | | 2 | /ListSearchIndex | POST | Schema, Keyword | ------------ | /ListSearchIndex | ------------ | | 2 | /RefreshSchemaSearchIndex | POST | Schema | Guid | /RefreshSchemaSearchIndex | ------------ | | 2 | /RemoveData | POST | | 2 | /RestoreData | POST | Schema, Guid, Version | ------------ | /RestoreData | ------------ | | 2 | /Search | POST | Schema, Keyword | (SearchFields), recordFrom, recordUpto, Guid | /Search | Schema, (SearchFields) can be single value or can be comma (,) seperated or can be ranged (~). Either (SearchFields) or Guid should be given else This API cannot used in big data (Server will crash)| | 2 | /SearchDataByProperty | POST | Schema, Keyword, (SearchFields) | ------------ | /SearchMultipleData | (SearchFields) can be single value or can be comma (,) seperated or can be ranged (~) | | 2 | /SearchMultipleData | POST | Schema, Keyword | PropertyField, PropertyValue, Guid | /SearchDataByProperty | Schema, PropertyField, PropertyValue can be single value or can be comma (,) seperated or can be ranged (~). Either (SearchFields) or Guid should be given else This API cannot used in big data (Server will crash) | | 2 | /SearchTopRecords | POST | Schema, Keyword, Nextbatch, BATCHCOUNT, field | ------- | /SearchTopRecords | Nextbatch will be first element of response data | | 2 | /SetCacheData | POST | Schema, Key, Data | Timeout | /SetCacheData | If Timeout is not given, Data will be stored in redis permanently | | 2 | /ShowConnectionStatus | GET | ------------ | ------------ | ------------ | ------------ |

Sample Input

createConnection

POST : createConnection() → {undefined}

This will Create Redis Connection pool object.

Post Data:

{
    "host": "localhost",
    "dbname": "0",
    "port": "1607", // Optional
    "password": "YourPassword" // Optional
}

closeConnection

GET : closeConnection() → {undefined}

This will close already created Redis connection pool if any.

showStatus

GET : showStatus() → {JSON}

Returns the Status of Redis connection

Response:

{
  "Status": "TP Redis Module Loaded and Ready to Rock and Roll...",
  "Message": "DataLake - Licensed by SkunkworxLab, LLC."
}

SetupSearchHash

POST : SetupSearchHash() → {JSON}

Defining Redis Schema index keys

Post Data:

{
    "ShortCodes": [
        {
            "sc": "Name",
            "type": "string"
        },
        {
            "sc": "Email",
            "type": "string"
        },
        {
            "sc": "Phone",
            "type": "integer"
        },
        {
            "sc": "Joined",
            "type": "date"
        },
        {
            "sc": "WorkLocation",
            "type": "string",
            "csv": "true"
        }
    ],
    "Schema": "company",
    "Keyword": "employee"
}

InsertData

POST : InsertData() → {JSON}

This will insert Key value pair against the Schema given

Post Data:

{
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003", // optional
    "Schema": "company",
    "Keyword": "employee",
    "MetaData": "{ \"Name\": \"Aswin\",  \"Email\": \"[email protected]\", \"Age\": \"25\", \"Phone\": \"9876543210\", \"Joined\": \"06-March-2016\", \"WorkLocation\": [\"California\", \"India\"] }" // MetaData will be stringified json object.
}

GetAllSchemaData

POST : GetAllSchemaData() → {JSON}

This will search records for given Keyword, Schema and returns all Key-value pair available in the schema. Result can be filtered with particular Guid

Post Data:

{
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003", // optional
    "Schema": "companies",
    "Keyword": "employee"
}

SearchTidalPoolHash

POST : SearchTidalPoolHash() → {JSON}

This will search record for given PropertyField, PropertyValue, Keyword and SchemaName including comma(,) separated PropertyValues and range of PropertyValue

Implementation

Post Data:

{
    "Schema":"companies",
    "Keyword": "employee",
    "Name": "Aswin",
    "Email": "[email protected]",
    "Joined": "6-March-2018"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.SearchTidalPoolHash(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success",
  "TotalCount": 1,
  "items": [
    {
      "Guid": "ced293b4-23d8-490f-8944-9495bad5b003",
      "Keyword": "Label",
      "MetaData": "{ \"FirstName\": \"Aswin\",  \"City\": \"New York\", \"Age\": \"25\" }"
    }
  ]
}

GetDatafromSchemas

POST : GetDatafromSchemas() → {JSON}

This Method is the combination of GetSearchHashSchema, GetTidalPoolSchema, and SearchTidalPoolHash. This will search the records for given PropertyField, PropertyValue, Keyword and SchemaName. Multi-value search of records using comma(,) separated list of PropertyValue(s) or range between the PropertyValue(s) or it can be filtered with particular Guid. It can also performs multiple comma(,) separated Schema Name search.

Implementation

Post Data:

{
    "Schema": "Test",
    "PropertyField": ["FirstName"],
    "PropertyValue": ["Aswin"]
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetDatafromSchemas(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Message": "Success",
    "items": {
        "Test": [
            {
                "Guid": "ced293b4-23d8-490f-8944-9495bad5b003",
                "Keyword": "Label",
                "MetaData": "{ \"FirstName\": \"Aswin\",  \"City\": \"New York\", \"Age\": \"25\" }"
            }
        ]
    }
}

GetSchemaList

GET : GetSchemaList() → {undefined}

GetSchemaList will return list of Schema Names available in the Selected Redis DB.

Implementation

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetSchemaList(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Message": "Success",
    "items": [
        "Test"
    ]
}

GetSearchHashSchema

POST : GetSearchHashSchema() → {JSON}

This will return the defined Redis Schema index keys for the given Schema Name.

Implementation

Post Data:

{
    "VESShortCode": "Test",
    "Keyword": "Label"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetSearchHashSchema(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "VESShortCode": "Test",
  "Keyword": "Label",
  "SearchHash": [
    {
      "sc": "FirstName",
      "type": "string"
    },
    {
      "sc": "City",
      "type": "string"
    },
    {
      "sc": "Age",
      "type": "integer"
    }
  ]
}

GetTpSearchHash

POST : GetTpSearchHash() → {JSON}

This will return the defined Redis Schema index keys for the given Schema Name.

Implementation

Post Data:

{
    "Keyword": "Label",
    "VESShortCode": "Test"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetTpSearchHash(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success",
  "items": [
    {
      "sc": "FirstName",
      "type": "string"
    },
    {
      "sc": "City",
      "type": "string"
    },
    {
      "sc": "Age",
      "type": "integer"
    }
  ]
}

SetKeyData

POST : SetKeyData() → {JSON}

Set cache data into the Redis DB for the given key-value pair with TTL(Time to live) provided. If TTL(Time to live) not provided, it will set cache data as permanent data.

Implementation

Post Data:

{
    "Schema": "doshArnCache:12345",
    "Key": "Data",
    "Data": "{ \"FirstName\": \"Aswin\",  \"City\": \"New York\", \"Age\": \"25\" }",
    "TimeOut": 100
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.SetKeyData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success"
}

GetKeyData

POST : GetKeyData() → {JSON}

Get the value from cache memory for given search key before the TTL(Time to live) expire

Implementation

Post Data:

{
    "Schema": "doshArnCache:12345",
    "Key": "Data"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetKeyData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Data": [
        "{ \"FirstName\": \"Aswin\",  \"City\": \"New York\", \"Age\": \"25\" }"
    ]
}

RefreshTidalPoolData

POST : RefreshTidalPoolData() → {JSON}

Refresh/Re-Build all the search index for the given Schema Name

Implementation

Post Data:

{
    "VESShortCode": "Test"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.RefreshTidalPoolData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Message": "Success"
}

UpdateTidalPoolHash

POST : UpdateTidalPoolHash() → {JSON}

Refresh/Re-Build particular search index for the given Guid, Schema Name.

Implementation

Post Data:

{
    "VESShortCode": "Test",
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.UpdateTidalPoolHash(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success"
}

SnapshotTidalPoolData

POST : SnapshotTidalPoolData() → {JSON}

It Creates Backup copy of given Guid, Schema Name.

Implementation

Post Data:

{
    "VESShortCode": "Test",
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.SnapshotTidalPoolData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Message": "Success",
    "Version": "1"
}

RemoveTidalPoolData

POST : RemoveTidalPoolData() → {JSON}

It Creates Backup copy of given Guid, Schema Name and Removes Key-value pair from the Schema.

Implementation

Post Data:

{
    "Keyword": "Label",
    "VESShortCode": "Test",
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.RemoveTidalPoolData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success"
}

RollbackTidalPoolData

POST : RollbackTidalPoolData() → {JSON}

It Creates Backup copy of current Key-Value pair for the Given Guid-Schema and Replaces old Key-Value pair from Archive.

Implementation

Post Data:

{
    "Version": "1",
    "VESShortCode": "Test",
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.RollbackTidalPoolData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success"
}