valpha

freetext filter for sql-esque data

About

valpha is a refractor and update to the library json-ctrl by combining several of the smaller dependencies (json-cnf, json-tkn, json-tim, and jsqlon). As such there are many moving pieces for which this README.md will try to explain and address.

Assumptions

A core assumption of this library is that the stored data is SQL-eqsue. In other words, that the JSON object's outer most keys correspond to record ids and that each record (an object) is structured as if in a SQL table i.e. each record contains the same fields:

json = {
  id1: {
    field1: value1f1,
    field2: value1f2,
    field3: value1f3,    
  },
  // ...
  idn: {
    field1: valueNf1,
    field2: valueNf2,
    field3: valueNf3,    
  }
}

It may be the case that users need to overwrite the default behavior of how field values are treated. For this three objects exists in which overriding functions can be set. These include:

• fieldRenderFunctions: how a given field's value should be altered when rendering it on the page.
• fieldSortByFunctions: how a given field's value should be altered when comparing against other values for sorting.
• fieldFilterFunctions: how a given field's value should be altered when comparing against specified filters.

To overwrite a field's function (e.g. fieldX), a function with the following signature is expected:

function overwriteFieldX(id, field, record) {
  let value = record[field]
  // manipulate value to be returned.
  return value
}

then set the corresponding object (e.g. fieldRenderFunctions) to have this function with the field name as the key:

fieldRenderFunctions = {
  fieldX: overwriteFieldX
}

Filtering

Filtering of the records in the stored JSON occurs in four steps:

1. records are passed through the specified conjunctive normal form filters. (This utilizes specified fieldFilterFunctions).
2. the records passing step 1 are passed through the global RegEx (This utilizes specified fieldRenderFunctions, as this is what the user sees).
3. the records are sorted via TimSort from the given sortSpecification (This utilizes the specified fieldSortByFunctions).
4. the records of the specified "page" are returned.

Sorting

Sorting is handled via the state variable sortSpecification. sortSpecification is an array of objects, where each object has the following structure:

sortSpec = {
  field: fieldName,
  isAscending: boolean
}

This specification allows for the fields, in the order provided in sortSpecification to undergo TimSort.

Pagination

Pagination is provided for controlling which records are show to the user. It is worthwhile that this is local pagination, therefore all the data is already in memory. This stems from the combination of functions applied to the data outlined in the section regarding filtering. Consider the question "what are the five elements on page two?". If filters, RegEx and sorting are in play, there is no way to know this without finding all records that pass and sorting them to know which will be on page two.

What's Inside?

Components

• VAlphaToggle: the upper row of buttons in VRecordsTable (filter, sort, regex and possibly config buttons)
• VConfig: handles defaults of user settable configs and is used in VRecordsTable inside the navigation drawer
• VFilterView: handles the rendering of the CNF filters, displayed underneath the text input
• VRecordsTable: the whole shebang
• VTimSort: the autocomplete handling the sort specifications (in addition to clicking the column names on VRecordsTable)

Utilities

• valpha: all of the utility functions
• config: default configurations for the utility functions

Configuration

Utility Functions

There is a lot which can be configured to achieve the desired behavior. As these configurations are mostly independent, they are split up into sub-configs. These are passed to the utility functions from the components.

I already touched on the fieldRenderFunctions, fieldSortByFunctions, fieldFilterFunctions above. In addition there are:

1. configCNFLogic: The logical options for CNF.
2. configCNFFunctions: The functions which can be applied during CNF.
3. configCNFConditionals: The conditionals (or tests) which can be applied during CNF.
4. configCNFExtractors:
5. configTKNFunctionMap: A mapping of tokens to functions in the CNF.
6. configTKNConditionalMap: A mapping of tokens to conditionals in the CNF.

configCNFLogic

The two logical operations supported are and and or. You can change how they are displayed in VFilterView as so:

{
  and: {
    display: "∧"
  },
  or: {
    display: "∨"
  }
}

configCNFFunctions

These are the known functions and corresponding types they work on for the free text filter.

{
  identity: {
    display: "x → x",
    function: (x) => x,
    types: ["number", "string", "array:number", "undefined"]
  },
  abs: {
    display: "abs",
    function: (x) => Math.abs(x),
    types: ["number"]
  },
  mean: {
    display: "mean",
    function: (nums) => (nums.reduce((add, val) => add + val) / nums.length),
    types: ["array:number"]
  },
  max: {
    display: "max",
    function: (nums) => Math.max(...nums),
    types: ["array:number"]
  },
  min: {
    display: "min",
    function: (nums) => Math.min(...nums),
    types: ["array:number"]
  },
  length: {
    display: "len",
    function: (arr) => arr.length,
    types: ["array", "array:number", "array:string", "array:object"]
  }
}

configCNFConditionals

Here you find which conditionals exist

{
  eq: {
    display: "=",
    conditional: (a, b) => {
      // intentional == rather than === to allow for 1 == "1" to be true
      return a == b;
    },
    types: [
      "array", "array:string", "array:object", "array:number",
      "number", "string", "undefined"
    ]
  },
  neq: {
    display: "≠",
    types: [
      "array", "array:string", "array:object", "array:number",
      "number", "string", "undefined"
    ],
    conditional: (a, b) => a != b,
  },
  lt: {
    display: "<",
    types: ["number"],
    conditional: (a, b) => a < b,
  },
  gt: {
    display: ">",
    types: ["number"],
    conditional: (a, b) => a > b,
  },
  lte: {
    display: "≤",
    types: ["number"],
    conditional: (a, b) => a <= b,
  },
  gte: {
    display: "≥",
    types: ["number"],
    conditional: (a, b) => a >= b,
  },
  ss: {
    display: "⊂",
    types: ["array", "array:string", "array:number", "string"],
    conditional: (a, b) => {
      let includes = false
      if (Array.isArray(a)) {
        for (let i = 0; i < a.length; i++) {
          if (a[i] == b) includes = true
          if (includes) return includes
        }
      } else if (typeof a === 'string') {
        return a.includes(b)
      } else { return includes }
      return includes
    }
  },
  nss: {
    display: "⟈",
    types: ["array", "array:string", "array:number", "string"],
    conditional: (a, b) => {
      let includes = true
      if (Array.isArray(a)) {
        return a.every(e => e != b)
      } else if (typeof a === 'string') {
        return !a.includes(b)
      }
      return includes
    }
  }
}

configCNFExtractors

{

}

configTKNFunctionMap

There are the default tokens that map tokens (the keys) to the corresponding values in configCNFFunctions.

{
  max: "max",
  min: "min",
  maximum: "max",
  minimum: "min",
  length: "length",
  len: "length",
  mean: "mean",
  median: "median",
  average: "mean",
  ave: "mean",
  identity: "identity"
}

configTKNConditionalMap

There are the default tokens that map tokens (the keys) to the corresponding values in configCNFConditionals.

{
  eq: "eq",
  is: "eq",
  equal: "eq",
  "=": "eq",
  "!=": "neq",
  "≠": "neq",
  ">": "gt",
  "≥": "gte",
  ">=": "gte",
  "<": "lt",
  "≤": "lte",
  "<=": "lte",
  "is not": "neq",
  neq: "neq",
  "not equal to": "neq",
  gt: "gt",
  "greater than": "gt",
  "less than": "lt",
  lt: "lt",
  "less than or equal to": "lte",
  "greater than or equal to": "gte",
  "member of": "ss",
  substring: "ss",
  contains: "ss",
  includes: "ss"
}

Extra config

There are some other things which are used both in the components and utility functions, these include:

1. fieldOrder
2. configFieldMap
3. regexForPunctuation

fieldOrder

fieldOrder[String]

the fieldOrder is an optional array of strings specifying the order in which columns should be displayed. In addition, if provided only these fields are considered valid. By default all fields will be used.

configFieldMap

{
  fieldNameAsInRecord: 'Field Name As Shown To User',

}

The default behavior uses the field name as found in a record as the token corresponding to the field. There are times however when the internal field name may be not be desired (e.g. _some_field or aWeirdField)

The configFieldMap is a way to overwrite this. Both the internal name and the overriding name will be recognized as valid tokens.

NOTE: all other configs correspond to the original field name.

regexForPunctuation

This is a regex pattern used for scrubbing the user provided input for invalid characters. By default it is:

/(\w+(?:[.,-]\w+)*)|[-.,()&$#![\]{}"']+/g;

Components

The above configurations are passed to the utility functions. What follows are the configuration options just for the components:

1. configOptions
2. functionLookup

configOptions

The configOptions are used by VConfig to expand the provided defaulting fieldRenderFunctions, fieldFilterFunctions, and fieldSortByFunctions. If provided it allows the user to swap "profiles" of how a field should be treated in the table. Since this may change due to state of the application, it also has a corresponding functionLookup. That dictionary holds functions that when called produce render / filter and sortBy functions.

{
  field: {                    // <--- field which is configurable
    profileName: {            // <--- name of the configuration
      renderConfig: {         // <--- how this config renders the field
        functionName: String, // <--- name of function to call  in functionLookup
                              // to produce the render function
        params: {             // <--- params to pass the function
          key:value,
          ...
        }
      },
      filterConfig: {         // same as above but for filter
        functionName: String,
        params: {
          key:value,
          ...
        }
      },
      sortByConfig: {
        functionName: String, // same as above but for sortBy
        params: {
          key:value,
          ...
        }
      }
    },
    ...
  },
  ...
}

functionLookup

This is used explicitly by VConfig to generate the corresponding render / filter and sortBy functions. It is done this way as you can define all the functions you need to produce the functions you want in advance and then using an API get which fields are configurable and how.

{
  functionName: function
}

Build Setup

# install dependencies
$ npm run install

# serve with hot reload at localhost:3000
$ npm run dev

# build for production and launch server
$ npm run build
$ npm run start

# generate static project
$ npm run generate

For detailed explanation on how things work, check out Nuxt.js docs.