@kalisio/feathers-import-export
v1.2.0
Published
Import/Export data with FeathersJS
Downloads
304
Readme
feathers-import-export
feathers-import-export
provides convenient methods to import/export to/from Feathers services.
feathers-import-export
has been specially designed to process large volumes of data and to overcome data transfer problems during import and export, it relies on the capabilities of the S3 API.
[!WARNING] Consequently, the use of this library requires being able to operate a store compatible with the S3 API.
[!NOTE] To deal with the objects in the stores,
feathers-import-export
relies on the feathers-s3 library. It is highly recommended to read a little more about this library upfront.
Principle
The following sections illustrate the different processes implemented by feathers-import-export
:
Import
Export
Usage
Installation
npm install @kalisio/feathers-import-export --save
or
yarn add @kalisio/feathers-import-export
Example
Setup the service
Assuming you have setup a Feathers app:
// Import Feathers stufff
import { Service } from '@kalisio/feathers-import-export'
// Setup Feathers app
const options = {
s3Options: {
s3Client: {
credentials: {
accessKeyId: process.env.S3_ACCESS_KEY_ID,
secretAccessKey: process.env.S3_SECRET_ACCESS_KEY
},
endpoint: process.env.S3_ENDPOINT,
region: process.env.S3_REGION,
signatureVersion: 'v4'
},
bucket: process.env.S3_BUCKET,
prefix: 'tmp' // a folder used to store imported/exporter files
},
app,
workingDir: process.env.TMP_DIR,
}
app.use('import-export', new Service(options))
The s3Options
options are described in feathers-s3
Import data from a file
Import data from a data.csv
file into the my-service
service:
const response = await app.service('import-export').create({
method: 'import',
id: 'key/to/the/file/in/the/bucket' // file key into the bucket
servicePath: 'path/to/service' // path to the service where to import the data
})
[!NOTE] This method assumes that you have already uploaded the file.
Export data to a file
Export data from the my-service
service into the data.csv
file:
const response = await app.service('import-export').create({
method: 'export',
servicePath: 'path/to/my-service',
filename: 'data.csv',
format: 'csv'
})
API
feathers-import-export
consists in a single service that provides the following methods:
constructor (app, options)
Create an instance of the service with the given options:
| Parameter | Description | Required |
|---|---|---|
|s3Options
| the options to configure the S3 service. Refer to feathers-s3 API. | yes |
| app
| the feathers app. | yes |
| workingDir
| the working directory to process temporary files. Default value is /tmp
. | no |
registerTransform (key, transform)
Register a transformation function for the given key.
| Parameter | Description | Required |
|---|---|---|
|key
| the key assigend to the transformation function. | yes |
| transform
| the transformation function. | yes |
create (data, params)
Shortcut method that calls import or export according the value of the method
property.
The payload data
must contain the following properties:
| Argument | Description | Required |
|---|---|---|
| method
| the method to call, either ìmport
or export
. | yes |
Concerning the other properties, refer to the description of the different methods.
import (data, params)
Imports the content of a file that is stored on a S3 compatible storage.
The payload data
must contain the following properties:
| Argument | Description | Required |
|---|---|---|
| id
| the object key. Note that the final computed key takes into account the prefix
option of the service. | yes |
| servicePath
| the path to the service into which to import the data. | yes |
| transform
| the transformation to apply before importing the data. Defaut is undefined
| no |
export (data, params)
Exports the result of a query into a JSON, CSV or GeoJson file that it stored on an S3 compatible storage. The file can be compressed using GZip. By default It returns a Presigned URL to the file.
The payload data
must contain the following properties:
| Argument | Description | Required
|---|---|---|
| servicePath
| the path to the service to be queried.| yes |
| query
| the query to apply. Default value is {}
| no |
| chunkPath
| the path to the data when processing the query response. Default value is data
| no |
| chunkSize
| the number of objects to be processed by chunk. Defaut value is 500
| no |
| transform
| the transformation to apply before expoting the data. Defaut is undefined
| no |
| format
| the output format. Defaut value is json
| no |
| zip
| whether to zip the output or not. Default value is true
| no |
| signedUrl
| whether to return a signed url. Default value is true
| no |
| expiresIn
| the expiration delay of the returned signed url. Default value is 300
| no |
[!WARNING] The
chunkSize
must be less than themax
property of thepaginate
options assigned to the service.
Transformation
As illustrated in the previous sections, feathers-import-export
allows you to apply a transformation before importing or exporting the data.
The transformation can be carried out via a transform object or via a function.
Transform object
The transform object can be declared with the following specifications:
toArray
: boolean indicating if the JSON object will be transformed into an array using Lodash, defaults to falsetoObjects
: if your input JSON objects are flat arrays it will be transformed into objects according to the given indexed list of property names to be used as keys, not defined by defaultfilter
: a filter to be applied on the JSON object using any option supported by siftmapping
: a map between input key path and output key path supporting dot notation, the values of the map can also be a structure like this:path
: output key pathvalue
: a map between input values and output valuesdelete
: boolean indicating if the input key path should be deleted or not after mapping
unitMapping
: a map between input key path supporting dot notation and from/to units to convert using math.js for numbers or moment.js for dates, a value of the map is a structure like this:from
: the unit or date format to convert from, e.g. feet or YYYY-MM-DD HH:mm:ss.SSSto
: the unit or date format to convert to, e.g. m or MM-DD-YYYY HH:mm:ss.SSS, if given for a date the date object will be converted back to stringasDate
: mandatory to indicate if the value is a date, could be utc or local to interpret it as UTC or Local Time asString: mandatory to convert numbers to strings, indicates the radix to be used if anyasNumber
: mandatory to convert strings to numbersasCase
: target case to be used as the name of a Lodash (e.g.lowerCase
) or JS string (e.g. toUpperCase) case conversion function (e.g. lowerCase)empty
: value to be set if the input value is empty
pick
: an array of properties to be picked using Lodashomit
: an array of properties to be omitted using Lodash merge: an object to be merged with each JSON objects using LodashasObject
: this boolean indicates if the output should be transformed into an object if the array contains a single object, defaults to falseasArray
: this boolean indicates if the output should be transformed into an array containing the object, defaults to false.
transform: {
toArray: true, // The following input object { 1: { property: 'a' }, 2: { property: 'b' } } will be transformed into [{ property: 'a' }, { property: 'b' }]
toObjects: ['1', '2'], // The following input object ['a', 'b'] will be transformed into { 1: 'a', 2: 'b' }
mapping: {
sourceProperty: 'targetProperty',
sourceProperty: {
path: 'targetProperty',
values: {
'a': 'c' // Will map { xxx: 'a' } to { yyy: 'c' }
}
},
'source.property': 'target.property',
sourceProperty: 'targetArrayProperty[0]'
},
unitMapping: {
property: { from: 'feet', to: 'm' } // This one will be converted from feet to meters
},
pick: ['onlyThisPropertyWillBeKept'],
omit: ['onlyThisPropertyWillBeRemoved'],
merge: { newProperty: 'will be added to the final objects' }
}
[!TIP] The transformations are applied in the order of the documentation, e.g. filtering occurs before mapping.
Transform function
The transformation function must be registered in the service.
The function must have the following signature: function myTrasnform (chunk, options)
where
chunk
represents an array of JSON objects.options
represents the options passed to theimport
orexport
methods. It allows you to retrieve some contextual data if needed when processing the chunk.
function myTransform (chunk, options) {
chunk.forEach(object => {
// mutate object
})
return chunk
}
To specify the transformation function within the import or export payload, you must assign to the transform
property the key used to register the function
Assuming you have registered the myTransform
function with the my-transform
key, then you can declare the transformation function as below:
transform: 'my-transform'
Hooks
As mentionned before feathers-import-export
relies on feathers-s3, particularly on the methods getObjectCommand and uploadFile wich are used respectively by the import
and export
methods. Consequently, you have the flexibility to register hooks on these methods to incorporate additional processing steps. For instance, it might be practical to include a before hook on the uploadFile
method to execute preprocessing on the entire file before transferring it to the storage, such as converting it to another file format.
Registering hooks
Hooks can be registered by accessing the internal S3 service, as demonstrated below:
app.use('path-to-service', new Service(Object.assign(options, { app })))
service = app.service('path-to-service')
service.s3Service.hooks({
before: {
uploadFile: [myHook]
}
})
Predefined hooks
geojson2shp
This hook converts exported GeoJSON data to ESRI Shapefile format using ogr2ogr. The output file is a compressed archive containing .shp
, .shx
, .dbf
and other side-car files of one or several layers.
[!NOTE] To be executed the
filename
option must have the extentionshp.zip
. Otherwise the hook is skipped.
License
Copyright (c) 2017-20xx Kalisio
Licensed under the MIT license.
Authors
This project is sponsored by