faros-feeds-sdk
v1.4.5
Published
Utilities for feed development
Downloads
255
Keywords
Readme
feeds-sdk
An SDK that simplifies feed development by handling all Faros API interactions. A feed is a tool that extracts data from an external data source, typically an IaaS or PaaS API, and loads it into Faros.
Uploading data
The SDK includes two functions for uploading data to the Faros platform:
withEntryUploader
and entryUploader
. They have slightly different signatures,
but they both wrap all the necessary API calls to the Faros platform, so you can
focus on what's unique to your feed. Choosing one comes down to whether your feed
requires state to be persisted between runs.
Stateless feeds
Feeds which do not need to persist state between runs can use entryUploader
.
This will give you a
stream.Writable
whose entries are uploaded to the API.
Example of using entryUploader
:
import {entryUploader} from 'faros-feeds-sdk';
const writer = await entryUploader({
name: 'EC2',
// Other required parameters...
});
// Written entries will be streamed to the API
writer.write({
ec2_Instance: {
instanceId: 'i-0ef827cbfe244614f',
instanceType: 't2.micro',
state: {
code: 80,
name: 'stopped',
},
launchedAt: new Date(1607473249000),
},
});
// Write more entries and finally:
writer.end();
Stateful feeds
When possible, it's good practice to support incremental data ingestion.
In order to do so, use withEntryUploader
, which comes with a callback you can
use to read the existing feed state, and return the new state.
Example of using withEntryUploader
:
import {withEntryUploader} from 'faros-feeds-sdk';
export interface FeedState {
readonly lastLaunchedAt: number;
}
const uploaderCfg = {
name: 'EC2',
// Other required parameters...
};
await withEntryUploader<FeedState>(uploaderCfg, (writer, state) => {
// Write entries to writer using the old state...
return /* new state */;
});
Lambda runner
The feeds SDK also includes a convenience function called createLambdaHandler
for generating a lambda handler if you would like to run your feed in Faros's
Lambda environment.
:clipboard: Note: The resulting lambda handler is only compatible with feeds that have been set up and scheduled via Faros's /accounts API, because of the very specific way in which it handles feed params.
The lambda handler decrypts secrets, params, and environment variables provided
by the Faros scheduler, and passes them on to a user defined runFeed
method
that in turn uses them to run the feed. Example usage:
import {createLambdaHandler, FarosEnv} from 'faros-feeds-sdk';
async function runFeed(env: FarosEnv): Promise<void> {
// validate input params
// run the feed
}
export const lambdaHandler = createLambdaHandler(runFeed);
An example FarosEnv
for the GitHub feed might look something like this:
{
"token": "myGitHubToken",
"orgLogin": "acme",
"repoList": ["repo1", "repo2"],
"farosApiUrl": "https://prod.api.faros.ai",
"farosApiKey": "myFarosApikey",
"runMode": "Root",
"logger": Logger
}
Worker Feeds
The createLambdaHandler
also takes in an optional parameter partition,
which is a function will takes the FarosEnv
and returns a list of FarosEnv
to enable parallel execution of the feed. The root feed should adapt each
sub FarosEnv
with the appropriate param the feed needs to run.
:clipboard: Note: Create unique values of the param the feed uses as the origin to ensure correct handling of the worker's feed state.
import {createLambdaHandler, FarosEnv} from 'faros-feeds-sdk';
async function runFeed(env: FarosEnv): Promise<void> {
// validate input params
// run the feed
}
async function partition(env: FarosEnv): Promise<Map<string, FarosEnv>> {
// return map of farosEnv with split params
// e.g. return new Map([['partition1', farosEnv]])
}
export const lambdaHandler = createLambdaHandler(runFeed, partition);
Logging
The FarosEnv
passes a logger to the feed. This logger should be used by the
feed for all logging to ensure logs are correctly stored.
Links
- Canonical models.
This library can be used for the
sdl
property ofEntryUploaderConfig
. See the docs for details.