Hull Connector Template

TL;DR: start with Five seconds overview.

This is a scaffolding tool for creating and maintaining Hull connectors. It comes with base directory and file structure, default configuration and documentation files templates.

The complete boilerplate template is in template directory. During generation it's being run through lodash template function to replace <%= var %> with values from configuration and copied to a selected working path.

Installation

Install it as a global node package:

npm i -g hull-connector-template

Now you can use the hull-connector-template command to generate or update connector base code.

Creating new connector

hull-connector-template [working-path]

You can execute this command over new directory which will be created, or run it inside an empty, already existing directory.

When run script will ask you couple of questions interactively to fill in templates with data.

Updating existing connnector

When you have an existing connector already generated by hull-connector-template or not, but you want to make it compatible with the template see below two options how to peform the difference comparison:

Simple update

Running hull-connector-template over an existing connector directory will pick configuration values from package.json and manifest.json files, so you won't have to answer configuration questions again, unless you want to change any value.

It will also detect changes and conflicts over all files and will allow you to decide if overwrite file or keep old one. This method works for files which are not modified on connector level, such as .eslintrc, .babelrc etc. since it allows to update only whole file.

Advanded update

For files which are usually updated on the connector level - such as package.json, manifest.json or JS source code files etc. - simple method of overwriting/not overwriting whole file would not work. In this case using dedicated GUI diffing tool is the best option to decide line-by-line what to update.

To get the filesystem path to the hull-connector-template directory which should be used for this diff execute following command:

hull-connector-template --path

This will return something like this:

/usr/local/lib/node_modules/hull-connector-template/template

Then use it to run your diffing tool and compare template with actual connector code to peform more complex update:

difftool /usr/local/lib/node_modules/hull-connector-template/template ./connector-working-path

Connector development

This section describes how the further connector development process looks like in details. The boilerplate code comes with README.md file which contains high level overview of this section and links here for full information.

This document also include some information about Hull's official development process which should allow 3rd party developers to contribute to our open-source projects.

Five seconds overview

Step #1: generate new connector and install deps

$ hull-connector-template new-connector
# answer couple of questions from the generator
# and enter the newly generated connector directory
$ cd new-connector
# and install all dependencies
$ yarn

This will install both production and development dependencies of your base connector. The post install script will also build the front-end and back-end application with babeljs. But you don't need to worry about that. This is a pre-deployment script.

To learn more about the basic structure of the code, please go to Repository structure.

Step #2: now start the application

# start the application
$ yarn start:dev

Your back-end server from server directory will be listening by default on 8082 port and it will be serving your front-end application from src directory. It's using babel-watch and hot reload component to not make you restart the whole thing everytime you do a change.

To learn more about the development scripts, please go to Developing.

Step #3: code and verify!

# open the files in your favorite editor
# and start coding
$ edit server/server.js
# then perform whole test script to check
# if you didn't break anything
$ yarn test
# or if you want to be more specific,
# run only linting
$ yarn test:lint
# or unit tests only
$ yarn test:unit

To learn more about the testing scripts, please go to Testing/Debugging.

Steph #4: deploy or contribute

If you want to deploy your connector on your own you can run it on production like this:

yarn start

or if you want to contribute to one of Hull's official connectors please fork our repository and issue a PR when work is done.

Now you are good to jump into details below. Good luck!

Repository structure

This is a reference of how the common boilerplate code structure looks like:

root/

  # development tools configuration
  .circleci/        - Configuration for CircleCI, it runs all tests on each build and enforce test coverage
  .babelrc          - the server side only babeljs configuration
  src/.babelrc      - the client side only babeljs configuration
  .eslintrc         - the server side eslintrc configuration
  src/.eslintrc.    - the client side eslint configuration which extends the one at root
  .prettierrc       - global prettier configuration, applied for the whole code base
  .jest.config.js   - test runnner configuration file
  webpack.config.js - 


  # Project definitions and docs
  manifest.json - The main file telling Hull how this connector works
  package.json  - npm project defition
  CHANGELOG.md  - each release changes are descibed here
  README.md     - technical overview of the connector

  # Code base
  assets/ - Images, Logos and User Guide (readme.md)
    readme.md - The User Guide which is served in the hull dashboard and linked from technical README.md
    docs/     - Images and other assets for User Guide

  flow-typed/ - Flow type definitions

  server/    - Server-side code for the connector
    actions/ - Route handlers for express application
    lib/     - Business logic of the connector
      sync-agent.js

  src/ - Front-end application which will be served by the backend application

  test/
    integration/ - Integration tests
      fixtures/  - Fixtures for notifications, payloads, etc.
      helper/    - Mocking helpers for tests
      scenarios/ - Expectations and inputs for various test scenarios
    unit/        - Unit tests

Developing

To successfully build the sources on your machine, make sure that you have the correct version of node along with one package manager installed. See engines in package.json for details.

Testing/Debugging

Execute yarn run test or npm run test in the repository root to run all tests.

If you want to run the connector on your local machine, execute yarn run start:dev or npm run start:dev which will start a new node server. Make sure to set the proper environment variables when running the code locally.

Running and writing tests

There are two sets of tests, unit tests and integration tests. Please use unit tests for all features testing. The purpose of integration tests is just end-to-end validation of functionality on sample applications.

Integration tests for the SyncAgent are organized in scenarios. Please see the Test Scenarios Guide for a detailed description of the scenarios.

Integration-testing for logs and connector responses

Mockr is a testing addition to make it easy to simulate calls and settings and write assertions on the connector's responses to Hull

This boilerplate comes with mocha/chai/sinon/nock already setup for server tests. It also includes test/integration/support/mockr package which sets up some mocks and minihull, which is a stripped down version of hull that's able to send messages to connectors and offer expectations on what the connector should send to the Firehose.

Here's how:

const { expect } = require("chai");
const mockr = require("./support/mockr");

// Your server's entry point, with the same format as the one `hull-connector` bundles.
// Options will be passed to it.
const server = require("../../server/server");

describe("Test Group", () => {
  // Start the mocks. they will run `beforeEach` and `afterEach` cleanups for you,
  // Start a development server
  const mocks = mockr({
    server
    beforeEach,
    afterEach,
    port: 8000,
    segments: [{ id: "1", name: "A" }], // Segments that should exist on the server
  });

  it("should behave properly", done => {
    const myNock = mocks
      .nock("https://api.myremote.test.com")
      .get("/test")
      .query({ foo: "bar" })
      .reply(200, [{ email: "[email protected]", id: "foobar" }]);

    // Optional, if you want to stub more things that your connector
    // will access during it's flow.
    mocks.minihull.stubApp("/api/v1/search/user_reports").respond({
      pagination: { total: 0 },
      aggregations: {
        without_email: { doc_count: 0 },
        by_source: { buckets: [] },
      },
    });

    // Send a `user:update` call to the connector.
    mocks.minihull.userUpdate(
      {
        // Connector Settings
        connector: {
          id: "123456789012345678901234",
          private_settings: {
            api_key: "123",
            handle_accounts: true,
            prospect_enabled: true,
            prospect_segments: ["1"],
            prospect_filter_titles: ["foo"],
            prospect_limit_count: 2,
          },
        },
        // Message payload
        messages: [
          {
            user: { id: "abc", "traits/clearbit/source": "reveal" },
            account: { id: "ACCOUNTID", domain: "domain.com" },
            segments: [{ id: "1" }],
          },
        ],
      },
      // This is what the Firehose receives.
      ({ batch, logs }) => {
        const [first, second, third, fourth] = batch;
        expect(batch.length).to.equal(4);
        expect(logs[1].message).to.equal("outgoing.user.start");
        myNock.done();
        done();
      }
    );
  });
});

Releasing

We follow the Git Flow model, semver for versioning and we maintain CHANGELOG.md for each release of the production build.

Once you have the prerequisites installed, execute yarn run build or npm run build in the repository root to build the project locally.