soda-ts

functional SoQL wrapper to interact with Open Data API

| branch | coverage | CI | | :------ | :----------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------: | | staging | | | | master | | |

install

# for js
yarn add ramda rxjs soda-ts
# for the adventurous 🤷🏼‍♂️
yarn add https://github.com/data-depot/soda-ts.git
# for ts
yarn add -D @types/ramda

use

soda-ts was designed with composition in mind. To start of, we need to create a query. Dataset src id must be provided while the domain defaults to NYC Open Data

createQuery

import { createQuery } from 'soda-ts'

const query = createQuery({ src: 'w7w3-xahh' })

// to pull from other domains
const query = createQuery({
  domain: 'data.cityofchicago.org',
  src: 'ydr8-5enu'
})

A raw query is the most primitive form of query. To make it more interesting, we need to compose it with clauses

import { createQuery, where } from 'soda-ts'
import { pipe } from 'ramda'

const query = pipe(
    createQuery
    where``
)({ src: 'w7w3-xahh' })

createRunner$

Now, we can run the query. We have a Promise and Observable wrappers. For simple requests, createRunner, which is Promise based, is quite useful. But we'd recommend the Observable api.

import { createQuery, where, createRunner$ } from 'soda-ts'
import { pipe } from 'ramda'

const authOpts = {
    appToken: '', // REQUIRED for authenticated req
    keysCamelCased: false // whether to serialize keys to camelCase from snake_case
}

pipe(
    createQuery
    where`something > 12`
    createRunner$(authOpts)
)({ src: '' })
    .subscribe(
        next: (rawData) => {},
        error: ...,
        complete: ...
    )

autoPaginator$

To grab large paginated datasets, we can take advantage of query managers. They handle all aspects of pagination automagically and uses Subjects from rxjs as an eventbus. Combining ramda and rxjs composition, we can write declarative code to handle all the side effects cleanly.

import { createQuery, where, autoPaginator$, createManagerCreator } from 'soda-ts'
import { pipe } from 'ramda'
import { Subject } from 'rxjs'

export const managerOpts = {
  limit: 5,
  offset: 0,
  authOpts: {
    appToken: '',
    keysCamelCased: true
  }
}

const paginatorSub$ = new Subject()

paginatorSub$.subscribe({
    next(val) {
        // whatever you'd like to do with each page of data
    },
    error(e) {
        // handle error from individual req
    },
    complete() {
        //  handle end of all pagination reqs
    }
})

pipe(
  createQuery
  where`SOMETHING > 0`
  createManagerCreator<RawData>(managerOpts),
  autoPaginator$(paginatorSub$)
)({ src: 'w7w3-xahh' }).subscribe({
  error: () => {
    // handle if auto paginator breaks
},
  complete: () => {
    // pagination completed
}})