@lcdev/store: MobX state modelling library

A lightweight entity container library that leverages common patterns in our apps. Roughly equivalent to mobx-state-tree, but a lot lighter on features. We don't typically need the snapshots, lifecycles and fancy features it provides - and it comes with a relatively high learning cost. This package is more like a grab bag of features that have been found to be useful across all of apps, and is opinionated as such.

yarn add @lcdev/store@VERSION

Structure:

• You will probably want to separate your stores in a foo-stores package
  • This makes them very easily testable and consumable from multiple frontends
  • See dura for an example of this structure
• You'll probably want to use @lcdev/fetch in tandem with this package
  • See dura for an example of how to tie authentication tokens into it
• Setting up your stores and APIs should be done in one place, added to react context, and usable through a hook
  • See dura for an example of this

Guide Level Explanation

This package mostly centers around the notion of what we call entities. You might call this a 'model', 'record', or some other vocab. At launchcode, we're trying to stay consistent with Model on the backend, and Entity on the frontend (there is no formal or good reason, we just needed to pick).

If you come from an object-oriented background, the notion of Entities (we'll use the capitalization like this to formally refer to it) should be natural. An Entity is the representation of some concrete object in the system - a User, Employee, BlogCategory, Timecard, Student, whatever works as an analogy.

You might start to assume that Entity is a class - that's not technically the case. But if it comes natural, it's not the worst way to think about it (many previous iterations of this package were this way).

You might also wonder why this is necessary at all. "I already have JSON objects, what more do I need?". That's fair actually, it's entirely true that complicated frontends can be built out from raw data, mutating it in that same form. With things like Redux being popular in this problem space, it's very likely you're used to solving problems this way.

const someBackendData = await (await fetch('/my-api/data')).json();

... later on ...

someBackendData.updatedValue = 42;
await fetch('/my-api/post-data', { data: someBackendData });

A bit contrived, but you should get the idea here. Everything is a POJO (plain old javascript object). There isn't much room for complication at all - it's nice and simple.

One Tuesday, you look at the backlog, and see that next up is to add some date information of every shift card within your scheduling web app. Sounds easy, we'll just do some moment magic and be on our way.

function ShiftCard({ shift }) {
  return (
    <div className="contrived example totally not inspired by SOL">
      {... some info about our shift ...}

      {/* and voila */}
      <span>{moment(shift.startDate).format('HH:mm')}</span>
    </div>
  );
}

You've done this a thousand times before. And with mobx, accessing stores is easy.

Well, it turns out that we've actually created a pretty big problem for ourselves. At this point, we'll admit that this example is entirely real - SOL ran into this exact conundrum early on. It turns out that re-doing the work of date parsing and formatting on ever render is really really bad (renders were on the order of seconds).

So what's the reaction here? Just parse any dates upfront? We're done then. Well it's not quite that easy. We've ran into an issue that scales up. We don't have a canonical definition of 'Shift' (in this example). All we have is the JSON data coming back, and a type that loosely correlates (to an extent, we're trusting the backend here too).

We aren't convinced yet though. POJOs are nice and easy to grok, no type system involved. Let's continue our little example...

Our next feature to implement is on that same ShiftCard component. This time, we want to show what area of the building that this shift occurs in. Again, sounds pretty simple.

First question to answer is of course backend data. We'll want to include area data on shifts.

[
  { id: 1282, startDate: "2020-01-01T07:00:00Z", area: { id: 2, description: "Area LTC-1", department: { id: 1, description: "LTC" } } },
  { id: 1283, startDate: "2020-01-02T07:00:00Z", area: { id: 2, description: "Area LTC-1", department: { id: 1, description: "LTC" } } },
  { id: 1284, startDate: "2020-01-03T07:00:00Z", area: { id: 2, description: "Area LTC-1", department: { id: 1, description: "LTC" } } },
  { id: 1285, startDate: "2020-01-04T07:00:00Z", area: { id: 2, description: "Area LTC-1", department: { id: 1, description: "LTC" } } },
  ...
]

Something stands out of course. We don't need that much redundant data. And it turns out that most shifts in one schedule happen in the same area. With a POJO, we're almost out of luck. We'll leave it as an exercise to figure out how to share data here without any additional mechanisms.

If it's a Friday and you're feeling particularly lazy, you'd be tempted to just YOLO here and keep everything in memory as it is returned from the backend. But it turns out that this can have a really really bad effect - memory usage climbs (GBs were seen for this particular example) - but worst of all, we lose the definition of "area 2"...

Let me explain what I mean - since there are multiple copies of the area { id: 2, ... } in memory, we have no way of changing that area. So any changes that you want to propagate to other parts of the app that reference that particular thing just 'have to know' to re-fetch that data (not to mention the useless data usage of doing that).

This problem is well known, well solved and the solution has a name: Identity Maps. You'll see it in ORMs a lot. It's a simple concept at heart - store one and only one of every 'thing'.

So you're sold - we want some amount of transformation of input data, we want a canonical source for 'things'. Well it turns out to be really difficult to do that without a categorization system. You might see where we're going with this. The 'area #1' is not the same as 'department #1', but in fact they may both look like { id: 1, label: 'LTC' }. We need a way to identify the object as a specific type.

Here comes that Entity part. This package provides a flexible but barebones way to create and manage these objects.

Before going on though, a point to make. Our API response should really look like this:

[
  { id: 1282, startDate: "2020-01-01T07:00:00Z", area: { id: 2 } },
  { id: 1283, startDate: "2020-01-02T07:00:00Z", area: { id: 2 } },
  { id: 1284, startDate: "2020-01-03T07:00:00Z", area: { id: 2 } },
  { id: 1285, startDate: "2020-01-04T07:00:00Z", area: { id: 2 } },
  ...
]

Turns out that this can actually be a lot faster too (think about the foreign keys here). The obvious following question is "well how do we get those areas then?" - the answer is pretty simple, just fetch them (individually, in batch, all at once, it's up to your application). You might criticize this and say "there's no way we can allow N+1 fetches like that in my app, we need it to be fast!". And sure, that's potentially true. Try it yourself and see. You can always go back to including references in the first response, and form Entities from them. How this could be related to GraphQL is left as a research exercise (that's definitely where we're heading).

Okay, lots of theoreticals and examples here. Let's show some real code.

import { buildEntity } from '@lcdev/store';

const Area = buildEntity('Area')
  .fields({
    description: String,
  })

const Shift = buildEntity('Shift')
  .fields({
    startDate: InstanceOf(Date),
    area: Area.Reference,
  })

There's our area and shift again. This time though, we're actually constructing something entirely separate from the runtime objects for those things.

import { EntityReference } from '@lcdev/store';

const ltc1 = Area.create(2, { description: 'Area LTC-1' });

const shift1 = Shift.create(1282, {
  startDate: new Date('2020-01-01T07:00:00Z'),
  area: new EntityReference(Area, 2),
});

Okay, you've now seen some code. Let's explore a bit.

1. buildEntity('EntityName') creates a new definition for EntityName - this is like creating a class
2. .fields({ ... }) defines the type of properties that any EntityName will contain
3. EntityName.create(id, fields) creates a new EntityName - IDs are always assumed to be integer primary keys

You should at this point do a little reading:

• runtypes this is what we use for .fields({ ... in here ... })
• mobx-state-tree notice the similarities to this API
• orbit this library and similar ones served a lot of inspiration

An entity usually has more than just fields - let's see actions and computed properties.

const Shift = buildEntity('Shift')
  .fields({
    startDate: InstanceOf(Date),
    endDate: InstanceOf(Date),
    area: Area.Reference,
  })
  .computed(self => ({
    get isOver12Hours() {
      // this is wrong, but just ignore that for this example
      return self.endDate.getHours() - self.startDate.getHours() > 12;
    },
  }))
  .actions({
    changeDate(dayOfTheMonth: number) {
      // again, this is very wrong, but ignore that for example's sake
      this.startDate.setDate(dayOfTheMonth);
      this.endDate.setDate(dayOfTheMonth);
    },
  });

Any computed properties are memoized, and all actions are mobx actions (meaning they are transactional and efficiently batched).

Stores

So of course, entities don't exist in a vacuum. We need a container to store them.

You might be tempted to do this ad-hoc. You might set up a store class for every entity, and do something like:

import { Instance } from '@lcdev/store';

class ShiftStore {
  @observable shifts: Instance<typeof Shift>[] = [];

  async fetchShift(id) {
    const shift = await fetch(...);
    ... transformation ...

    this.shifts.push(shift);
  }
}

Seems harmless? Now red flags should be going off - what if we called fetchShift(1) twice? Fine, we'll add checks.

import { Instance } from '@lcdev/store';

class ShiftStore {
  @observable shifts: Instance<typeof Shift>[] = [];

  async fetchShift(id) {
    const shift = await fetch(...);
    ... transformation ...

    const foundIndex = this.shifts.findIndex(s => s.id === id);

    if (foundIndex !== -1) {
      // or is it slice?
      this.shifts.splice(foundIndex, 1);
    }

    this.shifts.push(shift);
  }
}

I'll bet you'd get tired of this really quick. So would your CPU. Cycling through every value for an ID lookup is crazy.

This is of course where @lcdev/store comes in. There's a naturally ocurring set of functions that all 'entity stores' will encounter in their lifetime, so we've done them for you.

import { EntityStore, Instance } from '@lcdev/store';

class ShiftStore extends EntityStore<Instance<typeof Shift>> {
  async fetchShift(id) {
    const shift = await fetch(...);
    ... transformation ...

    this.add(shift);
  }
}

Every EntityStore has a backing hashmap of ID -> Entity. So lookups and adds are closer to O(1) than O(n ^ 2).

• get(id): Entity | undefined - finds the entity by ID
• getAnd(id, callback) - runs callback if the entity was there, returning result
• forceGet(id): Entity - get, but errors when not there
• take(id): Entity - like get, but also removes from the store
• has(id): boolean - does the store contain this entity
• remove(id): boolean - removes by id
• removeWhere(predicate) - removes all that match predicate
• add(entity) - adds to the store
• addAll(entities) - adds to the store

There are some more not listed here. Importantly, you can treat an EntityStore as though it's a list of values. In fact, the following will work.

for (const shift of shiftStore) { ... }

Which is to say that entity stores are 'just' an iterator.

References between Entities

A quick note - there is a tiny bit of boilerplate to enable references between entities.

Normally, you'd initialize all of your entity stores in one place at the startup of your app.

import { registerStore } from '@lcdev/store';

const blogStore = new BlogStore();
const categoryStore = new CategoryStore();
const userStore = new UserStore();

// all we need is this
registerStore(Blog, blogStore);
registerStore(Category, categoryStore);
registerStore(User, userStore);

Once we're 'registered', we can resolve references between entities. Let's revisit the example above with our shifts and areas.

import { EntityReference } from '@lcdev/store';

const ltc1 = Area.create(2, { description: 'Area LTC-1' });

const shift1 = Shift.create(1282, {
  startDate: new Date('2020-01-01T07:00:00Z'),
  area: new EntityReference(Area, 2),
});

If ltc1 and shift1 are in EntityStores that were registered, we can do:

// foundArea here is `ltc1` above - if it was actually in a store
const foundArea = shift1.area.resolve();

Unlike mobx-state-tree, this process it explicit. You call resolve() and get back Entity | undefined. With this way of modelling data, we get a lot of benefits (single source of truth) and no technical drawbacks (lookups are instant, and return the full object).

FilterViews - computed data views

One extremely common operation is to filter your data. You've done it many times.

A FilterView creates a logical grouping of single or multidimensional filters. In english? You write a filter in one place, and you can use it in many places.

Better, let's see an example:

import { EntityStore, Instance, FilterView } from '@lcdev/store';

type Filters = {
  hardcover?: boolean;
  soldRange?: [number, number];
};

class BookStore extends EntityStore<Instance<typeof Book>> {
  view = new FilterView(
    () => this.values,
    () => ({} as Filters),
    (cf, filters) => {
      if (filters.hardcover !== undefined) {
        cf.dimension(book => book.hardcover).filterExact(filters.hardcover);
      }

      if (filters.soldRange) {
        cf.dimension(book => book.sold).filterRange(filters.soldRange);
      }

      return cf;
    },
  );
}

This dimension stuff is crossfilter, a library for multidimensional filters. It has a lot of features that we won't explore here, but suffice to say it handles simple and complex workloads.

There is a lot to be said about the FilterView concept, and it can be composed well. Without going too far in depth, we'll look at once last feature that makes it extremely useful.

Oftentimes you have some base set of filters, with something more static layered on top. For example, you might have a page that shows two lists (let's call these 'past' and 'pending', another example from SOL). You might even want to show these two lists at once.

  baseView = new FilterView(
    () => this.values,
    () => ({} as Filters),
    (cf, filters) => {
      if (filters.pastOrPresent) {
        // contrived way to filter our either Past or Present values
        cf.dimension(v => v.pastOrPresent).filterExact(filters.pastOrPresent);
      }

      return cf;
    },
  );

  pastView = baseView.fork(
    // here, we override the base filters
    (filters) => ({ ...filters, pastOrPresent: 'past' }),
  );

  presentView = baseView.fork(
    // here, we override the base filters
    (filters) => ({ ...filters, pastOrPresent: 'present' }),
  );

The cool thing about this is, any updates to the baseView filters get reflected in the two forked views. Only overriden filters stay. There is even an option to define more filters in a fork, which we'll leave to you to discover.

Structure of a Store

A normal entity store looks like this:

import { EntityStore, Instance, Errors, createAction } from '@lcdev/store';

export class BlogStore extends EntityStore<Instance<typeof Blog>> {
  err = new Errors<BlogStore.Err>();

  // any number of loading states that compound into isLoading
  @observable isFetching = false;
  @observable isSaving = false;
  @observable isCreating = false;
  @observable isDeleting = false;

  private loading = loadingState(
    () => this.isFetching || this.isSaving || this.isCreating || this.isDeleting, {
      // debounce for isLoading vs showLoading
      loadingDebounce: 100,
    },
  );

  get isLoading() {
    return this.loading.isLoading;
  }

  get showLoading() {
    return this.loading.showLoading;
  }

  fetchBlogs = createAction(
    loading => (this.isFetching = loading),
    this.err.setter(BlogStore.Err.Fetching),
    async () => {
      // do the fetching and call 'addAll' here
    },
  );
}

export namespace BlogStore {
  export enum Err {
    Fetching = 'fetching',
  }
}

Errors

One of the important lessons we've learnt is that errors are not global. This is a costly mistake that's hard to backtrack. The ShiftStore error should not be shown in the creation, deletion and edit dialogs - there needs to be distinct states. Clearing error messages at the right time is difficult to reason about, depending on your component structure (open prop vs conditional rendering, etc).

See the Errors container for more about this. The example store in here gives you an idea of how it's used.

In a component, you'd be interested in showing any current errors. You can do so easily with the Errors structure:

function BlogView() {
  const { blogStore } = useStores();

  React.useEffect(() => {
    blogStore.fetchBlogs();

    return () => {
      blogStore.clearError(BlogStore.Err.Fetching);
    };
  }, []);

  // do this if you know what error to show
  const error = blogStore.getErrorMessage(BlogStore.Err.Fetching);

  // do this if you don't (try to avoid it)
  const error = blogStore.anyError;

  // do this if you want to show every error type - is an array
  const errors = blogStore.allErrors;

  return (
    <>
      <MyErrorMessage>{error}</MyErrorMessage>

      {... normal component things ...}
    </>
  );
}

Pagination

We have built in support for pagination state, since it's a common problem. See the PaginationView container for a convenient wrapper.

In your store, it'll likely look like this:

import { EntityStore, Instance, PaginationView, createAction } from '@lcdev/store';

type PaginationFilters = {
  year?: number;
};

class BookStore extends EntityStore<Instance<typeof Book>> {
  public readonly pagination = new PaginationView(
    () => this.values,
    () => ({} as PaginationFilters),
  );

  fetchBooksPage = createAction(
    () => {}, () => {}, // ignore loading and errors, for the sake of example
    async (pageNum: number, filters: PaginationFilters) => {
      const { books, totalPages } = await magicBookFetch(pageNum, filters);

      // add to our pagination state - only stores IDs
      this.pagination.addAll({
        items: books,
        page,
        totalPages,
        filters,
        // ordering is required, to order within each page
        orderBy: 'insertDatetime',
        orderByDirection: 'desc',
      });

      // add to our BookStore
      return this.addAll(books);
    },
  );
}

Then, in a component:

function BookView() {
  const { bookStore } = useStores();

  const {
    pagination: {
      filtered, // this is the filtered list of books for the current page
      totalPages,
      currentPage,
      setCurrentPage,
    },
  } = bookStore;

  React.useEffect(() => {
    bookStore
      .fetchBooksPage(1, {})
      .then(() => {
        // typically, you want to do this after the action is complete
        setCurrentPage(1);
      });
  }, []);

  return (
    ... normal component things ...
  );
}

Entity Extension

Entity types are 'inheritable', meaning their behavior can be extended.

const MyExtendedEntity = MyEntity.extendAs('MyExtendedEntity')
  .fields({
    otherProperty: Boolean,
  });

This is sort of similar to classical OO inheritance.

New IDs - Offline and Draft States

The newID() function returns an ID that's negative (this could be changed, it's an implementation detail). Use isNewID(id) to check if a number was generated by newID(). We can use negative numbers because primary keys are positive. Also see stripNewID, which removes the id property from an object if it isNewID. This is useful for "create/save" or "upsert" routes.

API Reference

Most public functions, classes and types are jsdoc annotated and should show up in VS Code. Below is a listing of public exports that you can use, with a brief description.

• Entity<Name, Fields, Computed, Actions>: container for backend data, with behavior and computed properties
• EntityBuilder<Name, Fields, Computed, Actions>: class-like object that creates instances of Entities
• EntityReference<Name, E>: a reference to a real Entity (E) - use .resolve() to get the referenced E
• EntityStore<E>: container of many Es (Entities), which are unique by ID
• FilterView<V, Filters>: a multi-dimensional way to filter V[]
• PaginationView<E, Filters>: a FilterView with backing storage for pagination state to simplify logic in your app
• PaginationStore<Filters>: raw storage for pages of entities - you'll want to use PaginationView instead
• Errors<ErrorNames>: granular error state, to be initialized per store
• loadingState: sets up debounced computed loading state for a store
• ID, HasID, WithID, toNum: primary key type and conversion to raw integer
• newID, isNewID, stripNewID: for 'new' entities that have no canonical ID yet
• createAction: utility to create an async action (usually on stores) that handles errors and loading state
• shareConcurrentCalls: avoids two concurrent calls for fetchMyThing(1), caching running action

Tips and Tools

• use Instance<typeof MyEntity> to extract Entity type for MyEntity builder
• use typeof MyEntityStore['EntityType'] to extract Entity type for MyEntity store