npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

@cubejs-backend/cubestore

v1.1.5

Published

Cube.js pre-aggregation storage layer.

Downloads

48,527

Readme

WebsiteDocsExamplesBlogSlackTwitter

npm version GitHub Actions FOSSA Status

Cube Store

Cube.js pre-aggregation storage layer.

Motivation

Over the past year, we've accumulated feedback around various use-cases with pre-aggregations and how to store them. We've learned that there are a set of problems where relational databases as a storage layer has significant performance and functionality issues.

These problems include:

  • Performance issues with high cardinality rollups (1B and more)
  • Lack of HyperLogLog support
  • Degraded performance for big UNION ALL queries
  • Poor JOIN performance across rolled up tables
  • Table/schema name length issues across different database types
  • SQL type differences between source and external database

Over time, we realized that if we try to fix these issues with existing database engines, we'd end up modifying these databases' codebases in one way or another.

We decided to take another approach and write our own materialized OLAP cache store, designed solely to store and serve rollup tables at scale.

Approach

To optimize performance as much as possible, we went with a native approach and are using Rust to develop Cube Store, utilizing a set of technologies like RocksDB, Apache Parquet, and Arrow that have proven effectiveness in solving data access problems.

Cube Store is fully open-sourced and released under the Apache 2.0 license.

Plans

We intend to start distributing Cube Store with Cube.js, and eventually make Cube Store the default pre-aggregation storage layer for Cube.js. Support for MySQL and Postgres as external databases will continue, but at a lower priority.

We'll also update all documentation regarding pre-aggregations and include usage and deployment instructions for Cube Store.

Supported architectures and platforms

If your platform/architecture is not supported, you can launch Cube Store using Docker.

| | linux-gnu | linux-musl | darwin | win32 | | -------- | :---------: | :----------: | :------: | :-----: | | x86 | N/A | N/A | N/A | N/A | | x86_64 | ✅ | ✅ | ✅ | ✅ | | arm64 | ✅ | | ✅[1] | |

[1] It can be launched using Rosetta 2 via the x86_64-apple binary.

Usage

With Cube.js

Starting with v0.26.48, Cube.js ships with Cube Store enabled when CUBEJS_DEV_MODE=true. You don't need to set up any CUBEJS_EXT_DB_* environment variables or externalDriverFactory inside your cube.js configuration file.

For versions prior to v0.26.48, you should upgrade your project to the latest version and install the Cube Store driver:

yarn add @cubejs-backend/cubestore-driver

After starting up, Cube.js will print a message:

🔥 Cube Store (0.26.64) is assigned to 3030 port.

With Docker

Start Cube Store in a Docker container and bind port 3030 to 127.0.0.1:

docker run -d -p 3030:3030 cubejs/cubestore:edge

Configure Cube.js to use the above connection for an external database via the .env file:

CUBEJS_EXT_DB_TYPE=cubestore
CUBEJS_EXT_DB_HOST=127.0.0.1

With Docker Compose

Create a docker-compose.yml file with the following content:

version: '2.2'
services:
  cubestore:
    image: cubejs/cubestore:edge

  cube:
    image: cubejs/cube:latest
    ports:
      - 4000:4000  # Cube.js API and Developer Playground
      - 3000:3000  # Dashboard app, if created
    env_file: .env
    depends_on:
      - cubestore
    links:
      - cubestore
    volumes:
      - ./schema:/cube/conf/schema

Configure Cube.js to use the above connection for an external database via the .env file:

CUBEJS_EXT_DB_TYPE=cubestore
CUBEJS_EXT_DB_HOST=cubestore

Build

docker build -t cubejs/cubestore:latest .
docker run --rm cubejs/cubestore:latest

Development

Debian prerequisites (incomplete): apt-get install lld libssl-dev pkg-config cmake

When changing Datafusion or Arrow:

Check out https://github.com/cube-js/arrow-rs/tree/cube and https://github.com/cube-js/arrow-datafusion/tree/cube and add the following to the current directory's Cargo.toml. (But remember to exclude this from your PR!)


[patch.'https://github.com/cube-js/arrow-rs']
arquet = { path = "../../../arrow-rs/parquet" }
arrow = { path = "../../../arrow-rs/arrow" }

[patch.'https://github.com/cube-js/arrow-datafusion']
datafusion = { path = "../../../arrow-datafusion/datafusion" }

Of course, you can use absolute paths or adjust the paths to your chosen checkout location.

It is possible that uncommenting the arrow-datafusion .cargo/config.toml path line works for you too, but it might not, if you are making changes in arrow-rs.

License

Cube Store is Apache 2.0 licensed.