npm package discovery and stats viewer.

Discover Tips

  • General search

    [free text search, go nuts!]

  • Package details

    pkg:[package-name]

  • User packages

    @[username]

Sponsor

Optimize Toolset

I’ve always been into building performant and accessible sites, but lately I’ve been taking it extremely seriously. So much so that I’ve been building a tool to help me optimize and monitor the sites that I build to make sure that I’m making an attempt to offer the best experience to those who visit them. If you’re into performant, accessible and SEO friendly sites, you might like it too! You can check it out at Optimize Toolset.

About

Hi, 👋, I’m Ryan Hefner  and I built this site for me, and you! The goal of this site was to provide an easy way for me to check the stats on my npm packages, both for prioritizing issues and updates, and to give me a little kick in the pants to keep up on stuff.

As I was building it, I realized that I was actually using the tool to build the tool, and figured I might as well put this out there and hopefully others will find it to be a fast and useful way to search and browse npm packages as I have.

If you’re interested in other things I’m working on, follow me on Twitter or check out the open source projects I’ve been publishing on GitHub.

I am also working on a Twitter bot for this site to tweet the most popular, newest, random packages from npm. Please follow that account now and it will start sending out packages soon–ish.

Open Software & Tools

This site wouldn’t be possible without the immense generosity and tireless efforts from the people who make contributions to the world and share their work via open source initiatives. Thank you 🙏

© 2024 – Pkg Stats / Ryan Hefner

flatgeobuf

v3.35.0

Published

A performant binary encoding for geographic data

Downloads

15,445

Readme

layout FlatGeobuf

CI npm Maven Central Nuget Crates.io Discord Chat

A performant binary encoding for geographic data based on flatbuffers that can hold a collection of Simple Features including circular interpolations as defined by SQL-MM Part 3.

Inspired by geobuf and flatbush. Deliberately does not support random writes for simplicity and to be able to cluster the data on a packed Hilbert R-Tree enabling fast bounding box spatial filtering. The spatial index is optional to allow the format to be efficiently written as a stream, support appending, and for use cases where spatial filtering is not needed.

Goals are to be suitable for large volumes of static data, significantly faster than legacy formats without size limitations for contents or metainformation and to be suitable for streaming/random access.

The site switchfromshapefile.org has more in depth information about the problems of legacy formats and provides some alternatives but acknowledges that the current alternatives has some drawbacks on their own, for example they are not suitable for streaming.

FlatGeobuf is open source under the BSD 2-Clause License.

Examples

Specification

layout

  • MB: Magic bytes (0x6667620366676201)
  • H: Header (variable size flatbuffer)
  • I (optional): Static packed Hilbert R-tree index (static size custom buffer)
  • DATA: Features (variable size flatbuffers)

The fourth byte in the magic bytes indicates major specification version. The last byte of the magic bytes indicate patch level. Patch level is backwards compatible so an implementation for a major version should accept any patch level version.

Any 64-bit flatbuffer value contained anywhere in the file (for example coordinates) is aligned to 8 bytes to from the start of the file or feature to allow for direct memory access.

Encoding of any string value is assumed to be UTF-8.

A changelog of the specification is available here.

I recommend these blog posts by Horace Williams provides more details and explanations:

  • https://worace.works/2022/02/23/kicking-the-tires-flatgeobuf/
  • https://worace.works/2022/03/12/flatgeobuf-implementers-guide/

Performance

Preliminary performance tests has been done using road data from OSM for Denmark in SHP format from download.geofabrik.de, containing 906602 LineString features with a set of attributes.

| | Shapefile | GeoPackage | FlatGeobuf | GeoJSON | GML | |-----------------------|-----------|------------|------------|---------|-----| | Read full dataset | 1 | 1.02 | 0.46 | 15 | 8.9 | | Read w/spatial filter | 1 | 0.94 | 0.71 | 705 | 399 | | Write full dataset | 1 | 0.77 | 0.39 | 3.9 | 3.2 | | Write w/spatial index | 1 | 1.58 | 0.65 | - | - | | Size | 1 | 0.72 | 0.77 | 1.2 | 2.1 |

The test was done using GDAL implementing FlatGeobuf as a driver and measurements for repeated reads using loops of ogrinfo -qq -oo VERIFY_BUFFERS=NO runs and measurements for repeated writes was done with ogr2ogr conversion from the original to a new file with -lco SPATIAL_INDEX=NO and -lco SPATIAL_INDEX=YES respectively.

Note that for the test with spatial filter a small bounding box was chosen resulting in only 1204 features. The reason for this is to primarily test the spatial index search performance.

As performance is highly data dependent I've also made similar tests on a larger dataset with Danish cadastral data consisting of 2511772 Polygons with extensive attribute data.

| | Shapefile | GeoPackage | FlatGeobuf | |-----------------------|-----------|------------|------------| | Read full dataset | 1 | 0.23 | 0.12 | | Read w/spatial filter | 1 | 0.31 | 0.26 | | Write full dataset | 1 | 0.95 | 0.63 | | Write w/spatial index | 1 | 1.07 | 0.70 | | Size | 1 | 0.77 | 0.95 |

Optimizing Remotely Hosted FlatGeobufs

If you're accessing a FlatGeobuf file over HTTP, consider using a CDN to minimize latency.

In particular, when using the spatial filter to get a subset of features, multiple requests will be made. Often round-trip latency, rather than throughput, is the limiting factor. A caching CDN can be especially helpful here.

Fetching a subset of a file over HTTP utilizes Range requests. If the page accessing the FGB is hosted on a different domain from the CDN, Cross Origin policy applies, and the required Range header will induce an OPTIONS (preflight) request.

Popular CDNs, like Cloudfront, support Range Requests, but don't cache the requisite preflight OPTIONS requests by default. Consider enabling OPTIONS request caching . Without this, the preflight authorization request could be much slower than necessary.

Features

Supported applications / libraries

Documentation

TypeScript / JavaScript

Prebuilt bundles (intended for browser usage)

Node usage

See this example for a minimal how to depend on and use the flatgeobuf npm package.

FAQ

Why not use WKB geometry encoding?

It does not align on 8 bytes so it not always possible to consume it without copying first.

Why not use Protobuf?

Performance reasons and to allow streaming/random access.

Why not use compression as part of the format?

Separation of concerns and to allow random access.

Why am I not getting expected performance in GDAL?

Default behaviour is to assume untrusted data and verify buffer integrity for safety. If you have trusted data and want maximum performance make sure to set the open option VERIFY_BUFFERS to NO.

What about MapBox Vector Tiles?

FlatGeobuf does not aim to compete with MapBox Vector Tiles. MVTs are great for rendering but they are relatively expensive to create and is a lossy format, where as FlatGeobuf is lossless and very fast to write especially if a spatial index is not needed.

Why does it not work with create-react-app?

See https://github.com/flatgeobuf/flatgeobuf/issues/244 for root cause and workaround.

Does FlatGeobuf support mixing features with and without geometry with spatial index?

Currently it likely does not but could in the future, see https://github.com/flatgeobuf/flatgeobuf/discussions/260.