tile-maker

TopoJSON Tile Maker

The TopoJSON Tile Maker uses Node.js web services and an example browser front end to convert a collection of one or more ESRI shape files into a complete administrative geography. Multiple layers are supported so that an integrated set of census administrative geography layers can be created. To use USA as an example states, counties, census tracts, census block groups into are combined into layers of TopoJSON tiles optimised for a series to map zoom levels. Typically zoom levels 6-11 are supported. The web services contain a very simple tile server for testing; the user then downloads the tiles for deploying a OSM 'standard' style compliant map server.

Tile Maker is integrates a number of commonly used Node.js modules to create a simple to use, yet powerful application for creating administrative geographies tile sets typically used in large scale Epidemiological studies. It has been deliberately built to handle very large shape files; the principle test suites being US census geography 2014 to census block group and England and Wales 2011 census geography. A Node.js web architecture was chosen deliberately to avoid breaking web browsers with very large shape files.

Tile Maker was developed as part of the Rapid Inquiry Facility or RIF; a freely available software application that supports two types of environmental health activities: disease mapping studies and risk analysis studies. The RIF was designed to help epidemiologists and public health researchers to rapidly investigate potential environmental hazards, especially those related to industrial sites. The tool uses health, environmental, socio-economic, population and geographic data to calculate risks in relation to sources of exposure and to generate maps. The RIF, and Tile Maker, has been developed by the Small Area Health Statistics Unit at Imperial College, London.

SAHSU has developed a rapid inquiry facility (RIF) http://www.sahsu.org/content/rapid-inquiry-facility to permit fast and versatile exploration of the small area data held in national databases in response to urgent queries about environmental hazards. The RIF has been widely disseminated in Europe (with EU funding) and the USA (as part of the CDC Environmental Public Health Tracking (EPHT) Program).

This repository is currently a stub; the actual code will be transferred from the RIF in Q3/Q4 2016.

The RIF

Graphical User Interface

Note: a much better JQuery-UI prototype that has the RIF look and feel will appear shortly.

Work flow

1. Select a set of shape files. Zip files are supported;
2. GUI unpacks zip files, processing DBF and ESRI extended attribute files (.shp.ea.iso.xml) to extract meta data;
3. The name of the administrative geography is defined (e.g. United States 2014 Census);
4. For each shape file select the field containing the administrative code (e.g. MT) and the administrative name (e.g. Montana);
5. The quantization, zoom level range and the inter zoom level simplification factor are defaulted. These can be changed;
6. The files are uploaded to the tile maker Node.js service for processing;
7. The GUI provides status on Node.js service processing until the initial simplification is complete;
8. The untiled topoJSON is fetched and displayed using Leaflet: http://leafletjs.com/; below zoom level 6 only shape files with less than 200 areas are displayed; shape files with more than 10,000 area are suppressed. This is prevent browser crashes before tile production;
9. If all the require meta data is present then the Node.js service continues to process to produce the tiles for the required zoom levels;
10. When complete the GUI displays the tiles

The processing setup is savable to an XML file; this can be used in a "batch" mode to the Node.js web service and provide the interface for the RIF's own Java based data cleaning and loading tool or other tools.

The GUI and the web service will accept geoJSON files; if the XML file is provided the processing will also produce tiles. This allows very large shape files to be pre-shrunk using Mike Bostock's topojson program and there to reduce the server memory requirements.

Node.js web service processing

To be added.

Limits

1. Memory; the Node.js back end requires approximately 7x the uncompressed size of the shape files
2. Browsers; Internet Explorer 10+, the latest Chrome and Firefox will be tested. a) Leaflet on IE has severe problerms with >20M of JSON data; it does work - it is just abysmally slow! b) Firefox 47.0 has a bad bug (1274010) which causes large xmlhttprequest POSTs to fail randomly . It apperars to be fix in beta. Previous Firefox releases could handle multi GB file loads c) Chrome will barf with very large files, but is fine to 300-500 MB.
3. There is a hard limit of 255M per shape file record. This is the Node.js string limit.