Spatial analytics with GeoJSON in BigQuery

[ad_1]

A simple method to inform the distinction between the 2 codecs is to verify if there’s a “FeatureCollection” JSON object within the file – then it’s GeoJSON. If there may be precisely one “Function” JSON object on every line, then it’s GeoJSON-NL.

Why geoJSON-NL as a spatial format?

GeoJSON is a good spatial file format as a result of it really works in a broad variety of purposes throughout many platforms, and is a subset of JSON. Nonetheless, the issue with GeoJSON is that each one options are saved as a “FeatureCollection” object in a big array. Parsing a big array with giant JSON strings is a efficiency bottleneck. All options saved within the array have to be loaded and parsed into reminiscence by a single CPU thread from the supply file, resulting in ETL jobs that fail attributable to out-of-memory errors. If there are points with anyone function in a FeatureCollection, the complete parse & load job could fail.

Newline-delimited geoJSON solves the issues of normal geoJSON information by storing every function on a singular line as an alternative of in an array, enabling increased efficiency and effectivity for purposes working with a considerable amount of spatial knowledge, similar to streaming IoT (instance). Options might be learn and written from a number of threads or a distributed course of, and every function might be parsed on it’s personal, reducing the reminiscence footprint and permitting extra strong schema and geometry validation.

Changing spatial knowledge to geoJSON-NL

You may convert your spatial knowledge to geoJSON-NL format utilizing quite a lot of open supply and proprietary instruments. The part under contains code examples for changing geoJSON, shapefiles, GPX, KML, and CSV information to geoJSON-NL. You need to use the code examples under utilizing the GCP Cloud Shell (best to start out), your native growth machine (Linux or MacOS based mostly), or as a GCP Cloud Perform utilizing a Node.js or Python runtime setting.