ElasticSearch: Geographically Encoded Objects for ElasticSearch

(Driver available in GDAL 1.10 or later)
Driver is WRITE Only

ElasticSearch is an Enterprise-level search engine for a variety of data sources. It supports full-text indexing and geospatial querying of those data in a fast and effecient manor using a predefined REST API. This driver serializes all of the supported OGR file formats in to an ElasticSearch index.

Field definitions

Fields are dyamically mapped from the input OGR data source. However, the driver will take advatage of advanced options within ElasticSearch as defined in a field mapping file.

The mapping file allows you to modify the mapping according to the ElasticSearch field-specific types. There are many options to choose from, however, most of the functionality is based on all the different things you are able to do with text fields within ElasticSearch.

ogr2ogr -progress --config ES_WRITEMAP /path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 my_shapefile.shp

The ElasticSearch writer supports the following Configuration Options:

ES_WRITEMAP=/path/to/mapfile.txt Creates a mapping file that can be modified by the user prior to insert in to the index.
ES_META=/path/to/mapfile.txt Tells the driver to the user-defined field mappings.
ES_BULK=10000 Identifies the number of records to be inserted at a time. Lower record counts help with memory consumption within ElasticSearch but take longer to insert.
ES_OVERWRITE=1 Overwrites the current index by deleting an existing one.

It is possible to apply several options at a time. The following use case takes advantage of a predefined mapping file as well as a limited BULK insert count.

ogr2ogr -progress --config ES_OVERWRITE 1 --config ES_BULK 10000 --config ES_META /path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 PG:"host=localhost user=postgres dbname=my_db password=password" "my_table" -nln thetable

Examples

Basic Transform:

ogr2ogr -progress -f "ElasticSearch" http://localhost:9200 my_shapefile.shp

Create a Mapping File:
The mapping file allows you to modify the mapping according to the ElasticSearch field-specific types. There are many options to choose from, however, most of the functionality is based on all the different things you are able to do with text fields.

ogr2ogr -progress --config ES_WRITEMAP /path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 my_shapefile.shp

Read the Mapping File:
Reads the mapping file during the transformation

ogr2ogr -progress --config ES_META /path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 my_shapefile.shp

Bulk Uploading (for larger datasets):
Bulk loading helps when uploading a lot of data. The integer value is the number of bytes that are collection before being inserted.

ogr2ogr -progress --config ES_BULK 10000 -f "ElasticSearch" http://localhost:9200 PG:"host=localhost user=postgres dbname=my_db password=password" "my_table" -nln thetable

Overwrite the current Index:
If specified, this will overwrite the current index. Otherwise, the data will be appended.

ogr2ogr -progress --config ES_OVERWRITE 1 -f "ElasticSearch" http://localhost:9200 PG:"host=localhost user=postgres dbname=my_db password=password" "my_table" -nln thetable

Specify several at a time:
Several flags can be set at the same time.

ogr2ogr -progress --config ES_OVERWRITE 1 --config ES_BULK 10000 --config ES_META /path/to/file/map.txt -f "ElasticSearch" http://localhost:9200 PG:"host=localhost user=postgres dbname=my_db password=password" "my_table" -nln thetable

ElasticSearch: Geographically Encoded Objects for ElasticSearch

Field definitions

Examples

See Also