= GDAL/OGR 2.1.4 Release Notes = The 2.1.4 release is a bug fix release. == Build == * configure: remove bashism (#6791) * configure: fix --without-mrf (#6811) * Add HDF4_INCLUDE option to MSVC builds (#6805) * DWG: fix compilation problems (#6843) * configure: fix detection of 64bit file API with clang 5 (#6912) * configure: use .exe extension when building with mingw64* toolchains (#6919) * mongoDB: compilation fix on Windows == Port == * CPLSerializeXMLNode(): fix potential buffer overflow (#6851) */vsicurl/: fix occasional inappropriate failures in Read() with some combinations of initial offset, file size and read request size (#6901) == Algorithms == * gdal_rasterize: fix crash when rasterizing empty polygon (#6844) * GDALRasterize(): avoid hang in some cases with all_touched option (#5580) == GDAL utilities == * Man pages: remove redundant utility name (#6822) * gdaldem: backport r37897 (#6847) * gdalwarp: fix crash when -multi and -to RPC_DEM are used together (#6869) == GDAL drivers == AIGRID driver: * fix handling on raw 32-bit AIG blocks (#6886) DTED driver: * correctly create files at latitudes -80, -75, -70 and -50 (#6859) GeoRaster driver: * support for JP2-F compression (#6861) * support direct loading of JPEG-F when blocking=no (#6861) * default blocking increased from 256x256 to 512x512 (#6861) * fix handling of memory allocation failure (#6884) OpenJpeg driver: * support direct extracting of GeoRaster JP2-F BLOB (#6861) GPKG driver: * set matrix_width/matrix_height in gpkg_tile_matrix in conformance with req 45 (#6826) * avoid corruption of gpkg_tile_matrix when building overviews, down to a level where they are smaller than the tile size (#6932) GTiff driver: * map D_North_American_1927 datum citation name to OGC North_American_Datum_1927 so that datum is properly recognized (#6863) Idrisi driver: * make Create() zero-initialize the .rst file (#6873) MBTiles driver: * on opening if detecting 3 bands, expose 4 bands since there might be transparent border tiles (#6836) netCDF driver: * fix crash on int64/uint64 dimensions and variables, and add support for them (#6870) NWT_GRC driver: * Fix handling of alpha values in GRC color table (#6905) SRTMHGT driver: * handle files for latitude >= 50 (#6840) VRT driver: * fix 2.1 regression that can cause crash in VRTSimpleSource::GetFileList() (#6802) WMTS driver: * accept tiles of small dimensions (https://github.com/OSGeo/gdal/pull/210) == OGR core == * OGR SQL: fix IN filtering on MapInfo indexed columns (2.0 regression, #6798) * Fix issue when getting MULTIPOINT(0 0,1 1) envelope (#6841) * OGRPolygon: make IsEmpty() return TRUE if the polygon is made of an empty ring (#6844) * OGRParse: fix parsing logic to avoid false positive detection of string as datetime (#6867) == OGR utilities == * ogr2ogr/GDALVectorTranslate(): fix crash when using -f PDF -a_srs (fixes #6920) == OGR drivers == Amigocloud driver: * backport part of change of trunk r36639 that should have been backported in 2.1 in r36640 but was missed for some reason (https://github.com/OSGeo/gdal/pull/137) GeoJSON driver: * ESRIJson: avoid endless looping on servers that don't support resultOffset (#6895) GPKG driver: * make GetFeature() works on non conformant tables that have no integer primary key field (#6799), and be robust to non standard column types * GPKG: declare feature id column of features tables and tile pyramid user data tables as NOT NULL (#6807, relates to ​https://github.com/opengeospatial/geopackage/issues/282) * GPKG/SQLite: fix ExecuteSQL() to work with a statement with 2 SELECT and ORDER BY clause (#6832) * GPKG: make driver robust to difference of cases between table_name in gpkg_contents/gpkg_geometry_columns and name in sqlite_master (#6916) ILI2 driver: * assign FID to features (#6839) LIBKML driver: * emit style related errors as warnings to make datasets openable by SWIG bindings (#6850) OCI driver: * support for long identifiers (up to 128 long) when running of 12.2 or + (#6866) ODS driver: * fix FID filtering (#6788) OpenFileGDB driver: * fix bug when field description offset is beyond 4GB (#6830) PG driver: * avoid errors with field default expressions like 'foo'::text (#6872) == Java bindings == * Fix crash on GetDefaultHistogram() if the C++ method returns an error (#6812) == Perl bindings == * Backport r37220 for fixing #6796 * Backport r37222 == Python bindings == * fix 'import osgeo.gdal_array' with python3 and SWIG 3.0.10 (#6801) * allow gdal.FileFromMemBuffer() to use buffer > 2GB (fixes #6828) == Security oriented fixes == Note: this is only a very partial backport of more extensive fixes done in GDAL trunk. Credit to OSS-Fuzz for all of them (some have been found locally, so no related ticket) * Fix CPLErrorSetState to ensure it does not write beyond DEFAULT_LAST_ERR_MSG_SIZE and correctly null-terminates last message. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1529. * OpenAIR: fix potential out-of-buffer read if we need to ingest 30000 bytes. Credit to OSS Fuzz * GDALDataset and GDALRasterBand::ReportError(): fix crash if dataset name has a % character * NASAKeywordHandler::SkipWhite(): fix out of bounds read * MITAB: ParseTABFileFields(): fix out of bounds read * MITAB: avoid potentially veryyyy long loop when stroking arcs. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1644 * MITAB: avoid heap-buffer-overflow in MITABCoordSys2TABProjInfo(). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1653 * DXF: prevent null ptr deref on corrupted file * DXF: fix out-of-bounds read on corrupted files * DXF: fix memory leak in case of corrupted file * KML::unregisterLayerIfMatchingThisNode(): use memmove() instead of memcpy(). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1580 * KML: fix crash on weird structure with recursively empty folders. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1591 * KML: fix null ptr dereference on corrupted file. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1634 * OGRCurveCollection::importBodyFromWkb(): fix potential crash if the curve in the collection has not the same dimension has the collection dimension. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1584 * TIGER: fix potential stack buffer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1586 * VFK: avoid out-of-bounds read. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1596 * VFK: fix out-of-bounds read. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2074 * CPLHexToBinary(): avoid reading outside of hex2char array on on-ASCII input. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1606 * OGR PDS: avoid int32 overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1607 * GeoRSS: fix null pointer dereference on corrupted files. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1611 * VSIArchiveFilesystemHandler::SplitFilename(): improve performance. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1597 * OGRGeometryFactory::organizePolygons(): fix crash on empty polygons. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1620 * JML: fix null pointer dereference on invalid files. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1622 * Shape: prevent null ptr deref on truncated MultiPointM geometry. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1638 * /vsisubfile/: avoid Tell() to return negative values. And make VSIIngestFile() more robust to unsigned overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1642 * GTM: avoid useless recursive opening of files when provided with a gzip-compressed input. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1650 * GTiff: fix heap-buffer-overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1677 * GTiff: avoid heap-buffer-overfow on corrupted State Plane citation. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2050 * GTiff: avoid potential stack buffer overflow on corrupted Imagine citation. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2105 * GTiff: prevent heap overflow and fix reading of multi-band band-interleaved 16/24-bit float datasets. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2264 * GTiff: fix potential infinite loop when parsing some 24-bit float denormalized numbers. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2315 * Internal libjson-c: fix stack buffer overflow. Submitted upstream as https://github.com/json-c/json-c/pull/325. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1682 * ILI1/ILI2: fix null pointer dereference when opening filename ','. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1689 * ILI1: fix various crashes and memleaks on corrupted files (including, but not limited to, https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1760 , 1784, 1926) * ILI2: use proper cast operator. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1813 * ILI2: fix crash due to unhandled exception. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2191 * OGRSpatialReference::morphFromESRI(): fix heap-use-after-free issue. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1690 * GML / NAS: fix memory leak in error code path, and potential heap-buffer-read-overflow * morphFromESRI(): prevent potential null pointer dereference. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1783 * NTF: fix various issues: heap & stack buffer-overflow, null ptr derefs, memory leaks. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1819 , 1820 , 1823, 1833, 1854, 1862, 1910, 1931, 1961, 1982, 1990, 1993, 1995, 1996, 2003, 2033, 2052, 2077, 2084, 2103, 2130, 2135, 2146, 2166, 2185, 2187 * morphFromESRI(): avoid potential null pointer dereference. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1867 * GXF: validate nGType to avoid later out-of-bound read in GXFReadRawScanlineFrom(). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=1975 * GXF: fix int overflow and avoid excessive memory allocation. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2207 * TIGER: avoid stack buffer overflows. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2020 * ISO8211: prevent stack buffer oveflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2022 * VICAR: fix null pointer dereference. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2116 * GPKG: fix potential heap-buffer overflow in GPkgHeaderFromWKB(). Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2150 * GPKG: fix potential null ptr deref. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2240 * REC: fix potential stack buffer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2165 * E00GRID: fix heap buffer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2182 * E00GRID: avoid index-out-of-bounds write access. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2237 * E00GRID: fix index out of bounds access. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2327 * VICAR: avoid use-after-free and heap-buffer-overflow. Fixes https://oss-fuzz.com/v2/testcase-detail/4825577427042304 * OGRCompoundCurve::importFromWkb(): avoid potential stack overflow. Fixes https://oss-fuzz.com/v2/testcase-detail/5192348843638784 * Selafin: fix double frees. Fixes https://oss-fuzz.com/v2/testcase-detail/6429713822121984 * netCDF: avoid stack buffer overflow. Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=2302 = GDAL/OGR 2.1.3 Release Notes = The 2.1.3 release is a bug fix release. == Build == * Windows: include plugins in clean, oci: also clean dll * windows: makegdal_gen.bat: update script to generate most recent Visual C++ project files (#6635) == Port == * CPLHTTPFetch(): add a CAINFO option to set the path to the CA bundle file. As a fallback also honour the CURL_CA_BUNDLE and SSL_CERT_FILE environment variables used by the curl binary, which makes this setting also available for /vsicurl/, /vsicurl_streaming/, /vsis3/ and /vsis3_streaming/ file systems (#6732) * Fix GetDiskFreeSpace() on 32bit Linux to avoid 32bit overflow when free disk space is above 4 GB (#6750) == GDAL algorithms == * Warper: avoid undefined behaviour when doing floating point to int conversion, that may trigger exception with some compilers (LLVM 8) (#6753) * OpenCL wrapper: do not use CL_MEM_COPY_HOST_PTR flag as it is forbidden by the opencl spec for clGetSupportedImageFormats() use case. Fix for NVidia use case (#6624) * OpenCL wrapper: fix wrapping of Float32 data type with NVIDIA OpenCL (#6669) == Utilities == * gdalinfo -json: fix order of points in wgs84Extent.coordinates (github #166) * gdal_rasterize: fix segfaults when rasterizing into an ungeoreferenced raster, or when doing 'gdal_rasterize my.shp my.tif' with a non existing my.tif (#6738) * gdal_calc.py: fix usage message == GDAL drivers == ADRG driver: * handle north and south polar zones (ZNA 9 and 18) (#6783) DODS driver: * fix crash on URL that are not DODS servers (#6718) HDF4 driver: * include hdf4compat.h in hdf4dataset.cpp GRASS driver: * frmts/grass/pkg/configure: add support for GRASS 7.2 (#6785) * frmts/grass/pkg/Makefile.in: do not clone datum tables and drivers (#2953) GTiff driver: * fix creating an image with the Create() interface with BLOCKYSIZE > image height (#6743) * don't check free disk space when outputting to /vsistdout/ (#6768) Idrisi/RST driver: * on creation: use geotransform of source dataset even if it doesn't have a SRS (#6727) netCDF driver: * fix erroneous detection of a non-longitude X axis as a longitude axis that caused a shift of 360m on the georeferencing (#6759) OpenJPEG driver: * for single-line organized images, such as found in some GRIB2 JPEG2000 images, use a Wx1 block size to avoid huge performance issues (#6719) PCIDSK driver: * handle Exceptions returned from destructor and check access rights in setters (github #183) RMF driver: * fix reading nodata for non-double data type (github #174) Sentinel2 driver: * add support for new "Safe Compact" encoding of L1C products (#6745) VRT driver: * Fix linking error on VRTComplexSource::RasterIOInternal() (#6748) == OGR core == * GML geometry parsing: avoid 'Cannot add a compound curve inside a compound curve' error (#6777) * Avoid assertion when stroking degenerated arcs (#6778) * Coordinate transformation: prevent unnecessary coordinate transformations (github #184) == OGR drivers == AmigoCloud driver: * add option to receive an AmigoCloud API key in the connection string, fix page size (github #137) GeoJSON driver: * fix segfaults on writing on NULL geometry with -lco WRITE_BBOX=YES (#6698) * support multilinestring from esriGeometryPolyline (#6773) GPKG driver: * don't show Spatialite vgpkg_ virtual tables (#6707) ILI1 driver: * make polygon reconstruction in Surface layers robust to curves not in natural order (#6728) MSSQL driver: * build optional mssql plugin with SQL Native Client support for MSSQL Bulk Copy NAS driver: * fix crash with enabled debugging SDE driver: * rename driver to OGR_SDE, and rename ICreateLayerFromRegInfo() as CreateLayerFromRegInfo() (#6714) Shape driver: * when rewriting the geometry of the last record in the .shp, do it at the file offset it previously used (#6787) SQLite driver: * Spatialite: do not report some BLOB columns as geometry columns of tables/views (when found before the geometry column(s)) (#6695, #6659) * fix update of features with multiple geometry columns (#6696) * Spatialite: avoid spatial views to cause layers 'layer_name(geometry_name)' to be publicly listed (#6740) * SQLite/GPKG: add explicit error message when trying to open a read-only WAL-enabled database, and make sure when closing a WAL-enabled database opened in read-only mode to reopen it in read-write mode so that the -wal and -shm files are removed (#6776) WFS driver: * fix potential nullptr dereference on dataset without layer (github #179) XLSX driver: * fix GDAL 2.1.2 regression regarding reading XLSX files whose sheet filenames are internally stored as 'absolute' (#6733) == Perl bindings == * fix setting the geometry of a MultiLineString and MultiPolygon (#6786) == Python bindings == * add outputType option to gdal.Rasterize() (#6710) * fix gdal.DEMProcessingOptions(zeroForFlat=True) (#6775) * swig/python/setup.py: fix build issues when CXX is defined in the environment (#6765) = GDAL/OGR 2.1.2 Release Notes = The 2.1.2 release is a bug fix release. == Port == * Fix deadlock at CPLWorkerThreadPool destruction (#6646) * CPLGetSymbol() on Windows: honour GDAL_FILENAME_IS_UTF8 setting to call LoadLibraryW() (#6650) * /vsicurl/: take CPL_VSIL_CURL_ALLOWED_EXTENSIONS into account even if GDAL_DISABLE_READDIR_ON_OPEN is defined (#6681) * Fix regression in PLScenes V1 raster random access. Add a CPL_VSIL_CURL_ALLOWED_FILENAME config option to solve that (fixes #6691) * /vsimem/: fix potential crash when closing -different- handles pointing to the same file from different threads (fixes #6683) == GDAL Core == * Nodata comparison: fix test when nodata is FLT_MIN or DBL_MIN (#6578) * External overview: fix incorrect overview building when generating from greater overview factors to lower ones, in compressed and single-band case (#6617) * Make sure that GDALSetCacheMax() initialize the raster block mutex (#6611) * Fix potential deadlock in multithreaded writing scenarios (#6661) * Make multi-threaded reading/writing when using GetLockedBlock()/MarkDirty()/ DropLock() lower level interfaces (#6665) * Fix multi-threading issues in read/write scenarios (fixes #6684) == Utilities == * gdal_translate & gdalwarp: lower the default value of GDAL_MAX_DATASET_POOL_SIZE to 100 on MacOSX (#6604) * gdalwarp: when -to RPC_DEM is specified, make -et default to 0 as documented (#6608) * gdalwarp: fix failure with cutline on a layer of touching polygons (#6694) * gdal_translate: when using -projwin with default nearest neighbour resampling, align on integer source pixels (#6610) * gdal2xyz: use %d formatting for Int32/UInt32 data types (#6644) == GDAL drivers == AIGRID driver: * fix 2.1.0 regression when reading statistics (.sta) file with only 3 values, and fix <2.1 behaviour to read them in LSB order (#6633) ENVI driver: * on closing, pad image file with trailing nul bytes if needed (#6662) GeoRaster driver: * fix for geotransform matrix with rotation terms (#6593) GPKG driver: * check if transaction COMMIT is successful (#6667) * fix crash on overview building on big overview factor (#6668) GTiff driver: * fix saving and loading band description (#6592) * avoid reading external metadata file that could be related to the target filename when using Create() or CreateCopy() (#6594) * do not generate erroneous ExtraSamples tag when translating from a RGB UInt16, without explicit PHOTOMETRIC=RGB (#6647) HDF4 driver: * Fixed erroneous type casting in HDF4Dataset::AnyTypeToDouble() breaking reading georeference data and some other metadata from HDF4-EOS datasets (#6618) HDF5 driver: * correct number of GCPs to avoid dummy trailing (0,0)->(0,0,0) and remove +180 offset applied to GCP longitude. Add instead a heuristics to determine if the product is crossing the antimeridian, and a HDF5_SHIFT_GCPX_BY_180 config option to be able to override the heuristics (#6666) ISCE driver: * make parsing of properties case insensitive (#6637) netCDF driver: * fix crash on datasets with 1D variable with 0 record (#6645) RMF driver: * fix wrong counter decrement that caused compressed RMF to be incorrectly decompressed (https://github.com/OSGeo/gdal/pull/153) * fix load/store inversion of cm and dm units in MTW files (https://github.com/OSGeo/gdal/pull/162) ROI_PAC driver: * fix computation of line offset for multi-band BIP files, and warn if detecting a wrong file produced by GDAL >= 2.0.0 (#6591) SRTMHGT driver: * accept filenames like NXXEYYY.SRTMGL1.hgt (#6614) VRT driver: * make CheckCompatibleForDatasetIO() return FALSE on VRTDerivedRasterBands (#6599) * fix performance issue when nodata set at band level and non-nearest resampling used (#6628) * VRTComplexSource: do temp computations on double to avoid precision issues when band data type is Int32/UInt32/CInt32/Float64/CFloat64 (#6642) == OGR utilities == * GDALVectorTranslate(): imply update mode if access mode not explicitly specified and hDstDS != NULL (#6612) == OGR drivers == Multi driver fix: * GPKG/SQLite/PG/FileGDB/MEM: properly set/reset field subtype with AlterFieldDefn() (fixes #6689) Carto driver: * rebranded from CartoDB to Carto. "CartoDB" is still a valid alias to Carto GPKG: * update schema to reflect CURRENT_TIMESTAMP -> 'now' changes (https://github.com/OSGeo/gdal/pull/155) * do not emit error when running ExecuteSQL() with a spatial filter on an empty layer (#6639) * list (non spatial) layers not registered in gpkg_contents (#6679) * robustified layer deletion and renaming (#6679) * implementation of DeleteField, AlterFieldDefn and ReorderFields (#6679) * GetExtent() to save computed extent in gpkg_contents if empty (#6679) * add 'RECOMPUTE EXTENT ON' special SQL (#6679) * update last modified date on tiles update (#6679) * check identifier unicity on layer creation (#6679) * add possibility to disable foreign key check by setting OGR_GPKG_FOREIGN_KEY_CHECK=NO config option (#6679) * add HasSpatialIndex(tblname,geomcolname) SQL function (#6679) GML driver: * remove hack for CityGML regarding forcing srsDimension to 3 if not specified. Instead add a GML_SRS_DIMENSION_IF_MISSING config option that can be set to 3 if needed (#6597) * consider srsName with URL like 'http://www.opengis.net/def/crs/EPSG/0/' as following EPSG axis order. Add SWAP_COORDINATES=YES/NO/AUTO (and GML_SWAP_COORDINATES configuration option) to override all auto guessing (#6678) ILI1 driver: * fix crash in OGRILI1Layer::JoinSurfaceLayer() when the multicurve of the feature of the poSurfaceLineLayer layer is empty (fixes #6688) MSSQLSpatial driver: * Fix select layer to recognize geometry column with sqlncli (#6641) NAS driver: * Fix to handle that 'anlass' is optional OCI driver: * 3 new options: MULTI_LOAD, MULTI_LOAD_COUNT, FIRST_ID (#6606) OSM driver: * fix issue garbage content when laundering tag name with column (https://github.com/OSGeo/gdal/pull/161) PG driver: * fix errors caused by missing geometry_columns/spatial_ref_sys tables in non PostGIS databases, that prevent reading more than 500 features (http://hub.qgis.org/issues/10904) PLScenes driver: * add HTTP retry logic (#6655) * V0 API: workaround limitations on filtering on image_statistics.image_quality (#6657) S57 driver: * fix GDAL 2.1.1 regression where open options are ignored if OGR_S57_OPTIONS config option is not defined (#6609) Shapefile driver: * change REPACK implementation on Windows to be robust to remaining file descriptors opened on the .shp/.shx/.dbf (#6672) SQLite driver: * do not emit error when running ExecuteSQL() with a spatial filter on an empty layer (#6639) * Spatialite: avoid crash when creating layer with geom_type = wkbCurve (#6660) * don't touch nSoftTransactionLevel in DoTransactionCommand() in case of error, since it can cause later CPLAssert() to trigger (#6667) XLSX driver: * only list worksheets (and no charts) as layers (#6680) = GDAL/OGR 2.1.1 Release Notes = The 2.1.1 release is a bug fix release. == Build == * Unix: Fix compilation in std=c++11 mode with Ubuntu 16.04 (#6490) * Unix: fix libqhull include path when it is /usr/local/include/libqhull (#6522) * Fix build failure on Solaris 11 on MRF driver (#6559) == Port == * Export VSICreateCachedFile() as CPL_DLL so as to enable building JP2KAK as a plugin (https://github.com/OSGeo/gdal/pull/126) * Unix filesystem: make error message about failed open to report the filename (#6545) * Fix VSIErrorV() buffer reallocation (#6545) == GDAL algorithms == * Fix GDAL_CG_Create() to call GDALContourGenerator::Init() (#6491) * GDALContourGenerate(): handle the case where the nodata value is NaN (#6519) * GDALGridCreate(): fix hang in multi-threaded case when pfnProgress is NULL or GDALDummyProgress (#6552) * GDAL contour: fix incorrect oriented contour lines in some rare cases (#6563) == GDAL Core == * DIMAP/Pleiades metadata reader: take into tiling to properly shift RPC (#6293) == Utilities == * gdal_polygonize.py: explicitly set output layer geometry type to be polygon (#6530) * gdalwarp: do not densify cutlines by default when CUTLINE_BLEND_DIST is used (#6507) == GDAL drivers == ASRP driver: * fix georeferencing of polar arc zone images (#6560) ECW driver: * Add configuration option ECW_ALWAYS_UPWARD=TRUE/FALSE to work around issues with "Downward" oriented images (#6516) GTiff driver: * fix RasterIO() reported when downsampling a RGBA JPEG compressed TIFF file (#6943) * fix suboptimal behaviour before regarding flushing of working buffer (#6580) * fix race between empty block filling logic and background compression threads when using Create() interface and NUM_THREADS creation option (#6582) * Internal libtiff: fix 1.11 regression that prevents from reading one-strip files that have no StripByteCounts tag (#6490) ISCE driver: * fix computation of line offset for multi-band BIP files, and warn if detecting a wrong file produced by GDAL 2.1.0 (#6556) JP2KAK driver: * add support for Kakadu v7.8 (https://github.com/OSGeo/gdal/pull/138) MRF driver: * Fix plugin building (#6498) * Fix crashes on MRF using JPEG or PNG crashes due to dandling pointer (#6554) * Fix crashes when NoData is set after create (#6555) PostgisRaster driver: * fix potential crash when one tile has a lower number of bands than the max of the table (fixes #6267) SAFE driver: * fix handling of SLC Products by providing access to measurements as subdatasets (#6514) VRT driver: * avoid useless floating point values in SrcRect / DstRect (#6568) * VRT warp: fix issue with partial blocks at the right/bottom and dest nodata values that are different per band (#6581) == OGR core == * Fix assertion in OGRGeometryFactory::GetCurveParmeters() on degnerated CIRCULARSTRING (#6553) * Fix crash/corrupted values when running importFromWkb() on POLYGON M/POLYGON ZM geometries (#6562) == OGRSpatialReference == * importFromESRI(): fix import of multi line MERCATOR SRS (#6523) * morphToESRI(): correctly compute standard_parallel_1 of Mercator(2SP) projection from scale factor of Mercator(1SP) (#6456, #4861) == OGR utilities == * ogr2ogr: keep -append and -overwrite when -update follows (backport trunk r34341) * ogr2ogr: do not return error on ogr2ogr --utility_version (backport of trunk r34340) * ogr2ogr: fix heuristics to detect likely absence of '-f' (#6561) == OGR drivers == CartoDB driver: * fix CartoDB'fication() by removing manual creation of the_geom_webmercator mercator, and also attach created sequence to table.cartodb_id (#6565) GeoJSON driver: * fix field type detection when first value of a field is null (#6517) * ESRI Json reader: do not set field width of 2147483647 (#6529) * fix wrong behaviour when there's a 'id' at Feature level and 'id' or 'ID' field in properties (#6538) * in case top level id is a negative integer, put the value in a 'id' attribute (#6538) LIBKML driver: * fix crash when reading or elements (#6518) MITAB driver: * limit (width, precision) of numeric fields on creation to (20,16) for compatibility with MapInfo (#6392) MSSQLSpatial driver: * Fix MSSQL bulk insert with table names containing spaces (#6527) NAS driver: * support multiple 'anlass' in updates (cherry picked from r34461) OpenFileGDB driver: * transcode UTF-16 strings found in column names, alias, etc... to UTF-8 (instead of using only their ASCII byte) (#6544) * do not error out on geometries that have a declared M array but that is missing (#6528) * do not emit an error on a empty table whose declaration has M settings (#6564) PG driver: * fix insertion of binary/bytea content in non-copy mode (#6566) ROI_PAC driver: * add support for reading/writing .flg files (#6504) S57 driver: * Fix ogr2ogr -S57 (#6549) VFK driver: * recreate DB automatically when it's outdated (#6511) VRT driver: * add support for 'm' attribute in PointFromColumns mode (#6505) SOSI driver: * make registration of driver work again (2.1 regression) (#6500) * Update to latest version of ​https://github.com/kartverket/gdal. Make SOSI driver support more geometry types (including curved geometries) plus provides some improvements on attribute-type mapping. Fix some memory errors/leaks. Disable by default non working creation code (#6503) WFS driver: * invalidate underlying layer when SetIgnoredFields() is called (QGIS #15112) * don't crash on empty declaration (#6586) == Perl bindings == * Fix GetGeomFieldIndex() * Fix Dataset.Warp (#6521) = GDAL/OGR 2.1.0 Release Notes = == In a nutshell... == * New GDAL/raster drivers: - CALS: read/write driver for CALS Type I rasters - DB2 driver: read/write support for DB2 database (Windows only) - ISCE: read/write driver (#5991) - MRF: read/write driver (#6342) - SAFE: read driver for ESA SENTINEL-1 SAR products (#6054) - SENTINEL2: read driver for ESA SENTINEL-2 L1B/LC1/L2A products - WMTS: read driver for OGC WMTS services * New OGR/vector drivers: - AmigoCloud: read/write support for AmigoCloud mapping platform - DB2 driver: read/write support for DB2 database (Windows only) - MongoDB: read/write driver - netCDF: read/write driver - VDV: read/write VDV-451/VDV-452 driver, with specialization for the Austrian official open government street graph format * Significantly improved drivers: - CSV: new options, editing capabilities of existing file - ElasticSearch: read support and support writing any geometry type - GeoJSON: editing capabilities of existing file, "native data" (RFC 60) support - MBTiles: add raster write support. fixes in open support - PDF: add PDFium library as a possible back-end. - PLScenes: add support for V1 API - VRT: on-the-fly pan-sharpening - GTiff: multi-threaded compression for some compression methods * Port library: add /vsis3/, /vsis3_streaming/, /vsicrypt/ virtual file systems * RFC 26: Add hash-set band block cache implementation for very larger rasters (WMS, WMTS, ...) http://trac.osgeo.org/gdal/wiki/rfc26_blockcache * RFC 48: Geographical networks support (GNM) https://trac.osgeo.org/gdal/wiki/rfc48_geographical_networks_support * RFC 58: Add DeleteNoDataValue(): https://trac.osgeo.org/gdal/wiki/rfc58_removing_dataset_nodata_value * RFC 59.1: Make GDAL/OGR utilities available as library functions: https://trac.osgeo.org/gdal/wiki/rfc59.1_utilities_as_a_library For gdalinfo, gdal_translate, gdalwarp, ogr2ogr, gdaldem, nearblack, gdalgrid, gdal_rasterize, gdalbuildvrt Available in C, Python, Perl and Java bindings. * RFC 60: Improved round-tripping in OGR https://trac.osgeo.org/gdal/wiki/rfc60_improved_roundtripping_in_ogr Implemented in GeoJSON driver * RFC 61: Support for measured geometries. https://trac.osgeo.org/gdal/wiki/rfc61_support_for_measured_geometries Implemented in Shapefile, PostgreSQL/PostGIS, PGDump, MEM, SQLite, GeoPackage, FileGDB, OpenFileGDB, CSV, VRT * Upgrade to EPSG database v8.8 * General sanitization pass to clean-up code, fix a lot of compiler warnings, as well as issues pointed by static code analyzers, such as Coverity Scan (credits to Kurt Schwehr for tackling a large part of them) or CLang Static Analyzer. * Fixes in a number of drivers to be more robust against corrupted files (most found with American Fuzzy Lop): RIK, INGR, Northwood, HF2, CEOS, GTiff, GXF, BMP, NITF, HFA, VRT, FIT, CEOS2, NWT_GRD/NWT_GRC, MITAB, RPFTOC, DBF/Shape, XYZ, VFK, DXF, NAS, GSAG, GS7BG, OpenFileGDB, RMF, AIGRID, OpenAIR, EHDR, ISO8211, FAST, USGSDEM, DGN, SGI, OpenJPEG, PCRaster, BSB, ADRG, SRP, JPEG, Leveller, VICAR, PCIDSK, XPM as well as in portability library (CPL), algorithms and raster core. * Driver removed: - Google Maps Engine (GME) (#6261) == New installed files == * gdal_pansharpen.py * data/vdv452.xml * data/vdv452.xsd * data/netcdf_config.xsd * /path/where/bash-completion/scripts/are/installed/gdal-bash-completion.sh == Backward compatibility issues == See MIGRATION_GUIDE.TXT == GDAL/OGR 2.1.0 - General Changes == Build(Unix): * Allow plugin for HDF4, HDF5, GeoRaster, FileGDB, OCI and PG * Add a --with-gnm option to enable GNM * Add --enable-lto switch to turn on link time optimization (GCC >= 5) * Add --with-spatialite=dlopen --with-spatialite-soname=libspatialite.so[.X] syntax to allow linking against spatialite through dlopen() mechanism (#6386) * Add a lot of warning options when available in the compiler. * Change detection test of podofo to work with podofo 0.9.3 * Fix gdalserver compilation with recent GNU libc (such as in Arch Linux) (#6073) * Add pkg-config support for libkml fork (#6077) * Update config.sub and config.guess to their latest upstream versions so as to be able to detect new architectures (android, ppcle64, etc..) (#6101) * Fix python package installation in custom prefixes (#4563) * configure: check that CXX is really a working compiler (#4436) * Build support for Kakadu 7.7 * thinplatespline.cpp: avoid using optimized version of VizGeorefSpline2DBase_func4() with ICC versions that fail on it (#6350) * Add bash completion for GDAL/OGR utilities and scripts (#6381) * Add support for ODBC autodetection with mingw64 (#6000) * Remove macos 9 and older support (#6133). * Remove dist_docs, burnpath and pszUpdableINST_DATA in gcore/gdaldrivermanager.cpp as they are have not been used for a while (#6140). * Removed --without-ogr configure flag (#6117). Always build with OGR. Build(Windows): * Add support for Visual Studio 2015 (a.k.a MSVC_VER=1900 or VC 14) * Add KAKFLAGS to nmake.opt with KDU_{MAJOR,MINOR,PATCH}_VERSION define-s. * nmake.opt: make it less error prone to define SETARGV when paths include spaces (#6032) * nmake.opt: add CL.EXE compiler option /FC to display full path of source code file in diagnostics * Remove WinCE support (#6133) * nmake.opt: add /I flag to the INSTALL command so that xcopy will be smart enough to create a directory when copying files and avoid prompting for user input (https://github.com/OSGeo/gdal/pull/95) * Enable mssql spatial blugin build, use BCP as the default option for the sqlncli enabled builds * Add support to find MrSID 9.5 dll * Fix Windows build with recent MySQL versions and Visual Studio 2015 (#6457) Build(all): * Compatibility with C++11 and C++14 * Optional crypto++ dependency for /vsicrypt/ support * Optional mongocxx dependency for MongoDB support Other: * Disable copy constructor and assignment operators in classes OGRFieldDefn, OGRGeomFieldDefn, OGRFeature, GDALMultiDomainMetadata, GDALDefaultOverviews, GDALOpenInfoGDALDataset, GDALRasterBlock, GDALRasterBand and GDALDriver (#6100) == GDAL 2.1.0 - Overview of Changes == Port: * Add /vsicrypt/ virtual file system for reading/creating/update encrypted files on the fly, with random access capabilities * Add /vsis3/ and /vsis3_streaming/ virtual file systems to read/write objects from AWS S3 buckets * /vsizip/: avoid returning the previous file list of an already opened .zip if it has changed afterwards (#6005) * /vsizip/: use CP437 on Windows when ICONV support is available (#6410) * /vsimem/: implement append mode (#6049) * /vsistdin/: fix caching of first 1MB / VRT driver: read XML content from /vsistdin/ in a streaming compatible way (#6061) * /vsistdout/: flush when closing the handle (#6149) * Add VSIReadDirEx() with a limit on the number of files before giving up and corresponding VSIFilesystemHandler::ReadDirEx() virtual function * cpl_vsil_cache: rework to be able to work on very large files on 32bit systems * Add CPLThreadLocaleC class to use thread-specific locale settings (when available) * VSIWin32Handle::Flush(): add VSI_FLUSH config option that can be set to TRUE to force FlushFileBuffers(). (hack related to #5556) * Fix VSIL append mode in Windows (#6065) * Make CPLCreateMutexEx() support CPL_MUTEX_REGULAR; and fix CPL_MUTEX_ADAPTIVE to avoid continuing in code path for creation of recursive mutex * Add CPLWorkerThreadPool API * Add CPLGetThreadLocalConfigOption() * Fix CPL_LSBINT16PTR() and CPL_LSBINT32PTR() to work with non-byte pointer (#6090) * CPLRecodeStub(): add special case for CP437 -> UTF-8 when input is only printable ASCII * CPLHTTPFetch(): add LOW_SPEED_TIME and LOW_SPEED_LIMIT options * CPLGetValueType(): improve heuristics to avoid detecting some WKB strings as real numbers (#6128) * Add CPL_SHA256* and CPL_HMAC_SHA256 functions * Make CPLVirtualMemFileMapNew() work on all systems having mmap(), and thus GeoTIFF GTIFF_VIRTUAL_MEM_IO optimization too * Add VSI_MALLOC_VERBOSE() and similar macros to output an error message in case of failed alloc * CPLParseXMLString(): limit depth of elements to 10000 * Win32 file management: handle files whose total file path length is greater than 255 characters in Open() and Stat() * Make CPLFormFilename(absolute_path, "..", NULL) truncate when possible * Add VSIGetDiskFreeSpace() * Implement CPLIsInf() for Solaris 11 * Add a CPL_FINAL macro that expands to C++11 final keyword when C++11 is enabled, and use it in a few places * Avoid CPLEscapeString (CPLES_URL) encoding some characters unnecessarily (#5526) * Add CPLTestBool(), CPLTestBoolean(), CPLFetchBool() * Fix CPLGetValueType() to recognize D1 as a string and not a real number (#6305) * CPLFormFilename(): use '/' directory separator on Windows on /vsicurl_streaming/ files (#6310) * Add hack enabled by -DDEBUG_BOOL to detect implicit int->bool conversions that dislike MSVC (#6325) * Add hack to '#define NULL nullptr' when NULL_AS_NULLPTR is defined. Must be used together with -std=c++11 -Wzero-as-null-pointer-constant with GCC to detect misuses of NULL/nullptr (#6327) * Add VSIError mechanism to store errors related to filesystem calls, and use it for /vsis3/. Add new CPLE_ error numbers. (https://github.com/OSGeo/gdal/pull/98) * Fix CPLsscanf() to be conformant regarding how space/tab/... separators are handled, so as to fix OGR GMT to be able to read coordinates separated by tabulations (#6453) Core: * Change default value of GDAL_CACHEMAX to 5% of usable physical RAM * Allow open options name to be prefixed by @ to be silently ignored when not existing in driver options (RFC 60) * Honour NBITS metadata item when doing RasterIO() with non-nearest resampling (#6024) * GDALClientServer: pass open options to INSTR_Open call * Improve performance of GDALCopyWords() float->byte/int16/uint16 by using SSE2 * Decorate prototypes of RasterIO() related operations with CPL_WARN_UNUSED_RESULT * Avoid deadlock when writing 2 datasets in 2 threads (#6163) * Remove obsolete symbols __pure_virtual(), GDALCreateProjDef(), GDALReprojectToLongLat(), GDALReprojectFromLongLat() and GDALDestroyProjDef() * Remove obsolete non-template-based implementation of GDALCopyWords(). We don't support anymore such compilers * GDALJP2Box::ReadBoxData(): do not abort if memory allocation fails * Add GDALAdjustValueToDataType() in C API, and use it in GDALTranslate() and GDALWarp() * GDALDefaultOverviews::CreateMaskBand(): fix logic error related to writing per-band mask * Block cache: fix excessive memory consumption when dealing with datasets with different block sizes (#6226) * EXIFExtractMetadata(): fix potential 1-byte write buffer stack overflow * MDReader: do no attempt reading side-car files on /vsisubfile/ (#6241) * GDALCheckBandCount(): allow by default a maximum number of bands to 65536 * GDALOpenInfo: add StealSiblingFiles() and AreSiblingFilesLoaded() methods * GDALOpenInfo::GetSiblingFiles(): give up after GDAL_READDIR_LIMIT_ON_OPEN (default=1000) files in the directory * GDALDefaultOverviews: add TransferSiblingFiles() method * GDALDriver::DefaultCreateCopy(): fix logic related to propagation of NBITS / PIXELTYPE metadata items as creation options * Reorder driver registration a bit so that formats with efficient identification are probed first * Add GDALIdentifyEnum (GDAL_IDENTIFY_UNKNOWN, GDAL_IDENTIFY_FALSE or GDAL_IDENTIFY_TRUE) for return values of Identify() * GDALLoadRPCFile(): load .rpc files from Ikonos products that have extra end-of-line character (#6341) * Export GDALRegenerateOverviewsMultiBand() symbol, but mostly for plugins (#6342) * Add GDAL_NO_AUTOLOAD to compile out the body of AutoLoadDriver (#6380) * Avoid ComputeStatistics(), GetHistogram() and ComputeRasterMinMax() to use only the first column of blocks in approximation mode for a raster whose shape of blocks is a square (#6378) * Add GDALGetDataTypeSizeBits() and GDALGetDataTypeSizeBytes(). * GDALReadOziMapFile(): fix reading Ozi external files from virtual file systems (https://github.com/OSGeo/gdal/pull/114) * Add CPLSetCurrentErrorHandlerCatchDebug() to enable custom error handlers not to intercept debug messages Algorithms: * RPC transformer: provide SSE2 accelerated transformer * RPC: fix off-by-half pixel computation of (pixel, line), and in bilinear and bicubic RPC DEM interpolation; fix off-by-one pixel registration for Pleiades RPC (#5993) * RPC: make RPCInverseTransformPoint() check convergence has been reached (#6162) * RPC DEM: optimize when DEM is in NAD83 or any other geodetic CS that transform as a no-op to WGS 84 * RPC DEM: take into account vertical datum of the DEM when present to convert DEM elevations to ellipsoidal heights (#6084) * RPC DEM: do fallback cubic -> bilinear and bilinear -> near on DEM edges * RPC DEM: improve inverse transformer to validate error threshold and improve convergence (#6162, #6377) * RPC: fix issues with dateline (#6472) * TPS solver: discard duplicated GCP to avoid avoidable error, warning when 2 GCPs have same (pixel,line) but different (X,Y) or the reverse * Warper: rework multithreaded computations to use a thread pool rather than forking threads each time * Warper: avoid really excessive processing time for some warping with target areas completely off the source raster (especially when involving RPC) (#6182) * Warper: CreateKernelMask(): fix potential 32 bit integer overflow when using warp memory value > 2GB (#6448) * gdal_grid: add linear interpolation algorithm * gdal_grid: add invdistnn algorithm, variation on the existing inverse distance weighting algorithm with quadtree to search for points only in the neighborhood (#6038) * gdal_grid: fix crash in optimized mode with GCC 4.4 on 64bit (#5987) * gdal_grid: compile gdalgrid AVX optimization for Windows when supported by compiler * Add GDALTriangulationXXXX() API through libqhull * Sieve filter: fix crash on nodata polygons (#6096) * Sieve filter: improvement to walk through the biggest neighbour chain until we find a polygon larger than the threshold (#6296) * GDALFPolygonize(): factor implementation with integer case * GDALComputeMedianCutPCT(): fix to make it work with rasters with more than 2 billion pixels (#6146) * Overview: Make average and gauss methods aware of transparent color table entries (#6371) Utilities: * gdalinfo: workaround bug in proj <= 4.9.1 on datasets with a SRS with a vertical shift grid (#6083) * gdal_translate: detect more reliably if specified bands are not in default order * gdal_translate: fix -a_nodata with negative values on rasters that have PIXELTYPE=SIGNEDBYTE; fix preserving PIXELTYPE=SIGNEDBYTE with VRT * gdal_translate: fix random behaviour when -scale is used without source bounds (#6455) * gdal_rasterize: do on-the-fly reprojection of input vector onto output raster * gdal_rasterize: rasterize: always create output with 1/2 cell buffer of input geometry envelope (#6058) * gdal_rasterize: add the -dialect option * gdal_rasterize: accept NaN as a valid value for -init / -burn (#6467) * gdalwarp: add -doo option to specify open options of (existing) output dataset * gdalwarp: if RPC_DEM warping option is specified, use exact transformer by default (#5993) * gdalwarp: make it honour nodata value of existing dataset (if -dstNoData isn't explicitly specified) * gdalwarp: do not propagate STATISTICS_ of second or following source datasets * gdalwarp: do not emit warning when using -cutline with a SRS and the source raster has RPC or GEOLOCATION * gdalwarp: check that the cutline is valid after transformation/reprojection * gdalwarp: better deal when dealing with a mix of RGB and RGBA datasets as input * gdalwarp: fix -srcnodata to not put garbage values as target dstnodata (#6315) * gdalwarp: densify cutline to avoid invalid geometry after reprojection to source raster, especially in the RPC case (#6375) * gdalserver: add a -nofork mode (Unix only for now), so that multiple clients can connect to the same dataset. Useful for safe 'concurrent' updates * add gdal_pansharpen.py script * gdal2tiles.py: fix error on a raster with less than 3 bands that resulted in a 'IndexError: list index out of range' * gdal2tiles.py: Change EPSG:3785 / EPSG:900913 into EPSG:3857 (#5622) * gdal2tiles.py: add Leaflet template (https://github.com/OSGeo/gdal/pull/71) * gdal2tiles.py: add -q switch for quiet mode * gdaldem: correctly deal with NaN as nodata value (#6066) * gdaldem color-relief: deal with the case of repeated entries with the same value and the input raster has pixels that match that value exactly (#6422) * gdaladdo: emit error message if passed an invalid dataset name (#6240) * gdaladdo: do not silence warnings when opening in update mode, if the open is successful * gdalbuildvrt: fix potential crash when using -b switch (#6095) * gdalbuildvrt: accept nan as value for -srcnodata and -vrtnodata * gdalbuildvrt: return non zero return code if the flush of the VRT to disk failed * gdal_merge.py: takes again into account -n flag (#6067) * gdalbuildvrt / gdal_translate / VRT: use floating point values for source and destination offsets and sizes (#6127) * gdalmove.py: fix to run with GDAL 2.0 gdal.InvGeoTransform() signature * gdal_calc.py: Add * from gdalnumeric to gdal_calc.py eval namespace again, to fix 2.0 regression that made for example 'log10(A)' to no longer work (https://github.com/OSGeo/gdal/pull/121) Python samples: * gdalpythonserver.py: update to protocol 3 AAIGRID: * when writing with floating-point values, ensure at least one value has a decimal point to avoid issues with some readers (#6060) BMP driver: * BMP: avoid too big color table allocation in case of wrong iClrUsed value BLX driver: * converted to support VirtualIO ECRG driver: * change subdataset definition to make sure that they only consist of frames of same scale (#6043) * fix base34 decoding and Windows filename handling (#6271) ECW driver: * use wide char Open API on Windows when GDAL_FILENAME_IS_UTF8=YES (https://github.com/OSGeo/gdal/pull/70) * fix reading of ECW in /vsi file systems (with SDK >= 4) (#6482) GeoRaster driver: * fix deflate read error depending on endianness (#6252) GIF driver: * libgif: partial resync with giflib master (but keep ABI of 4.1.6) * fix crash when CreateCopy a source with one color in the color table GMT driver: * validate raster dimensions on opening, and acquire mutex in dataset destructor GPKG driver: * write support: fix various issues in update scenarios when interacting with the GDAL block cache that could result in lost/corrupted band data to be written in tiles (#6309) * write support: fix potential use of freed sqlite temporary DB handle when generating overviews with partial tiles (#6335) * write support: fix potential crash in scenarios when block cache is full (#6365) * write support: fix inversion of row/column in one of the SQL request involved in partial tiles management (#6339) * fix generation of files with 1-band paletted input dataset. Also improve the logic to retrieve color palette when forcing BAND_COUNT=1 on opening (#6450) GTiff driver: * add NUM_THREADS creation and open options to enable multi-threaded compression * fix GTiffDataset::IsBlockAvailable() wrong behaviour when compiling against internal libtiff, when a BigTIFF file has a TileByteCounts with LONG/4-byte counts and not LONG8 (#6001) * Correctly take into account overridden linear units for a geotiff with a EPSG PCS code (#6210, #4954) * make VirtualMemIO() work with non native endianness * improve VirtualMemIO() performance in tiled Contig read to single band copy * improve single band tiled VirtualMemIO reading * improve DirectIO() to work on tiled uncompressed geotiff, for non-resampling and nearest resampling cases. Also improve performance of resampling cases on un-tiled files * fix DirectIO() mode with complex types and inverted endianness (#6198) * optimize writing of 12-bit values * implement lazy loading of .aux.xml and .tfw/.wld * Internal libtiff: update to CVS HEAD post libtiff 4.0.6 * make SetColorInterpretation(GCI_AlphaBand) work on a 5 band or more GeoTIFF (#6102) * automatically set PHOTOMETRIC=RGB if manually assigning color interpretation Red,Green,Blue to band 1,2,3 before directory crystallization (#6272) * add GEOTIFF_KEYS_FLAVOR=ESRI_PE creation option to write EPSG:3857 in a ESRI compatible way (#5924) * call XTIFFInitialize() in LibgeotiffOneTimeInit() as the former isn't thread-safe, so better call it from the later which is thread-safe (#6163) * fix reading and writing angular units different from degree (namely arc-second, arc-minute, grad, gon and radian) (#6171) * do not use VirtualMemIO optimization on compressed /vsimem/ files (#6195) * correctly set GTRasterTypeGeoKey=RasterPixelIsPoint if AREA_OR_POINT=Point but there is no SRS set (#6225) * improve a bit error detection when writing * check free space before writing (only for big, non sparse, uncompressed) * do not read large 'one row' JBIG compressed files with the scanline API (#6264) * Fix SetMetadata() to properly clear existing PAM metadata (complement to #5807) * prevent potential out of bounds read/write to TIFFTAG_EXTRASAMPLES (#6282) * do not use first directory as potential mask, to avoid assertion in GTiffDataset::SetDirectory() (#6287) * reject files with strips/tiles/scanlines bigger than 2 GB to avoid 32 bit integer overflow. Also in case of files with Contig PlanarConfiguration do not make reading one block for band 2 OK when reading for band 1 issued an error (#6288) * GTIFFWriteDirectory(): avoid memory leak of codec related memory (#2055) * Make ALPHA=NO in CreateCopy() cancel alpha color interpretation even if present in source raster * fix problem with implicit overviews of JPEG-compressed files (#6308) * fix compilation problem with internal libtiff if DEFER_STRILE_LOAD isn't defined (which is not the default configuration) (https://github.com/OSGeo/gdal/pull/90) * use more appropriate error message when 4GB threshold is reached with external overviews, and try to make doc of BIGTIFF=IF_NEEDED/IF_SAFER clearer (#6353) HDF4 driver: * Skip quotation mark when parsing HDF-EOS metadata. HDF5 driver: * add Komsat Mission ID to possible value for HDF5 SAR product (https://github.com/OSGeo/gdal/pull/103) HF2 driver: * fix reading side of the driver to work on architectures where char is unsigned, like PPC or ARM (#6082) HFA driver: * when reading projection, preserve EPSG code if AutoIdentifyEPSG() identified the SRS, even if a PE string is present (#6079) * keep TOWGS84 even when using ESRI PE string (#6158) * fix crashes on corrupted files (#6208, #6286) INGR driver: * check that RLE decoding produced the expected number of bytes and error out otherwise; test that 'random' line seeking actually works JPEGLS driver: * fix build error (#6430) JP2ECW driver: * honour psExtraArg->eResampleAlg when upsampling (#6022) JP2KAK driver: * honour psExtraArg->eResampleAlg when upsampling (#6022) * try rounded dimensions to decide if the data is being requested exactly at a sub-resolution * support v7.7 on Unix (additional changes potentially needed on Windows) JP2OpenJPEG driver: * Safer multi-threaded use JPEG driver: * add USE_INTERNAL_OVERVIEWS open option (default to TRUE as in recent versions, can be set to FALSE to hide internal overviews KMLSuperOverlay driver: * fix 2.0 regression with some RasterIO() requests involving resampling (#6311) * recognize datasets that have a intermediate that forms a structure (#6343) LCP driver: * fix wrong use of endianness macros to fix behaviour on big-endian hots Leveller driver: * update to v9 read support (#5632,) MBTiles driver: * add raster write support * fix so that datasets whose lowest min zoom level is 16 or above are recognized * be robust to invalid 'bounds' at dataset opening (#6458) MEM driver: * avoid 32 bit overflows NetCDF driver: * add support for reading NC4 unsigned short attributes and variables (#6337) * read correctly nodata values in [128,255] range for (unsigned) Byte data type (#6175) * implement Get/SetUnitType() using the standard units attribute (https://github.com/OSGeo/gdal/pull/96) * optimize IReadBlock() and CheckData() handling of partial blocks in the x axis by re-using the GDAL block buffer instead of allocating a new temporary buffer for each block (#5950) * full read/write support for new NetCDF4 types NC_UBYTE, NC_USHORT, NC_UINT and NC_STRING for variables (except for NC_STRING) and attributes (https://github.com/OSGeo/gdal/pull/99) * add support for the geostationary projection (#6030) * fix one byte heap write overflow in NCDFTokenizeArray() (#6231) * fix potential buffer overflows with uses of nc_inq_varname(), nc_inq_attname() and nc_get_att_text() (#6227) * validate that gridmapping:GeoTransform has 6 values (#6244) * fix wrong use of deallocator when writing a GEOLOCATION array, and other issues * limit number of bands reported to 32768 by default * validate raster dimensions * validate content of NC_GLOBAL#GDAL variable (#6245) NGSGEOID driver: * make it work on > 2GB file NITF driver: * data/nitf_spec.xml: Add CSCCGA, MENSRB, SENSRB, STREOB, ENGRDA, EXPLTB and PATCHB TREs (https://github.com/OSGeo/gdal/pull/81, #6285, https://github.com/OSGeo/gdal/pull/86) * fix parser to properly deal with variable length items not in first nesting level Northwoord driver: * fix computation of intermediate color table values on non-Intel platforms (#6091) NTv2 driver: * support reading/writing/appending to files with big-endian order (cf https://github.com/OSGeo/proj.4/issues/345) OpenJPEG driver: * do not expose block dimensions larger than dataset dimensions to avoid wasting memory (#6233) PCIDSK driver: * Remove the old driver (#6172) PDF driver: * add PDFium library as a possible back-end. Initial support contributed by Klokan Technologies GmbH (http://www.maptiler.com/) * workaround a bug of PoDoFo 0.9.0 by avoiding loading of vector content in raster-only mode (doesn't prevent the crash if reading the vector layers) (#6069) * make OGC BP registration work with media box where bottom_y is negative and top_y = 0 (in non rotated case) * make OGC BP registration work with media box where min_x != 0 (in non-rotated case) * correctly take into account non-meter linear units with OGC BP encoding, and add support for US FOOT (#6292) PDS driver: * change default values of PDS_SampleProjOffset_Shift and PDS_LineProjOffset_Shift to 0.5 (#5941) * fix nodata value for UInt16 to be 0 (#6064) * accept 'ODL_VERSION_ID = ODL3' in header (#6279) PGChip driver: * driver removed from sources PLScenes driver: * PLScenes V0: avoid opening raster to generate dummy .aux.xml PNG driver: * Support writing 1, 2 or 4 bit single band * Add NBITS creation option * fix XML of creation option list * Update internal libpng to 1.2.56 PostgisRaster driver: * avoid Identify() to recognize OGR PostgreSQL connection strings with schemas option and cause loud 'QuietDelete' (#6034) Raw drivers: * better support for direct read of more than 2GB in single gulp (untested though) RIK driver: * fix Identify() method to recognize again non-RIK3 RIK datasets (#6078) RMF driver: * implement GetNoDataValue() * add read/write access to new RMF format for files larger than 4 Gb (version=0x201) (https://github.com/OSGeo/gdal/pull/11) ROIPAC driver: * Support offset and scale band (#6189) VICAR driver: * change PDS_SampleProjOffset_Shift and PDS_LineProjOffset_Shift default values to 0.5 (#5941) * fix loss of precision in scale and offset VRT driver: * expose implicit 'virtual' overviews for VRT whose bands are made of a single SimpleSource/ComplexSource * gdalvrt.xsd: Add capitalized versions of true and false (#6014) * GetSingleSimpleSource(): check there's a single source (#6025) * honour VRTRasterBand NBITS metadata with SimpleSource and ComplexSource * properly take into account nodata value declared at VRT band level when doing resampling with non-nearest * honour relativeToVRT when using AddBand() to add a VRTRawRasterBand (https://github.com/OSGeo/gdal/pull/67) * VRT warp: fix crash with implicit overviews and destination alpha band (#6081) * make GetDefaultHistogram() on a sourced raster band save the result in the VRT (#6088) * serialize NODATA and NoDataValue items with %.16g, e.g. so as to be able to hold large int32 nodata values (#6151) * VRTSourcedRasterBand: make ComputeRasterMinMax() and ComputeStatistics() forward bApproxOK to overview band (useful for implicit overviews) * make CreateCopy() preserve NBITS metadata item * avoid loading sibling file list if not available * VRT raw: don't truncate last figure of ImageOffset if there are left space padding (#6290) * VRTWarpedDataset::SetMetadataItem(): fix crash when calling with name=SrcOvrLevel and value=NULL (#6397) * Warped VRT: fix deadlock in situation where warped VRT datasets are read in multiple threads and the block cache reaches saturation. Also add a GDAL_ENABLE_READ_WRITE_MUTEX config option that can be set to NO to disable the read/write mutex mechanism in cases where it would deadlock (#6400) WMS driver: * add a IIP (Internet Imaging Protocol) minidriver * limit number of zoom levels for ArcGIS MapServer JSon (#6186) * determine a resolution that will not result in a number that is larger than the maximum size of an integer. Any value that exceeds the maximum size of an integer will raise an invalid dataset dimensions error. (https://github.com/OSGeo/gdal/pull/89) XYZ driver: * be more robust to not exactly equal X and Y spacing (#6461) == OGR 2.1.0 - Overview of Changes == Core: * Add OGREditableLayer class to add editing capabilities to drivers with none or limited editing capabilities * OGRGeometry: add DelaunayTriangulation() method (GEOS >= 3.4) * OGRGeometry and derived classes: implement copy constructor and assignment operator (#5990) * OGRGeometry: Fix result of Equals on POINT EMPTY with POINT(0 0) * OGRFeature SetField(): more type conversions allowed, particularly with array types * OGRFeature::SetGeometry()/SetGeometryDirectly(): make it work when passed geometry is the currently installed geometry (#6312) * OGR SQL: do not silently skip NULL values in the first records when evaluating a SELECT DISTINCT (#6020) * OGR SQL: correctly sort NULL values in first positions (#6155) * OGR SQL: fix CAST(x AS bigint) to return an evaluated int64 node, and not int32 (#6479) * OGR SQL: handle 'fid' as Integer64 in where clause, and allow CAST(fid AS bigint) in selected columns (#6484) * Add OGRUpdateFieldType() * Decorate a few functions/methods of the OGR layer API with CPL_WARN_UNUSED_RESULT * WKT export: use 15 significant figures, instead of 15 figures after decimal point (#6145) * WKT export: do not append .0 after non-finite values (#6319) * Fix typo in definition of name of OGR_FD_ReorderFieldDefns (final s was missing) * OGRLayer::SetIgnoredFields(): properly reset state of non first geometry fields (#6283) * Make OGRLayer::SetSpatialFilter(GetSpatialFilter()) work with non empty spatial filter (#6284) * OGRLayerDecorator: add missing CreateGeomField() * OGRLayer::Erase(): do not discard input geometries that have no intersection with method layer (#6322) * OGRLayer::Erase(): Speedup = ~70% * Add OGRPreparedGeometryContains() * Use prepared geometry intersects as pretest in layer Intersection, Union, and Identity methods if requested. Use prepared geometry containment as pretest in layer Intersection method if requested. * Bail out from layer algebra methods if GEOS calls fail and not SKIP_FAILURES. * OGR_G_SetPoints(): error out if padfX or padfY == NULL, do not change coordinate dimension to 3D when pabyZ == NULL, fix optimization on linestring to call setPoints() only if the strides are the ones of a double, not 0 as incorrectly done before (#6344) * OGRParseDate(): more strict validation to reject invalide dates (#6452) OGRSpatialReference: * Upgrade to EPSG database v8.8 * Add support for SCH (Spherical Cross-track Height) projection * Optimize reprojection typically between WGS84 based SRS and WebMercator * Correctly transform Mercator_2SP and _1SP to ESRI Mercator, and back from ESRI Mercator to Mercator_2SP (#4861) * No longer enforce C locale if running against latest proj that is locale safe (4.9.2 or later) * EPSGGetPCSInfo(): use pcs.override.csv in priority over pcs.csv to read projection name, UOM, UOMAngle, GeogCS, etc... (#6026) * morphToESRI(): use GCS_WGS_1972 as GCS name for EPSG:4322 (#6027) * morphToESRI(): use Mercator_Auxiliary_Sphere projection for EPSG:3857. morphFromESRI(): map Mercator_Auxiliary_Sphere to EPSG:3857 (#5924) * Align hard-coded WKT of well known GCS definitions of WGS84, WGS72, NAD27 and NAD83 with the WKT of their EPSG def (#6080) * morphFromESRI(): special case with PROJCS name 'WGS_84_Pseudo_Mercator' (#6134) * OSR C API: fix declarations of OSRSetAxes() and OSRSetWagner(), and add missing OSRSetHOMAC(), OSRSetMercator2SP() and OSRSetTPED() (#6183) * Recognize EPSG 9835 method (Lambert Cylindrical Equal Area (Ellipsoidal)), needed for EPSG:6933 PCS for example * importFromProj4/exportToProj4(): rework linear unit conversion between WKT name/values and proj4 unit name, and extend its scope in WKT to proj4 conversions * OSR ESRI .prj: add support for reading custom ellipsoid in Parameters line * on import of +proj=geos, if +sweep=x is used then store it as a proj4 extension node (#6030) Utilities: * ogrinfo / ogr2ogr: implement @filename syntax for -sql and -where * ogr2ogr: prevent the -gt setting from overriding transaction group size of 1 set by skipfailures earlier (#2409) * ogr2ogr: warn if -zfield field does not exist in source layer * ogr2ogr -skip: rollback dataset transaction in case of failure (#6328) * ogr2ogr: fix -append with a source dataset with a mix of existing and non existing layers in the target datasource (#6345) * ogr2ogr: imply quiet mode if /vsistdout/ is used as destination filename * ogr2ogr: make -dim and -nlt support measure geometry types CartoDB: * fix GetNextFeature() on a newly create table (#6109) * defer 'CartoDBfycation' at layer closing * optimize feature insertion with multiple rows INSERT CSV driver: * add editing capabilities of existing files * add X_POSSIBLE_NAMES, Y_POSSIBLE_NAMES, Z_POSSIBLE_NAMES, GEOM_POSSIBLE_NAMES and KEEP_GEOM_COLUMNS open options * add HEADERS open option to force OGR to handle numeric column names. (PR #63) * add EMPTY_STRING_AS_NULL=YES/NO open option * implement compatibility enhancements for GeoCSV specification (#5989) * fix detection of TAB delimiter in allCountries.csv when the first line has a comma (#6086) * fix issues with leading single quote, and missing first line after ResetReading(), when parsing allCountries.txt (#6087) * speed-up GetFeatureCount() on allCountries.txt * on CreateDataSource() with a .csv name, do not try to open other existing .csv files in the directory * make CreateGeomField() returns OGRERR_NONE in case of success instead of OGRERR_FAILURE (#6280) * avoid adding trailing comma in header line when writing 'WKT,a_single_field' DGN driver: * add partial 3D transformation support for cell headers DXF driver: * detect files without .dxf extension (#5994) * fix handling of ELLIPSE with Z extrusion axis = -1 (#5705) * take into account full definition of spline entity (degree, control points, weights and knots) when stroking splines (#6436) * better handling of various object coordinate systems found in dxf files for point, line, polyline, spline and ellipse entities. Add anchor position to text styles. Remove polygon/polyface mesh parsing from polyline entity (#6459) ElasticSearch driver: * use get /_stats instead of /_status for ElasticSearch 2.0 compatibility (#6346) FileGDB driver: * make CreateFeature() honour user set FID, and implement more fine grained transaction for Linux/Unix * give a hint of using FileGDB SDK 1.4 is FileGDB compression is used GeoJSON driver: * Add editing capabilities of existing files * Add ARRAY_AS_STRING=YES open option * Use '%.17g' formatting by default for floating-point numbers and add SIGNIFICANT_FIGURES layer creation option (#6291) * add a json_ex_get_object_by_path() function * fix crash on null / non-json object features (#6166) * serialize string values that are valid JSon dictionary or array as it (ie do not quote them) * make sure there's enough space to write the FeatureCollection bbox (#6262). Also avoid duplicating FeatureCollection bbox if source has one (trunk only) * Export POINT EMPTY as having a null geometry, instead as of being POINT(0 0) (#6349) * Do not 'promote' a null field to OFTString type if it had another type before (#6351) GME driver: * Driver removed. Maps Engine being shut down at the end of January 2016. GML driver: * VFR: add new attribute DatumVzniku (v1.6) * VFR: fix ST_UVOH type handling * VFR: fix ZpusobyOchrany attributes (data types and names) * VFR: fix CisloDomovni attributes (Integer->IntegerList) * VFR: fix TEA attributes of StavebniObjekty * add NAMESPACE_DECL=YES option to OGR_G_ExportToGMLEx() to add xmlns:gml=http://www.opengis.net/gml or http://www.opengis.net/gml/3.2 declaration; Also accept GML2 or GML32 as valid valiues for FORMAT option (#6214) * serialize in .gfs file the name of the geometry element when it is 'geometry' since this is a particular case (#6247) * fix logic error in BuildJointClassFromScannedSchema() (#6302) GPKG driver: * make it accept files with non standard extension if they still have the correct application_id (#6396); also accept the .gpkx extension that may be used for extended geopackages * emit warning when generating a database without .gpkg/.gpkx extension (#6396) * as GPKG 1.1 uses a different application_id, emit a more specific warning if the application id starts with GPxx (but is not GP10). Add GPKG_WARN_UNRECOGNIZED_APPLICATION_ID config option to avoid the warning * correct scope of gpkg_geom_XXXXX extensions to be read-write, and allow reading geometry types CURVE or SURFACE * avoid trying to insert a gpkg_geom_XXXX extension if already done (#6402) * writer: implement strategy to flush partial_tiles temporary database when it becomes too big (#6462) * writer: when writing to GoogleMapsCompatible tiling scheme, better deal with source rasters in EPSG:4326 with latitude = +/-90 (#6463) * fix generation of files with 1-band paletted input dataset. Also improve the logic to retrieve color palette when forcing BAND_COUNT=1 on opening (#6450) GPX driver: * fix crash when parsing a 'time' extension element at route/track level (2.0 regression, #6237) ILI driver: * ILI1: Support for Surface polygon rings spread over multiple geometry records * ILI1: add string TID support (https://github.com/OSGeo/gdal/pull/91) * Fix crash with models using types derived from INTERLIS * Fix memory leaks (#6178) JML driver: * remove arbitrary limitation preventing from reading geometries with larger than 10 MB (#6338) KML driver: * fix crash on KML files without content but with nested folders (#6486) LIBKML driver: * for documents without folder, use document name when available as name of layer (#6409) Memory driver: * add support for sparse feature IDs * add ADVERTIZE_UTF8 layer creation option MITAB driver: * Add support for block sizes other than 512 bytes in .map files, for MapInfo 15.2 compatibility (#6298) * write correct datum id for EPSG:3857 * read MID files with TAB delimiter and empty first field (#5405) * use projection code 29 when exporting non-Polar Lambert Azimuthal Equal Area (#5220) * fix crashes when parsing invalid MIF geometries (#6273) MSSQLSpatial driver: * Implement MSSQL bulk insert (#4792) * do not treat a primary key that is not of integer type as the FID (#6235) NTF driver: * fix potential buffer overflows when reading too short lines (#6277) ODBC driver: * remove limitations to 500 columns ODS driver: * fix loss of precision in formula computation OpenFileGDB driver: * do not emit warning if SDC/CDF table detected and that FileGDB driver is present * fix min/max on columns without indices (#6150) * build correct geometry for a multi-part wkbMultiLineStringZ (#6332) * add support for reading SHPT_GENERALPOINT (#6478) OSM driver: * correct fields ids for the (non frequently used) Node message * do not override 'our' osm_id (the node, way or relation id) with a tag named 'osm_id' (#6347) * properly deal with polygons in other_relations geometrycollection (#6475) PG driver: * Add PRELUDE_STATEMENTS and CLOSING_STATEMENTS open option to be for example able to specify options, like statement_timeout, with pg_bouncer * Fix 2.0 regression when overwriting several existing PostGIS layers with ogr2ogr (#6018) * Update PG, PGDump and CartoDB drivers to correctly export POINT EMPTY for PostGIS 2.2 * avoid resetting error potentially emitted by ExecuteSQL() (#6194) * sanitize management of quoting for FID column at layer creation * fix to get SRID on result layer with PostGIS 2.2 * in copy mode (the default on layer creation), do not truncate the concatenated string list to the field width (#6356) * make such that GEOMETRY_NAME layer creation option is honoured in ogr2ogr when the source geometry field has a not-null constraint (#6366) * read and set DESCRIPTION metadata item from/into pg_description system table; add DESCRIPTION layer creation option * support int2[] and numeric[] types, better map float4[] type * remove code that was intended to handled binary cursors as it cannot be triggered * fix append of several layers in PG_USE_COPY mode and within transaction (ogr2ogr -append use case) (#6411) PGDump driver: * fix issue with case of ogc_fid field in case the FID layer creation option is not set by user or by ogr2ogr (related to #6232) * in copy mode (the default on layer creation), do not truncate the concatenated string list to the field width (#6356) * make such that GEOMETRY_NAME layer creation option is honoured in ogr2ogr when the source geometry field has a not-null constraint (#6366) * set DESCRIPTION metadata item from/into pg_description system table; add DESCRIPTION layer creation option Shapefile driver: * accept opening standalone .dbf files whose header length is not a multiple of 32 bytes (#6035) * fix REPACK crash on shapefile without .dbf (#6274) * add capability to restore/build a missing .shx file when defining SHAPE_RESTORE_SHX to TRUE (#5035) * avoid CreateLayer() to error out when passed wkbUnknown | wkb25D (#6473) SQLite/Spatialite driver: * support file:xxx URI syntax (derived from patch by joker99, #6150) * fix heuristics in OGRSQLiteSelectLayer::GetExtent() to not be used when there's a sub SELECT (#6062) * fix crash on GetLayerByName('non_existing_table(geom_column)') (#6103) * fix OGRSQLiteSelectLayerCommonBehaviour::GetBaseLayer() to no longer 'eat' consecutive characters in layer name (#6107) * Spatialite: turn debug messages warning about update not being supported because of missing or too old spatialite version as errors, and return NULL to the caller (#6199) * fix memleak in OGRSQLiteTableLayer destructor when updating geometry_columns_time * VFS: increase mxPathname to 2048 by default, and provide OGR_SQLITE_VFS_MAXPATHNAME config option to be able to configure that higher if that would be needed. Useful when dealing with very long names like /vsicurl/.... with AWS S3 security tokens * VFS: do not probe -wal files on /vsicurl/ SXF driver: * fix wrong use of endianness macros to fix behaviour on big-endian hots * add recoding from CP1251 for TEXT attribute that is now decoded * fix various issues (#6357) VRT driver: * implement CloseDependentDatasets() * fix editing with 'direct' geometry mode which could cause attribute column to be empty (#6289) * fix crash with a OGRVRTWarpedLayer using a source layer that would have non geometry column (unlikely to happen currently as this would require explicit disabling it, but more likey with following commit that createe VRT non-spatial layer implictly when the source is non-spatial, linked to #6336) * avoid creating an implicit wkbUnknown geometry field when the source has no geometry column and there's no XML elements related to geometry fields (#6336) XLSX driver: * fix reading sheets with more than > 26 columns and 'holes' (#6363) XPlane driver: * extend ICAO identifiers to 5 digits (#6003) == SWIG Language Bindings == All bindings: * add a options parameter to gdal.ReprojectImage() to pass warp options * Change ReadRaster and WriteRaster to use GIntBig and the *IOEx-methods * prevent NULL file pointer from being passed to VSIF*L functions * make gdal.Rename() accept Unicode strings * add SpatialReference.GetAxisName() and SpatialReference.GetAxisOrientation() (#6441) * add SpatialReference.GetAngularUnitsName() (#6445) Java bindings: * Fix typemap for input parameter of type GIntBig (fixes GetFeature(long), DeleteFeature(long), etc...) (#6464) * Bump minimal java version to 1.5 in case SWIG generates anotations (#6433, patch by Bas Couwenberg) * GNUmakefile: add -f in rm commands * GNUmakefile: add support for all hardening buildflags Perl bindings: * Fix #6050: string formatting in croak. * Perl Makefile.PL: add support for all hardening buildflags (#5998) * use strict and warnings in overridden constructors. * add $VERSION to ogr_perl.i (OGR.pm), which is required by pause.perl.org. * Add some basic module info for CPAN. * Bugfix for Geo::OGR::Feature->new(). * Add many utility level algorithms as methods to various classes. * New class for XML stuff * New Makefile.PL, which can download and build GDAL. This allows automatic testing of the CPAN module. * Wrap VSIStdoutSetRedirection and allow creating datasets via an object, which can write and close. * Geo::OGR::Driver and Geo::OGR::DataSource are now Perl wrappers for respective GDAL classes. * Add to the error stack also errors from the bindings * Fix sending utf8 from Perl to GDAL. Should also remove some "uninitialized value" warnings. * ReadTile and WriteTile methods for Dataset, ReadTile accepts now tile size and scaling algorithm. * Improved Parent - Child management. * Improved support for 64bit ints. * Measures support in Geometry class. * Many new tests Python bindings: * make Feature.ExportToJson() output boolean value for a boolean field * support floating point coordinates for the source windows of Band.ReadRaster() and Band.ReadAsArray() * fix build with SWIG 3.0.6 (#6045) * make gdal.OpenEx() throw a Python exception in case of failed open when exceptions are enables with gdal.UseExceptions() (#6075) * Disable opening a NumPy dataset with a filename returned by gdal_array.GetArrayFilename(() unless GDAL_ARRAY_OPEN_BY_FILENAME is set to TRUE * disable the warning about using deprecated wkb25DBit constant as it uses a trick that prevents the bindings from being used by py2exe / pyinstaller (#6364) = GDAL/OGR 2.0 Release Notes = == In a nutshell... == * New GDAL drivers: - BPG: read-only driver for Better Portable Graphics format (experimental, no build support) - GPKG: read/write/update capabilities in the unified raster/vector driver - KEA: read/write driver for KEA format - PLMosaic: read-only driver for Planet Labs Mosaics API - ROI_PAC: read/write driver for image formats of JPL's ROI_PAC project (#5776) - VICAR: read-only driver for VICAR format * New OGR drivers: - Cloudant: read/write driver for Cloudant service - CSW: read-only driver for OGC CSW (Catalog Service for the Web) protocol - JML: read/write driver for OpenJUMP .jml format - PLScenes: read-only driver for Planet Labs Scenes API - Selaphin: read/write driver for the Selaphin/Seraphin format (#5442) * Significantly improved drivers: CSV, GPKG, GTiff, JP2OpenJPEG, MapInfo file, PG, SQLite * RFC 31: OGR 64bit Integer Fields and FIDs (trac.osgeo.org/gdal/wiki/rfc31_ogr_64) In OGR core, OGR SQL, Shapefile, PG, PGDump, GeoJSON, CSV, GPKG, SQLite, MySQL, OCI, MEM, VRT, JML, GML, WFS, CartoDB, XLSX, ODS, MSSQLSpatial, OSM, LIBKML, MITAB * RFC 46: GDAL/OGR unification ( http://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification) - GDAL and OGR PDF drivers are unified into a single one - GDAL and OGR PCIDSK drivers are unified into a single one * RFC 49: Add support for curve geometries (http://trac.osgeo.org/gdal/wiki/rfc49_curve_geometries) In OGR core, and GML, NAS, PostgreSQL, PGDUMP, GPKG, SQLite, VFK, VRT, Interlis drivers * RFC 50: Add support for OGR field subtypes (http://trac.osgeo.org/gdal/wiki/rfc50_ogr_field_subtype) In OGR core, OGR SQL, swig bindings, CSV, FileGDB, GeoJSON, GML, GPKG, OpenFileGDB, PG, PGDump, SQLite, VRT * RFC 51: RasterIO() improvements : resampling and progress callback (http://trac.osgeo.org/gdal/wiki/rfc51_rasterio_resampling_progress) * RFC 52: Stricter SQL quoting (http://trac.osgeo.org/gdal/wiki/rfc52_strict_sql_quoting) * RFC 53: OGR not-null constraints and default values (http://trac.osgeo.org/gdal/wiki/rfc53_ogr_notnull_default) In OGR core, OGR SQL, PG, PGDump, CartoDB, GPKG, SQLite, MySQL, OCI, VRT, GML, WFS, FileGDB, OpenFileGDB and MSSQLSpatial * RFC 54: Dataset transactions (https://trac.osgeo.org/gdal/wiki/rfc54_dataset_transactions) In PG, GPKG, SQLite, FileGDB and MSSQLSpatial * RFC 55: refined SetFeature() and DeleteFeature() semantics. In GPKG, Shape, MySQL, OCI, SQLite, FileGDB, PG, CartoDB, MITAB and MSSQL * RFC 56: OFTTime/OFTDateTime millisecond accuracy ( https://trac.osgeo.org/gdal/wiki/rfc56_millisecond_precision ) * RFC 57: 64bit histogram bucket count ( https://trac.osgeo.org/gdal/wiki/rfc57_histogram_64bit_count ) * Upgrade to EPSG v8.5 database * Fix locale related issues when formatting or reading floating point numbers (#5731) == New installed files == * data/gdalvrt.xsd: XML schema of the GDAL VRT format == Backward compatibility issues == See MIGRATION_GUIDE.TXT == GDAL/OGR 2.0 - General Changes == Build(Unix): * Fix for cpl_recode_iconv.cpp compilation error on freebsd 10 (#5452) * Fix pthread detection for Android * Fix in Armadillo detection test (#5455) * Fix detection of OCI by changing linking order to please modern GCC (#5550) * Fix test to accept MariaDB 10.X as valid MySQL (#5722) * Make sure $(GDAL_INCLUDE) is first to avoid being confused by GDAL headers of a previous version elsewhere in the include path (#5664) * Always use stat rather than stat64 for Mac OSX in AC_UNIX_STDIO_64. (#5780, #5414). * Add support for ECW SDK 5.1 (#5390) * Do not enable Python bindings if PYTHON env variable is set without --with-python being explicitly specified (#5956) Build(Windows): * PDF: fix compilation issue with Visual Studio 2012 (#5744) * PDF: Add support to compile the pdf driver as plugin (#5813) * Add support for MrSID 9.1 SDK (#5814) * when building netCDF, HDF4, HDF5 as plugins, call registration of 'sub-drivers' GMT, HDF4Image and HDF5Image (#5802) Build(all): * Ruby bindings: disable autoconf and makefile support (#5880) * Fix compilation errors with json-c 0.12 (#5449) * Fix compilation error in alg/gdalgrid.cpp when AVX is available, but not SSE (#5566) == GDAL 2.0 - Overview of Changes == Port: * Introduce a more generic lock API (recursive mutex, adaptive mutex, spinlock) * Add types for CPLMutex, CPLCond and CPLJoinableThread (only enforced in -DDEBUG mode) * Add CPLGetPhysicalRAM() and CPLGetUsablePhysicalRAM() * CPLSpawn() on Windows: quote arguments with spaces in them (#5469) * /vsigzip/: avoid infinite loop when reading broken .gml.gz file (#5486) * /vsizip/ : fix bug that caused premature end of file condition with some read patterns (#5530) * /vsizip/ on >4GB zips: accept .zip declare 0 disks (#5615) * /vsitar/: remove useless validation test that prevents from opening valid .tar files (#5864) * /vsistdout/: add VSIStdoutSetRedirection() for compatibility with MapServer FCGI (https://github.com/mapserver/mapserver/pull/4858) * /vsimem/: update st_mtime and return it with Stat() * /vsimem/: in update mode, when seeking after end of file, only extend it if a write is done * /vsimem/: Make Rename() on a directory also rename filenames under that directory (#5934) * /vsicurl/: manage redirection from public URLs to redirected AWS S3 signed urls, with management of the expiration. Enabled by default. Can be disabled if CPL_VSIL_CURL_USE_S3_REDIRECT=NO (#6439) * /vsicurl/: avoid reading after end-of-file and fix failure when reading more than 16MB in a single time (#5786) * /vsicurl/: by default do not use HEAD request when detecting a AWS S3 signed URL * Allow CPL_VSIL_CURL_ALLOWED_EXTENSIONS to be set to special value {noext} * VSIWin32Handle::Flush(): no-op implementation is sufficient to offer same guarantee as POSIX fflush() (#5556) * Unix VSIL: reset eof in all cases in Seek() * Windows plugins: complementary fix to #5211 to avoid error dialog box when there are dependency problems (#5525) * Fix VSIReadDirRecursive() recursing on the parent or current directory (#5535) * cpl_error: obfuscate password * HTTP: set CURLOPT_NOSIGNAL if available (#5568) * Add COOKIE option to CPLHTTPFetch() (#5824) * CPLHTTPFetch(): add retry logic in case of 502, 503 and 504 errors with the GDAL_HTTP_MAX_RETRY (default: 0)and GDAL_HTTP_RETRY_DELAY (default: 30 s) config options (#5920) * Fix stack corruption upon thread termination with CPLSetThreadLocalConfigOption on Windows 32 bit (#5590) * cpl_csv: Stop probing for csv/horiz_cs.csv. (#5698) * vsipreload: implement clearerr() and readdir64() (#5742) * CPLsetlocale(): return a string that is thread-locale storage to avoid potential race in CPLLocaleC::CPLLocaleC() (#5747) * CPLHexToBinary(): faster implementation (#5812) * CPLAcquireMutex(): improve performance on Windows (#5986) Core: * Add imagery (satellite or aerial) metadata support (Alos, DigitalGlobe, Eros, GeoEye, OrbView, Landsat, Pleiades, Resurs-DK1, Spot/Formosat). * Reduce lock contention on the global cache mutex and make it possible to use spin lock instead with GDAL_RB_LOCK_TYPE=SPIN * Block cache: make block cache manager safe with respect to writing dirty blocks (#5983) * EXIF reader: fix memleak in error code path * EXIF reader: add missing validation for some data types (#3078) * Fix crash in GDALPamRasterBand::SerializeToXML() when saving an empty RAT (#5451) * ComputeStatistics(): use Welford algorithm to avoid numerical precision issues when computing standard deviation (#5483) * Fix crashing issue with TLS finalization on Unix (#5509) * GDALJP2Metadata::CreateGMLJP2(): use EPSGTreatsAsLatLong() and EPSGTreatsAsNorthingEasting() to determine if axis swapping is needed (#2131) * GDALJP2AbstractDataset: implement GetFileList() to report .wld/.j2w if used * GMLJP2: be robust when parsing GMLJP2 content that has nul character instead of \n (#5760) * GMLJP2: add missing rangeParameters element to validate against GMLJP2 schema (#5707) * GMLJP2: write non null bounding box at root of FeatureCollection (#5697) * GMLJP2: SRS export as GML: output XML definition of a SRS as a GML 3.1.1 compliant Dictionary (#5697) * GMLJP2: when setting GDAL_JP2K_ALT_OFFSETVECTOR_ORDER=TRUE write it as a XML comment so that we can interpret the OffsetVector elements correctly on reading * GMLJP2: when parsing a GMLJP2 box, accept srsName found on gml:RectifiedGrid if not found on origin.Point, so as to be compatible with the example of DGIWG_Profile_of_JPEG2000_for_Georeferenced_Imagery.pdf (#5697) * GMLJP2: add compatibility with GMLJP2 v2.0 where SRS is expressed as CRS URL * GMLJP2: on reading, don't do axis inversation if there's an explicit axisName requesting easting, northing order (#5960); also strip axis order in reported SRS * JP2Boxes: add null terminated byte to GDAL XML, XML or XMP boxes * Add GDALGetJPEG2000Structure() (#5697) * GDALMultiDomainMetadata::XMLInit(): when importing XML metadata, erase the existing document to replace it with the new one * Metadata: fix correct sorting of StringList / metadata (#5540, #5557) * Make GetMaskBand() work with GDT_UInt16 alpha bands (#5692) * Fix 32bit overflow in GDALRasterBand::IRasterIO() and GDALDataset::BlockBasedRasterIO() (#5713) * RasterIO: small optimization in generic RasterIO() implementation to avoid loading partial tiles at right and/or bottom edges of the raster when they are going to be completely written * Fix crash when calling GetTiledVirtualMem() on non-Linux platform (#5728) * Add GDAL_OF_INTERNAL flag to avoid dataset to be registered in the global list of open datasets * GDALDriver::CreateCopy(): accept _INTERNAL_DATASET=YES as creation option, so as to avoid the returned dataset to be registered in the global list of open datasets * Implement GDALColorTable::IsSame() * GDALPamDataset: do not serialize dataset metadata unless it has been set through GDALDataset::SetMetadata() or GDALDataset::SetMetadataItem() * GDALLoadTabFile: add TAB_APPROX_GEOTRANSFORM=YES/NO configuration option to decide if an approximate geotransform is OK (#5809) * Optimize copy efficiency from tiled JPEG2000 images * Avoid fetching remote non-existing resources for sidecar files, when using /vsicurl/ with a URL that takes arguments (#5923) * Use GDALCanFileAcceptSidecarFile() in GDALMDReaderManager::GetReader() Algorithms: * RPC transformer: fix near interpolation in RPC DEM (#5553) * RPC transformer: take into account nodata in RPC DEM (#5680) * RPC transformer: add RPC_DEM_MISSING_VALUE transformer option to avoid failure when there's no DEM at the transformed point (#5730) * RPC transformer: in DEM mode, implement optimization, in specific conditions (input points at same longitude, DEM in EPSG:4326) to extract several elevations at a time * TPS transformer: fix crash if the forward or backward transform cannot be computed (#5588) * OpenCL warper: remove unused variable in bilinear resampling that can cause compilation error (#5518) * OpenCL wraper: fix code compilation with NVIDIA OpenCL (#5772) * Overview: Fix and speed-up cubic resampling in overview computation to take into account scaling factor (#5685) * Overview: ignore alpha=0 values when compute an average overview of an alpha band; and also avoid memory errors when calling GetMaskBand()/GetMaskFlags() after overview computation if GetMaskXXX() has been called before (#5640) * Overview: avoid crash when computing overview with a X dimension much smaller than Y dimension (#5794) * GDALRegenerateOverviewsMultiBand(): fix stride calculation error with certain raster dimensions (#5653) * Warper: numerous speed optimizations (SSE2 specific code, more fast code paths, ...) * Warper: fix Cubic and Bilinear resampling to work correctly with downsizing (#3740) * Warper: fix and optimize CubicSpline * Warper: regardless of the warping memory limit, add heuristics to determine if we must split the target window in case the 'fill ratio' of the source dataset is too low (#3120) * Warper: accept warping options METHOD=NO_GEOTRANSFORM and DST_METHOD=NO_GEOTRANSFORM to run gdalwarp on ungeoreferenced images * Warper: fix GDALSuggestedWarpOutput() wrong extent in some circumstances (e.g. dataset of big dimension with world coordinates) (#5693) * Warper: fix integer overflow when reprojecting into an area with (part of) bounds completely outside of the source projection (#5789) * Warper: add min,max,med,q1 and q3 resampling algorithms (#5868) * Warper: add a SRC_COORD_PRECISION warping option to help getting more reproducible output when -wm parameter changes (#5925) * Warper: fix failure in GDALSuggestedWarpOut2() when top-left and bottom-right corners transform to the same point (#5980) * GDALReprojectImage(): takes into account nodata values set on destination dataset * Median cut and dithering: optimizations and enhancements to deal with 8-bit precision (only if using internal interface for now) * rasterfill: add option to specify driver to use for temporary files * Polygonize: speed optimization: do not try to build the polygon for pixels that are masked by the mask band (i.e. alpha, nodata, etc...). Can considerably speed-up processing when the nodata outline forms a very complex polygon Utilities: * gdalinfo: display extra metadata domains attached to band, and refactor code a bit (#5542) * gdalinfo: add -oo option per RFC 46 * gdalinfo: add -json switch (partial implementation of RFC 44) * gdaladdo: add -oo option per RFC 46 * gdaladdo: add warning when subsampling factor 1 specified * gdal_translate: add -oo option per RFC 46 * gdal_translate: add -r and -tr options per RFC 51 * gdal_translate: add a -projwin_srs option to be able to express -projwin coordinates in another SRS than the one of the dataset * gdal_translate: support -'outsize avalue 0' or '-outsize 0 avalue' to preserve aspect ratio * gdal_translate: avoid preserving statistics when changing data type in situations where clamping can occur * gdal_translate: adjust RPC metadata (pixel/line offset/scale) when subsetting/rescaling, instead of just discarding it * gdal_translate: don't recopy band units if rescaling or unscaling is involved (#3085) * gdal_translate: increase GDAL_MAX_DATASET_POOL_SIZE default value to 450. (#5828) * gdal_translate: preserve NBITS image structure metadata when possible * gdalwarp: add -oo option per RFC 46 * gdalwarp: add -te_srs option to specify -te in a SRS which isn't the target SRS * gdalwarp: add a -ovr option to select which overview level to use, and default to AUTO. Also add a generic OVERVIEW_LEVEL=level open option, and make it available in standard VRT (#5688) * gdalwarp: initialize destination dataset to no_data value when automatically propagating source nodata (#5675) * gdalwarp: only apply INIT_DEST when processing the first input dataset (#5387) * gdalwarp: increase GDAL_MAX_DATASET_POOL_SIZE default value to 450. (#5828) * gdalwarp: do not preserve NODATA_VALUES metadata item in output dataset if adding an alpha channel with -dstalpha * gdalwarp: fix '-dstnodata none' to avoid read of uninitialized values (#5915) * gdalwarp: make -crop_to_cutline densify cutline in source SRS before reprojecting it to target SRS (#5951) * gdaldem: avoid too large files to be produced when using -co COMPRESS=xxxx -co TILED=YES (#5678) * gdallocationinfo: add -oo option * gdaltransform: add a -output_xy flag to restrict output coordinates to 'x y' only * gdal_grid: use nodata= parameter in the algorithm string to determine the nodata value to set on the band (#5605) * gdal_grid: fix crash in optimized mode with GCC 4.4 on 64bit (#5987) * gdalbuildvrt: add a -r option to specify the resampling algorithm * gdal_edit.py: add -unsetstats option (and fix -a_nodata to run on all bands, ant not just first one) * gdal_edit.py: add -stats and -approx_stats flags (patch by mwtoews, #5805) * gdal_edit.py: change -mo add metdata to existing one; add new option -unsetmd to clean existing metadata * gdal_edit.py: add -oo to specify open options * gdal_retile.py: fix to make it work with input images of different resolutions (#5749) * gdal_retile.py: implement progress bar (#5750) * gdal_merge.py: add timing information in verbose output * gdal_merge.py: take into account alpha band to avoid writing zones of source images that are fully transparent (#3669) * gdal2tiles.py: fix inverted long/lat in BoundingBox and Origin elements of tilemapresource.xml (#5336) * pct2rgb.py: make it work with color tables with less than 256 entries (#5555) * gdal_fillnodata.py: FillNodata: copy no data value to destination band when creating a dataset (if available) (#4625) * gdal_proximity.py: add a -use_input_nodata flag * gdalcompare.py: add options to suppress selected comparisons * gdalcompare.py: takes into account differences in overview bands * gdalcompare.py: compute difference on float to avoid integer underflow * epsg_tr.py: change to make it possible to export GEOCCS and COMPD_CS to proj.4 epsg and PostGIS spatial_ref_sys.sql files Python samples: * Added swig/python/samples/jpeg_in_tiff_extract.py * Added dump_jp2.py * Added validate_jp2.py * Added build_jp2.py * Added gcps2ogr.py * tolatlong.py: report error when operating on a non-georeferenced dataset AAIGRID: * Fix formatting string (#5731) BAG driver: * change nodata value for uncertainty band to 1e6 (#5482) BMP driver: * back out r17065 change that inferred georeferencing based on the resolution information in the BMP header (#3578) DIMAP driver: * DIMAP 2: handle the case where the Raster_Data element is in main file (#5018, #4826) * DIMAP 2: fix to extract geodetic SRS (#5018, #4826) * DIMAP 2: fix to extract geotransform from JPEG2000 file if not available in XML (#5018, #4826) DDS driver: * Add ETC1 compression format support * Header correction for worldwind client ECW driver: * correctly assign color interpretation to bands if order is unusual ENVI driver: * avoid generating potentially corrupted .hdr files when opening in update mode; Write 'Arbitrary' instead of 'Unknown' as the projection name for an undefined SRS (#5467) * when writing, consider that LOCAL_CS SRS is like ungeoreferenced (#5467) ERS driver: * reset RasterInfo.RegistrationCellX/Y if setting a new geotransform on an updated .ers file (#5493) * fix SetProjection() (#5840) GeoRaster driver: * fix Oracle SRID authority (#5607) * fix user-defined SRID issue (#5881) * new SRID search (#5911) GIF driver: * add compatibility with giflib 5.1 (#5519) * fix crash on images without color table (#5792) * fix reading of interlaced images with giflib >= 5.0 * validate the size of the graphic control extension block (#5793) * implement GetFileList() to report worldfile GRASS driver: * GRASS 7.0.0 support (#5852) GRIB driver: * avoid divide by zero while setting geotransform on 1xn or nx1 grib file (#5532à * allow writing PDS template numbers to all bands (#5144) GTiff driver: New capabilities: * for JPEG-in-TIFF, use JPEG capabilities to decompress fast overview levels 2,4 and 8, to generate 'hidden' overviews used by RasterIO() * add DISCARD_LSB creation option (lossy compression) to be best used with PREDICTOR=2 and LZW/DEFLATE compression * when GTIFF_DIRECT_IO=YES is enabled, performance improvements in GTiffRasterBand::DirectIO() with Byte dataset and Byte buffer * implement dataset DirectIO() * add GTIFF_VIRTUAL_MEM_IO=YES/NO/IF_ENOUGH_RAM configuration option so that RasterIO() can rely on memory-mapped file I/O (when possible and supported by the OS) * support reading and created streamable files * for JPEG-compressed TIFF, avoid quantization tables to be emitted in each strip/tile and use optimized Huffman coding by default * avoid SetNoDataValue() to immediately 'crystallize' the IFD * allow unsetting TIFFTAG_SOFTWARE, TIFFTAG_DOCUMENTNAME, etc... by removing them from metadata list or passing None as a value of SetMetadataItem() (#5619) * allow lossless copying of CMYK JPEG into JPEG-in-TIFF * set alpha on target by default when translating from Grey+Alpha * Internal overviews: for near, average, gauss and cubic, and pixel interleaving, make sure to use the same code path for compressed vs uncompressed overviews (#5701) * add RPCTXT=YES creation option to write sidecar _RPC.TXT file * internal libtiff updated to upstream libtiff 4.0.4beta * internal libgeotiff updated to upstream libgeotiff SVN head * speed optimization on write (at least in Vagrant) (#5914) * use importFromEPSG() when ProjectedCSTypeGeoKey is available (#5926) * on reading better deal with a few ESRI formulations of WebMercator (#5924) Fixes: * when overriding metadata in update mode, make sure to clear it from PAM file (#5807) * fix handling of Mercator_2SP (#5791) * avoid TIFF directory to be written (at end of file) when creating a JPEG-in-TIFF file * for a paletted TIFF with nodata, set the alpha component of the color entry that matches the nodata value to 0, so as gdal_translate -expand rgba works properly * fix to make band SetMetadata(NULL) clear band metadata (#5628) * fix error message when requesting a non existing directory * check data type with PHOTOMETRIC=PALETTE * be robust to out-of-memory conditions with SplitBand and SplitBitmapBand * avoid using optimized JPEG --> JPEG-in-TIFF path if INTERLEAVE=BAND is specified with a 3-band JPEG * to make Python bindings happy, avoid emitting CE_Failure errors due to libtiff errors when we still manage to open the file (#5616) * avoid crash when reading GeoTIFF keys if the stored key type isn't the one expected * make sure to call libgeotiff gtSetCSVFilenameHook() method when linking against external libgeotiff * avoid/limit DoS with huge number of directories * clean spurious spaces when reading values from _RPC.TXT * serialize RPC in PAM .aux.xml file if using PROFILE != GDALGeoTIFF and RPB = NO * fix clearing of GCPs (#5945) * avoid generated corrupted right-most and bottom-most tiles for 12-bit JPEG-compressed (#5971) * make sure to use scanline write API when writing single-band single-strip 1-bit datasets GRASS driver: * fix compilation issues against GRASS 7 HDF4 driver: * Add configuration support to be able to open more files simultaneously * Add class suffix to the parameter name when parsing HDF-EOS objects. * Fix AnyTypeToDouble() to use proper type (int instead of long) to work with DFNT_INT32/DFNT_UINT32 on 64-bit Linux (#5965) * MODIS: Set more correct values for PIXEL_/LINE_ OFFSET/STEP by comparing longitude and latitude subdatasets dimensions with main subdataset dimensions HDF5 driver: * avoid opening BAG files in the case HDF5 and BAG are plugins, and HDF5 is registered before BAG HFA driver: * fix recognition of Hotine Mercator Azimuth Center in Imagine format (and Swisstopo GeoTIFF) (#5551) * fix various hangs on invalid files * read projection even when it does not contain datum information (#4659) HTTP driver/wrapper: * make it work with vector files too * fix handling of non VSI*L file on Windows IRIS driver: * add support for the SHEAR data type (#5549) JPEG2000 driver: * do expansion of 1-bit alpha channel to 8-bit by default. Can be controlled with the 1BIT_ALPHA_PROMOTION open option (default to YES) * add GMLJP2 creation option * add GMLJP2V2_DEF creation option to create a GMLJP2 v2 box * Add capability of reading GMLJP2 v2 embedded feature collections and annotations * Add read/write support for RPC in GeoJP2 box (#5948) JP2ECW driver: * add metadata about JPEG2000 codestream and boxes (#5408) * Add 1BIT_ALPHA_PROMOTION open option (default to YES) to control expansion of 1-bit alpha channel to 8-bit * correctly assign color interpretation to bands if order is unusual * add WRITE_METADATA and MAIN_MD_DOMAIN_ONLY creation options to write GDAL metadata, JP2 XML boxes or XMP box * add GMLJP2V2_DEF creation option to create a GMLJP2 v2 box * Add capability of reading GMLJP2 v2 embedded feature collections and annotations * Add read/write support for RPC in GeoJP2 box (#5948) JP2KAK driver: * Compatibility with Kakadu v7.5 (#4575, #5344) * Handle Kakadu version 7 allocator.finalize (#4575) * fix bug in vsil_target::end_rewrite() that prevented TLM index to be generated (#5585) * Add FLUSH in creation option XML (#5646) * Do expansion of 1-bit alpha channel to 8-bit by default. Can be controlled with the 1BIT_ALPHA_PROMOTION open option (default to YES) * add GMLJP2V2_DEF creation option to create a GMLJP2 v2 box * Add capability of reading GMLJP2 v2 embedded feature collections and annotations * Add read/write support for RPC in GeoJP2 box (#5948) JP2OpenJPEG driver: * Support writing arbitrary number of bands. (#5697) * Generate cdef box when transparency is needed. Add NBITS, 1BIT_ALPHA and ALPHA creation options (#5697) * add INSPIRE_TG (for conformance with Inspire Technical Guidelines on Orthoimagery), PROFILE, JPX and GEOBOXES_AFTER_JP2C creation option (#5697) * add GMLJP2V2_DEF creation option to create a GMLJP2 v2 box * Add capability of reading GMLJP2 v2 embedded feature collections and annotations * add PRECINCTS creation option (#5697) * allow several quality values to be specified with QUALITY creation option. Add TILEPARTS, CODEBLOCK_WIDTH and CODEBLOCK_HEIGHT options (#5697) * support reading&writing datasets with unusual order of band color interpretation (#5697) * add WRITE_METADATA and MAIN_MD_DOMAIN_ONLY creation options to write GDAL metadata, JP2 XML boxes or XMP box (#5697) * add support for reading/writing/updating IPR box (from/into xml:IPR metadata domain) (#5697) * add YCC creation option to do RGB->YCC MCT, and turn it ON by default (#5634) * add USE_SRC_CODESTREAM=YES experimental creation option, to reuse the codestream of the source dataset unmodified * support reading & writing images with a color table (#5697) * support update mode for editing metadata and georeferencing (#5697) * add compatibility with OpenJPEG 2.1 (#5579) * fix warning when reading a single tile image whose dimensions are not a multiple of 1024 (#5480) * Add 1BIT_ALPHA_PROMOTION open option (default to YES) to control expansion of 1-bit alpha channel to 8-bit * Add read/write support for RPC in GeoJP2 box (#5948) * limit number of file descriptors opened JPEG driver: * use EXIF overviews if available * add EXIF_THUMBNAIL creation option to generate an EXIF thumbnail * use optimized Huffman coding to reduce file size * add support for reading and writing COMMENT * optimize whole image reading with dataset IRasterIO() * report non-fatal libjpeg errors as CE_Warning (or CE_Failure if GDAL_ERROR_ON_LIBJPEG_WARNING = TRUE) (#5667) * in compressor, increase default val of max_memory_to_use to 500MB JPIPKAK driver: * avoid symbol collision with kdu_cpl_error_message from JP2KAK driver * reset the bNeedReinitialize flag after a timeout (#3626) KMLSuperOverlay driver: * fix truncated raster on 32 bit builds (#5683) * make Identify() more restrictive to avoid false positives L1B driver: * add support for little-endian LRPT datasets (#5645) * expose band mask when there are missing scanlines (#5645) * expose WGS-84 or GRS-80 datum if read from header record (#5645) MAP driver: * retrieve the image filename in a case insensitive way (#5593) MBTiles driver: * better detection of 4 bands dataset and take into account alpha component of color table in RasterIO() (#5439) * avoid wrong detection of 3 bands when finding paletted PNG in /vsicurl mode (#5439) * fix dimension computation when opening a single tile dataset * better computation of extent from min/max of tile coordinates, for a single zoom level * use standard EPSG:3857 origin to fix a ~8m shift (#5785) MEM driver: * implement optimized versions of raster band and dataset IRasterIO() MSG driver: * fix compilation problem (#5479) * fix memory leaks (#5541) NetCDF driver: * Force block size to 1 scanline for bottom-up datasets if nBlockYSize != 1 (#5291) * Fix computation of inverse flattening (#5858) * In case the netCDF driver is registered before the GMT driver, avoid opening GMT files * Fix crash on opening a NOAA dataset (#5962) NGSGEOID driver: * make Identify() more restrictive NITF driver: * deal correctly with JPEG2000 NITF datasets that have a color table inboth Image Subheader and JP2 boxes, and for drivers that don't do color table expension * HISTOA TRE: put definition of TRE in conformance with STDI-0002 (App L page 14) and STDI-0006 (Page 57) (#5572) OGDI driver: * Remove OGDIDataset::GetInternalHandle (#5779). OZI driver: * remove .map header detection from Identify() since this is actually handled by the MAP driver PCIDSK driver: * close dataset in case of exception in PCIDSK2Dataset::LLOpen() (#5729) PCRaster driver: * Align libcsf code with PCRaster raster format code (#5843) * Implement Create() (#5844) * Improve handling of no-data value (#5953) PDF driver: * add compatibility with Poppler 0.31.0 * in the OGC Best practice case, switch rotational terms of the geotransform matrix (gt[2] and gt[4]) * in the OGC Best practice case, handle rotations of 90 and 270 degrees. * advertize LAYERS metadata domain * deal with OHA- datum (Old Hawaiian) * fix compilation problem with Podofo on Windows (patch by keosak, #5469) * add sanity check on page count PDS driver: * Added support for SPECTRAL_QUBE objects used e.g. by THEMIS instrument of Mars Odyssey spacecraft. PNG driver: * add creation options to write metadata in TEXT/iTXt chunks * optimize whole image reading with dataset IRasterIO() * Internal libpng: update to 1.2.52 PostgisRaster driver: * Fix read of metadata for tables with multiple raster cols (#5529) Rasterlite driver: * accept space in filename RPFTOC driver: * add tweak for weird relative directory names in the A.TOC file (#5979) VRT driver: * add handling of a shared='0' attribute on to open sources in non-shared mode, and VRT_SHARED_SOURCE config option that can be set to 0 in case the shared attribute isn't there (#5992) * VRT warp: make selection of source overview work (#5688) * VRT warp: expose as many overviews in warped dataset as there are in source dataset, and make warped VRT honour -ovr parameter of gdalwarp (#5688) * make sure nodata value set on VRT raster band is taken into account in statistics computation (#5463) * fix ComputeStatistics() on VRT that are a sub-window of source dataset (#5468) * VRT raw: fix corrupted serialization on Windows (#5531) * implement heuristics to determine if GetMinimum()/GetMaximum() should use the implementation of their sources of not. Can be overridden by setting VRT_MIN_MAX_FROM_SOURCES = YES/NO (#5444) * VRT warp: avoid to warp truncated blocks at right/bottom edges, so that scale computation is correct * fix RasterIO() to be able to fill buffers larger than 2GB (#5700) * fix performance problem when serializing into XML a big number of sources * do not output empty node on VRTDataset and VRTRasterBand elements * fix rounding of output window size on VRTSimpleSource (#5874) * add trick to make relativeToVRT works for a VRT-in-VRT * add more checks to CheckCompatibleForDatasetIO() to avoid issues with overview bands (#5954) * preserves relative links on reserialization of existing VRT (#5985) USGSDEM driver: * take into account horizontal unit = ft in the UTM case (#5819) TIL driver: * fix half pixel shift in geo registration (#5961) WEBP driver: * Lazy uncompressed buffer allocation and optimize band-interleaved IRasterIO() for whole image reading WMS driver: * Add support for ArcGIS server REST API * fix to make GDAL_DEFAULT_WMS_CACHE_PATH configuration option work as expected (#4540) * move the WMS layer name encoding to be done before the sub datasets URLs are created. XYZ driver: * fix back line seeking with datasets that have not the same number of values per lines (#5488) * deal with lines that have missing values (but still regularly spaced) == OGR 2.0 - Overview of Changes == Core: * OGRPolygon::importFromWkt(): fix memleak when importing broken 2.5D polygon * Fix OGRFeature::SetGeometryDirectly() and SetGeomFieldDirectly() to free the passed geometry even if the method fails (#5623) * OGR SQL: Add hstore_get_value(hstore, key) function * OGR SQL: sanitize how we deal with field names expressed as table_name.field_name and "fieldname.with_point_inside". By default, use standard quoting rules, and be tolerant when there's no ambiguity * OGR SQL: support arbitrary boolean expression on ON clause of a JOIN * OGR SQL: accept AS keyword in 'FROM table_name AS alias' clause * OGR SQL: don't consider backslash-doublequote as an escape sequence when inside a single-quoted string literal * Add OGR_API_SPY mechanism (http://www.gdal.org/ograpispy_8h.html) * Make OGRParseDate() recognize ISO 8601 format * ogr_core.h: only ignore -Wfloat-equal for IsInit() and not for the rest of the file and files that include it (#5299) * OGR layer algebra: properly initialize field maps to avoid Valgrind warnings in OGRLayer::Update() (#5778) * Make OGR_F_SetFieldBinary() set OFTString fields, mostly for testing purposes * OGR_G_CreateGeometryFromJson(): attach a WGS84 SRS to the returned geometry if the json object has no 'crs' member (#5922) OGRSpatialReference: * Upgrade to EPSG v8.5 database * Proj.4 import: for HOM, make sure +no_off/no_uoff is preserved, and change default value of gamma parameter to be the same as alpha (#5511) * Proj.4 export: export Aitoff, Winkel I, Winkel II, Winkel-Tripel, Craster, Loximuthal, Quartic Authalic * Adding support for Mercator_Auxiliary_Sphere without AUTHORITY SECTIONS (#3962) * Add QSC (Quadrilateralized_Spherical_Cube) projection, compatible with PROJ 4.9 * Various fixes to put EXTENSION node before AUTHORITY and make it pass Validate() (#5724) * importFromEPSG()/exportToProj4(): avoid precision loss in TOWGS84 parameters, e.g. on Amersfoort / RD EPSG:4289 (https://trac.osgeo.org/proj/ticket/260) * Add OSRCalcInvFlattening() and OSRCalcSemiMinorFromInvFlattening(), and use them in various places (#5858) * Remove deprecated variant of OGRSpatialReference::importFromOzi() (#5932) Utilities: * ogrinfo: add -oo option per RFC 46 * ogrinfo: display dataset and layer metadata. Add -nomdd, -listmdd, -mdd all|domain options, like in gdalinfo. OGR VRT: add dataset and layer metadata support * ogrinfo: add -nocount and -noextent options * ogr2ogr: add -oo and -doo options per RFC 46 * ogr2ogr: add -spat_srs option * ogr2ogr: turn string value to one element list if destination field is stringlist * ogr2ogr: fix problem with SRS when copying layers with multiple geometry columns with different SRS (#5546) * ogr2ogr: add special case for -update and GPKG and input=output * ogr2ogr: when copying a layer that has a source integer/integer64 field with same name as target FID column, avoid creating it into target layer and use its content as the FID value (#5845) * ogr2ogr: in non-append mode, if target layer has a FID creation option, reuse source FID name and source FIDs, unless -unsetFid is specified (#5845) * ogr2ogr: copy source dataset and layer metadata, unless disabled by -nomd. Additional dataset metadata can be set with -mo KEY=VALUE * ogr2ogr: add -ds_transaction to force dataset transactions, mainly for FileGDB driver * ogr2ogr: fix crash with -clipdst when a reprojection fails before (#5973) * ogrlineref: fix project if reper lies on first point or last point of line * ogr_layer_algebra.py: for Update, Clip and Erase, only creates attribute of input layer by default (#5976) Other: * OGR WCTS removed from tree Cross driver changes: * MSSQLSpatial and GPKG: use standardized 'GEOMETRY_NAME' option name. Add GEOMETRY_NAME to SQLite (#5816) * FileGDB and MySQL: use standardized 'FID' option name. SQLite: add a FID layer creation option (#5816) * SQLite, GPKG, PG, PGDump: in a newly created table, allow to create a integer field with same name of FID column (#5845) BNA driver: * fix segfault when calling GetNextFeature() on a write-only layer CartoDB: * add CARTODBFY layer creation option * launder layer and column names by default (#5904) * enable by default batch insertion of features in update mode * on a newly created layer, send new features created by CreateFeature() by chunks of a maximum size of 15 MB (configurable through CARTODB_MAX_CHUNK_SIZE). * implement deferred field creation * support boolean type * register tables with cdb_cartodbfytable() * fix creation of features with Date/DateTime/Time values * fix for multi-user account, and optimization for SQL layers * implement TestCapability() and CreateDataSource() similarly to PostgreSQL, i.e. redirect to Open() in update mode * accept a user column to have the same name of the FID (cartodb_id) * do automatic polygon->multipolygon promotion at creation time * in authenticated mode, retrieve all column information, including spatial info, default value and primary key in one single statement * use integer primary key of tables, when available, to scroll faster among features instead of using OFFSET/LIMIT (#5906) CSV driver: * add optional field type detection with AUTODETECT_TYPE=YES open option * add QUOTED_FIELDS_AS_STRING open option that default to NO. So by default, if AUTODETECT_TYPE=YES, the content of quoted fields will be tested for real, integer,... data types * fix to avoid truncation of WKT geometries to 8000 characters (#5508) * fix segfault when reading allCountries.txt of geonames.org (#5668) * accept space as separator as input/output, and add MERGE_SEPARATOR=YES/NO open option DXF driver: * improve TestCapability(ODsCCreateLayer) FileGDB driver: * add layer creation option to set CONFIGURATION_KEYWORD * avoid error message when failing to import SRS from WKID code (might be an ESRI code for example) * do not reject features with null geometry * use LatestWKID when available (#5638) * avoid emitting error when opening a FileGDB v9, so that OpenFileGDB can be tried to open it, in the case FileGDB is a plugin (#5674) * fix CreateFeature() to work when a esriFieldTypeGlobalID field is not set * report width of string fields (#5806) * add compatibility with FileGDB SDK v1.4 * enable bulk load on newly created layers GeoJSON driver: * implement Date/Time/DateTime field type detection * expose a 'id' object, of type string, directly on Feature object (not in its properties) as a field * add FLATTEN_NESTED_ATTRIBUTES and NESTED_ATTRIBUTE_SEPARATOR open options * TopoJSON: establish layer schema from objects properties (#5870) * implement automatic scrolling through result sets of ArcGIS GeoServices Feature Service (#5943) * accept and skip UTF-8 BOM (#5630) * ESRIJson: parse correctly rings of esriGeometryPolygon objects to build correct Polygon or MultiPolygon (#5538) * avoid truncation of real numbers on reading (#5882) * internal libjson-c: Fix to read floating point numbers in non C locale (#5461) * improve TestCapability(ODsCCreateLayer) * make string comparison for authority name case insensitive so as to recognize lowercase 'epsg' (#4995) * support reading Feature without geometry field GeoRSS driver: * fix to parse ATOM feed documents with atom: namespace (#5871) GME driver: * Added fixes discovered while using v.in.ogr and v.out.ogr in GRASS GML driver: * add XSD=filename open option * add FORCE_SRS_DETECTION, INVERT_AXIS_ORDER_IF_LAT_LONG, CONSIDER_EPSG_AS_URN, READ_MODE, EXPOSE_GML_ID, EXPOSE_FID, DOWNLOAD_SCHEMA and REGISTRY open options * Fix bug that prevented multiple instances of the reader with Xerces backend (#5571) * parse correctly GML geometries whose srsDimension attribute is on top-level geometry element and not on posList (#5606) * add datasource option SRSDIMENSION_LOC=GEOMETRY to be able to write srsDimension attribute on top level geometry element, default on posList unchanged (#5066) * add support for reading layers resulting from a WFS 2.0 join query * read/write top and as DESCRIPTION and NAME metadata items. Also add GML_ID, DESCRIPTION and NAME creation options * support to reader response to CSW GetRecords queries * Fix incorrect geometry cast when reading GML topogeometries (#5715) * VFR: fix ST_Z type (changes) -- list all layers * VFR: include also non-spatial (removed) features (ZaniklePrvky) in type ST_Z (changes) * VFR: use String when 32-bit integer was not wide enough * VFR: add support for UVOH file type * VFR: add missing support for OriginalniHraniceOmpv geometry * VFR: update GFS files to RFC31 (OGR 64bit Integer Fields and FIDs) * update RUIAN GFS files: add missing GMLFeatureClasses to OB type (SpravniObvody, Mop, Momc) * add support for parsing .xsd with a of polygonProperty and multiPolygonProperty * remove wrong case insensitive comparison related to gml_registry.xml use * various fixes to better deal with ArcByCenterPoint() as found in FAA AIXML files * make Expat parser accept trailing nul characters * correctly record path to attribute in case of attribute located on a nested element when .gfs is created with GML_ATTRIBUTES_TO_OGR_FIELDS=YES * fix GML_ATTRIBUTES_TO_OGR_FIELDS=YES to work correctly with xlink:href too (#5970 * make GML_EXPOSE_GML_ID to be honoured on WFS documents GPKG driver: * add support for non-spatial layers via the gdal_aspatial extension (#5521) * add support for creating spatial index * add layer metadata read/write support * implement ST_GeometryType(), GPKG_IsAsisgnable() and ST_SRID() to be compatible with Geometry Type Triggers and SRS ID Triggers Extensions * on creation, use GEOMCOLLECTION (instead of GEOMETRYCOLLECTION) (#5937) * make SELECT expressions passed to ExecuteSQL() be evaluated by SQLite * make it possible to use spatialite 4.2.0 SQL functions * add a 'INDIRECT_SQLITE' dialect that goes through the VirtualOGR mechanism (e.g. for compat with older Spatialite) * allow table names with dash character (#5472) * emit warning when required extensions are not implemented * disable PRAGRAM integrity_check by default, since it can be expensive on big files * read-only support for tables without integer primary key * fix Date and DateTime support * implement TEXT(maxwidth) type in read and creation * implement deferred table creation * fix reporting of geometry type for 2.5D (previous behaviour had the effect to turn to wkbUnknown) * put correct value (1) in gpkg_geometry_columns for 2.5D tables (#5481) * fix component geometry type of 3D MultiGeometries (#5629) * fix GetExtent() crash on layers without extent set in gpkg_contents (#5471) * avoid leak when a table has more than one FID column * accept spatial tables whose geometry field is declared as BLOB * recognize both GeomCollection and GeometryCollection as possible values, until GeoPackage SWG clears what is the official value * escape all column names in SQL (#5520) * accept geometries with Spatialite format, that can be returned with issuing a SQL request using spatialite functions * enable Spatialite 4.3 'amphibious' mode to avoid explicit cast to Spatialite geometries GPSBabel driver: * automatically open .igc files, implement Identify() and add open options * advertize creation option GTM driver: * declare OLCCreateField and OLCSequentialWrite capabilities IDRISI driver: * fix support for multi-ring polygons (#5544) ILI driver: * Use Ili1TransferElement written by ili2c 4.5.5 and newer * Fix crash in polygon geometry reading * Fix reading SURFACE polygons with multiple rings * Fix reading tables with polygon type * Support curve geometries for ILI1 and ILI2. * Add a MODEL open option ISIS3 driver: * fix to recognize IsisCube.Mapping.LatitudeType = Planetocentric (#5717) KML driver: * fix segfault when calling GetNextFeature() on a write-only layer LIBKML driver: * add support for reading gx:MultiTrack * rework libkml singleton factory management (#5775) MITAB driver: * add support for append/update/delete operations on .tab files (#5652) * add support for CreateField() on non empty file, AlterFieldDefn() and DeleteField() for .TAB (#5652) * implement SyncToDisk() for TAB layers (#5652) * convert to use of VSI*L FILE API (#5558) * don't write field width for integer fields in .mif, which is incompatible with MapInfo (#3853) * report OLCCreateField for .mif files (#5477) * fix opening .mif file without .mid file (#5570) * swap StdParallel1 and stdParallel2 if necessary on LCC projections (https://github.com/mapgears/mitab/issues/1) * take into account scale/bounds to properly round coordinates (https://github.com/mapgears/mitab/issues/2) * add MITAB_BOUNDS_FILE configuration option to specify a file with projection bounds (https://github.com/mapgears/mitab/issues/3) * add BOUNDS layer creation option (#5642) * refactor import/export of MIF coordsys to use the TAB code; take into account MITAB_BOUNDS_FILE to add Bounds to the CoordSys string * close polygon rings when reading Region from MIF file (#5614) * fix segfault in CreateFeature() if passing an invalid OGR feature style string (#1209) MSSQLSpatial driver: * Implement SPATIAL_INDEX layer creation option for MSSQL (#5563) * Implement support for WKB geometry upload (#5682) * Fix schema handling in MSSQL driver (#5401) * Fix spatial geometry field handling (#5474) * Bind string fields to unicode string columns in the database (#5239) * Fix recognizing image columns as geometry columns for the select layers. (#5498) * Fix issue when creating non-spatial table (#5696) * Fix to read metadata if the tables are specified in the connection string (#5796) * Fix crash if the tablename is specified in the connection string (#5826) * Include geometry column name in Update statement (#5930) * Implement FID layer creation option (#5816) * Fix issue when removing an MSSQL spatial layer MySQL: * thread-safe initialization of mysql client library (#5528) NAS driver: * implement wfs:update (adds new context 'update' and fields "endet" and "anlass" to "delete" layer). * also assign xlink:href attributes as layer attribute (not only in "alkis_beziehungen" layer; #5372) * fix filtering on OGR_GEOMETRY special field * make chevrons configurable by NAS_INDICATOR OCI driver: * add a ADD_LAYER_GTYPE=YES/NO layer creation option that defaults to YES to enforce a layer geometry type and is used to retrieve the layer geometry type when listing layers (#3754) * Fix FID (multi_load=off, OGRNullFID) - start with 1 (not -1) (#5454) * use VARCHAR2 instead of VARCHAR for unsupported types * Fix "ORA-00972: identifier is too long" error (#5466) * Fix memory leaks (#5599) * Fix creation of date and datetime fields (#5600) * initialize member variable to avoid UpdateLayerExtents() to be called randomly on non spatial tables (#5376) * avoid spatial index to be created each time SyncToDisk() is called * fix memory leak in DeleteLayer(const char*) * fix reading of 2D geometries that were always turned as 3D * in layers returned by ExecuteSQL(), only expose geometry column if there's one * force NLS_NUMERIC_CHARACTERS to ". " (#5709) ODBC driver: * try alternate DSN template for 64bit ODBC * make ODBC driver honour PGEO_DRIVER_TEMPLATE config. option (and also MDB_DRIVER_TEMPLATE in case PGEO_DRIVER_TEMPLATE isn't defined) (#5594) ODS driver: * fix export of OFTDate fields that were exported as string OpenAir driver: * tweak detection logic to read beyond first 10KB bytes when needed (#5975) OpenFileGDB driver: * add compatibility with .gdbtable files bigger than 4 GB (#5615) * support opening files with ConfigurationKeyword=MAX_FILE_SIZE_4GB or MAX_FILE_SIZE_256TB (#5615) * fix occasional write-after-end-of-buffer (#5464) * avoid error message when failing to import SRS from WKID code (might be an ESRI code for example) * fix spatial filter with GeneralPolygon shapes (#5591) * fix for reading GDB with string fields with a default value length > 127 (#5636) * better handling of certain definitions of raster columns * use LatestWKID when available (#5638) * increase accepted size for field description zone up to 10 MB (#5660) * fix ResetReading() on SQL layer with ORDER BY on indexed column (#5669) * add support for non spatial GDB v9 tables (#5673) * improve error reporting when file exists but cannot be opened due to permission problem (#5838) * report width of string fields (#5806) * try to deal more gracefully with inconsistent nValidRecordCount vs nTotalRecordCount values (#5842) * report 25D layer geometry type on FileGDB v9 tables when relevant * optimize sequential reading of sparse layers * avoid warning when opening a00000004.gdbtable * disable feature count optimization with IS NOT NULL on an index column OSM driver: * add mechanism to compute fields from other fields/tags with SQL expressions. Apply it for z_order on lines layer * fix random crash, particularly on MacOSX (#5465) * add CONFIG_FILE, USE_CUSTOM_INDEXING, COMPRESS_NODES, MAX_TMPFILE_SIZE and INTERLEAVED_READING open options PG driver: * use COPY mode by default (unless PG_USE_COPY is set to NO) when inserting features in a newly create table (#5460) * add UNLOGGED=YES/NO layer creation option to create unlogged tables (improved version of patch by Javier Santana, #4708) * implement deferred loading of table list, to optimize ExecuteSQL() (#5450) * implement optimization for spatial table listing for PostGIS 2.x * implement deferred creation of tables to capture all attribute and geometry column creations into a single CREATE TABLE statement (#5547) * change "No field definitions found" from fatal error to debug * when creating a table and filling it, avoid re-reading the table definition from PG system tables (#5495) * better handling of SRS authority name different than EPSG (authority code must still be integral) * fix crash when writing a StringList with 0 element (#5655) * emit errors instead of debug messages when postgres issues an error (#5679) * fix to make ExecuteSQL('CREATE DATABASE foo') work * fix regression that prevented to retrieve more than 500 features from a connection with tables= parameter and on a SQL result layer (#5837) * PG/PGDump: fix truncation of fields to work with multi-byte UTF-8 characters (#5854) PGDump driver: * switch to DROP_TABLE=IF_EXISTS by default (#5627) * fix crash when writing a StringList with 0 element (#5655) PGeo driver: * try alternate DSN template for 64bit ODBC REC driver: Shapefile driver: * add SPATIAL_INDEX layer creation option (#5562) * support .prj files with UTF-8 BOM * fill 'date of last update' header with current time instead of dummy date, and add a DBF_DATE_LAST_UPDATE layer creation option to override this with a fixed date (#3919) * fix reading of shapefiles whose .shx is non conformant (#5608) * fix writing values up to 2^53 in OFTReal fields with 0 decimal places (#5625) * delete implicit FID field as soon as we CreateField a real one * GetExtent(): don't trust extent in header if it contains Not-A-Number values (#5702) * make REPACK compact .shp if SetFeature() is called and changes one geometry size (#5706) * add check not to cut unicode character while cut the string lengnt for field max length during SetFeature * avoid reading whole .shx at open time for /vsicurl/ * add SHAPE_REWIND_ON_WRITE configuration option that can be set to NO to disable correction of ring winding order on write. Useful when dealing with MultiPolygon that are MultiPatch objects in fact (#5888) * Make ENCODING layer creation option proprietary over SHAPE_ENCODING config. option SOSI driver: * remove error noise (#5710) S57 driver: * various compliance fixes in ISO8211 and S57 writer (#5798) * make it possible to set LNAM_REFS=OFF as advertized in the doc SQLite/Spatialite driver: * SQLite/Spatialite: add support for multiple geometry column tables, accordingly with RFC 41 (#5494) * SQLite SQL dialect: Add hstore_get_value(hstore, key) function * remove 'T' suffix when formatting the content of a Date field (#5672) * optimize CreateFeature() when fields can be null or not null from one feature to another one * Spatialite: improve insertion performance by disabling triggers and doing the job ourselves * Fix segmentation fault when executing OGR2SQLITE_Register() when compiling against sqlite 3.8.7 (#5725) * make GetFIDColumn() work when run as first method call (#5781) * emit warning when reading text values in a integer/real field (possible since SQLite has no strong typing) * support reading date/datetime from Julian day floating point representation * SQLite dialect: fix insertion in geometry_columns of table names that are not in upper-case (#6483) SUA driver: * tweak detection logic to read beyond first 10KB bytes when needed (#5975) SXF driver: * Fix SXF file version check (#5456) * Fix wrong Miller Cylindrical projection string * Fix encoding issues (#5647) * Fix extract z value to OGRGeometry * Fix case sensitivity of RSC file VFK driver: * recode also header values * process DKATUZE from header properly * handle also duplicated records * check attribute 'parametry_spojeni' * speed-up GetFeatureCount() * fix reading properties. Escape characters for SQL * change SRS from EPSG 2065 to 5514 * fix reading SBP datablock (fix mismatch when reading from file and db) VRT driver: * do not propagate ignoring of x and y cols of a PointFromColumns to the source layer (#5777) * add an optional 'name' attribute on FID element, so as to be able to force the report of a FID column name even if it is not exposed as a regular field (related to #5845) * handle optional value to specify open options * do not enable passthrough filtering if redefining (#6480) Tiger driver: * Fix potential buffer underflow when providing /vsistdin/ to Tiger driver (#5567) WAsP driver: * added options and changed output precision to match WAsP Map Editor * improve TestCapability(ODsCCreateLayer) WFS driver: * automatically enable paging if WFS 2.0 capabilities report paging support * evaluate SELECT with JOIN on server-side for a Join-capable WFS 2.0 server * add a TRUST_CAPABILITIES_BOUNDS open option, that can be set to YES to trust layer bounds declared in GetCapabilities response, for faster GetExtent() runtime (#4041) * add INVERT_AXIS_ORDER_IF_LAT_LONG, CONSIDER_EPSG_AS_URN and EXPOSE_GML_ID open options * add capability to use spatial functions ST_xxxxx() as server-side filters * add dataset and layer metadata * allow SELECT with several ORDER BY columns * report name of geometry column * Add COOKIE option (#5824) * when parsing a layer schema without geometry from the GML .xsd, do not expose a geometry field at the WFS layer level (#5834) XLSX driver: * fix column numbering when there are more than 26 columns (#5774) XPlane driver: * add support for Taxi Location 1300 record found in V1000 == SWIG Language Bindings == All bindings: * bind GDALGetBandDataset() as Band.GetDataset() * add Feature.GetFieldAsBinary() Java bindings: * Pass eRWFlag to allow both reading or writing. Write was broken in DatasetRasterIO(). * updates to generate maven artifacts Perl bindings: * The breaking changes are described in more detail in swig/perl/Changes-in-the-API-in-2.0. * More comprehensive use of strings as constants (such as capabilities); they are also taken from bindings, which added new ones, and not hard-coded. * New classes (e.g., VSIF, GeoTransform, GeomFieldDefn) and new methods (e.g., constant lists, Dataset::SpatialReference). * Much improved documentation and switch to Doxygen::Filter::Perl. * New test codes. * Errors are confessed with stack trace and often caught earlier with better messages. * Use of attributes is deprecated and methods have been added to replace them. * Multiple geometry fields have necessitated some changes in schema and field related methods. * More support for named parameters (i.e., hashes as arguments). * NoDataValue: set max float if undef is given. * Unit: set '' if undef is given. * Also other changes that will also remove some "use of uninitialized value in subroutine entry" warnings. * Automatic handling of SQL result layers. * Fix issue with index attribute for field meta data (schema) (#5662) * Warn if attempt to create non-integer column for colors. * Remove prefix GCP from GCP swig made attributes. Python bindings: * add optional buf_xsize, buf_ysize and buf_type parameters to Dataset.ReadAsArray() and Dataset.LoadFile(), and use dataset RasterIO for better efficiency * avoid generating Python exception when PyString_FromStringAndSize() fails and GDAL errors as Python exceptions are disabled * Band.ReadRaster() and Dataset.ReadRaster(): clear the buffer in case there are holes in it due to odd spacings specified by the user * Fix hang of Python in case of repeated call to gdal/ogr.UseExceptions() and CE_Warning emitted (#5704) * for Python 2, accept unicode string as argument of Feature.SetField(idx_or_name, value) (#4608) * for Python 2, accept Unicode strings to be passed as key and/or value of the dictionary passed to SetMetadata() (#5833) * fix processing error of ogr_python.i with SWIG 3 (#5795) * NUMPY driver: avoid returning CE_None in GetGeoTransform() when there's no geotransform set (#5801) * Make GetFieldAsBinary() work with OFTString fields * For Python3 compat, make Feature.GetField() use GetFieldAsBinary() if GetFieldAsString() fails (#5811) = GDAL/OGR 1.11.0 Release Notes = == In a nutshell... == * New GDAL drivers: - KRO: read/write support for KRO KOKOR Raw format * New OGR drivers: - CartoDB : read/write support - GME (Google Map Engine) : read/write support - GPKG (GeoPackage): read-write support (vector part of the spec.) - OpenFileGDB: read-only support (no external dependency) - SXF driver: read-only support - WALK : read-only support - WasP : read-write support * Significantly improved drivers: GML, LIBKML * RFC 40: enhanced RAT support (#5129) * RFC 41: multiple geometry fields support * RFC 42: OGR Layer laundered field lookup * RFC 43: add GDALMajorObject::GetMetadataDomainList() (#5275) * RFC 45: GDAL datasets and raster bands as virtual memory mapping * Upgrade to EPSG 8.2 database == New installed files == * bin/ogrlineref * lib/pkgconfig/gdal.pc * gdalcompare.py * data/gml_registry.xml * data/inspire_cp_CadastralParcel.gfs * data/inspire_cp_BasicPropertyUnit.gfs * data/inspire_cp_CadastralBoundary.gfs * data/inspire_cp_CadastralZoning.gfs * data/ruian_vf_ob_v1.gfs * data/ruian_vf_st_v1.gfs * data/ogrvrt.xsd == Backward compatibility issues == == GDAL/OGR 1.11.0 - General Changes == Build(Unix): * add Unix configure support for SOSI * remove pointers to old ver of ingres library files * add --with-libjson-c configure option to build against external libjson-c (>= 0.11) (#4676) * compilation fixes for iOS (#5197, #5198) * update to autoconf 2.69 * add pkg-config gdal.pc (#3470) * configure for FileGDB: add explicit linking to libfgdbunixrtl (requires FileGDB SDK >= 1.2) (#5215); also try .dylib extension (#5221) * fix so that Java installs are found on the MAC to enable the MDB driver (#5267) * fix compilation with recent MySQL versions (5.6 for example) (#5284) * support --with-jp2mrsid with standalone Kakadu with MRSID v8 or later * Fix parallel build in Python bindings (#5346) * PCIDSK: don't link against libjpeg if configured --without-jpeg * Update configure script to pick up ECW JP2 SDK 5.1 (#5390) * add a 'make install' target for the Java bindings (#5424) * add Vagrant configuration Build(Windows): * add option to generate VC project for x64 on makegdal_gen.bat * nmake.opt: add WITH_PDB=1 option to optionally generate .pdb file on Release builds (#5420) * add support for building the OGR SOSI driver as a plugin (#3638) * add support for building the HDF4 driver as plugin (#5294) * add support for MrSID v9 * Remove makegdalXX.bat generated files == GDAL 1.11.0 - Overview of Changes == Port: * vsisubfile: fix Eof() behaviour to be POSIX compliant, so that the shapefile reader can read the last feature when using /vsitar (#5093) * vsicache: fix for 32bit binaries when file size is over 2GB (#5170) * vsicache: add optional nChunkSize and nCacheSize parameters to VSICreateCachedFile() * vsicurl: add CPL_VSIL_CURL_USE_HEAD config option to disable use of CURL HEAD for other services like mapbox (likely lame python http implementations) * vsitar: avoid infinite loop in case of invalid .tar structure * vsizip: fix path separator in CPLFormFilename * vsizip: allow additional extensions listed in CPL_VSIL_ZIP_ALLOWED_EXTENSIONS config option. * vsizip: improve UTF-8 support of filenames inside ZIP file (#5361) * vsizip: fix ZIP64 support * vsigzip: reset EOF flag when doing a Seek() to be POSIX compliant * curl: add .netrc support * Windows CPLGetSymbol(): avoid dialog boxes to pop up when a DLL or one of its dependencies does not exist (#5211) * Add CPLOPrintf() and CPLOvPrintf() functions for easy CPLString formatting * CPLBase64DecodeInPlace() : fix to be robust to malformed base64 strings * CPLQuadTree: add CPLQuadTreeInsertWithBounds() where the pfnGetBounds is not needed. * CPLQuadTree: fix potential infinite recursion when inserting several points with identical coordinates in the mode with limited bucket size * Protect concurrent calls to setlocale() by a mutex (#5366) Core: * RFC 45: GDAL datasets and raster bands as virtual memory mapping * GDALRasterBand::GetHistogram(): ignore nodata values (#4750, #5289) * allow auto loading of drivers to be disabled via config option * PAM .aux.xml and VRT: serialize Z component of a GCP as 'Z' attribute, for consistency, instead of GCPZ that could not be read back previously. In reading code, try reading 'Z' and if not found try 'GCPZ' (#5326) * JPEG2000: Add GDALGeorefPamDataset and GDALJP2AbstractDataset classes and use them in JP2KAK, JP2ECW, JP2OpenJPEG, JPEG2000 and MrSID drivers so that PAM georeferencing consistently overrides internal georeferencing * GDALDataset::IRasterIO(): don't use BlockBasedRasterIO() when INTERLEAVE=PIXEL if the request band count is just 1 * CopyWholeRaster(): make default GDAL_SWATH_SIZE to 1/4 of GDAL_CACHEMAX instead of hard-coded value of 10 MB * don't report empty RAT on GDALGetDefaultRAT() (#5232) * modify GDALGCPsToGeotransform() to do the regression in normalized coordinates to make the math more stable. * expose new GDALComposeGeoTransforms() function. * GDALDefaultOverviews::HaveMaskFile(): avoid fetching .ovr file * JPEG2000: Fix reading georeferencing from some JPEG2000 files with duplicated GeoTIFF JP2Box (#5249) * Cleanup raster block mutex (#5296) * Driver registration: move JPEG2000 (Jasper based) after MrSID JPEG2000 support Algorithms: * warper: fix regression with lanczos resampling when yradius > xradius (#5058) * warper: Make GDALCreateGenImgProjTransformer2() and GDALCreateGenImgProjTransformer3() fail when the creation of the reprojection transformer fails * warper: Fix warping when input pixel size is too close to 0 (#5190) * warper: revise formula of cubic resampling kernel, and a few optimizations (#5209) * warper: added DST_METHOD and support for GCP and TPS dest * warper: add support for DST_METHOD=RPC * warper: fix mode and near resampling corner computation (#5311) * warper: GDALGenImgProjTransform(): don't set panSuccess[i] to 1 in the middle of the function, if an intermediate transform before has set the flag to 0 * warper: fix cutline blending (#5343) * warper: Average/mode kernels: make them less sensitive to numerical precision issues (#5350) * warper: Average/mode kernels: avoid 'holes' when the source coordinates are in a reversed order from the target coordinates (#5433) * warper: provide prototypes and work around strict compiler requirements on some opencl platforms (#5400) * RPC: fix for computation of adfGTFromLL (#5395) * TPS: optimization for GCC x86_64 that make computation about twice faster with huge number of GCPs * TPS: when using Armadillo to solve the coefficients, use solve(A,B) instead of inv(A)xB to faster resolution * TPS: compute direct and inverse transformations in parallel when warping option NUM_THREADS or GDAL_NUM_THREADS config. options are set to > 1 * Geoloc: fix wrong bilinear interpolation in GDALGeoLocTransform() (#5305) * Geoloc: fail transformation of coordinates that is located on a nodata place of the geoloc array * rasterize: preliminary support for MERGE_ALG=ADD for heatmaps * gdal_grid: Add AVX optimized version of GDALGridInverseDistanceToAPower2NoSmoothingNoSearch * fill_nodata: GDALFillNodata(): Fix use of uninitialized memory and integer overflows (#4010, #5203) * rpc: Fix out-of-bounds read in RPC dem cubic interpolation Utilities: * gdalinfo: add -listmdd and -mdd all options (#5275) * gdal_translate: add a -exponent option to be used with -scale * gdal_translate: fix output file naming scheme in gdal_translate -sds (#5119) * gdal_translate: fix logic in detection non-gray color table level (#5245) * gdal_translate: add a -norat option * gdal_translate: don't add 0.1 when -scale is used with a dstmin equal to dstmax (useful to generate a raster with uniform color, i.e. scaleRatio = 0) * gdal_translate: use floor() to compute image coordinates from world coordinates when specifying -projwin (useful when extracting from left or top of upper-left corner, which generate negative image coordinates) (#5367) * gdaltindex: remove annoying warning 'Warning 1: Field location of width 255 truncated to 254' (#5121) * gdaltindex: add -src_srs_name and -src_srs_format to go with MapServer RFC100; add also a -f and -lyr_name options to be able to create a non-shapefile tileindex * gdalwarp: Fix segfault where metadata values were not being nullchecked properly during conflict resolution (#5069) * gdalwarp: honor -s_srs when using cutline (#5081) * gdalwarp: copy nodata values from source to dest if -dstnodata is not given ; add option to not set dest nodata with -dstnodata None (#5087) * gdalwarp: do not return a non-zero exit status for warnings * gdalwarp: prevent from copying statistics metadata (#5319) * gdal_rasterize: set the progress bar to 100% even when there's nothing to do * gdal_grid: add support for different types of geometries (#5341) * gdal_grid: add -z_increase and -z_multiply options * gdaldem: check that value of -z, -s, -az and -alt is numeric * gdalbuildvrt: validate values of -srcnodata and -vrtnodata arguments * gdal2tiles.py: Corrected OpenLayers code to reflect fix to geodetic resolution factor * gdal2tiles.py: add --tmscompatible flag so as to produce 2 tiles at zoom level 0 in geodetic profile * rgb2pct.py: Use python tempfile logic to avoid permissions issues with cwd (#5079) * gdal_edit.py: add a -ro option for drivers refusing to use the dataset in update-mode. * gdal_calc.py: add --allBands options (#5388) * Add vsipreload.cpp that can be compiled as a shared library that can be LD_PRELOAD'ed as an overload of libc to enable VSI Virtual FILE API to be used with binaries using regular libc for I/O * Add the wcs_virtds_params.py sample utility to be able to set the MapServer WCS virtual dataset parameters from a tileindex with rasters of mixed SRS (linked to MapServer RFC100) * gdalcompare.py: move to scripts * gdalcompare.py: ensure image dimensions match * gdal_ls.py: Fix issue with UTF-8 characters Multi driver changes: * JPEG2000 drivers: take into account PixelIsPoint in GeoJP2 boxes, and expose AREA_OR_POINT=Point (#5437) * JP2KAK, JP2ECW, JP2OpenJPEG, JPEG2000 CreateCopy(): take into account AREA_OR_POINT=Point if present to write GeoJP2 box (#5437) AAIGRID: * revert DECIMAL_PRECISION and add SIGNIFICANT_DIGITS to CreateCopy() (#3732) AIGRID: * Turn off errors that can be triggered if the info has no VAT table related with this coverage (#3031) BAG driver: * Recognise falseNorthing=10000000 as UTM South (#5152) DIMAP driver: * fix memleak in error-code path DTED driver: * Speed optimization to be more friendly with CPU cache in GDAL_DTED_SINGLE_BLOCK=YES mode ECW driver: * fix crash in GDALDeregister_ECW() with ECW SDK 5 called from GDALDestroy() (#5214) * fix issue with ECW_CLEVER optimization when nPixelSpace != sizeof eBufDataType (#5262) Envisat driver: * implement more reliable way of extracting GCPs from Meris tie-points (#5423) * add DEM corrections of TP-ADS products when present (#5423) * workaround dateline discontinuity in GCPs so they can be used with GDAL warping transformers (#5423) ERS driver: * fix wrong interpretation of RegistrationCellX/RegistrationCellY (#2612, #3056, #5075) GeoRaster driver: * fix RPC support (#4038) * fix read error when reading from pyramids (#5076) * make regular table and secure file a default for RDT (#5127) * fix error when reading NBIT pyramid levels (#5199) * show the VAT as RAT (#5200) * fix reading and writing of statistics metadata (#5237) * add generate pyramid create options (#5288) * fix incorrect geotransform interpretation when there is no SRS (#5323) GRASS driver: * fix compilation issues for GRASS 7 GRIB driver: * display temperature unit as deg Celcius in metadata (#3606) GTiff driver: * when compiling against internal libtiff, in read-only mode, optimization to avoid fetching the whole Strip/TileCounts and Strip/TileOffsets arrays * add validation of source overview characteristics with COPY_SRC_OVERVIEWS (#5059) * convert invalid TIFFTAG_RESOLUTIONUNIT=0 to 1(Unknown) (#5069) * fix potential issues in gt_citation.cpp / CheckUTM() * upgrade internal libtiff to latest CVS * implement reading and writing of ICC profiles (#5246) * make SetColorInterpretation() round-trip with GetColorInterpreation(); read color interpretation from PAM if it exists (overrides internal tiff color interpretation); set TIFFTAG_PHOTOMETRIC=PHOTOMETRIC_RGB if calling SetColorInterpretation() with R,G,B and no explicit PHOTOMETRIC creation option defined * gt_wkt_srs.cpp: fix compilation with external libgeotiff. The file is dependent of quite a few CPL stuff, don't try to pretend otherwise * implement GetVirtualMemAuto() for some formulations of TIFF files (RFC 45) * fix reading a single-strip TIFF file where the single strip is bigger than 2GB (32bit builds only) (#5403) * look for .tab file before .wld/.tfw GTX driver: * Add nodata support (#4660) HDF4 driver: * Skip "SceneLineNumber" table if present in the list of geolocation fields of ASTER L1A dataset. HDF5 driver: * add support for ODIM H5 georeferencing method (#5032) * set SRS GEOGCS in all cases (reverts r25801 and closes #4160) * support HDF5 NATIVE_SCHAR type, subdatsets without PAM (#5088) * release all opened handles so the file is closed at dataset closing (#5103) * better deal with dimensions of CSK-L1A HDF5 subdatasets (#4227) * avoid segmentation fault when H5Sget_simple_extent_ndims() returns negative value (#5291) HFA driver: * add minimally tested support for u2 and u4 data in basedata * use direct binning for thematic layers and real instead of integer for values (#5066) * add a HFA_COMPRESS_OVR config option to select whether to create compressed overviews (#4866) * fix rewriting of statistics in existing HFA file where base data value is 8-bit (#5175) * implement re-writing existing histogram in HFA file, after raster editing (#5176) * avoid segfaults when creating a Imagine dataset with an invalid WKT (#5258) * expose color columns in RAT as Integer with values in range [0-255] instead of Real with values [0-1] (#5362) * report histogram column as GFU_PixelCount instead of GFU_Generic (#5359) * ensure histogram column written as float for HFA when using RAT API (#5382) Idrisi driver: * Improve coordinate system handling and min/max statistics (#4980) IRIS driver: * add height information on bands; rename dataset metadata item CAPPI_HEIGHT --> CAPPI_BOTTOM_HEIGHT (#5104) * IRIS: add support for two bytes data (#5431) JP2ECW driver: * fix problem with JP2 write with SDK v5 * fix issue with ECW_CLEVER optimization when nPixelSpace != sizeof eBufDataType (#5262) * avoid writing dummy GeoJP2 box when source dataset has no georeferencing (#5306) JP2KAK driver: * preliminary support for Kakadu V7.x * fix creation of unsigned int16 with reversible compression (#4050) * on Windows, use VSI cache for I/O by default, instead Kakadu own I/O layer * remove extension from 12bit to 16bit (#5328) JP2OpenJPEG driver: * avoid 'Empty SOT marker detected: Psot=12.' warning to be repeated several times * add support for encoding GCPs in a GeoJP2 box (#5279) * avoid writing dummy GeoJP2 box when source dataset has no georeferencing (#5306) JPEG driver: * add autodetection of bitmasks that are msb ordered (#5102) * avoid memory leak when GDALOpen'ing() a JPEG through a http:// URL, and make it possible to access its overviews * return YCbCrK raw data for YCbCrK JPEG in GDAL_JPEG_TO_RGB = NO mode (instead of CMYK as before) (#5097) * implement reading and writing of ICC profiles (#5246) * internal libjpeg: apply patch for CVE-2013-6629 * allow fallback to PAM to read GCPs * give priority to PAM GeoTransform if it exists and other source of geotransform (.wld, .tab) also exists (#5352) KMLSuperOverlay driver: * recognize an alternate structure for raster KMZ file made of a single doc.kml and tiles whose name pattern is kml_image_L{level}_{j}_{i}.{png|jpg} * fix horrible speed performance in Open() (#5094) * fix crash at dataset closing and inability to read some big PNG tiles (#5154) * fix to generate files validating against OGC KML 2.2 schema * put Style into conformity with ATC 7 * remove Region in root KML (ATC 41) * add NAME and DESCRIPTION creation options; read them back as metadata * add ALTITUDE and ALTITUDEMODE creation options * directly write into .kmz file (instead of in temporary location) * correctly write directories entry in .kmz file * add progress callback L1B driver: * report correct values for GCP (#2403) * report more GCPS than before * implement geolocation array * add fetching of record metadata in .csv file * add subdatasets with solar zenith angles, cloud coverage * recognize NOAA-9/14 datasets whose dataset name in TBM header is encoded in EBCDIC and not in ASCII (#2848) * support opening a few NOAA <= 9 datasets that have no dataset name in the TBM header LCP driver: * better handling of projections (#3255) * add CreateCopy() (#5172) MBTiles driver: * add write support * avoid failure when there's no tile at the center of the maximum zoom level (#5278) * add capability to open /vsicurl/https:// signed AWS S3 URLs MEM driver: * Create(): use calloc() instead of malloc()+memset() for faster creation of huge in-memory datasets NetCDF driver: * fix to read netcdf-4 files with UBYTE data (#5053) * fix reading large netcdf-4 files with chunking and DEFLATE compression * fix netcdf chunking when creating file with > 2 dims ; add CHUNKING creation option (#5082 ) * fix duplicate nodata metadata when using CreateCopy() (#5084) * fix copying large metadata in netcdf driver (#5113) * fix netcdf geotransform detection (#5114) * fix netcdf driver irregular grids management (#5118 and #4513) * only call nc_close on a valid netcdf id when closing dataset * try and identify .grd (and .nc3) files in netcdf-4 format (#5291), so they are identified before the hdf5 driver NITF driver: * fix to support reading horizontal and/or vertical mono-block uncompressed images, even when the number of columns is <= 8192 (#3263) * update NITF Series list with new entries from MIL-STD-2411_1_CHG-3.pdf (#5353) * allow JP2KAK to be used as the JPEG2000 compression engine in the CreateCopy() case (#5386) PDF driver: * Avoid reporting a Poppler error as a GDAL error on some newer USGS GeoPDF files (#5201) * PDF writing: automatically adjust DPI in case the page dimension exceeds the 14400 maximum value (in user units) allowed by Acrobat (#5412) PDS driver: * Parse correctly MISSING_CONSTANT = 16#FF7FFFFB# as a IEEE754 single precision float expressed in hexadecimal; add support for ENCODING_TYPE = ZIP (data file compressed in a ZIP); recognize IMAGE_MAP_PROJECTION as an object included in UNCOMPRESSED_FILE object (#3939) PNG driver: * Implement reading and writing of ICC profiles (#5246) PostgisRaster driver: * Speed-up dataset opening (#5046). * Multi-tile multi-band caching added. * Smarter use of the information advertized in raster_columns view. * Avoid full table scan in situations without PKID/GIST indices. * Use of quadtree. Rasdaman driver: * caching of tiles for datasets with more than one band (#5298) * connections are now kept for a whole session (#5298) * fixing connection-string regex (#5298) * fixing possible memory leaks (#5298) Rasterlite driver: * fix resolution check typo in rasterlite driver Raw drivers: * implement GetVirtualMemAuto() (RFC 45) * IRasterIO(): add special behaviour to avoid going to block based IO when the dataset has INTERLEAVE=PIXEL and is eligible to direct I/O access pattern * allow direct I/O access even if a small proportion of scanlines are loaded (improve QGIS use case where the overview display will load sparse scanlines, which would prevent direct I/O at full resolution afterwards) * fix optimized RasterIO() when doing sub-sampling with non standard buffer pixel offset (#5438) RMF driver: * fix decompression of 24-bit RMF DEM (#5268) RPFTOC driver: * fix potential crash on some datasets when selecting the color palette (#5345) SAGA driver: * add read/write support for .prj files (#5316) SRP driver: * read TRANSH01.THF file to establish subdatasets (#5297) VRT driver: * Implement non-linear scaling with a power function (addition of Exponent, SrcMin, SrcMax, DstMin, DstMax sub-elements in ) * Preserve 64bit integer image offsets (#5086) * Make sure that VRTSourcedRasterBand::AddMaskBandSource() takes into account specified window (#5120) * Make GDALAutoCreateWarpedVRT() return NULL when GDALSuggestedWarpOutput() fails * VRTDataset::IRasterIO(): use source DatasetRasterIO even if band count is 1 * VRTWarped: avoid setting up relative paths for things that aren't file-like * make relativeToVRT=1 work with NITF_IM:, NETCDF:, HDF5:, RASTERLITE: WCS driver: * ensure C locale is enforced before parsing floating point values WMS driver: * accept 'WMS:http://server/?SRS=EPSG:XXXX' syntax to select the preferred SRS in which to fetch layers * CPLHTTPFetchMulti(): avoid doing a timeout-only select when there are no file descriptor to wait on (can happen when doing a file:// URL) * allow cache location to be specified with GDAL_DEFAULT_WMS_CACHE_PATH configuration option if not provided in the XML (#4540) * Update to be able to understand slight changes in formatting of JSon output of ArcGIS mapserver protocol XYZ driver: * accept datasets that have missing values at beginning and/or end of lines, such as MNT250_L93_FRANCE.XYZ * fix detection when there are only integral values with comma field separator * reopen with 'rb' flags for Windows happyness == OGR 1.11.0 - Overview of Changes == Core: * GEOS support: require GEOS >= 3.1.0 and use the _r API of GEOS to avoid issues with the global GEOS error handlers * exportToWkb(): ISO WKB generation with wkbVariant option (#5330) * geocoding: when getting several answers from server for a query, report geometries on second, third, etc.. feature, and not only first one (#5057) * allow auto loading of drivers to be disabled via config option * remove obsolete OGRGeometryFactory::getGEOSGeometryFactory() * OGRGeometryFactory::organizePolygons() in DEFAULT method: fix a case with 2 outer rings that are touching by the first point of the smallest one * OGRGeometryFactory::organizePolygons(): optimization in ONLY_CCW case * OGRGeometryFactory::organizePolygons(): Add an experimental mode : CCW_INNER_JUST_AFTER_CW_OUTER * OGRLineString::segmentize() : do not set 0 as z for interpolated points, but the z from the previous point * OGRLineString::setNumPoints(): add an optional argument to avoid zeroing the arrays * Add OGRLineString::setZ() * Add OGRLineString::Project() and OGRLineString::getSubline() * OGRPolygon: add stealExteriorRing() and stealInteriorRing(int iRing) * OGRLinearRing::isClockwise(): optimizations and make it work in a degenerated case when a vertex is used several times in the vertex list (#5342) * OGRLinearRing::isPointOnRingBoundary() : optimizations and take into account bTestEnvelope * Add OGR_G_SetPointCount and OGR_G_SetPoints functions to API C (#5357) * OGREnvelope3D::Contains(): fix incorrect test * Layer algebra: fix handling of method field mapping to output fields when output fields are precreated (#5089) * Layer algebra: when an error condition is skipped, call CPLErrorReset() (#5269) * OGRLayer::GetFeature(): make sure that the behaviour is not influenced by attribute or spatial filters in the generic implementation; upgrade OGDI, PG, MySQL, MSSQLSpatial, OCI, SDE, PGeo, ODBC, WALK, IDB, SQLite and Ingres driver (#5309) * introduce OGRLayer::FindFieldIndex() / OGR_L_FindFieldIndex() to lookup potentially laundered field names (RFC 42) * OGR SQL: upgrade to support RFC 41 (multiple geometry fields) * OGR SQL: more stricter checks * OGR SQL: make parsing error report a useful hint where the syntax error occurred * OGR SQL: fix thread-safety of swq_op_registrar::GetOperator() (#5196) * OGR SQL: support not explicitly specifying AS keyword for aliasing a column spec * OGR SQL: don't call CONCAT(a_column ...) or SUBSTR(a_column ...) as a_column * OGR SQL: validate that arguments of MAX, MIN, AVG, SUM, COUNT are columns and not any expression since this is not supported * OGR SQL: make AVG field definition a OFTReal * OGR SQL: implement MIN(), MAX() and AVG() on a date (#5333) * OGR SQL: fix SELECT * on a layer with a field that has a dot character (#5379) * SQL SQLITE dialect: Make it available to all OGR drivers that have a specialized ExecuteSQL() implementation OGRSpatialReference: * Upgrade to EPSG 8.2 database * identify LCC_2SP instead of LCC_1SP if lat_0==lat_1 and lat_2 is present (#5191) * add a variety of linear units to proj4 parsing (#5370) * Fix crash in CleanupESRIDatumMappingTable() if it is called twice (#5090) * fix order of AXIS and UNIT nodes in a VERT_CS node (#5105) * ecw_cs.wkt: add missing TOWGS84[-168,-60,320,0,0,0,0] to NTF datum (#5145) * fix OGRSpatialReference::importFromProj4() to work with non-C locale (#5147) * morph central_latitude to latitude_of_origin in morphFromESRI() (#3191) * OGRProj4CT: avoid using proj when the 2 projections are actually identical (#5188) * add sanity checks in OGR_SRSNode::importFromWkt() (#5193) * VERT_CS: when importing from proj.4 put AXIS node after UNIT; COMPD_CS: when importing from EPSG:x+y, set a more meaningful name for the COMPD_CS node * OGRSpatialReference::Validate() : in addition to hand-validation, use WKT grammar from OGC 01-009 CT * preserve authority when importing +init=auth_name:auth_code (e.g. +init=IGNF:LAMB93) Utilities: * ogrlineref: new utility to deal with linear geometries. * ogrinfo: upgrade to support RFC 41 (multiple geometry fields) * ogr2ogr: upgrade to support RFC 41 (multiple geometry fields) * ogr2ogr: bump default value for -gt from 200 to 20000 (#5391) * ogr2ogr: add -addfields option to add new fields found in a source layer into an existing layer ; add -unsetFieldWidth option to unset field with and precision; add -dim layer_dim option to force the coordinate dimension of geometries to match the one of the layer geometry type * ogr2ogr: Check that -t_srs is also specified when -s_srs is specified * ogr2ogr: add an explicit error message to report FID of feature that couldn't be inserted when CreateFeature() fails * ogr2ogr: make relaxed lookup optional and add a switch -relaxedFieldNameMatch to allow it (RFC 42) * ogr2ogr: make sure that the progress bar reaches 100% when converting OSM * ogr2ogr: make sure that target dataset is properly closed when a CreateFeature() fails (so that truncated shapefiles have their header file properly updated) * ogr_dispatch.py: Sample Python script to dispatch features into layers according to the value of some fields or the geometry type * ogrinfo.py: sync with ogrinfo (RFC 41) * ogr2ogr.py: port -nlt PROMOTE_TO_MULTI option from ogr2ogr.cpp (#5139) CSV driver: * avoid erroneously reset of file content when opening in update mode a file without header (#5161) * upgrade to support RFC 41 in read/write (multiple geometry fields) * allow backslash doublequote to load (#5318) DGN driver: * DGN writing: added polygon inner ring (holes) writing and MSLink writing (#5381) DXF driver: * fix writing of 25D linestring where z is not constant (#5210) * fix writing of POLYLINE objects (#5217, #5210) * accept reading files starting with a TABLES section (#5307) * support reading 3DFACE and SOLID (#5380) entities * fix an error when processing clockwise circle arc (#5182) * avoid building an invalid polygon when edges cannot be reassembled: turn it into a multilinestring * use CPLAtof() instead of atof() to avoid issues with locales * fix linear approximation of circular and elliptic arc in HATCH boundaries (#5182) DWG driver: * add support for reading AcDb3dPolyline (#5260) * fix linear approximation of circular and elliptic arc in HATCH boundaries (#5182) FileGDB driver: * implement IgnoreFields API to speed-up a bit the conversion of a sub-set of fields when there's a huge amount of them (e.g. Tiger database). * when writing of an attribute, use size in bytes (#5192) * implement ref counting of the FileGDB SDK API' Geodatabase* object to avoid issues on Linux 64bit with interleaved opening and closing of databases (#4270) * honour update flag to determine which operations are allowed or not * add a driver global mutex to protect all calls as the FileGDB API SDK is not thread-safe at all * add a COLUMN_TYPES layer creation option to override default column types; support reading/writing XML column types * optimize GetFeatureCount() and GetExtent() when there are filters set * set the default width for string fields to 65536. The width can be configured with the FGDB_STRING_WIDTH configuration option * fix creation and writing of Binary fields; enable reading * add a CREATE_MULTIPATCH creation option FME driver: * fix Linux compilation GeoJSON driver: * recognize alternate formats such as the ones of https://code.google.com/p/election-maps/ * add read support for TopoJSON * upgrade internal libjson-c to json-c 0.11 (#4676) * report integer values that are int64 as strings * add 3d support to esri geojson reader (#5219) * be less strict on looking for esri field type tag (#5219) * fix sometimes incorrect result (significant digit lost...) when using -lco COORDINATE_PRECISION=0 * fix handling of huge coordinates when writing (#5377) GeoRSS driver: * advertize OLCCreateField capability GFT driver: * switch http to https for the oauth2 link to improve security GML driver: * add support for multiple geometry columns (RFC 41) * add support for reading Finnish National Land Survey Topographic data (MTK GML) * add support for support Finnish NLS cadastral data and Inspire cadastral data. * add support for Czech RUIAN VFR format * add data/gml_registry.xml file to associate feature types with schemas. * extend .gfs syntax to be able to fetch OGR fields from XML attributes. * extend .gfs syntax to support multiple geometry columns, and define a geometry property name * autodiscover all XML attributes as OGR fields when creating .gfs file if GML_ATTRIBUTES_TO_OGR_FIELDS is set to YES (#5418) * allow the in .gfs to have several components that give the full XML path * fix writing of .xsd file to avoid fid/gml_id being written as regular fields (#5142) * fix writing of global srsName attribute on the global boundedBy.Envelope when all layers have same SRS (#5143) * support for writing .gml/.xsd with fields of type StringList, RealList, IntegerList and support for parsing such .xsd files * when writing .xsd for a datasource that has fields of type StringList, RealList or IntegerList, advertize SF-1 profile in the .XSD schema * recognize xsd:boolean in XSD parsing and map it to String (#5384) * add STRIP_PREFIX and WRITE_FEATURE_BOUNDED_BY dataset creation option to help minimizing the size of GML files * don't write top in GML files with multiple layers of different SRS * fix segfault when reading a GML file with huge coordinates (#5148) * avoid opening our own .xsd files as valid datasources (#5149) * make driver thread-safe with Xerces * open successfully GML datasources with 0 layers (#249, #5205) * fix tweaking of DescribeFeatureType requests * support reading WFS 2.0 GetFeature documents with wfs:FeatureCollection as a wfs:member of the top wfs:FeatureCollection * fix for crash on certain xlink:href with GML_SKIP_RESOLVE_ELEMS=NONE (#5417) * GML geometry: fix duplicated points in GML_FACE_HOLE_NEGATIVE=YES mode (TopoSurface) (#5230) * GML geometry: accept CompositeSurface as a child of surfaceMembers (#5369) * GML geometry: join multilinestrings to linestrings in rings * GML geometry: correctly deal with MultiSurface of Surface of PolygonPatch where a PolygonPatch has only interior ring(s) and no exterior ring (#5421) * GML geometry: accept formulations of 'MULTIPOINT EMPTY, MULTILINESTRING EMPTY, MULTIPOLYGON EMPTY and GEOMETRYCOLLECTION EMPTY that are valid GML 3 (and accepted by PostGIS) * GML geometry: make use of cs, ts and decimal attributes of (deprecated) gml:coordinates element * GML geometry: accept XML header and comments GPX driver: * advertize OLCCreateField capability ILI driver: * add support for multiple geometry columns (RFC 41) * use IlisMeta model reader/writer instead of IOM * add layers for surface and area geometries KML driver: * output KML that validates the ogckml22.xsd schema by placing elements under the level (#5068) * in writing mode, avoid defining an extending schema for the name and description fields (related to #5208) LIBKML driver: * various checks, fixes and improvements related to OGC KML 2.2 Abstract Test Suite * add support for reading as a LINESTRING (#5095) * add support for writing and reading * add support for writing atom:author, atom:link, phonenumber, Region, ScreenOverlay, 3D model, StyleMap * add support for reading and generating Camera object * add layer creation options to generate a LookAt element at layer level * if UPDATE_TARGETHREF dataset creation option is defined, a NetworkLinkControl/Update document will be created * add dataset creation options to generate a NetworkLinkControl element * add dataset and layer creation options LISTSTYLE_ICON_HREF and LISTSTYLE_TYPE * add support for writing a NetworkLink * add support for creating PhotoOverlay objects * add support for creating BalloonStyle elements * offer LIBKML_USE_SIMPLEFIELD configuration option can be set to NO to use Data element instead of SimpleField * add layer creation option FOLDER to optionally write layers as Folder instead of Document * add dataset and layer creation options NAME, VISIBILITY, OPEN, SNIPPET and DESCRIPTION * workaround bugs in pretty serializers * when writing a .kmz file, put layers .kml docs into a layers/ subdirectory * fix mem leaks, and use after free in kml2FeatureDef() (#5240) * create document with default namespace set to http://www.opengis.net/kml/2.2 * when writing, consider empty strings as unset (useful when converting from CSV) * don't write empty