.. _mapcache_caches:

*****************************************************************************
Cache Types
*****************************************************************************

:Author: Thomas Bonfort
:Contact: tbonfort at terriscope.fr

This document details the different cache backends that can be used to store tiles

.. _mapcache_cache_disk:

Disk Caches
--------------------------------------------------------------------------------

The disk based cache is the simplest cache to configure, and the one with the
the fastest access to existing tiles. It is ideal for small tile repositories,
but may cause trouble for sites hosting millions of tiles, as the number of 
files or directory may rapidly overcome the capabilities of the underlying
filesystem. Additionaly, the block size chosen for the filesystem must closely
match the mean size of a stored tile: ideally, any given tile should just fit
inside a filesystem block, so as not to waste storage space inside each block,
and not have to use up multiple blocks per tile.

The location of the files/directories has to be readable and writable by the
user running the tile server.


There are two types of disk caches, that create a different hierarchy of files:

Default Structure
================================================================================

The default disk cache stores tiles in a structure nearly identical to the
file/directory hierarchy used by TileCache. The only change is that a top level
directory corresponding to the name of the grid is added (as MapCache supports
multiple grids per tileset)

This cache is capable of detecting blank (i.e. uniform color) tiles and using
a symbolic link to a single blank tile to gain disk space.


.. code-block:: xml
   
   <cache name="disk" type="disk">
      <base>/tmp</base>
      <symlink_blank/>
   </cache>

The two only configuration keys are the root directory where the tiles will be
stored, and a key to activate the symbolic linking of blank tiles


Template Structure
================================================================================

The template based disk cache allows you to create (or reuse an existing) tile
structure that you define in advance. The <template> parameter takes a string
argument where various template entries will be replaced at runtime by the correct
value for each tile to store.

.. code-block:: xml
   
   <cache name="tmpl" type="disk">
      <!-- template

          string template that will be used to map a tile (by tileset, grid name, dimension,
          format, x, y, and z) to a filename on the filesystem.
          the following replacements are performed:
          - {tileset} : the tileset name
          - {grid} : the grid name
          - {dim} : a string that concatenates the tile's dimension
          - {ext} : the filename extension for the tile's image format
          - {x},{y},{z} : the tile x,y,z values
          - {inv_x}, {inv_y}, {inv_z} : inverted x,y,z values (inv_x = level->maxx - x - 1). This
               is mainly used to support grids where one axis is inverted (e.g. the google schema)
               and you want to create on offline cache.

         * note that this type of cache does not support blank-tile detection and symlinking.
      
         * warning: it is up to you to make sure that the template you chose creates a unique
           filename for your given tilesets. e.g. do not ommit the {grid} parameter if your
           tilesets reference multiple grids. Failure to do so will result in filename
           collisions ! 

      -->
      <template>/tmp/template-test/{tileset}#{grid}#{dim}/{z}/{x}/{y}.{ext}</template>
   </cache>


.. _mapcache_cache_bdb:

BerkeleyDB Caches
--------------------------------------------------------------------------------

The BerkeleyDB cache backend stores tiles in a key-value flat-file database.
and therefore does not have the disadvantages of disk caches with regards to
the number of files stored on the filesystem. As the image blobs are stored
contiguously, the block size chosen for the filesystem has no influence on the
storage capacity of the volume.

Note that for a given bdb cache, only a single database file is created, which
will store the tiles of its associated tilesets (i.e. there is not a database
file created per tileset, grid, and or dimension). If you need to store different
tilesets to different files, then use multiple dbd cache entries. It is not
possible to use multiple database files for tileset grids or dimensions.

The berkeleyDB based caches are a bit faster than the disk based caches during
reads, but may be a bit slower during concurrent writes if a high number of
threads all try to insert new tiles concurrently.

.. code-block:: xml

   <cache name="bdb" type="bdb">
      <!-- base (required)
         absolute filesystem path where the berkeley db database file is to be stored.
         this directory must exist, and be writable
      -->
      <base>/tmp/foo/</base>
      
      <!-- key_template (optional)
         string template used to create the key for a tile entry in the database.
         defaults to the value below. you should include {tileset}, {grid} and {dim} here
         unless you know what you are doing, or you will end up with mixed tiles
      <key_template>{tileset}-{grid}-{dim}-{z}-{y}-{x}.{ext}</key_template>
      -->
   </cache>


.. _mapcache_cache_sqlite:

Sqlite Caches
--------------------------------------------------------------------------------

There are three different sqlite caches that vary by the database schema they
create and query. Sqlite caches have the advantage that they store tiles as
blobs inside a single database file, and therefore do not have the
disadvantages of disk caches with regards to the number of files stored. As the
image blobs are stored contiguously, the block size chosen for the filesystem
has no influence on the storage capacity of he volume.

The sqlite based caches are a bit slower than the disk based caches, and may
have write-locking issues at seed time if a high number of threads all try to
insert new tiles concurrently.


Default Schema
================================================================================

There are two different configuration options for sqlite caches, which drive how
the database files are named on the filesystem for each tileset. You can either
supply a base directory, where multiple database files will be created, or 
supply a template that will be populated with the name of the tileset and grid.

Note 

.. code-block:: xml

   <cache name="sqlite" type="sqlite3">
      <base>/tmp/sqlitecachedir</base>
   </cache>


.. code-block:: xml
   
   <cache name="sqlitetemplate" type="sqlite3">
      <dbname_template>/tmp/{tileset}-{grid}.db</dbname_template>
   </cache>


The default sqlite schema is created by the call

.. code-block:: sql

   create table tiles(x integer, y integer, z integer, data blob, dim text, ctime datetime, atime datetime, hitcount integer default 0, primary key(x,y,z,dim))



Custom Schema
================================================================================

This cache can use any database schema, it is up to you to supply the SQL that
will be exectuted to select or insert a new tile.

This cache type is not fully implemented yet (there is no way to configure it
from the mapcache.xml configuration file yet)


MBTiles Schema
================================================================================

This cache type is a shortcut to the previous custom schema sqlite cache, with
pre-populated SQL queries that correspond to the MBTiles specification.

Note that most mbtiles file you download cannot be populated on demand as the
spec-mandated table names are actually views and not simple tables.


.. code-block:: xml
   
   <cache name="mbtiles" type="mbtiles">
      <dbname_template>/Users/XXX/Documents/MapBox/tiles/natural-earth-1.mbtiles</dbname_template>
   </cache>


.. _mapcache_cache_memcache:

Memcache Caches
--------------------------------------------------------------------------------

This cache type stores tile to an external memcached server running on the local
machine or accessible on the network. This cache type has the advantage that 
memcached takes care of expiring tiles, so the size of the cache will never
exceed what has been configured in the memcache instance.

Memcache support requires a rather recent version of the apr-util library. Note
that under very high loads (usually only atainable on synthetic benchmarks on
localhost), the memcache implementation of apr-util may fail and start dropping
connections for some intervals of time before coming back online afterwards.

You can add multiple <server> entries.


.. code-block:: xml

   <cache name="memcache" type="memcache">
      <server>
         <host>localhost</host>
         <port>11211</port>
      </server>
   </cache>

.. note::
   Tiles stored in memcache backends are configured to expire in 1 day by
   default. This can be overriden at the tileset level with the <auto_expire>
   keyword.



.. _mapcache_cache_tiff:

(Geo)TIFF Caches
--------------------------------------------------------------------------------

TIFF caches are the most recent addition to the family of caches, and use the 
internal tile structure of the TIFF specification to access tile data. Tiles can
be stored in JPEG only (TIFF does not support PNG tiles).

As a single tiff file may contain many tiles, there is a drastic reduction in
the number of files that have to be stored on the filesystem, which solves the
major shortcomings of the disk cache. Another advantage is that the same TIFF
files can be used by programs or WMS servers that only understand regular GIS
raster formats, and be served up with high performance for tile access.

The TIFF cache should be considered read-only for the time being. Write access
is already possible but should be considered experimental as there might be
some file corruption issues, notably on network file-systems. Note that until
all the tiles in a given tiff file have been seeded/created, the TIFF file is
said to be "sparse" in the sense that it is missing a number of jpeg tiles. As
such most non-gdal based programs will have problems opening these incomplete
files.

Note that the tiff tile structure must
exactly match the structure of the grid used by the tileset, and the tif file
names must follow strict naming rules.

There are three mandatory parameters to configure a TIFF cache:

- <xcount> the number of tiles stored along the x (horizontal) direction of
  the TIFF file.
- <ycount> the number of tiles stored along the y (vertical) direction of
  the TIFF file.
- <template> the template to use when looking up a TIFF file name given the
  x,y,z of the requested tile

  - {x} is replaced by the x value of the leftmost tile inside the TIFF file
    containing the requested tile
  - {inv_x} is replaced by the x value of the rightmost tile.
  - {y} is replaced by the y value of the bottommost tile.
  - {inv_y} is replaced by the y value of the topmost tile.
  - {div_x} is replaced by the index of the TIFF file starting from the left
    of the grid (i.e. {div_x} = {x}/<countx>)
  - {inv_div_x} same as {div_x} but starting from the right.
  - {div_y} is replaced by the index of the TIFF file starting from the bottom
    of the grid (i.e. {div_y} = {y}/<county>)
  - {inv_div_y} same as {div_y} but starting from the top.

  Note that {inv_x} and {inv_div_x} will probably be rarely used, whereas
  {inv_y} and {inv_div_y} will find some usage for people who prefer to index
  their TIFF files from top to bottom rather than bottom to top.

An additional optional parameter defines which JPEG compression should be 
applied to the tiles when saved into the TIFF file:

- <format> the name of the (jpeg) format to use

.. seealso:: :ref:`mapcache_jpeg_format`

.. code-block:: xml

   <cache name="tiff" type="tiff">
      <template>/data/tiffs/{tileset}/{grid}/L{z}/R{inv_y}/C{x}.tif</template>
      <xcount>64</xcount>
      <ycount>64</ycount>
      <format>myjpeg</format>
   </cache>

In this example, assuming a grid using 256x256 tiles, the files that are read
to load the tiles are tiled TIFFs with jpeg compression, who's size are
16384x16384. The number of files to store on disk is thus reduced 4096 times
compared to the basic disk cache.


If compiled with GeoTiff and write support, MapCache will add referencing
information to the TIFF files it creates, so that the TIFF files can be used
in any geotiff- enabled software. Write support does not produce full geotiffs
with the definition of the projection used, but only the pixel scale and
tie-points, i.e.  what is usually found in .tfw files.

For reference, here is the gdalinfo output on a TIFF file created by MapCache
when compiled with geotiff support:

::

   LOCAL_CS["unnamed",
       UNIT["metre",1,
           AUTHORITY["EPSG","9001"]]]
   Origin = (-20037508.342789247632027,20037508.342789247632027)
   Pixel Size = (156543.033928040997125,-156543.033928040997125)
   Metadata:
     AREA_OR_POINT=Area
   Image Structure Metadata:
     COMPRESSION=YCbCr JPEG
     INTERLEAVE=PIXEL
     SOURCE_COLOR_SPACE=YCbCr
   Corner Coordinates:
   Upper Left  (-20037508.343,20037508.343) 
   Lower Left  (-20037508.343,-20037508.343) 
   Upper Right (20037508.343,20037508.343) 
   Lower Right (20037508.343,-20037508.343) 
   Center      (   0.0000000,   0.0000000)