Details Ticket 3974


Comment | Reply | Take | Resolve


Serial Number 3974
Subject r.mapcalc/libgis failure with LFS enabled
Area grass6
Queue grass
Requestors flor@itc.it,merler@itc.it,neteler@itc.it
Owner none
Status open
Last User Contact Wed Jan 11 19:47:11 2006 (3 yr ago)
Current Priority 70
Final Priority 70
Due No date assigned
Last Action Thu Jan 12 22:20:55 2006 (3 yr ago)
Created Tue Jan 10 12:16:19 2006 (3 yr ago)

Transaction History Ticket 3974


Tue, Jan 10 2006 12:16:19    Request created by guest  
Subject: r.mapcalc/libgis failure with LFS enabled

Platform: GNU/Linux/x86_64
grass obtained from: CVS
grass binary for platform: Compiled from Sources

Hi,

we have discovered some severe problems in the GRASS raster map
library (within libgis) in certain cases. The problem was first
identified in August/December 2005 (see other mails). We really
need to fix it.

For a test, I have generated a XY location with 3x3 pixels
and generated a simple map:

GRASS 6.1.cvs (xy):~ > r.mapcalc "one=1"
 100%

GRASS 6.1.cvs (xy):~ > r.univar one
Processing ..  100%

total null and non-null cells: 9
total null cells: 0

Of the non-null cells:
----------------------
n: 9
minimum: 99
maximum: 156
range: 57
mean: 125
standard deviation: 23.5372
variance: 554
variation coefficient: 18.8298 %
sum: 1125


GRASS 6.1.cvs (xy):~ > r.out.ascii one
north: 3
south: 0
east: 3
west: 0
rows: 3
cols: 3
120 156 99
120 156 99
120 156 99

My proposal is to add some tests to libgis/ to verify
if the data are wrongly written or read or both.

Here some tests:
* Machine: 64bit PENTIUM, RHEL4, gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)
- 64bit enabled
  - LFS enabled
  -> above problems

* Machine: 64bit PENTIUM, RHEL4, gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)
- 64bit enabled
  - LFS disabled
  -> maps generated *here* are read correctly
  -> maps generated with LFS enabled are read wrongly (so write error with LFS
enabled?)

* Machine: 32bit Mandriva2006
  - LFS enabled
  Here various raster modules crash with LFS enabled
  (r.sun, r.mapcalc + d.rast - error reading compressed row)

Apparently something is wrong with LFS (off_t issues?).
Is anyone able to write some low level test for libgis which
will be performed during compilation (as already exists for
libvect)?

Markus
Tue, Jan 10 2006 14:27:38    Mail sent by mneteler  
An addition,

it probably only happens when having set
 export GRASS_INT_ZLIB=1
before starting GRASS. In this case it would be
the combination of ZLIB compression for Integer files
and LFS enabled.

Markus
Tue, Jan 10 2006 18:15:07    Mail sent by glynn@gclements.plus.com  
Return-Path <glynn@gclements.plus.com>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn@gclements.plus.com>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <17347.60311.386863.420555@cerise.gclements.plus.com>
Date Tue, 10 Jan 2006 17:15:03 +0000
To Request Tracker <grass-bugs@intevation.de>
Cc grass5@grass.itc.it
Subject Re: [GRASS5] [bug #3974] (grass) r.mapcalc/libgis failure with LFS enabled
In-Reply-To <20060110111619.D905E1005CB@lists.intevation.de>
References <20060110111619.D905E1005CB@lists.intevation.de>
X-Mailer VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Request Tracker wrote:

> this bug's URL: http://intevation.de/rt/webrt?serial_num=3974
> -------------------------------------------------------------------------
> 
> Subject: r.mapcalc/libgis failure with LFS enabled

One thing which I note about the implemention of LFS in GRASS is that
it works by defining the macros in config.h.

Consequently, every file which uses off_t must include config.h before
including any other headers. Otherwise, you will have different source
files assuming different sizes for off_t.

Historically, files have typically only included config.h if they need
to explicitly use a specific macro which it defines.

In adding the _FILE_OFFSET_BITS setting to config.h, you need to
re-evaluate the set of files which need to include config.h.

If I intended to enable LFS globally, I would have done so by adding
-D_FILE_OFFSET_BITS=64 to the compiler flags in Grass.make. That
guarantees that the same settings will be used throughout.

-- 
Glynn Clements <glynn@gclements.plus.com>


Tue, Jan 10 2006 18:44:39    Mail sent by neteler@itc.it  
Return-Path <neteler@itc.it>
Delivered-To grass-bugs@lists.intevation.de
Date Tue, 10 Jan 2006 18:44:28 +0100
From Markus Neteler <neteler@itc.it>
To grass developers list <grass5@grass.itc.it>
Cc grass-bugs@intevation.de
Subject Re: [bug #3974] (grass) r.mapcalc/libgis failure with LFS enabled
Message-ID <20060110174427.GA17205@bartok.itc.it>
Mail-Followup-To grass developers list <grass5@grass.itc.it>, grass-bugs@intevation.de
References <20060110171507.28F091005D5@lists.intevation.de>
Mime-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Disposition inline
In-Reply-To <20060110171507.28F091005D5@lists.intevation.de>
User-Agent Mutt/1.4.1i
X-PGP-Key http://www.gdf-hannover.de/neteler/markus_gpgkey.asc
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
On Tue, Jan 10, 2006 at 06:15:07PM +0100, Glynn Clements via RT wrote:
> 
> Request Tracker wrote:
> 
> > this bug's URL: http://intevation.de/rt/webrt?serial_num=3974
> > -------------------------------------------------------------------------
> > 
> > Subject: r.mapcalc/libgis failure with LFS enabled
> 
> One thing which I note about the implemention of LFS in GRASS is that
> it works by defining the macros in config.h.
> 
> Consequently, every file which uses off_t must include config.h before
> including any other headers. Otherwise, you will have different source
> files assuming different sizes for off_t.
> 
> Historically, files have typically only included config.h if they need
> to explicitly use a specific macro which it defines.

The situation is:

find . -type f -name "*.c" -exec grep -l off_t {} \;
./raster3d/r3.in.v5d/v5d.c
./raster3d/r3.out.v5d/v5d.c
./raster/r.thin/io.c
./raster/r.support/modhead/modhead.c
./raster/r.support/modhead/row_addr.c
./lib/gis/get_row.c
./lib/gis/format.c
./lib/gis/put_row.c
./lib/gis/opencell.c
./lib/image/rdwr.c
./lib/segment/seek.c
./lib/segment/format.c

find . -type f -name "*.c" -exec grep -l off_t {} \; | xargs fgrep --files-without-match
'config.h'
./raster3d/r3.in.v5d/v5d.c
./raster3d/r3.out.v5d/v5d.c
./raster/r.thin/io.c
./lib/gis/opencell.c
./lib/image/rdwr.c
./lib/segment/seek.c

Are these the files to be fixed? Doesn't look like a big job.

The ./lib/segment/seek.c case would probably solve these bugs:
  http://www.intevation.de/rt/webrt?serial_num=3514&display=History
  (r.cost + r.walk as well, see at bottom of the page)
?

Markus


Wed, Jan 11 2006 10:09:52    Mail sent by mneteler  
As (indirectly) suggested, I have added
#include "config.h"

to:

./raster3d/r3.in.v5d/v5d.c
./raster3d/r3.out.v5d/v5d.c
./raster/r.thin/io.c
./lib/gis/opencell.c
./lib/image/rdwr.c
./lib/segment/seek.c

as 'off_t' is used in these files.

The problem seems to persist, though.

Markus
Wed, Jan 11 2006 19:47:11    Mail sent by glynn@gclements.plus.com  
Return-Path <glynn@gclements.plus.com>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn@gclements.plus.com>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <17349.21163.208883.736892@cerise.gclements.plus.com>
Date Wed, 11 Jan 2006 18:47:07 +0000
To Markus Neteler via RT <grass-bugs@intevation.de>
Cc flor@itc.it, merler@itc.it, neteler@itc.it, grass5@grass.itc.it
Subject Re: [GRASS5] [bug #3974] (grass) r.mapcalc/libgis failure with LFS enabled
In-Reply-To <20060111090952.987B01006AC@lists.intevation.de>
References <20060111090952.987B01006AC@lists.intevation.de>
X-Mailer VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Markus Neteler via RT wrote:

> As (indirectly) suggested, I have added
> #include "config.h"
> 
> to:
> 
> ./raster3d/r3.in.v5d/v5d.c
> ./raster3d/r3.out.v5d/v5d.c
> ./raster/r.thin/io.c
> ./lib/gis/opencell.c
> ./lib/image/rdwr.c
> ./lib/segment/seek.c
> 
> as 'off_t' is used in these files.
> 
> The problem seems to persist, though.

I note that gis.h includes config.h, but it does so after it includes
stdio.h. That may have an impact.

Also, it's possible that some code is assuming that an off_t is the
same size as a long.

I'd suggest that it's likely to be simpler to change the way that
large file support is implemented so that it only applies to source
files which specifically allow for it.

I.e. rather than setting _FILE_OFFSET_BITS in config.h, define a
separate macro (e.g. GRASS_LARGEFILE) and modify specific source files
to do:

	#ifdef GRASS_LARGEFILE
	#define _FILE_OFFSET_BITS 64
	#endif

before including any headers.

The most obvious candidates are the files containing the core raster
I/O code (opencell.c, get_row.c, put_row.c, format.c).

-- 
Glynn Clements <glynn@gclements.plus.com>


Thu, Jan 12 2006 16:39:19    Mail sent by neteler@itc.it  
Return-Path <neteler@itc.it>
Delivered-To grass-bugs@lists.intevation.de
Date Thu, 12 Jan 2006 16:39:10 +0100
From Markus Neteler <neteler@itc.it>
To Glynn Clements <glynn@gclements.plus.com>
Cc grass-bugs@intevation.de, grass5@grass.itc.it
Subject Re: [GRASS5] [bug #3974] (grass) r.mapcalc/libgis failure with LFS enabled
Message-ID <20060112153910.GF23366@vanga.itc.it>
Mail-Followup-To Glynn Clements <glynn@gclements.plus.com>, grass-bugs@intevation.de, grass5@grass.itc.it
References <20060111090952.987B01006AC@lists.intevation.de> <17349.21163.208883.736892@cerise.gclements.plus.com>
Mime-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Disposition inline
In-Reply-To <17349.21163.208883.736892@cerise.gclements.plus.com>
User-Agent Mutt/1.4.1i
X-PGP-Key http://www.gdf-hannover.de/neteler/markus_gpgkey.asc
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
On Wed, Jan 11, 2006 at 06:47:07PM +0000, Glynn Clements wrote:
> 
> Markus Neteler via RT wrote:
> 
> > As (indirectly) suggested, I have added
> > #include "config.h"
> > 
> > to:
> > 
> > ./raster3d/r3.in.v5d/v5d.c
> > ./raster3d/r3.out.v5d/v5d.c
> > ./raster/r.thin/io.c
> > ./lib/gis/opencell.c
> > ./lib/image/rdwr.c
> > ./lib/segment/seek.c
> > 
> > as 'off_t' is used in these files.
> > 
> > The problem seems to persist, though.
> 
> I note that gis.h includes config.h, but it does so after it includes
> stdio.h. That may have an impact.
> 
> Also, it's possible that some code is assuming that an off_t is the
> same size as a long.
> 
> I'd suggest that it's likely to be simpler to change the way that
> large file support is implemented so that it only applies to source
> files which specifically allow for it.
> 
> I.e. rather than setting _FILE_OFFSET_BITS in config.h, define a
> separate macro (e.g. GRASS_LARGEFILE) and modify specific source files
> to do:
> 
> 	#ifdef GRASS_LARGEFILE
> 	#define _FILE_OFFSET_BITS 64
> 	#endif
> 
> before including any headers.
> 
> The most obvious candidates are the files containing the core raster
> I/O code (opencell.c, get_row.c, put_row.c, format.c).
>

I will leave these fixes to the original author(s) of LFS
support in GRASS since I don't know enough about these details.

Markus


Thu, Jan 12 2006 22:20:55    Comments added by guest  
Imported LANDSAT damaged:

http://mpa.itc.it/markus/tmp/raster_corrupted/
Comment | Reply | Take | Resolve

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request