Tue, Jan 10 2006
12:16:19
|
|
Request created by guest
|
|
Subject: r.mapcalc/libgis failure with LFS enabled
Platform: GNU/Linux/x86_64
grass obtained from: CVS
grass binary for platform: Compiled from Sources
Hi,
we have discovered some severe problems in the GRASS raster map
library (within libgis) in certain cases. The problem was first
identified in August/December 2005 (see other mails). We really
need to fix it.
For a test, I have generated a XY location with 3x3 pixels
and generated a simple map:
GRASS 6.1.cvs (xy):~ > r.mapcalc "one=1"
100%
GRASS 6.1.cvs (xy):~ > r.univar one
Processing .. 100%
total null and non-null cells: 9
total null cells: 0
Of the non-null cells:
----------------------
n: 9
minimum: 99
maximum: 156
range: 57
mean: 125
standard deviation: 23.5372
variance: 554
variation coefficient: 18.8298 %
sum: 1125
GRASS 6.1.cvs (xy):~ > r.out.ascii one
north: 3
south: 0
east: 3
west: 0
rows: 3
cols: 3
120 156 99
120 156 99
120 156 99
My proposal is to add some tests to libgis/ to verify
if the data are wrongly written or read or both.
Here some tests:
* Machine: 64bit PENTIUM, RHEL4, gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)
- 64bit enabled
- LFS enabled
-> above problems
* Machine: 64bit PENTIUM, RHEL4, gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)
- 64bit enabled
- LFS disabled
-> maps generated *here* are read correctly
-> maps generated with LFS enabled are read wrongly (so write error with LFS
enabled?)
* Machine: 32bit Mandriva2006
- LFS enabled
Here various raster modules crash with LFS enabled
(r.sun, r.mapcalc + d.rast - error reading compressed row)
Apparently something is wrong with LFS (off_t issues?).
Is anyone able to write some low level test for libgis which
will be performed during compilation (as already exists for
libvect)?
Markus
|
|
Tue, Jan 10 2006
14:27:38
|
|
Mail sent by mneteler
|
|
An addition,
it probably only happens when having set
export GRASS_INT_ZLIB=1
before starting GRASS. In this case it would be
the combination of ZLIB compression for Integer files
and LFS enabled.
Markus
|
|
Tue, Jan 10 2006
18:15:07
|
|
Mail sent by glynn@gclements.plus.com
|
|
Return-Path |
<glynn@gclements.plus.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
From |
Glynn Clements <glynn@gclements.plus.com>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Transfer-Encoding |
7bit
|
Message-ID |
<17347.60311.386863.420555@cerise.gclements.plus.com>
|
Date |
Tue, 10 Jan 2006 17:15:03 +0000
|
To |
Request Tracker <grass-bugs@intevation.de>
|
Cc |
grass5@grass.itc.it
|
Subject |
Re: [GRASS5] [bug #3974] (grass) r.mapcalc/libgis failure with LFS enabled
|
In-Reply-To |
<20060110111619.D905E1005CB@lists.intevation.de>
|
References |
<20060110111619.D905E1005CB@lists.intevation.de>
|
X-Mailer |
VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Request Tracker wrote:
> this bug's URL: http://intevation.de/rt/webrt?serial_num=3974
> -------------------------------------------------------------------------
>
> Subject: r.mapcalc/libgis failure with LFS enabled
One thing which I note about the implemention of LFS in GRASS is that
it works by defining the macros in config.h.
Consequently, every file which uses off_t must include config.h before
including any other headers. Otherwise, you will have different source
files assuming different sizes for off_t.
Historically, files have typically only included config.h if they need
to explicitly use a specific macro which it defines.
In adding the _FILE_OFFSET_BITS setting to config.h, you need to
re-evaluate the set of files which need to include config.h.
If I intended to enable LFS globally, I would have done so by adding
-D_FILE_OFFSET_BITS=64 to the compiler flags in Grass.make. That
guarantees that the same settings will be used throughout.
--
Glynn Clements <glynn@gclements.plus.com>
|
|
Tue, Jan 10 2006
18:44:39
|
|
Mail sent by neteler@itc.it
|
|
Return-Path |
<neteler@itc.it>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Tue, 10 Jan 2006 18:44:28 +0100
|
From |
Markus Neteler <neteler@itc.it>
|
To |
grass developers list <grass5@grass.itc.it>
|
Cc |
grass-bugs@intevation.de
|
Subject |
Re: [bug #3974] (grass) r.mapcalc/libgis failure with LFS enabled
|
Message-ID |
<20060110174427.GA17205@bartok.itc.it>
|
Mail-Followup-To |
grass developers list <grass5@grass.itc.it>, grass-bugs@intevation.de
|
References |
<20060110171507.28F091005D5@lists.intevation.de>
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Disposition |
inline
|
In-Reply-To |
<20060110171507.28F091005D5@lists.intevation.de>
|
User-Agent |
Mutt/1.4.1i
|
X-PGP-Key |
http://www.gdf-hannover.de/neteler/markus_gpgkey.asc
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
On Tue, Jan 10, 2006 at 06:15:07PM +0100, Glynn Clements via RT wrote:
>
> Request Tracker wrote:
>
> > this bug's URL: http://intevation.de/rt/webrt?serial_num=3974
> > -------------------------------------------------------------------------
> >
> > Subject: r.mapcalc/libgis failure with LFS enabled
>
> One thing which I note about the implemention of LFS in GRASS is that
> it works by defining the macros in config.h.
>
> Consequently, every file which uses off_t must include config.h before
> including any other headers. Otherwise, you will have different source
> files assuming different sizes for off_t.
>
> Historically, files have typically only included config.h if they need
> to explicitly use a specific macro which it defines.
The situation is:
find . -type f -name "*.c" -exec grep -l off_t {} \;
./raster3d/r3.in.v5d/v5d.c
./raster3d/r3.out.v5d/v5d.c
./raster/r.thin/io.c
./raster/r.support/modhead/modhead.c
./raster/r.support/modhead/row_addr.c
./lib/gis/get_row.c
./lib/gis/format.c
./lib/gis/put_row.c
./lib/gis/opencell.c
./lib/image/rdwr.c
./lib/segment/seek.c
./lib/segment/format.c
find . -type f -name "*.c" -exec grep -l off_t {} \; | xargs fgrep --files-without-match
'config.h'
./raster3d/r3.in.v5d/v5d.c
./raster3d/r3.out.v5d/v5d.c
./raster/r.thin/io.c
./lib/gis/opencell.c
./lib/image/rdwr.c
./lib/segment/seek.c
Are these the files to be fixed? Doesn't look like a big job.
The ./lib/segment/seek.c case would probably solve these bugs:
http://www.intevation.de/rt/webrt?serial_num=3514&display=History
(r.cost + r.walk as well, see at bottom of the page)
?
Markus
|
|
Wed, Jan 11 2006
10:09:52
|
|
Mail sent by mneteler
|
|
As (indirectly) suggested, I have added
#include "config.h"
to:
./raster3d/r3.in.v5d/v5d.c
./raster3d/r3.out.v5d/v5d.c
./raster/r.thin/io.c
./lib/gis/opencell.c
./lib/image/rdwr.c
./lib/segment/seek.c
as 'off_t' is used in these files.
The problem seems to persist, though.
Markus |
|
Wed, Jan 11 2006
19:47:11
|
|
Mail sent by glynn@gclements.plus.com
|
|
Return-Path |
<glynn@gclements.plus.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
From |
Glynn Clements <glynn@gclements.plus.com>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Transfer-Encoding |
7bit
|
Message-ID |
<17349.21163.208883.736892@cerise.gclements.plus.com>
|
Date |
Wed, 11 Jan 2006 18:47:07 +0000
|
To |
Markus Neteler via RT <grass-bugs@intevation.de>
|
Cc |
flor@itc.it, merler@itc.it, neteler@itc.it, grass5@grass.itc.it
|
Subject |
Re: [GRASS5] [bug #3974] (grass) r.mapcalc/libgis failure with LFS enabled
|
In-Reply-To |
<20060111090952.987B01006AC@lists.intevation.de>
|
References |
<20060111090952.987B01006AC@lists.intevation.de>
|
X-Mailer |
VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Markus Neteler via RT wrote:
> As (indirectly) suggested, I have added
> #include "config.h"
>
> to:
>
> ./raster3d/r3.in.v5d/v5d.c
> ./raster3d/r3.out.v5d/v5d.c
> ./raster/r.thin/io.c
> ./lib/gis/opencell.c
> ./lib/image/rdwr.c
> ./lib/segment/seek.c
>
> as 'off_t' is used in these files.
>
> The problem seems to persist, though.
I note that gis.h includes config.h, but it does so after it includes
stdio.h. That may have an impact.
Also, it's possible that some code is assuming that an off_t is the
same size as a long.
I'd suggest that it's likely to be simpler to change the way that
large file support is implemented so that it only applies to source
files which specifically allow for it.
I.e. rather than setting _FILE_OFFSET_BITS in config.h, define a
separate macro (e.g. GRASS_LARGEFILE) and modify specific source files
to do:
#ifdef GRASS_LARGEFILE
#define _FILE_OFFSET_BITS 64
#endif
before including any headers.
The most obvious candidates are the files containing the core raster
I/O code (opencell.c, get_row.c, put_row.c, format.c).
--
Glynn Clements <glynn@gclements.plus.com>
|
|
Thu, Jan 12 2006
16:39:19
|
|
Mail sent by neteler@itc.it
|
|
Return-Path |
<neteler@itc.it>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Thu, 12 Jan 2006 16:39:10 +0100
|
From |
Markus Neteler <neteler@itc.it>
|
To |
Glynn Clements <glynn@gclements.plus.com>
|
Cc |
grass-bugs@intevation.de, grass5@grass.itc.it
|
Subject |
Re: [GRASS5] [bug #3974] (grass) r.mapcalc/libgis failure with LFS enabled
|
Message-ID |
<20060112153910.GF23366@vanga.itc.it>
|
Mail-Followup-To |
Glynn Clements <glynn@gclements.plus.com>, grass-bugs@intevation.de, grass5@grass.itc.it
|
References |
<20060111090952.987B01006AC@lists.intevation.de> <17349.21163.208883.736892@cerise.gclements.plus.com>
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Disposition |
inline
|
In-Reply-To |
<17349.21163.208883.736892@cerise.gclements.plus.com>
|
User-Agent |
Mutt/1.4.1i
|
X-PGP-Key |
http://www.gdf-hannover.de/neteler/markus_gpgkey.asc
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
On Wed, Jan 11, 2006 at 06:47:07PM +0000, Glynn Clements wrote:
>
> Markus Neteler via RT wrote:
>
> > As (indirectly) suggested, I have added
> > #include "config.h"
> >
> > to:
> >
> > ./raster3d/r3.in.v5d/v5d.c
> > ./raster3d/r3.out.v5d/v5d.c
> > ./raster/r.thin/io.c
> > ./lib/gis/opencell.c
> > ./lib/image/rdwr.c
> > ./lib/segment/seek.c
> >
> > as 'off_t' is used in these files.
> >
> > The problem seems to persist, though.
>
> I note that gis.h includes config.h, but it does so after it includes
> stdio.h. That may have an impact.
>
> Also, it's possible that some code is assuming that an off_t is the
> same size as a long.
>
> I'd suggest that it's likely to be simpler to change the way that
> large file support is implemented so that it only applies to source
> files which specifically allow for it.
>
> I.e. rather than setting _FILE_OFFSET_BITS in config.h, define a
> separate macro (e.g. GRASS_LARGEFILE) and modify specific source files
> to do:
>
> #ifdef GRASS_LARGEFILE
> #define _FILE_OFFSET_BITS 64
> #endif
>
> before including any headers.
>
> The most obvious candidates are the files containing the core raster
> I/O code (opencell.c, get_row.c, put_row.c, format.c).
>
I will leave these fixes to the original author(s) of LFS
support in GRASS since I don't know enough about these details.
Markus
|
|
Thu, Jan 12 2006
22:20:55
|
|
Comments added by guest
|
|
Imported LANDSAT damaged:
http://mpa.itc.it/markus/tmp/raster_corrupted/ |
|