Tue, May 30 2006
04:57:53
|
|
Request created by hbowman
|
|
Subject: r.series: error in row of input map is non-fatal giving wrong results
Hi,
a problem with r.series:
One of my input maps has a corruption in it (disk flaw??), using r.series
(method=average) I see this error:
WARNING: error reading compressed map [dbt_corrupt] in mapset
[sw2006f_icw], row 1465
but the process continues to completion and I get a resulting map which looks
ok, but is subtly wrong.
G61> r.univar dbt_mean_corrupted
100%
total null and non-null cells: 11160000
total null cells: 9626502
Of the non-null cells:
----------------------
n: 1533498
minimum: 0
maximum: 1.8731
range: 1.8731
mean: 0.810907
standard deviation: 0.361051
variance: 0.130358
variation coefficient: 44.5243 %
sum: 1243524.7052483591
G61> r.univar dbt_mean_good
100%
total null and non-null cells: 11160000
total null cells: 9629925
Of the non-null cells:
----------------------
n: 1530075
minimum: 0.2611
maximum: 1.8731
range: 1.612
mean: 0.812793
standard deviation: 0.359435
variance: 0.129194
variation coefficient: 44.2223 %
sum: 1243634.0932001742
As this was called from a script it is lucky I noticed..
The read error should trigger a G_fatal_error().
thanks,
Hamish
|
|
Tue, May 30 2006
14:45:43
|
|
Mail sent by glynn@gclements.plus.com
|
|
Return-Path |
<glynn@gclements.plus.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
From |
Glynn Clements <glynn@gclements.plus.com>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Transfer-Encoding |
7bit
|
Message-ID |
<17532.15979.943259.530642@cerise.gclements.plus.com>
|
Date |
Tue, 30 May 2006 13:45:31 +0100
|
To |
Request Tracker <grass-bugs@intevation.de>
|
Cc |
grass-dev@grass.itc.it
|
Subject |
Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
|
In-Reply-To |
<20060530025753.3F2821005C4@lists.intevation.de>
|
References |
<20060530025753.3F2821005C4@lists.intevation.de>
|
X-Mailer |
VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Request Tracker wrote:
> this bug's URL: http://intevation.de/rt/webrt?serial_num=4522
> -------------------------------------------------------------------------
>
> Subject: r.series: error in row of input map is non-fatal giving wrong results
>
> Hi,
>
>
> a problem with r.series:
>
> One of my input maps has a corruption in it (disk flaw??), using r.series
> (method=average) I see this error:
>
> WARNING: error reading compressed map [dbt_corrupt] in mapset
> [sw2006f_icw], row 1465
>
> but the process continues to completion and I get a resulting map which looks
> ok, but is subtly wrong.
> The read error should trigger a G_fatal_error().
Agreed. The question is: why does libgis leave this to individual
modules rather than doing it itself? Are there really modules which
might want to do something other than call G_fatal_error() in this
case?
FWIW, it isn't just r.series; the following all call G_get_raster_row
(and similar) without checking the return value:
display/d.rast.arrow
display/d.rast.edit
display/d.rast.num
display/d.rast
imagery/i.ifft
imagery/i.pca
lib/D
lib/ogsf
lib/rst/interp_float
paint/p.out.vrml
ps/ps.map
raster/r.bilinear
raster/r.carve
raster/r.contour
raster/r.flow
raster/r.grow2
raster/r.lake
raster/r.le/r.le.patch
raster/r.le/r.le.pixel
raster/r.le/r.le.setup
raster/r.le/r.le.trace
raster/r.neighbors
raster/r.param.scale
raster/r.random.cells
raster/r.random.surface
raster/r.recode
raster/r.series
raster/r.slope.aspect
raster/r.sum
raster/r.sun
raster/r.sunmask
raster/r.surf.area
raster/r.surf.contour
raster/r.texture
raster/r.to.vect
raster/r.volume
raster/r.water.outlet
raster/r.watershed/ram
raster/r.watershed/seg
raster/r.watershed/shed
raster/simwe/simlib
vector/v.vol.rst
--
Glynn Clements <glynn@gclements.plus.com>
|
|
Wed, May 31 2006
04:51:06
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Wed, 31 May 2006 14:50:48 +1200
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
Glynn Clements <glynn@gclements.plus.com>
|
Cc |
grass-bugs@intevation.de, grass-dev@grass.itc.it
|
Subject |
Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
|
Message-Id |
<20060531145048.0d85b79f.hamish_nospam@yahoo.com>
|
In-Reply-To |
<17532.15979.943259.530642@cerise.gclements.plus.com>
|
References |
<20060530025753.3F2821005C4@lists.intevation.de> <17532.15979.943259.530642@cerise.gclements.plus.com>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
|
X-Spam-Level |
|
> > The read error should trigger a G_fatal_error().
>
> Agreed. The question is: why does libgis leave this to individual
> modules rather than doing it itself? Are there really modules which
> might want to do something other than call G_fatal_error() in this
> case?
I doubt it.
Is there a possibility of many orphaned .tmp files if the module exits
during a read while an output map open? i.e. how important is it to exit
with clean-up at any given point of a typical raster [read; do stuff;
write] operation?
Hamish
|
|
Wed, May 31 2006
19:11:23
|
|
Mail sent by glynn@gclements.plus.com
|
|
Return-Path |
<glynn@gclements.plus.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
From |
Glynn Clements <glynn@gclements.plus.com>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Transfer-Encoding |
7bit
|
Message-ID |
<17533.52789.412070.442208@cerise.gclements.plus.com>
|
Date |
Wed, 31 May 2006 18:11:17 +0100
|
To |
Hamish <hamish_nospam@yahoo.com>
|
Cc |
grass-bugs@intevation.de, grass-dev@grass.itc.it
|
Subject |
Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
|
In-Reply-To |
<20060531145048.0d85b79f.hamish_nospam@yahoo.com>
|
References |
<20060530025753.3F2821005C4@lists.intevation.de> <17532.15979.943259.530642@cerise.gclements.plus.com> <20060531145048.0d85b79f.hamish_nospam@yahoo.com>
|
X-Mailer |
VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Hamish wrote:
> > > The read error should trigger a G_fatal_error().
> >
> > Agreed. The question is: why does libgis leave this to individual
> > modules rather than doing it itself? Are there really modules which
> > might want to do something other than call G_fatal_error() in this
> > case?
>
> I doubt it.
>
> Is there a possibility of many orphaned .tmp files if the module exits
> during a read while an output map open?
Yes.
> i.e. how important is it to exit
> with clean-up at any given point of a typical raster [read; do stuff;
> write] operation?
Almost nothing does this at present. Even modules which check the
return value from G_get_raster_row() etc usually just call
G_fatal_error() if it fails.
There are precisely 20 files in 15 modules which call G_unopen_cell(),
while there are rather more than that which create new raster maps.
The remainder simply don't bother to clean up upon errors (including
r.example).
I suggest that:
a) G_get_raster_row() etc should automatically call G_fatal_error() if
an error occurs, with a function to disable this behaviour for modules
which really need to do their own error handling.
b) It should be the library's responsibility to clean up incomplete
maps upon exit.
Having to manually check for error conditions and perform clean-up
whenever an error occurs is enough of a nuisance that it will
frequently be omitted (or, if error handling is added, those code
paths will seldom get tested and thus will frequently contain bugs).
--
Glynn Clements <glynn@gclements.plus.com>
|
|
Thu, Jun 1 2006
07:32:16
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Thu, 1 Jun 2006 17:31:36 +1200
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
Glynn Clements <glynn@gclements.plus.com>
|
Cc |
grass-bugs@intevation.de, grass-dev@grass.itc.it
|
Subject |
Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
|
Message-Id |
<20060601173136.06066da3.hamish_nospam@yahoo.com>
|
In-Reply-To |
<17533.52789.412070.442208@cerise.gclements.plus.com>
|
References |
<20060530025753.3F2821005C4@lists.intevation.de> <17532.15979.943259.530642@cerise.gclements.plus.com> <20060531145048.0d85b79f.hamish_nospam@yahoo.com> <17533.52789.412070.442208@cerise.gclements.plus.com>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
|
X-Spam-Level |
|
> I suggest that:
>
> a) G_get_raster_row() etc should automatically call G_fatal_error() if
> an error occurs, with a function to disable this behaviour for modules
> which really need to do their own error handling.
ok
> b) It should be the library's responsibility to clean up incomplete
> maps upon exit.
>
> Having to manually check for error conditions and perform clean-up
> whenever an error occurs is enough of a nuisance that it will
> frequently be omitted (or, if error handling is added, those code
> paths will seldom get tested and thus will frequently contain bugs).
does that mean you have to start sending the file descriptors, etc,
along to the lib fns? While one read error probably means many, waiting
for cleanup-on-GRASS-exit seems less messy somehow.
Should G_get_raster_row() do system("$GISBASE/etc/clean_temp") before
calling G_fatal_error() ?
Hamish
|
|
Thu, Jun 1 2006
15:27:23
|
|
Mail sent by glynn@gclements.plus.com
|
|
Return-Path |
<glynn@gclements.plus.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
From |
Glynn Clements <glynn@gclements.plus.com>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Transfer-Encoding |
7bit
|
Message-ID |
<17534.60196.780248.89602@cerise.gclements.plus.com>
|
Date |
Thu, 1 Jun 2006 14:27:00 +0100
|
To |
Hamish <hamish_nospam@yahoo.com>
|
Cc |
grass-bugs@intevation.de, grass-dev@grass.itc.it
|
Subject |
Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
|
In-Reply-To |
<20060601173136.06066da3.hamish_nospam@yahoo.com>
|
References |
<20060530025753.3F2821005C4@lists.intevation.de> <17532.15979.943259.530642@cerise.gclements.plus.com> <20060531145048.0d85b79f.hamish_nospam@yahoo.com> <17533.52789.412070.442208@cerise.gclements.plus.com> <20060601173136.06066da3.hamish_nospam@yahoo.com>
|
X-Mailer |
VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Hamish wrote:
> > a) G_get_raster_row() etc should automatically call G_fatal_error() if
> > an error occurs, with a function to disable this behaviour for modules
> > which really need to do their own error handling.
>
> ok
>
> > b) It should be the library's responsibility to clean up incomplete
> > maps upon exit.
> >
> > Having to manually check for error conditions and perform clean-up
> > whenever an error occurs is enough of a nuisance that it will
> > frequently be omitted (or, if error handling is added, those code
> > paths will seldom get tested and thus will frequently contain bugs).
>
> does that mean you have to start sending the file descriptors, etc,
> along to the lib fns? While one read error probably means many, waiting
> for cleanup-on-GRASS-exit seems less messy somehow.
>
> Should G_get_raster_row() do system("$GISBASE/etc/clean_temp") before
> calling G_fatal_error() ?
My comment was limited to cleaning up the temporary cell/fcell file
created when writing new raster maps. Essentially, G_open_raster_new()
etc would need to use atexit() to automatically "unopen" any
incomplete maps.
Applications would be responsible for cleaning up their own temporary
files.
--
Glynn Clements <glynn@gclements.plus.com>
|
|