Details Ticket 4522


Comment | Reply | Take | Resolve


Serial Number 4522
Subject r.series: error in row of input map is non-fatal giving wrong results
Area grass6
Queue grass
Requestors hamish_nospam@yahoo.com
Owner none
Status open
Last User Contact Thu Jun 1 15:27:23 2006 (2 yr ago)
Current Priority 60
Final Priority 70
Due No date assigned
Last Action Thu Jun 1 15:27:23 2006 (2 yr ago)
Created Tue May 30 04:57:53 2006 (2 yr ago)

Transaction History Ticket 4522


Tue, May 30 2006 04:57:53    Request created by hbowman  
Subject: r.series: error in row of input map is non-fatal giving wrong results
Hi,


a problem with r.series:

One of my input maps has a corruption in it (disk flaw??), using r.series
(method=average) I see this error:

WARNING: error reading compressed map [dbt_corrupt] in mapset
         [sw2006f_icw], row 1465

but the process continues to completion and I get a resulting map which looks
ok, but is subtly wrong.


G61> r.univar dbt_mean_corrupted
 100%
total null and non-null cells: 11160000
total null cells: 9626502

Of the non-null cells:
----------------------
n: 1533498
minimum: 0
maximum: 1.8731
range: 1.8731
mean: 0.810907
standard deviation: 0.361051
variance: 0.130358
variation coefficient: 44.5243 %
sum: 1243524.7052483591



G61> r.univar dbt_mean_good
 100%
total null and non-null cells: 11160000
total null cells: 9629925

Of the non-null cells:
----------------------
n: 1530075
minimum: 0.2611
maximum: 1.8731
range: 1.612
mean: 0.812793
standard deviation: 0.359435
variance: 0.129194
variation coefficient: 44.2223 %
sum: 1243634.0932001742


As this was called from a script it is lucky I noticed..

The read error should trigger a G_fatal_error().


thanks,
Hamish
Tue, May 30 2006 14:45:43    Mail sent by glynn@gclements.plus.com  
Return-Path <glynn@gclements.plus.com>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn@gclements.plus.com>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <17532.15979.943259.530642@cerise.gclements.plus.com>
Date Tue, 30 May 2006 13:45:31 +0100
To Request Tracker <grass-bugs@intevation.de>
Cc grass-dev@grass.itc.it
Subject Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
In-Reply-To <20060530025753.3F2821005C4@lists.intevation.de>
References <20060530025753.3F2821005C4@lists.intevation.de>
X-Mailer VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Request Tracker wrote:

> this bug's URL: http://intevation.de/rt/webrt?serial_num=4522
> -------------------------------------------------------------------------
> 
> Subject: r.series: error in row of input map is non-fatal giving wrong results
> 
> Hi,
> 
> 
> a problem with r.series:
> 
> One of my input maps has a corruption in it (disk flaw??), using r.series
> (method=average) I see this error:
> 
> WARNING: error reading compressed map [dbt_corrupt] in mapset
>          [sw2006f_icw], row 1465
> 
> but the process continues to completion and I get a resulting map which looks
> ok, but is subtly wrong.

> The read error should trigger a G_fatal_error().

Agreed. The question is: why does libgis leave this to individual
modules rather than doing it itself? Are there really modules which
might want to do something other than call G_fatal_error() in this
case?

FWIW, it isn't just r.series; the following all call G_get_raster_row
(and similar) without checking the return value:

display/d.rast.arrow
display/d.rast.edit
display/d.rast.num
display/d.rast
imagery/i.ifft
imagery/i.pca
lib/D
lib/ogsf
lib/rst/interp_float
paint/p.out.vrml
ps/ps.map
raster/r.bilinear
raster/r.carve
raster/r.contour
raster/r.flow
raster/r.grow2
raster/r.lake
raster/r.le/r.le.patch
raster/r.le/r.le.pixel
raster/r.le/r.le.setup
raster/r.le/r.le.trace
raster/r.neighbors
raster/r.param.scale
raster/r.random.cells
raster/r.random.surface
raster/r.recode
raster/r.series
raster/r.slope.aspect
raster/r.sum
raster/r.sun
raster/r.sunmask
raster/r.surf.area
raster/r.surf.contour
raster/r.texture
raster/r.to.vect
raster/r.volume
raster/r.water.outlet
raster/r.watershed/ram
raster/r.watershed/seg
raster/r.watershed/shed
raster/simwe/simlib
vector/v.vol.rst

-- 
Glynn Clements <glynn@gclements.plus.com>


Wed, May 31 2006 04:51:06    Mail sent by hamish_nospam@yahoo.com  
Return-Path <hamish_nospam@yahoo.com>
Delivered-To grass-bugs@lists.intevation.de
Date Wed, 31 May 2006 14:50:48 +1200
From Hamish <hamish_nospam@yahoo.com>
To Glynn Clements <glynn@gclements.plus.com>
Cc grass-bugs@intevation.de, grass-dev@grass.itc.it
Subject Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
Message-Id <20060531145048.0d85b79f.hamish_nospam@yahoo.com>
In-Reply-To <17532.15979.943259.530642@cerise.gclements.plus.com>
References <20060530025753.3F2821005C4@lists.intevation.de> <17532.15979.943259.530642@cerise.gclements.plus.com>
X-Mailer Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
X-Face M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
Mime-Version 1.0
Content-Type text/plain; charset=US-ASCII
Content-Transfer-Encoding 7bit
X-Spam-Status No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
X-Spam-Level
> > The read error should trigger a G_fatal_error().
> 
> Agreed. The question is: why does libgis leave this to individual
> modules rather than doing it itself? Are there really modules which
> might want to do something other than call G_fatal_error() in this
> case?

I doubt it.


Is there a possibility of many orphaned .tmp files if the module exits
during a read while an output map open? i.e. how important is it to exit
with clean-up at any given point of a typical raster [read; do stuff;
write] operation?


Hamish


Wed, May 31 2006 19:11:23    Mail sent by glynn@gclements.plus.com  
Return-Path <glynn@gclements.plus.com>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn@gclements.plus.com>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <17533.52789.412070.442208@cerise.gclements.plus.com>
Date Wed, 31 May 2006 18:11:17 +0100
To Hamish <hamish_nospam@yahoo.com>
Cc grass-bugs@intevation.de, grass-dev@grass.itc.it
Subject Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
In-Reply-To <20060531145048.0d85b79f.hamish_nospam@yahoo.com>
References <20060530025753.3F2821005C4@lists.intevation.de> <17532.15979.943259.530642@cerise.gclements.plus.com> <20060531145048.0d85b79f.hamish_nospam@yahoo.com>
X-Mailer VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Hamish wrote:

> > > The read error should trigger a G_fatal_error().
> > 
> > Agreed. The question is: why does libgis leave this to individual
> > modules rather than doing it itself? Are there really modules which
> > might want to do something other than call G_fatal_error() in this
> > case?
> 
> I doubt it.
> 
> Is there a possibility of many orphaned .tmp files if the module exits
> during a read while an output map open?

Yes.

> i.e. how important is it to exit
> with clean-up at any given point of a typical raster [read; do stuff;
> write] operation?

Almost nothing does this at present. Even modules which check the
return value from G_get_raster_row() etc usually just call
G_fatal_error() if it fails.

There are precisely 20 files in 15 modules which call G_unopen_cell(),
while there are rather more than that which create new raster maps. 
The remainder simply don't bother to clean up upon errors (including
r.example).

I suggest that:

a) G_get_raster_row() etc should automatically call G_fatal_error() if
an error occurs, with a function to disable this behaviour for modules
which really need to do their own error handling.

b) It should be the library's responsibility to clean up incomplete
maps upon exit.

Having to manually check for error conditions and perform clean-up
whenever an error occurs is enough of a nuisance that it will
frequently be omitted (or, if error handling is added, those code
paths will seldom get tested and thus will frequently contain bugs).

-- 
Glynn Clements <glynn@gclements.plus.com>


Thu, Jun 1 2006 07:32:16    Mail sent by hamish_nospam@yahoo.com  
Return-Path <hamish_nospam@yahoo.com>
Delivered-To grass-bugs@lists.intevation.de
Date Thu, 1 Jun 2006 17:31:36 +1200
From Hamish <hamish_nospam@yahoo.com>
To Glynn Clements <glynn@gclements.plus.com>
Cc grass-bugs@intevation.de, grass-dev@grass.itc.it
Subject Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
Message-Id <20060601173136.06066da3.hamish_nospam@yahoo.com>
In-Reply-To <17533.52789.412070.442208@cerise.gclements.plus.com>
References <20060530025753.3F2821005C4@lists.intevation.de> <17532.15979.943259.530642@cerise.gclements.plus.com> <20060531145048.0d85b79f.hamish_nospam@yahoo.com> <17533.52789.412070.442208@cerise.gclements.plus.com>
X-Mailer Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
X-Face M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
Mime-Version 1.0
Content-Type text/plain; charset=US-ASCII
Content-Transfer-Encoding 7bit
X-Spam-Status No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
X-Spam-Level
> I suggest that:
> 
> a) G_get_raster_row() etc should automatically call G_fatal_error() if
> an error occurs, with a function to disable this behaviour for modules
> which really need to do their own error handling.

ok
 
> b) It should be the library's responsibility to clean up incomplete
> maps upon exit.
> 
> Having to manually check for error conditions and perform clean-up
> whenever an error occurs is enough of a nuisance that it will
> frequently be omitted (or, if error handling is added, those code
> paths will seldom get tested and thus will frequently contain bugs).

does that mean you have to start sending the file descriptors, etc,
along to the lib fns? While one read error probably means many, waiting
for cleanup-on-GRASS-exit seems less messy somehow.

Should G_get_raster_row() do system("$GISBASE/etc/clean_temp") before
calling G_fatal_error() ?


Hamish


Thu, Jun 1 2006 15:27:23    Mail sent by glynn@gclements.plus.com  
Return-Path <glynn@gclements.plus.com>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn@gclements.plus.com>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <17534.60196.780248.89602@cerise.gclements.plus.com>
Date Thu, 1 Jun 2006 14:27:00 +0100
To Hamish <hamish_nospam@yahoo.com>
Cc grass-bugs@intevation.de, grass-dev@grass.itc.it
Subject Re: [GRASS-dev] [bug #4522] (grass) r.series: error in row of input map is non-fatal giving wrong results
In-Reply-To <20060601173136.06066da3.hamish_nospam@yahoo.com>
References <20060530025753.3F2821005C4@lists.intevation.de> <17532.15979.943259.530642@cerise.gclements.plus.com> <20060531145048.0d85b79f.hamish_nospam@yahoo.com> <17533.52789.412070.442208@cerise.gclements.plus.com> <20060601173136.06066da3.hamish_nospam@yahoo.com>
X-Mailer VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Hamish wrote:

> > a) G_get_raster_row() etc should automatically call G_fatal_error() if
> > an error occurs, with a function to disable this behaviour for modules
> > which really need to do their own error handling.
> 
> ok
>  
> > b) It should be the library's responsibility to clean up incomplete
> > maps upon exit.
> > 
> > Having to manually check for error conditions and perform clean-up
> > whenever an error occurs is enough of a nuisance that it will
> > frequently be omitted (or, if error handling is added, those code
> > paths will seldom get tested and thus will frequently contain bugs).
> 
> does that mean you have to start sending the file descriptors, etc,
> along to the lib fns? While one read error probably means many, waiting
> for cleanup-on-GRASS-exit seems less messy somehow.
> 
> Should G_get_raster_row() do system("$GISBASE/etc/clean_temp") before
> calling G_fatal_error() ?

My comment was limited to cleaning up the temporary cell/fcell file
created when writing new raster maps. Essentially, G_open_raster_new()
etc would need to use atexit() to automatically "unopen" any
incomplete maps.

Applications would be responsible for cleaning up their own temporary
files.

-- 
Glynn Clements <glynn@gclements.plus.com>


Comment | Reply | Take | Resolve

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request