Details Ticket 4248


Comment | Reply | Take | Resolve


Serial Number 4248
Subject r.surf.contour: Treats 0 as NULL and other archaic problems
Area grass6
Queue grass
Requestors hamish_nospam@yahoo.com,tutey@o2.pl
Owner none
Status open
Last User Contact Wed Apr 5 20:37:20 2006 (2 yr ago)
Current Priority 30
Final Priority 70
Due No date assigned
Last Action Wed Jul 26 18:23:15 2006 (2 yr ago)
Created Fri Nov 8 08:44:32 2002 (6 yr ago)

Transaction History Ticket 4248


Fri, Nov 8 2002 08:44:32    Request created by guest (as #1402)  
Subject: r.surf.contour,  0 == null ?

Platform: GNU/Linux/i386
grass obtained from: Mirror of Trento site
grass binary for platform: Compiled from Sources
GRASS Version: 5.0.0

re: [GRASSLIST:4859]


r.surf.contour seems to be treating the 0m contour line as null values, thus
obscuring all the low lying topography.

The man page seems to suggest this is by design, but I guess that the program
just hasn't been updated to deal with zeros as real values?

I consider this a bug as it is inconstant with other modern GRASS modules.

Negative valued isobaths seem to work.

Using r.mapcalc to make the 0m contour a 1m contour does get around the 
problem, and otherwise the module puts out very nice results.


thanks
Fri, Aug 12 2005 13:51:59    Request created by guest (as #3515)  
Subject: r.surf.contour: max output=10*max input

Platform: GNU/Linux/i386
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: 61 cvs 09.08.05

# I did the following:

r.surf.contour -f input=melio_poz_clear_rast output=melio_poz_clear_rast_10_dem
# in a region of:

GRASS 6.1.cvs (caves_utm33):~ > g.region -p
projection: 1 (UTM)
zone:       33
datum:      wgs84
ellipsoid:  a=6378137 es=0.00669438
north:      5679480
south:      5679190
west:       601920
east:       602250
nsres:      10
ewres:      10
rows:       29
cols:       33


# The max output is 10 times more than max input!!!, see:

# the input

r.info -r melio_poz_clear_rast
min=9250000.000000
max=10025000.000000

# the output

r.info -r melio_poz_clear_rast_10_dem
min=9250000.000000
max=131410297.000000



What's interesting, this *error* high values seem to be *concentrated* in the
lower-left corner of the output, *where no input data is present* see:

http://www.biol.uni.wroc.pl/sieczka/udostepnione/output_max_10times_too_high.png
The yellow background is the output DEM, the colour overlay is the input. Black
arrow indicates the error (red perdendicular lines).

Maciek

P.S.
Could it be raleted to the "dozens of warnings" http://intevation.de/rt/webrt?serial_num=3514&displa
y=History
issue?
Sat, Aug 13 2005 11:43:42    Area changed to grass6.1 by msieczka (as #3515)  
Wed, Aug 31 2005 20:54:29    Area changed to grass6 by mneteler (as #3515)  
Wed, Apr 5 2006 12:24:50    Request created by guest  
Subject: r.surf.contour: Treats 0 as NULL and other archaic problems

Hi,

- If you feed r.surf.contour an input file including 0 as an elevation (say a
coastal area with the sea set to 0), it treats all the 0s as NULL and takes
days to complete. Change the 0s to -1s and it goes by relatively quickly.
Zero hasn't been NULL in GRASS for a long long time now.


- Looking at the code I see support for a MASK is hardcoded, and the module
does all sorts of bitwise operations on unmasked cells.. "w.t.f.?"


- running some time/memory trials I see there is no use for not using the -f
flag. Memory use was at most 5mb RAM during the non-default "memory intensive"
run. I think we can handle that these days. With -f it was 4 times faster:

rows:       2900
cols:       3000
total null cells: 4048763
2.8GHz Pentium4


CELL input, with the -f flag
  real    42m23.483s
  user    32m34.690s
  sys     8m5.880s
   2563 pts/6    RN+   40:37    207    19  5484 3172  0.1
================
DCELL -f
  real    41m8.144s
  user    32m21.580s
  sys     8m8.930s
  10641 pts/6    RN+   40:27    217    19  5496 3224  0.1

================
DCELL no -f
  real    178m50.506s
  user    165m21.230s
  sys     9m46.040s
  13944 pts/6    RN+  175:06    279    19  3676 2460  0.1
================
CELL no -f
  real    178m54.097s
  user    165m11.190s
  sys     9m42.560s
   1071 pts/6    RN+  174:52    270    19  3664 2412  0.1


Let's get rid of non "-f" code in the module.


- I noticed that the memory slowly grows as 'r.surf.contour -f' ran. Minor
memory leak?


- output is always CELL, regardless of input map type. If this is an indelible
feature of the module(?), it should at least be noted on the help page.


Otherwise I really like the output this module creates. I never liked having
to throw away data (by hand) to stop v.surf.rst from segmenting and aliasing
itself along the high density contour lines verticies, especially in low lying
areas & valley floors where is lots of empty space between contours.

Along with the contour lines I am lucky enough to have trig station elevations
so I don't have problems with "flat hill tops".


merging some other bugs into this one as similar issues (#1402 and #3515).


thanks,
Hamish
Wed, Apr 5 2006 12:25:07    Request 3515 merged into 4248 by hbowman (as #3515)  
Wed, Apr 5 2006 12:25:20    Request 1402 merged into 4248 by hbowman (as #1402)  
Wed, Apr 5 2006 12:28:37    User changed to hamish_nospam@yahoo.com,werchowyna@epf.pl by hbowman  
Wed, Apr 5 2006 13:44:57    Mail sent by msieczka  
Hamish wrote:

> I really like the output this module creates. I never liked having
> to throw away data (by hand) to stop v.surf.rst from segmenting and aliasing
> itself along the high density contour lines verticies, especially in low
> lying areas & valley floors where is lots of empty space between contours.
Hamish,

I'm also having too much troubles with v.surf.rst to be able to produce a
decent DEM from elevation contours. Seeking for an alternative I found Pavel
Sakov's "nnbathy" - a natural neighbor interpolator, based on Jonathan
Shewchuk's "triangle". It is even better for me than r.surf.contour - it
handles non continous contour lines fine, as well as elevation points between
contour lines, huge areas of no data, rapid elevation gradients, and is darn
fast. The source code is available.

http://www.marine.csiro.au/~sakov/

It is quite complicated to make use of it for Grass. But since I have managed
to do it, everydoby else will too :).

What I was thinking about is, instead of fixing r.surf.contour, to bring
nnbathy to Grass. IMHO it is a good idea because it provides the same
functionality + much more (rapid gradients and clustered input/missing input
tolerant), and is efficient. Of course I'm not able do do it, but wanted to
trigger your interest.

So far, here's how I proceed to produce a DEM with nnbathy from Grass data and
import it back into Grass:

#contour lines into points
v.to.points

#add x,y,z columns into points vector
v.db.addcol 

# copy Z form layer1 to layer2 in the points vector
v.to.db

# populate x,y with points coordinates in layer2
v.to.db

# vector points into ASCII file
v.db.select -c points layer=2 column=x,y,z fs=" " > points.asc

# remove duplicates from points.asc; sort BTW
sort points.asc | uniq > points.asc.sort.uniq

# create the GRID on which nnbathy is supposed to interpolate
g.region as needed
r.mapcalc 'background=null()'
r.stats -g background nv="" > background.asc

# run nnbathy, eg.
./nnbathy -i points.asc.sort.uniq -o background.asc > dem.grd

# replace "nan" in nnbathy output with something Grass understands, we'll
# filter this out later:

cat dem.grd | sed 's/nan/999999.0/g' > dem.grd.nonan

# import it into Grass vector:
v.in.ascii -t -z input=dem.grd.nonan output=dem_vect fs=space x=1 y=2 z=3

# vector into raster:
v.to.rast input=dem_vect output=dem use=z

# change 999999.0 into null
r.null

I wish it wasn't all that complicated. Anybody interested in implementing
nnbathy in Grass :) ? I believe it is worth it.

Maciek
Wed, Apr 5 2006 13:53:47    Mail sent by neteler@itc.it  
Return-Path <neteler@itc.it>
Delivered-To grass-bugs@lists.intevation.de
Date Wed, 5 Apr 2006 13:54:24 +0200
From Markus Neteler <neteler@itc.it>
To Maciek Sieczka via RT <grass-bugs@intevation.de>
Subject Re: [GRASS5] [bug #4248] (grass) r.surf.contour: Treats 0 as NULL and other archaic problems
Message-ID <20060405115424.GA10019@bartok.itc.it>
References <20060405114457.75D121006D1@lists.intevation.de>
Mime-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Disposition inline
In-Reply-To <20060405114457.75D121006D1@lists.intevation.de>
X-PGP-Key http://www.gdf-hannover.de/neteler/markus_gpgkey.asc
X-PGP-Fingerprint D4D5 2F80 120E AD60 E2F6 2297 21B3 D02B E1E7 E789
User-Agent Mutt/1.5.11
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Maciek,

> The source code is available.
> http://www.marine.csiro.au/~sakov/

what is the license of his software?

Markus


Wed, Apr 5 2006 14:28:11    Mail sent by hamish_nospam@yahoo.com  
Return-Path <hamish_nospam@yahoo.com>
Delivered-To grass-bugs@lists.intevation.de
Date Thu, 6 Apr 2006 00:27:56 +1200
From Hamish <hamish_nospam@yahoo.com>
To Maciek Sieczka via RT <grass-bugs@intevation.de>
Cc grass5@grass.itc.it
Subject Re: [GRASS5] [bug #4248] (grass) r.surf.contour: Treats 0 as NULL and other archaic problems
Message-Id <20060406002756.6aff1678.hamish_nospam@yahoo.com>
In-Reply-To <20060405114457.75D121006D1@lists.intevation.de>
References <20060405114457.75D121006D1@lists.intevation.de>
X-Mailer Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
X-Face M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
Mime-Version 1.0
Content-Type text/plain; charset=US-ASCII
Content-Transfer-Encoding 7bit
X-Spam-Status No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
X-Spam-Level
> I'm also having too much troubles with v.surf.rst to be able to
> produce a decent DEM from elevation contours. Seeking for an
> alternative I found Pavel Sakov's "nnbathy" - a natural neighbor
> interpolator, based on Jonathan Shewchuk's "triangle". It is even
> better for me than r.surf.contour - it handles non continous contour
> lines fine, as well as elevation points between contour lines, huge
> areas of no data, rapid elevation gradients, and is darn fast. The
> source code is available.
> 
> http://www.marine.csiro.au/~sakov/


How does nnbathy outpt differ from a rasterized TIN?
For that matter, how does r.surf.contour differ from a rasterized TIN?

e.g. v.3Dtin by Jachym Cepicky on the Wiki add-ons page + v.to.rast
Does v.to.rast work on "faces"? Should it?

What other options?
idw, idw2, v.krige (add-ons), and I imagine numerous other Kriging and
other varieties from the R-statistics interface.


Hamish


Wed, Apr 5 2006 15:09:40    Mail sent by msieczka  
Hamish wrote:

> How does nnbathy outpt differ from a rasterized TIN?

It is a smooth surface, no sign of a triangles. I know traingulation and
natural neighbor have something in common, but they are not exactly the same
thing.

> For that matter, how does r.surf.contour differ from a rasterized TIN?

As to my experience, it does. You can check for yourself - SAGA gis can
produce a rasterized TIN. r.surf.contours flooding algorithm is different from
triangulation.

> What other options?
> idw, idw2,

As to my experience, IDW interpolation is no any good for clusterized and 
sparse input data like in case of elevation contour lines. Also, it is not
able to handle rapid gradients well.

> v.krige (add-ons), and I imagine numerous other Kriging and

I only played with krigging in Surfer a bit, and was not satisfied with the
resultant DEM - grainy surface, rapid gradients lead to errors in surface.

> other varieties from the R-statistics interface.

Dunno.

But one thing more. What I like abouth nnbathy is that the result is easy to
foresee - what I intuitevily expect is mostly what I get. No fiddling around.
It is not trying to make the surface fit some maths function - it just follows
the data. This makes it possible to interpolate over a highly diverse surface
and still preserve an ideally flat floodland and a 100 m high vertical wall in
the center of it, without artifacts.

nnbathy will connect distant input data with a straight, expected, continous
surface - no brakes (IDW), segmentaion (RST) or artifacts like a hollow or a
bulge over a supposedly flat area (RST,IDW) - unless you tell it to do it by
feeding appropropiate input into it.

All pretty much like triangulation, but better because the output grid is
continous and smooth.

For contour lines it's best of what I've seen.

Maciek
Wed, Apr 5 2006 18:46:48    Mail sent by werchowyna@epf.pl  
Return-Path <werchowyna@epf.pl>
Delivered-To grass-bugs@lists.intevation.de
Date Wed, 5 Apr 2006 18:46:39 +0200
From Maciek Sieczka <werchowyna@epf.pl>
To Markus Neteler via RT <grass-bugs@intevation.de>
Cc hamish_nospam@yahoo.com, neteler@itc.it
Subject Re: [bug #4248] (grass) r.surf.contour: Treats 0 as NULL and other archaic problems
Message-Id <20060405184639.a3d2cbb2.werchowyna@epf.pl>
In-Reply-To <20060405115347.271371006D2@lists.intevation.de>
References <20060405115347.271371006D2@lists.intevation.de>
X-Mailer Sylpheed version 2.1.1 (GTK+ 2.8.6; i486-pc-linux-gnu)
Mime-Version 1.0
Content-Type text/plain; charset=US-ASCII
Content-Transfer-Encoding 7bit
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
On Wed,  5 Apr 2006 13:53:47 +0200 (CEST)
Markus Neteler via RT <grass-bugs@intevation.de> wrote:

> Maciek,
> 
> > The source code is available.
> > http://www.marine.csiro.au/~sakov/
> 
> what is the license of his software?


>From nn/README, as downloaded from
http://www.marine.csiro.au/~sakov/nn.tar.gz


---

nn
Natural Neighbours interpolation library
Version 1.57

C library functions and two utilities for Natural Neighbours
interpolation.

Copyright 2000-2005 CSIRO Marine Research
GPO 1538 Hobart
TAS 7001
Australia
Please send comments and bugs to Pavel.Sakov@csiro.au

There is no warranty whatsoever.  Use at your own risk.

This code may be freely redistributed under the condition that the
copyright notices are not removed. You may distribute modified versions
of this code UNDER THE CONDITION THAT THIS CODE AND ANY MODIFICATIONS
MADE TO IT IN THE SAME FILE REMAIN UNDER COPYRIGHT OF CSIRO, BOTH
SOURCE AND OBJECT CODE ARE MADE FREELY AVAILABLE WITHOUT CHARGE, AND
CLEAR NOTICE IS GIVEN OF THE MODIFICATIONS.

---

But the license name is not mentioned if this matters. I will ask author
right away.

Maciek

--------------------
W polskim Internecie s? setki milion?w stron. My przekazujemy Tobie tylko najlepsze
z nich!
http://katalog.panoramainternetu.pl/


Wed, Apr 5 2006 20:37:20    Mail sent by michael.barton@asu.edu  
Return-Path <Michael.Barton@asu.edu>
Delivered-To grass-bugs@lists.intevation.de
Date Wed, 05 Apr 2006 11:35:09 -0700
From Michael Barton <michael.barton@asu.edu>
Subject Re: [GRASS5] [bug #4248] (grass) r.surf.contour: Treats 0 as NULL and other archaic problems
In-reply-to <20060405102450.8A8EB100160@lists.intevation.de>
To Paolo Cavallini via RT <grass-bugs@intevation.de>, grass5@grass.itc.it
Message-id <C0595BED.200BC%michael.barton@asu.edu>
MIME-version 1.0
Content-type text/plain; charset=US-ASCII
Content-transfer-encoding 7bit
User-Agent Microsoft-Entourage/11.2.3.060209
Thread-Topic [GRASS5] [bug #4248] (grass) r.surf.contour: Treats 0 as NULL and other archaic problems
Thread-Index AcZY36qc6ViMzsTSEdqPsAAUUSYxwg==
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
I discovered what seems to be a similar behavior in d.his. It appears to
treat 0's as  nulls. I promised Glynn to send a test data set but haven't
had a chance to do so yet. Maybe someone else can also check this.

Michael
__________________________________________
Michael Barton, Professor of Anthropology
School of Human Evolution & Social Change
Center for Social Dynamics and Complexity
Arizona State University

phone: 480-965-6213
fax: 480-965-7671
www: http://www.public.asu.edu/~cmbarton


> From: Request Tracker <grass-bugs@intevation.de>
> Reply-To: Request Tracker <grass-bugs@intevation.de>
> Date: Wed,  5 Apr 2006 12:24:50 +0200 (CEST)
> To: <grass5@grass.itc.it>
> Subject: [GRASS5] [bug #4248] (grass) r.surf.contour: Treats 0 as NULL and
> other archaic problems
> 
> this bug's URL: http://intevation.de/rt/webrt?serial_num=4248
> -------------------------------------------------------------------------
> 
> Subject: r.surf.contour: Treats 0 as NULL and other archaic problems
> 
> Hi,
> 
> - If you feed r.surf.contour an input file including 0 as an elevation (say
a
> coastal area with the sea set to 0), it treats all the 0s as NULL and takes
> days to complete. Change the 0s to -1s and it goes by relatively quickly.
> Zero hasn't been NULL in GRASS for a long long time now.
> 
> 
> - Looking at the code I see support for a MASK is hardcoded, and the module
> does all sorts of bitwise operations on unmasked cells.. "w.t.f.?"
> 
> 
> - running some time/memory trials I see there is no use for not using the -f
> flag. Memory use was at most 5mb RAM during the non-default "memory intensive"
> run. I think we can handle that these days. With -f it was 4 times faster:
> 
> rows:       2900
> cols:       3000
> total null cells: 4048763
> 2.8GHz Pentium4
> 
> 
> CELL input, with the -f flag
>   real    42m23.483s
>   user    32m34.690s
>   sys     8m5.880s
>    2563 pts/6    RN+   40:37    207    19  5484 3172  0.1
> ================
> DCELL -f
>   real    41m8.144s
>   user    32m21.580s
>   sys     8m8.930s
>   10641 pts/6    RN+   40:27    217    19  5496 3224  0.1
> 
> ================
> DCELL no -f
>   real    178m50.506s
>   user    165m21.230s
>   sys     9m46.040s
>   13944 pts/6    RN+  175:06    279    19  3676 2460  0.1
> ================
> CELL no -f
>   real    178m54.097s
>   user    165m11.190s
>   sys     9m42.560s
>    1071 pts/6    RN+  174:52    270    19  3664 2412  0.1
> 
> 
> Let's get rid of non "-f" code in the module.
> 
> 
> - I noticed that the memory slowly grows as 'r.surf.contour -f' ran. Minor
> memory leak?
> 
> 
> - output is always CELL, regardless of input map type. If this is an indelible
> feature of the module(?), it should at least be noted on the help page.
> 
> 
> Otherwise I really like the output this module creates. I never liked having
> to throw away data (by hand) to stop v.surf.rst from segmenting and aliasing
> itself along the high density contour lines verticies, especially in low lying
> areas & valley floors where is lots of empty space between contours.
> 
> Along with the contour lines I am lucky enough to have trig station elevations
> so I don't have problems with "flat hill tops".
> 
> 
> merging some other bugs into this one as similar issues (#1402 and #3515).
> 
> 
> thanks,
> Hamish
> 
> 
> -------------------------------------------- Managed by Request Tracker
> 


Thu, Apr 6 2006 03:42:44    Mail sent by hamish_nospam@yahoo.com  
Return-Path <hamish_nospam@yahoo.com>
Delivered-To grass-bugs@lists.intevation.de
Date Thu, 6 Apr 2006 13:42:20 +1200
From Hamish <hamish_nospam@yahoo.com>
To Maciek Sieczka <werchowyna@epf.pl>
Cc grass-bugs@intevation.de, grass5@grass.itc.it
Subject Re: [bug #4248] (grass) r.surf.contour: Treats 0 as NULL and other archaic problems
Message-Id <20060406134220.2c6de22a.hamish_nospam@yahoo.com>
In-Reply-To <20060405184639.a3d2cbb2.werchowyna@epf.pl>
References <20060405115347.271371006D2@lists.intevation.de> <20060405184639.a3d2cbb2.werchowyna@epf.pl>
X-Mailer Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
X-Face M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
Mime-Version 1.0
Content-Type text/plain; charset=US-ASCII
Content-Transfer-Encoding 7bit
X-Spam-Status No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD, UPPERCASE_25_50
X-Spam-Level
> ---
> 
> nn
> Natural Neighbours interpolation library
> Version 1.57
> 
> C library functions and two utilities for Natural Neighbours
> interpolation.
> 
> Copyright 2000-2005 CSIRO Marine Research
> GPO 1538 Hobart
> TAS 7001
> Australia
> Please send comments and bugs to Pavel.Sakov@csiro.au
> 
> There is no warranty whatsoever.  Use at your own risk.
> 
> This code may be freely redistributed under the condition that the
> copyright notices are not removed. You may distribute modified
> versions of this code UNDER THE CONDITION THAT THIS CODE AND ANY
> MODIFICATIONS MADE TO IT IN THE SAME FILE REMAIN UNDER COPYRIGHT OF
> CSIRO, BOTH SOURCE AND OBJECT CODE ARE MADE FREELY AVAILABLE WITHOUT
> CHARGE, AND CLEAR NOTICE IS GIVEN OF THE MODIFICATIONS.
> 
> ---

"BOTH SOURCE AND OBJECT CODE ARE MADE FREELY AVAILABLE WITHOUT CHARGE"

I don't know if that is GPL compatible; you can sell GPL binaries, you
just have to give your customer the source code if they want it (and if
the customer wants they can do what they want with it, e.g. publish that
source on the net for everyone else).

Also I don't like the fact that any changes I make to the code become
"property" of CSRIO, even if it is easy enough to keep nn as a function
in separate file.


> But the license name is not mentioned if this matters. I will ask
> author right away.

Maybe he is happy with a GPL or dual-GPL license agreement with GRASS.



Hamish


Tue, Apr 18 2006 07:25:19    Comments added by hbowman  
notes about "0 is NULL" and "CELL output only" added to the BUGS section of
the help page.


Hamish
Tue, Apr 18 2006 08:29:34    Comments added by hbowman  
Cc: grass5@grass.itc.it

r.surf.contour's fast mode is now the default.
Slow mode marked for future removal.
Module's options and flags have been left in a backwards compatible way.
(fast: ~5mb RAM use   slow: ~3mb RAM use  for a 3000x3000 cell region)

0 as NULL and CELL output only remain to be fixed.


Hamish
Wed, Jul 26 2006 18:22:59    User changed to hamish_nospam@yahoo.com,tutey2o2.pl@ by msieczka  
Wed, Jul 26 2006 18:23:15    User changed to hamish_nospam@yahoo.com,tutey@o2.pl by msieczka  
Comment | Reply | Take | Resolve

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request