Tue, Mar 22 2005
16:53:35
|
|
Request created by guest
|
|
Subject: bug in r.watershed
Platform: other
grass obtained from: Mirror of Trento site
grass binary for platform: Downloaded precompiled Binaries
GRASS Version: 5.4
I tried to use r.watershed with the spearfish data loaded on the website. I have
created the different maps needed (depression,amount of overland flow, terrain
blocking overland surface flow) and define the values (maximum size, maximum
length). The r.watershed command start but never stop. It is still written :
Running: /usr/local/grass54/etc/water/ram el="elevation.dem" de="essaidepression"
ov="essaiaccumulationflux" r="50" ob="essaiflowdirection" t=1000 ms=150 ac="essainbcells"
se="essaistreamsegment" di="essairesult" 98%
and stay at 98 %
Could il be a Mandrake 10.1 problem ? Or just a mistake in the data compilation
?
Thanks
Pierre-Antoine
|
|
Thu, Mar 31 2005
13:01:12
|
|
Comments added by guest
|
|
It realy seems to be bug in r.watershed. I was running r.watershed on my data
and it was realy fast until it get to 98%. I was waiting for tose 2% 3-4 times
longer than processing all 98% and then just killed process (I cant sleep when
my PC is on). Some maps where ready and usable but some not.
If there is still some job to do, then must be more feedback to user to be
shore, that r.watershed isnt just "hanging" and
it is not normal, that closing maps takes much longer time than rest of
processing. |
|
Wed, Jun 29 2005
09:16:58
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Wed, 29 Jun 2005 19:15:52 +1200
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
orkun <temiz@deprem.gov.tr>
|
Cc |
massimiliano.cannata@supsi.ch, grasslist@baylor.edu, grass-bugs@intevation.de
|
Subject |
Re: [GRASSLIST:7361] [bug #3112] grass has been running for three days
|
Message-Id |
<20050629191552.143d1b1d.hamish_nospam@yahoo.com>
|
In-Reply-To |
<42C2428C.8070704@deprem.gov.tr>
|
References |
<42C0E772.5090101@deprem.gov.tr> <42C195F7.3070904@supsi.ch> <42C2428C.8070704@deprem.gov.tr>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
|
X-Spam-Level |
|
> >> grass has been running for three days to create watershed related
> >> maps.
> >> At this moment, it has been in this stage for a long time (almost
> >> one day):
> >> ~~~~~~~~~~
> >> Running: /usr/lib/grass/etc/water/ram el="el_5" t=100 se="str_5"
> >> ha="hbas20_50"
> >> SECTION 1a (of 5): Initiating Memory.
> >> SECTION 1b (of 5): Determining Offmap Flow. Percent Complete:
> >> SECTION 2: A * Search. Percent complete:
> >> SECTION 3: Accumulating Surface Flow. Percent complete:
> >> SECTION 4: Watershed determination.
> >> SECTION 5: Closing Maps.
> >> ~~~~~~~~~
> >>
> >> I am not sure if grass will be able to complete it's job ?
> >> or is it a good idea to cancel the job and start again with
> >> smaller region?
> >> what do you suggest ?
...
> > prcessing very large maps is quite slow with r.whatershed (I guess
> > you are using this command!).
..
> as far as I see r.terraflow does not produce half basin.
> I need a module that creates half basin other than r.watershed.
Looks like the while loop in raster/r.watershed/ram/close_maps2.c
line 38 is never satisfied.
Maybe same endless call to close_array_seg() is what causes bug #3112?
Things often report 98% when actually done as G_percent() is often
called in the wrong place or doesn't get called after exit. (e.g. watch
r.terrflow run) So it is likely it finished & then got stuck on the
cleanup step.
-- how big is the test region?
-- r.info: CELL map, DCELL map?
(idea- try a CELL version [r.mapcalc 'newmap=int(oldmap)'])
that's about as deep as I can look into it right now.
Hamish
|
|
Thu, Oct 27 2005
01:56:46
|
|
Comments added by guest
|
|
This still seems to be an issue. On grass-6.1.cvs_src_snapshot_2005_09_10, I
ran
r.watershed el=Elev_srtm_finished th=100 st=stream.map ba=basin.map vi=visual.map
where r.info on Elev_srtm_finished gives
+----------------------------------------------------------------------------+
| Layer: Elev_srtm_finished Date: Thu Jun 16 16:00:03 2005
|
| Mapset: srtm Login of Creator: harri
|
| Location: Villas
|
| DataBase: /home/harri/grass
|
| Title: ( Elev_srtm_finished )
|
|----------------------------------------------------------------------------|
|
|
| Type of Map: raster Number of Categories: 255
|
| Data Type: FCELL
|
| Rows: 4286
|
| Columns: 4765
|
| Total Cells: 20422790
|
| Projection: Latitude-Longitude (zone 0)
|
| N: 44:12:23.99796N S: 40:38:06.00946N Res: 0:00:02.999997
|
| E: 13:11:59.987155E W: 9:13:44.99994E Res: 0:00:02.999997
|
| Range of data: min = -40.000000 max = 2199.000000
|
|
|
| Data Source:
|
|
|
|
|
|
|
| Data Description:
|
| generated by r.in.gdal
|
|
|
|
|
+----------------------------------------------------------------------------+
'Determinig offmap flow' took almost one day, the next phases went past during
the night, but 'Closing maps' never finished; I waited for two days on this
one. During the last phase, I kept observing the memory usage of the
ram-subprogram. The virtual memory kept steadily at 631M, but the currently
used share of physical memory (RES in top) decreased slowly but surely from
over 300M to around 2M, where it stayed for 24 hours. No files appeared
anywhere. At this point, I had to reboot my laptop for other reasons.
On smaller maps the 'Closing...' seems to finish, but the speed of it seems to
depend on the value of the threshold-option selected. A subset of the raster
above (n=42.5 s=42 w=12.5 e=13, 600 x 600) produced the rasters in 30 seconds
with threshold=100. With threshold=10 other operations were somewhat slower,
not much though, but the closing of the maps is still running after a few
minutes. Quite a difference, and I suspect that the maps may never be closed.
No raster files have appeared anywhere, so its not a case of "its really
finished, it just doesn't seem so".
If the routine is still "ok" and "this thing just takes time", some
informative output would be nice... Of course, a threshold=10 is quite
ridiculous in this case, but still.
Greetings,
Harri K.
harri._spamless_kiiskinen@nospam_utu.fi |
|
Thu, Oct 27 2005
20:12:47
|
|
Comments added by guest
|
|
This still seems to be an issue. On grass-6.1.cvs_src_snapshot_2005_09_10, I
ran
r.watershed el=Elev_srtm_finished th=100 st=stream.map ba=basin.map vi=visual.map
where r.info on Elev_srtm_finished gives
+----------------------------------------------------------------------------+
| Layer: Elev_srtm_finished Date: Thu Jun 16 16:00:03 2005
|
| Mapset: srtm Login of Creator: harri
|
| Location: Villas
|
| DataBase: /home/harri/grass
|
| Title: ( Elev_srtm_finished )
|
|----------------------------------------------------------------------------|
|
|
| Type of Map: raster Number of Categories: 255
|
| Data Type: FCELL
|
| Rows: 4286
|
| Columns: 4765
|
| Total Cells: 20422790
|
| Projection: Latitude-Longitude (zone 0)
|
| N: 44:12:23.99796N S: 40:38:06.00946N Res: 0:00:02.999997
|
| E: 13:11:59.987155E W: 9:13:44.99994E Res: 0:00:02.999997
|
| Range of data: min = -40.000000 max = 2199.000000
|
|
|
| Data Source:
|
|
|
|
|
|
|
| Data Description:
|
| generated by r.in.gdal
|
|
|
|
|
+----------------------------------------------------------------------------+
'Determinig offmap flow' took almost one day, the next phases went past during
the night, but 'Closing maps' never finished; I waited for two days on this
one. During the last phase, I kept observing the memory usage of the
ram-subprogram. The virtual memory kept steadily at 631M, but the currently
used share of physical memory (RES in top) decreased slowly but surely from
over 300M to around 2M, where it stayed for 24 hours. No files appeared
anywhere. At this point, I had to reboot my laptop for other reasons.
On smaller maps the 'Closing...' seems to finish, but the speed of it seems to
depend on the value of the threshold-option selected. A subset of the raster
above (n=42.5 s=42 w=12.5 e=13, 600 x 600) produced the rasters in 30 seconds
with threshold=100. With threshold=10 other operations were somewhat slower,
not much though, but the closing of the maps is still running after a few
minutes. Quite a difference, and I suspect that the maps may never be closed.
No raster files have appeared anywhere, so its not a case of "its really
finished, it just doesn't seem so".
If the routine is still "ok" and "this thing just takes time", some
informative output would be nice... Of course, a threshold=10 is quite
ridiculous in this case, but still.
Greetings,
Harri K.
harri._spamless_kiiskinen@nospam_utu.fi |
|
Thu, Oct 27 2005
20:13:17
|
|
Comments added by guest
|
|
This still seems to be an issue. On grass-6.1.cvs_src_snapshot_2005_09_10, I
ran
r.watershed el=Elev_srtm_finished th=100 st=stream.map ba=basin.map vi=visual.map
where r.info on Elev_srtm_finished gives
+----------------------------------------------------------------------------+
| Layer: Elev_srtm_finished Date: Thu Jun 16 16:00:03 2005
|
| Mapset: srtm Login of Creator: harri
|
| Location: Villas
|
| DataBase: /home/harri/grass
|
| Title: ( Elev_srtm_finished )
|
|----------------------------------------------------------------------------|
|
|
| Type of Map: raster Number of Categories: 255
|
| Data Type: FCELL
|
| Rows: 4286
|
| Columns: 4765
|
| Total Cells: 20422790
|
| Projection: Latitude-Longitude (zone 0)
|
| N: 44:12:23.99796N S: 40:38:06.00946N Res: 0:00:02.999997
|
| E: 13:11:59.987155E W: 9:13:44.99994E Res: 0:00:02.999997
|
| Range of data: min = -40.000000 max = 2199.000000
|
|
|
| Data Source:
|
|
|
|
|
|
|
| Data Description:
|
| generated by r.in.gdal
|
|
|
|
|
+----------------------------------------------------------------------------+
'Determinig offmap flow' took almost one day, the next phases went past during
the night, but 'Closing maps' never finished; I waited for two days on this
one. During the last phase, I kept observing the memory usage of the
ram-subprogram. The virtual memory kept steadily at 631M, but the currently
used share of physical memory (RES in top) decreased slowly but surely from
over 300M to around 2M, where it stayed for 24 hours. No files appeared
anywhere. At this point, I had to reboot my laptop for other reasons.
On smaller maps the 'Closing...' seems to finish, but the speed of it seems to
depend on the value of the threshold-option selected. A subset of the raster
above (n=42.5 s=42 w=12.5 e=13, 600 x 600) produced the rasters in 30 seconds
with threshold=100. With threshold=10 other operations were somewhat slower,
not much though, but the closing of the maps is still running after a few
minutes. Quite a difference, and I suspect that the maps may never be closed.
No raster files have appeared anywhere, so its not a case of "its really
finished, it just doesn't seem so".
If the routine is still "ok" and "this thing just takes time", some
informative output would be nice... Of course, a threshold=10 is quite
ridiculous in this case, but still.
Greetings,
Harri K.
harri._spamless_kiiskinen@nospam_utu.fi |
|
Mon, Jan 2 2006
21:24:41
|
|
Area changed to grass6 by msieczka
|
|
Mon, Jan 2 2006
21:28:37
|
|
Mail sent by msieczka
|
|
According To Helena this bug still hives in 6.1. |
|
Thu, Aug 17 2006
15:38:37
|
|
Subject changed to r.watershed: hangs at 'SECTION 5: Closing Maps.' with bigger input by msieczka
|
|
Mon, Oct 23 2006
04:35:03
|
|
Comments added by hbowman
|
|
Hint from the module's author:
----------------------------------------
From: "Charles Ehlschlaeger" <c.ehlschlaeger at insightbb,com>
Subject: RE: [GRASS-user] r.watershed hangs
Date: Sun, 22 Oct 2006 21:18:41 -0500
To: grassuser at grass,itc,it
I think the problem occurs with "threshhold=3" You are telling the software
to create exterior watershed basins only 3 grid cells in size. I can't
remember off the top of my head whether that will potentially cause stack
overflow problems or maybe I forgot to include a check to see whether an
array memory allocation returned a null pointer.
The solution is to include a more realistic threshold size. Ask yourself
"what is the smallest watershed basin you want to consider?" If the answer
is 10,000m^2, and grid cells are 10x10m, then threshold should be set to
100.
Sincerely, chuck
----------------------------------------
|
|
Wed, Oct 25 2006
10:02:19
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Wed, 25 Oct 2006 21:01:26 +1300
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
grass5 <grass-dev@grass.itc.it>
|
Cc |
hardeep@eml.cc, grass-bugs@intevation.de, c.ehlschlaeger@insightbb.com
|
Subject |
Re: [bug #3112] r.watershed hangs
|
Message-Id |
<20061025210126.4a3f0da8.hamish_nospam@yahoo.com>
|
In-Reply-To |
<20061023140326.7b516601.hamish_nospam@yahoo.com>
|
References |
<1161484146.15628.273909957@webmail.messagingengine.com> <20061023140326.7b516601.hamish_nospam@yahoo.com>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
multipart/mixed; boundary="Multipart=_Wed__25_Oct_2006_21_01_26_+1300_YD_uF2hcKL8oY1kj"
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-2.3 tagged_above=-999 required=3 tests=[BAYES_00=-5, FORGED_YAHOO_RCVD=2.7]
|
X-Spam-Level |
|
This is a multi-part message in MIME format.
--Multipart=_Wed__25_Oct_2006_21_01_26_+1300_YD_uF2hcKL8oY1kj
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Hardeep Singh Rai wrote:
> I was doing watershed analysis for spearfish dataset
> (spearfish_grass60data-0.1.tar.gz), and on giving command:
>
> r.watershed elevation=elevation.dem threshhold=3 basin=raiBasin
> accumulation=raiAccu it starts processing of 5 steps. After about 20
> minutes, it completes all and throw message: "cloing maps", but do
> nothing; prompt never comes. I used it with option -m and without it.
> With both cases, "top" commands show ram or sed running with 99 usage
> of CPU. Even I waited for 12 hours, but it fail to finish. I am using
> GRASS 6.0.1 on Ubuntu.
>
> Please suggest some solution.
Hamish wrote:
> It's a known bug (#3112). It gets stuck trying to make a huge set of
> colormap rules for the new map. If you cancel (^C) you might find the
> map was created anyway (???).
Chuck wrote:
> The solution is to include a more realistic threshold size.
Yes.
The attached patch shows the problem quite well. As the treshold gets
smaller, the number of areas (cats) grows exponentially. Not only that,
but as the loop within a loop runs, it takes longer and longer to do the
same thing.
The inner loop gets slower I think due to the increased number of color
rules set with G_set_color() that have to be read each time.
#spearfish
g.region rast=elevation.dem
r.watershed elev=elevation.dem basin=tmp_basin thresh=200
d.rast tmp_basin
thresh=200 # 1488 cats
real 0m38.558s
user 0m37.770s
sys 0m0.110s
thresh=100 # 2858 cats
real 0m41.296s
user 0m41.050s
sys 0m0.120s
thresh=50 # 5482 cats
real 0m57.495s
user 0m55.870s
sys 0m0.180s
thresh=37 # 7258 cats
real 1m20.985s
user 1m20.340s
sys 0m0.200s
thresh=25 # 10648 cats
real 3m5.185s
user 3m4.020s
sys 0m0.280s
there are a lot of loops here, but I think the main problem is with so
many calls to G_{get,set}_color():
raster/r.watershed/ram/close_maps2.c
..
G_init_colors(&colors);
G_make_random_colors(&colors, 1, max);
G_set_color((CELL)0, 0, 0, 0, &colors);
r = 1;
incr = 0;
while(incr >= 0) {
for (gr=130+incr; gr<=255; gr += 20) {
for (rd=90+incr; rd<=255; rd += 30) {
for (bl=90+incr; bl<=255; bl += 40) {
flag = 1;
while (flag) {
G_get_color(r,&red,&green,&blue, &colors);
if((blue*.11 + red*.30 + green*.59) < 100) {
G_set_color(r, rd, gr, bl, &colors);
flag = 0;
}
if (++r > max) {
gr = rd = bl = 300;
flag = 0;
incr = -1;
}
}
}
}
}
if (incr >= 0) {
incr += 15;
if (incr > 120)
incr = 7;
}
}
Hamish
--Multipart=_Wed__25_Oct_2006_21_01_26_+1300_YD_uF2hcKL8oY1kj
Content-Type: text/plain;
name="rwaters_loop.diff"
Content-Disposition: attachment;
filename="rwaters_loop.diff"
Content-Transfer-Encoding: 7bit
Index: raster/r.watershed/ram/close_maps2.c
===================================================================
RCS file: /home/grass/grassrepository/grass6/raster/r.watershed/ram/close_maps2.c,v
retrieving revision 2.0
diff -u -r2.0 close_maps2.c
--- raster/r.watershed/ram/close_maps2.c 9 Nov 2004 13:32:28 -0000 2.0
+++ raster/r.watershed/ram/close_maps2.c 25 Oct 2006 07:44:47 -0000
@@ -46,6 +46,7 @@
G_set_color(r, rd, gr, bl, &colors);
flag = 0;
}
+if(r % 200 == 0) G_debug(0,"r=%d\tof %d", r, max);
if (++r > max) {
gr = rd = bl = 300;
flag = 0;
--Multipart=_Wed__25_Oct_2006_21_01_26_+1300_YD_uF2hcKL8oY1kj--
|
|
Wed, Oct 25 2006
23:35:20
|
|
Mail sent by c.ehlschlaeger@insightbb.com
|
|
Return-Path |
<c.ehlschlaeger@insightbb.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Message-Id |
<5ag1pm$5chkh4@asav13.insightbb.com>
|
X-IronPort-Anti-Spam-Filtered |
true
|
X-IronPort-Anti-Spam-Result |
Ao8CACN1P0VKi5Lq/2dsb2JhbAA
|
From |
"Charles Ehlschlaeger" <c.ehlschlaeger@insightbb.com>
|
To |
"'Hamish'" <hamish_nospam@yahoo.com>, "'grass5'" <grass-dev@grass.itc.it>
|
Cc |
<hardeep@eml.cc>, <grass-bugs@intevation.de>
|
Subject |
RE: [bug #3112] r.watershed hangs
|
Date |
Wed, 25 Oct 2006 16:35:16 -0500
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset="us-ascii"
|
Content-Transfer-Encoding |
7bit
|
X-Mailer |
Microsoft Office Outlook, Build 11.0.6353
|
In-reply-to |
<20061025210126.4a3f0da8.hamish_nospam@yahoo.com>
|
X-MIMEOLE |
Produced By Microsoft MimeOLE V6.00.2900.2962
|
Thread-index |
Acb4C94HAaliOZvlT8m0PJ+dZdQb/wAcKLCQ
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-5 tagged_above=-999 required=3 tests=[BAYES_00=-5]
|
X-Spam-Level |
|
Dear Hamish,
Thanks for posting the patch of r.watershed. That snippet of code ensures
that colors allocated to watersheds are as different as possible. Back in
the early '90s, we were so worried about RAM use that we wrote convoluted
code that prevented making new arrays whenever possible.
The simpliest fix would be to insert a conditional after
G_make_random_colors(&colors, 1, max) as shown in the patch below (two lines
added). From a visualization perspective, 1000 basins are too many basins to
matter anyway, so 1000 might be good. If you look at the experiment you
performed, you can see 10000 basins took 3 minutes to run.
The best fix would be to write an algorithm that chooses category color
based on adjacent cell values. Here is a psuedo code solution:
random_colors( Raster map, Color[] colors) {
long adjacentColors[][] = new long[ colors.length][];
for( int r = 0; r < map.rows(); r++) {
for( int c = 0; c < map.cols(); c++) {
long cellValue = map.getValue( r, c);
if( c < map.cols() - 1) {
if( map.getValue( r, c + 1) < cellValue) {
// add map.getValue( r, c + 1) to adjacentColors[
cellValue][] if not already in array
}
}
if( r < map.rows() - 1) {
if( map.getValue( r + 1, c ) < cellValue) {
// add map.getValue( r + 1, c) to adjacentColors[
cellValue][] if not already in array
}
}
if( (r < map.rows() - 1) && c < map.cols() - 1) {
if( map.getValue( r + 1, c + 1 ) < cellValue) {
// add map.getValue( r + 1, c + 1) to adjacentColors[
cellValue][] if not already in array
}
}
}
}
for( int i = 0; i < colors.length; i++) {
// if no adjacentColors[i] == null, give any random color
// otherwise randomly create 100 colors
// choose color furthest from colors of adjacentColors[i] array
}
}
Sincerely, chuck
-----Original Message-----
From: Hamish [mailto:hamish_nospam@yahoo.com]
Sent: Wednesday, October 25, 2006 3:01 AM
To: grass5
Cc: hardeep@eml.cc; grass-bugs@intevation.de; c.ehlschlaeger@insightbb.com
Subject: Re: [bug #3112] r.watershed hangs
Hardeep Singh Rai wrote:
> I was doing watershed analysis for spearfish dataset
> (spearfish_grass60data-0.1.tar.gz), and on giving command:
>
> r.watershed elevation=elevation.dem threshhold=3 basin=raiBasin
> accumulation=raiAccu it starts processing of 5 steps. After about 20
> minutes, it completes all and throw message: "cloing maps", but do
> nothing; prompt never comes. I used it with option -m and without it.
> With both cases, "top" commands show ram or sed running with 99 usage
> of CPU. Even I waited for 12 hours, but it fail to finish. I am using
> GRASS 6.0.1 on Ubuntu.
>
> Please suggest some solution.
Hamish wrote:
> It's a known bug (#3112). It gets stuck trying to make a huge set of
> colormap rules for the new map. If you cancel (^C) you might find the
> map was created anyway (???).
Chuck wrote:
> The solution is to include a more realistic threshold size.
Yes.
The attached patch shows the problem quite well. As the treshold gets
smaller, the number of areas (cats) grows exponentially. Not only that,
but as the loop within a loop runs, it takes longer and longer to do the
same thing.
The inner loop gets slower I think due to the increased number of color
rules set with G_set_color() that have to be read each time.
#spearfish
g.region rast=elevation.dem
r.watershed elev=elevation.dem basin=tmp_basin thresh=200
d.rast tmp_basin
thresh=200 # 1488 cats
real 0m38.558s
user 0m37.770s
sys 0m0.110s
thresh=100 # 2858 cats
real 0m41.296s
user 0m41.050s
sys 0m0.120s
thresh=50 # 5482 cats
real 0m57.495s
user 0m55.870s
sys 0m0.180s
thresh=37 # 7258 cats
real 1m20.985s
user 1m20.340s
sys 0m0.200s
thresh=25 # 10648 cats
real 3m5.185s
user 3m4.020s
sys 0m0.280s
there are a lot of loops here, but I think the main problem is with so
many calls to G_{get,set}_color():
raster/r.watershed/ram/close_maps2.c
..
G_init_colors(&colors);
G_make_random_colors(&colors, 1, max);
// new code next line
if( max < 10000) {
G_set_color((CELL)0, 0, 0, 0, &colors);
r = 1;
incr = 0;
while(incr >= 0) {
for (gr=130+incr; gr<=255; gr += 20) {
for (rd=90+incr; rd<=255; rd += 30) {
for (bl=90+incr; bl<=255; bl += 40) {
flag = 1;
while (flag) {
G_get_color(r,&red,&green,&blue, &colors);
if((blue*.11 + red*.30 + green*.59) < 100) {
G_set_color(r, rd, gr, bl, &colors);
flag = 0;
}
if (++r > max) {
gr = rd = bl = 300;
flag = 0;
incr = -1;
}
}
}
}
}
if (incr >= 0) {
incr += 15;
if (incr > 120)
incr = 7;
}
}// new code next line
}
Hamish
|
|
Thu, Oct 26 2006
07:33:45
|
|
Comments added by hbowman
|
|
From: "Charles Ehlschlaeger"
Subject: RE: [bug #3112] r.watershed hangs
Date: Wed, 25 Oct 2006 16:35:16 -0500
Dear Hamish,
Thanks for posting the patch of r.watershed. That snippet of code ensures
that colors allocated to watersheds are as different as possible. Back in
the early '90s, we were so worried about RAM use that we wrote convoluted
code that prevented making new arrays whenever possible.
The simpliest fix would be to insert a conditional after
G_make_random_colors(&colors, 1, max) as shown in the patch below (two lines
added). From a visualization perspective, 1000 basins are too many basins to
matter anyway, so 1000 might be good. If you look at the experiment you
performed, you can see 10000 basins took 3 minutes to run.
The best fix would be to write an algorithm that chooses category color
based on adjacent cell values. Here is a psuedo code solution:
random_colors( Raster map, Color[] colors) {
long adjacentColors[][] = new long[ colors.length][];
for( int r = 0; r < map.rows(); r++) {
for( int c = 0; c < map.cols(); c++) {
long cellValue = map.getValue( r, c);
if( c < map.cols() - 1) {
if( map.getValue( r, c + 1) < cellValue) {
// add map.getValue( r, c + 1) to adjacentColors[
cellValue][] if not already in array
}
}
if( r < map.rows() - 1) {
if( map.getValue( r + 1, c ) < cellValue) {
// add map.getValue( r + 1, c) to adjacentColors[
cellValue][] if not already in array
}
}
if( (r < map.rows() - 1) && c < map.cols() - 1) {
if( map.getValue( r + 1, c + 1 ) < cellValue) {
// add map.getValue( r + 1, c + 1) to adjacentColors[
cellValue][] if not already in array
}
}
}
}
for( int i = 0; i < colors.length; i++) {
// if no adjacentColors[i] == null, give any random color
// otherwise randomly create 100 colors
// choose color furthest from colors of adjacentColors[i] array
}
}
Sincerely, chuck
raster/r.watershed/ram/close_maps2.c
..
G_init_colors(&colors);
G_make_random_colors(&colors, 1, max);
// new code next line
if( max < 10000) {
G_set_color((CELL)0, 0, 0, 0, &colors);
r = 1;
[...]
if (incr >= 0) {
incr += 15;
if (incr > 120)
incr = 7;
}
}// new code next line
}
|
|
Thu, Oct 26 2006
07:59:56
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Thu, 26 Oct 2006 18:59:37 +1300
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
"Charles Ehlschlaeger" <c.ehlschlaeger@insightbb.com>
|
Cc |
grass-dev@grass.itc.it, hardeep@eml.cc, grass-bugs@intevation.de
|
Subject |
Re: [bug #3112] r.watershed hangs
|
Message-Id |
<20061026185937.6ba81de6.hamish_nospam@yahoo.com>
|
In-Reply-To |
<5ag1pm$5chkh4@asav13.insightbb.com>
|
References |
<20061025210126.4a3f0da8.hamish_nospam@yahoo.com> <5ag1pm$5chkh4@asav13.insightbb.com>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-2.3 tagged_above=-999 required=3 tests=[BAYES_00=-5, FORGED_YAHOO_RCVD=2.7]
|
X-Spam-Level |
|
Charles Ehlschlaeger wrote:
> Thanks for posting the patch of r.watershed. That snippet of code
> ensures that colors allocated to watersheds are as different as
> possible. Back in the early '90s, we were so worried about RAM use
> that we wrote convoluted code that prevented making new arrays
> whenever possible.
IMO this is why GRASS has (in general) scaled so well for today's
massive datasets. Raster overheads are very low.
> The simpliest fix would be to insert a conditional after
> G_make_random_colors(&colors, 1, max) as shown in the patch below (two
> lines added).
thanks, applied to the 6.2 & 6.3(HEAD) branches.
Also took the opportunity to slop on lots of metadata in the output
maps (6.3cvs only).
> From a visualization perspective, 1000 basins are too
> many basins to matter anyway, so 1000 might be good. If you look at
> the experiment you performed, you can see 10000 basins took 3 minutes
> to run.
I went with 10000.
> The best fix would be to write an algorithm that chooses category
> color based on adjacent cell values. Here is a psuedo code solution:
>
> random_colors( Raster map, Color[] colors) {
> long adjacentColors[][] = new long[ colors.length][];
> for( int r = 0; r < map.rows(); r++) {
> for( int c = 0; c < map.cols(); c++) {
> long cellValue = map.getValue( r, c);
> if( c < map.cols() - 1) {
> if( map.getValue( r, c + 1) < cellValue) {
> // add map.getValue( r, c + 1) to adjacentColors[
> cellValue][] if not already in array
> }
> }
> if( r < map.rows() - 1) {
> if( map.getValue( r + 1, c ) < cellValue) {
> // add map.getValue( r + 1, c) to adjacentColors[
> cellValue][] if not already in array
> }
> }
> if( (r < map.rows() - 1) && c < map.cols() - 1) {
> if( map.getValue( r + 1, c + 1 ) < cellValue) {
> // add map.getValue( r + 1, c + 1) to adjacentColors[
> cellValue][] if not already in array
> }
> }
> }
> }
> for( int i = 0; i < colors.length; i++) {
> // if no adjacentColors[i] == null, give any random color
> // otherwise randomly create 100 colors
> // choose color furthest from colors of adjacentColors[i] array
>
> }
> }
The random case doesn't do too poorly. (256^3 color possibilities vs.
the four-color theorem)
> // choose color furthest from colors of adjacentColors[i] array
color distance in 3D RGB space isn't something I understand how to do
well. (I've wondered for a while how to find the "opposite" RGB color:
think of it as a vector ray from the origin inside a 3D sphere?)
Anyway, I'll have to leave the "better way" for someone who understands
color-space a bit better than I do.
thanks,
Hamish
|
|
Thu, Nov 9 2006
07:14:41
|
|
Status changed to resolved by hbowman
|
|
Thu, Nov 9 2006
07:14:41
|
|
Comments added by hbowman
|
|
fixed in CVS, closing bug
Hamish
|
|