Details Ticket 3112


Comment | Reply | Take | Open


Serial Number 3112
Subject r.watershed: hangs at 'SECTION 5: Closing Maps.' with bigger input
Area grass6
Queue grass
Requestors pierre-antoine.versini@lcpc.fr
Owner none
Status resolved
Last User Contact Thu Oct 26 07:59:56 2006 (2 yr ago)
Current Priority 30
Final Priority 70
Due No date assigned
Last Action Thu Nov 9 07:14:41 2006 (2 yr ago)
Created Tue Mar 22 16:53:35 2005 (3 yr ago)

Transaction History Ticket 3112


Tue, Mar 22 2005 16:53:35    Request created by guest  
Subject: bug in r.watershed

Platform: other
grass obtained from: Mirror of Trento site
grass binary for platform: Downloaded precompiled Binaries
GRASS Version: 5.4

I tried to use r.watershed with the spearfish data loaded on the website. I have
created the different maps needed (depression,amount of overland flow, terrain
blocking overland surface flow) and define the values (maximum size, maximum
length). The r.watershed command start but never stop. It is still written :
Running: /usr/local/grass54/etc/water/ram el="elevation.dem" de="essaidepression"
ov="essaiaccumulationflux" r="50" ob="essaiflowdirection" t=1000 ms=150 ac="essainbcells"
se="essaistreamsegment" di="essairesult"  98%

and stay at 98 %

Could il be a Mandrake 10.1 problem ? Or just a mistake in the data compilation
?

Thanks

Pierre-Antoine

Thu, Mar 31 2005 13:01:12    Comments added by guest  
It realy seems to be bug in r.watershed. I was running r.watershed on my data
and it was realy fast until it get to 98%. I was waiting for tose 2% 3-4 times
longer than processing all 98% and then just killed process (I cant sleep when
my PC is on). Some maps where ready and usable but some not. 
If there is still some job to do, then must be more feedback to user to be
shore, that r.watershed isnt just "hanging" and 
it is not normal, that closing maps takes much longer time than rest of
processing.
Wed, Jun 29 2005 09:16:58    Mail sent by hamish_nospam@yahoo.com  
Return-Path <hamish_nospam@yahoo.com>
Delivered-To grass-bugs@lists.intevation.de
Date Wed, 29 Jun 2005 19:15:52 +1200
From Hamish <hamish_nospam@yahoo.com>
To orkun <temiz@deprem.gov.tr>
Cc massimiliano.cannata@supsi.ch, grasslist@baylor.edu, grass-bugs@intevation.de
Subject Re: [GRASSLIST:7361] [bug #3112] grass has been running for three days
Message-Id <20050629191552.143d1b1d.hamish_nospam@yahoo.com>
In-Reply-To <42C2428C.8070704@deprem.gov.tr>
References <42C0E772.5090101@deprem.gov.tr> <42C195F7.3070904@supsi.ch> <42C2428C.8070704@deprem.gov.tr>
X-Mailer Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
X-Face M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
Mime-Version 1.0
Content-Type text/plain; charset=US-ASCII
Content-Transfer-Encoding 7bit
X-Spam-Status No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
X-Spam-Level
> >> grass has been running for three days to create watershed related
> >> maps. 
> >> At this moment, it has been in this stage for a long time (almost
> >> one day):
> >> ~~~~~~~~~~
> >> Running: /usr/lib/grass/etc/water/ram el="el_5" t=100 se="str_5" 
> >> ha="hbas20_50"
> >> SECTION 1a (of 5): Initiating Memory.
> >> SECTION 1b (of 5): Determining Offmap Flow. Percent Complete:
> >> SECTION 2: A * Search.                      Percent complete:
> >> SECTION 3: Accumulating Surface Flow.       Percent complete:
> >> SECTION 4: Watershed determination.
> >> SECTION 5: Closing Maps.
> >> ~~~~~~~~~
> >>
> >> I am not sure if  grass will be able to complete it's job ?
> >> or is it a good idea to cancel the job and start again with 
> >> smaller region? 
> >> what do you suggest ?
...
> > prcessing very large maps is quite slow with r.whatershed (I guess
> > you are using this command!).
..
> as far as I see r.terraflow does not produce half basin.
> I need a module that creates half basin other than r.watershed.


Looks like the while loop in raster/r.watershed/ram/close_maps2.c
line 38 is never satisfied.

Maybe same endless call to close_array_seg() is what causes bug #3112?

Things often report 98% when actually done as G_percent() is often 
called in the wrong place or doesn't get called after exit. (e.g. watch
r.terrflow run)  So it is likely it finished & then got stuck on the
cleanup step.


-- how big is the test region?
-- r.info:  CELL map, DCELL map?
    (idea- try a CELL version [r.mapcalc 'newmap=int(oldmap)'])


that's about as deep as I can look into it right now.


Hamish


Thu, Oct 27 2005 01:56:46    Comments added by guest  
This still seems to be an issue. On grass-6.1.cvs_src_snapshot_2005_09_10, I
ran    

r.watershed el=Elev_srtm_finished th=100 st=stream.map ba=basin.map vi=visual.map
where r.info on Elev_srtm_finished gives

 +----------------------------------------------------------------------------+
| Layer:    Elev_srtm_finished             Date: Thu Jun 16 16:00:03 2005   
|
| Mapset:   srtm                           Login of Creator: harri          
|
| Location: Villas                                                          
|
| DataBase: /home/harri/grass                                               
|
| Title:     ( Elev_srtm_finished )                                         
|
|----------------------------------------------------------------------------|
|                                                                           
|
|   Type of Map:  raster              Number of Categories: 255             
|
|   Data Type:    FCELL                                                     
|
|   Rows:         4286                                                      
|
|   Columns:      4765                                                      
|
|   Total Cells:  20422790                                                  
|
|        Projection: Latitude-Longitude (zone 0)                            
|
|            N: 44:12:23.99796N    S: 40:38:06.00946N   Res: 0:00:02.999997 
|
|            E: 13:11:59.987155E    W: 9:13:44.99994E   Res: 0:00:02.999997 
|
|   Range of data:    min =  -40.000000 max = 2199.000000                   
|
|                                                                           
|
|   Data Source:                                                            
|
|                                                                           
|
|                                                                           
|
|                                                                           
|
|   Data Description:                                                       
|
|    generated by r.in.gdal                                                 
|
|                                                                           
|
|                                                                           
|
+----------------------------------------------------------------------------+
'Determinig offmap flow' took almost one day, the next phases went past during
the night, but 'Closing maps' never finished; I waited for two days on this
one. During the last phase, I kept observing the memory usage of the
ram-subprogram. The virtual memory kept steadily at 631M, but the currently
used share of physical memory (RES in top) decreased slowly but surely from
over 300M to around 2M, where it stayed for 24 hours. No files appeared
anywhere. At this point, I had to reboot my laptop for other reasons.

On smaller maps the 'Closing...' seems to finish, but the speed of it seems to
depend on the value of the threshold-option selected. A subset of the raster
above (n=42.5 s=42 w=12.5 e=13, 600 x 600) produced the rasters in 30 seconds
with threshold=100. With threshold=10 other operations were somewhat slower,
not much though, but the closing of the maps is still running after a few
minutes. Quite a difference, and I suspect that the maps may never be closed.
No raster files have appeared anywhere, so its not a case of "its really
finished, it just doesn't seem so".

If the routine is still "ok" and "this thing just takes time", some
informative output would be nice... Of course, a threshold=10 is quite
ridiculous in this case, but still.

Greetings,

Harri K.
harri._spamless_kiiskinen@nospam_utu.fi
Thu, Oct 27 2005 20:12:47    Comments added by guest  
This still seems to be an issue. On grass-6.1.cvs_src_snapshot_2005_09_10, I
ran    

r.watershed el=Elev_srtm_finished th=100 st=stream.map ba=basin.map vi=visual.map
where r.info on Elev_srtm_finished gives

 +----------------------------------------------------------------------------+
| Layer:    Elev_srtm_finished             Date: Thu Jun 16 16:00:03 2005   
|
| Mapset:   srtm                           Login of Creator: harri          
|
| Location: Villas                                                          
|
| DataBase: /home/harri/grass                                               
|
| Title:     ( Elev_srtm_finished )                                         
|
|----------------------------------------------------------------------------|
|                                                                           
|
|   Type of Map:  raster              Number of Categories: 255             
|
|   Data Type:    FCELL                                                     
|
|   Rows:         4286                                                      
|
|   Columns:      4765                                                      
|
|   Total Cells:  20422790                                                  
|
|        Projection: Latitude-Longitude (zone 0)                            
|
|            N: 44:12:23.99796N    S: 40:38:06.00946N   Res: 0:00:02.999997 
|
|            E: 13:11:59.987155E    W: 9:13:44.99994E   Res: 0:00:02.999997 
|
|   Range of data:    min =  -40.000000 max = 2199.000000                   
|
|                                                                           
|
|   Data Source:                                                            
|
|                                                                           
|
|                                                                           
|
|                                                                           
|
|   Data Description:                                                       
|
|    generated by r.in.gdal                                                 
|
|                                                                           
|
|                                                                           
|
+----------------------------------------------------------------------------+
'Determinig offmap flow' took almost one day, the next phases went past during
the night, but 'Closing maps' never finished; I waited for two days on this
one. During the last phase, I kept observing the memory usage of the
ram-subprogram. The virtual memory kept steadily at 631M, but the currently
used share of physical memory (RES in top) decreased slowly but surely from
over 300M to around 2M, where it stayed for 24 hours. No files appeared
anywhere. At this point, I had to reboot my laptop for other reasons.

On smaller maps the 'Closing...' seems to finish, but the speed of it seems to
depend on the value of the threshold-option selected. A subset of the raster
above (n=42.5 s=42 w=12.5 e=13, 600 x 600) produced the rasters in 30 seconds
with threshold=100. With threshold=10 other operations were somewhat slower,
not much though, but the closing of the maps is still running after a few
minutes. Quite a difference, and I suspect that the maps may never be closed.
No raster files have appeared anywhere, so its not a case of "its really
finished, it just doesn't seem so".

If the routine is still "ok" and "this thing just takes time", some
informative output would be nice... Of course, a threshold=10 is quite
ridiculous in this case, but still.

Greetings,

Harri K.
harri._spamless_kiiskinen@nospam_utu.fi
Thu, Oct 27 2005 20:13:17    Comments added by guest  
This still seems to be an issue. On grass-6.1.cvs_src_snapshot_2005_09_10, I
ran    

r.watershed el=Elev_srtm_finished th=100 st=stream.map ba=basin.map vi=visual.map
where r.info on Elev_srtm_finished gives

 +----------------------------------------------------------------------------+
| Layer:    Elev_srtm_finished             Date: Thu Jun 16 16:00:03 2005   
|
| Mapset:   srtm                           Login of Creator: harri          
|
| Location: Villas                                                          
|
| DataBase: /home/harri/grass                                               
|
| Title:     ( Elev_srtm_finished )                                         
|
|----------------------------------------------------------------------------|
|                                                                           
|
|   Type of Map:  raster              Number of Categories: 255             
|
|   Data Type:    FCELL                                                     
|
|   Rows:         4286                                                      
|
|   Columns:      4765                                                      
|
|   Total Cells:  20422790                                                  
|
|        Projection: Latitude-Longitude (zone 0)                            
|
|            N: 44:12:23.99796N    S: 40:38:06.00946N   Res: 0:00:02.999997 
|
|            E: 13:11:59.987155E    W: 9:13:44.99994E   Res: 0:00:02.999997 
|
|   Range of data:    min =  -40.000000 max = 2199.000000                   
|
|                                                                           
|
|   Data Source:                                                            
|
|                                                                           
|
|                                                                           
|
|                                                                           
|
|   Data Description:                                                       
|
|    generated by r.in.gdal                                                 
|
|                                                                           
|
|                                                                           
|
+----------------------------------------------------------------------------+
'Determinig offmap flow' took almost one day, the next phases went past during
the night, but 'Closing maps' never finished; I waited for two days on this
one. During the last phase, I kept observing the memory usage of the
ram-subprogram. The virtual memory kept steadily at 631M, but the currently
used share of physical memory (RES in top) decreased slowly but surely from
over 300M to around 2M, where it stayed for 24 hours. No files appeared
anywhere. At this point, I had to reboot my laptop for other reasons.

On smaller maps the 'Closing...' seems to finish, but the speed of it seems to
depend on the value of the threshold-option selected. A subset of the raster
above (n=42.5 s=42 w=12.5 e=13, 600 x 600) produced the rasters in 30 seconds
with threshold=100. With threshold=10 other operations were somewhat slower,
not much though, but the closing of the maps is still running after a few
minutes. Quite a difference, and I suspect that the maps may never be closed.
No raster files have appeared anywhere, so its not a case of "its really
finished, it just doesn't seem so".

If the routine is still "ok" and "this thing just takes time", some
informative output would be nice... Of course, a threshold=10 is quite
ridiculous in this case, but still.

Greetings,

Harri K.
harri._spamless_kiiskinen@nospam_utu.fi
Mon, Jan 2 2006 21:24:41    Area changed to grass6 by msieczka  
Mon, Jan 2 2006 21:28:37    Mail sent by msieczka  
According To Helena this bug still hives in 6.1.
Thu, Aug 17 2006 15:38:37    Subject changed to r.watershed: hangs at 'SECTION 5: Closing Maps.' with bigger input by msieczka  
Mon, Oct 23 2006 04:35:03    Comments added by hbowman  
Hint from the module's author:

----------------------------------------
From: "Charles Ehlschlaeger" <c.ehlschlaeger at insightbb,com>
Subject: RE: [GRASS-user] r.watershed hangs
Date: Sun, 22 Oct 2006 21:18:41 -0500
To: grassuser at grass,itc,it

I think the problem occurs with "threshhold=3" You are telling the software
to create exterior watershed basins only 3 grid cells in size. I can't
remember off the top of my head whether that will potentially cause stack
overflow problems or maybe I forgot to include a check to see whether an
array memory allocation returned a null pointer.

The solution is to include a more realistic threshold size. Ask yourself
"what is the smallest watershed basin you want to consider?" If the answer
is 10,000m^2, and grid cells are 10x10m, then threshold should be set to
100.

Sincerely, chuck
----------------------------------------
Wed, Oct 25 2006 10:02:19    Mail sent by hamish_nospam@yahoo.com  
Return-Path <hamish_nospam@yahoo.com>
Delivered-To grass-bugs@lists.intevation.de
Date Wed, 25 Oct 2006 21:01:26 +1300
From Hamish <hamish_nospam@yahoo.com>
To grass5 <grass-dev@grass.itc.it>
Cc hardeep@eml.cc, grass-bugs@intevation.de, c.ehlschlaeger@insightbb.com
Subject Re: [bug #3112] r.watershed hangs
Message-Id <20061025210126.4a3f0da8.hamish_nospam@yahoo.com>
In-Reply-To <20061023140326.7b516601.hamish_nospam@yahoo.com>
References <1161484146.15628.273909957@webmail.messagingengine.com> <20061023140326.7b516601.hamish_nospam@yahoo.com>
X-Mailer Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
X-Face M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
Mime-Version 1.0
Content-Type multipart/mixed; boundary="Multipart=_Wed__25_Oct_2006_21_01_26_+1300_YD_uF2hcKL8oY1kj"
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-2.3 tagged_above=-999 required=3 tests=[BAYES_00=-5, FORGED_YAHOO_RCVD=2.7]
X-Spam-Level
This is a multi-part message in MIME format.

--Multipart=_Wed__25_Oct_2006_21_01_26_+1300_YD_uF2hcKL8oY1kj
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Hardeep Singh Rai wrote:
> I was doing watershed analysis for spearfish dataset
> (spearfish_grass60data-0.1.tar.gz), and on giving command:
> 
> r.watershed elevation=elevation.dem threshhold=3 basin=raiBasin
> accumulation=raiAccu it starts processing of 5 steps. After about 20
> minutes, it completes all and throw message: "cloing maps", but do
> nothing; prompt never comes. I used it with option -m and without it.
> With both cases, "top" commands show ram or sed running with 99 usage
> of CPU. Even I waited for 12 hours, but it fail to finish. I am using
> GRASS 6.0.1 on Ubuntu.
> 
> Please suggest some solution.

Hamish wrote:
> It's a known bug (#3112). It gets stuck trying to make a huge set of
> colormap rules for the new map. If you cancel (^C) you might find the
> map was created anyway (???).

Chuck wrote:
> The solution is to include a more realistic threshold size.

Yes.

The attached patch shows the problem quite well. As the treshold gets
smaller, the number of areas (cats) grows exponentially. Not only that,
but as the loop within a loop runs, it takes longer and longer to do the
same thing.

The inner loop gets slower I think due to the increased number of color
rules set with G_set_color() that have to be read each time.


#spearfish
g.region rast=elevation.dem
r.watershed elev=elevation.dem basin=tmp_basin thresh=200
d.rast tmp_basin

thresh=200         # 1488 cats
real    0m38.558s
user    0m37.770s
sys     0m0.110s

thresh=100         # 2858 cats
real    0m41.296s
user    0m41.050s
sys     0m0.120s

thresh=50          # 5482 cats
real    0m57.495s
user    0m55.870s
sys     0m0.180s

thresh=37          # 7258 cats
real    1m20.985s
user    1m20.340s
sys     0m0.200s

thresh=25          # 10648 cats
real    3m5.185s
user    3m4.020s
sys     0m0.280s




there are a lot of loops here, but I think the main problem is with so
many calls to G_{get,set}_color():


raster/r.watershed/ram/close_maps2.c
..
        G_init_colors(&colors);
        G_make_random_colors(&colors, 1, max);
        G_set_color((CELL)0, 0, 0, 0, &colors);
        r = 1;
        incr = 0;
        while(incr >= 0) {
            for (gr=130+incr; gr<=255; gr += 20) {
                for (rd=90+incr; rd<=255; rd += 30) {
                    for (bl=90+incr; bl<=255; bl += 40) {
                        flag = 1;
                        while (flag) { 
                            G_get_color(r,&red,&green,&blue, &colors);
                            if((blue*.11 + red*.30 + green*.59) < 100) {

                                G_set_color(r, rd, gr, bl, &colors);
                                flag = 0;
                            }
                            if (++r > max) {
                                gr = rd = bl = 300;
                                flag = 0;
                                incr = -1;
                            }
                        }
                    }
                }
            }
            if (incr >= 0) {
                incr += 15;
                if (incr > 120)
                    incr = 7;
            }
        }



Hamish

--Multipart=_Wed__25_Oct_2006_21_01_26_+1300_YD_uF2hcKL8oY1kj
Content-Type: text/plain;
 name="rwaters_loop.diff"
Content-Disposition: attachment;
 filename="rwaters_loop.diff"
Content-Transfer-Encoding: 7bit

Index: raster/r.watershed/ram/close_maps2.c
===================================================================
RCS file: /home/grass/grassrepository/grass6/raster/r.watershed/ram/close_maps2.c,v
retrieving revision 2.0
diff -u -r2.0 close_maps2.c
--- raster/r.watershed/ram/close_maps2.c	9 Nov 2004 13:32:28 -0000	2.0
+++ raster/r.watershed/ram/close_maps2.c	25 Oct 2006 07:44:47 -0000
@@ -46,6 +46,7 @@
 				G_set_color(r, rd, gr, bl, &colors);
 			        flag = 0;
 		            }
+if(r % 200 == 0) G_debug(0,"r=%d\tof %d", r, max);
 			    if (++r > max) {
 			        gr = rd = bl = 300;
 				flag = 0;

--Multipart=_Wed__25_Oct_2006_21_01_26_+1300_YD_uF2hcKL8oY1kj--


Wed, Oct 25 2006 23:35:20    Mail sent by c.ehlschlaeger@insightbb.com  
Return-Path <c.ehlschlaeger@insightbb.com>
Delivered-To grass-bugs@lists.intevation.de
Message-Id <5ag1pm$5chkh4@asav13.insightbb.com>
X-IronPort-Anti-Spam-Filtered true
X-IronPort-Anti-Spam-Result Ao8CACN1P0VKi5Lq/2dsb2JhbAA
From "Charles Ehlschlaeger" <c.ehlschlaeger@insightbb.com>
To "'Hamish'" <hamish_nospam@yahoo.com>, "'grass5'" <grass-dev@grass.itc.it>
Cc <hardeep@eml.cc>, <grass-bugs@intevation.de>
Subject RE: [bug #3112] r.watershed hangs
Date Wed, 25 Oct 2006 16:35:16 -0500
MIME-Version 1.0
Content-Type text/plain; charset="us-ascii"
Content-Transfer-Encoding 7bit
X-Mailer Microsoft Office Outlook, Build 11.0.6353
In-reply-to <20061025210126.4a3f0da8.hamish_nospam@yahoo.com>
X-MIMEOLE Produced By Microsoft MimeOLE V6.00.2900.2962
Thread-index Acb4C94HAaliOZvlT8m0PJ+dZdQb/wAcKLCQ
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-5 tagged_above=-999 required=3 tests=[BAYES_00=-5]
X-Spam-Level
Dear Hamish,

Thanks for posting the patch of r.watershed. That snippet of code ensures
that colors allocated to watersheds are as different as possible. Back in
the early '90s, we were so worried about RAM use that we wrote convoluted
code that prevented making new arrays whenever possible.


The simpliest fix would be to insert a conditional after
G_make_random_colors(&colors, 1, max) as shown in the patch below (two lines
added). From a visualization perspective, 1000 basins are too many basins to
matter anyway, so 1000 might be good. If you look at the experiment you
performed, you can see 10000 basins took 3 minutes to run.

The best fix would be to write an algorithm that chooses category color
based on adjacent cell values. Here is a psuedo code solution:

random_colors( Raster map, Color[] colors) {
    long adjacentColors[][] = new long[ colors.length][];
    for( int r = 0; r < map.rows(); r++) {
       for( int c = 0; c < map.cols(); c++) {
          long cellValue = map.getValue( r, c);
          if( c < map.cols() - 1) {
             if( map.getValue( r, c + 1) < cellValue) {
                // add map.getValue( r, c + 1) to adjacentColors[
cellValue][] if not already in array
             }
          }
          if( r < map.rows() - 1) {
             if( map.getValue( r + 1, c ) < cellValue) {
                // add map.getValue( r + 1, c) to adjacentColors[
cellValue][] if not already in array
             }
          }
          if( (r < map.rows() - 1) && c < map.cols() - 1) {
             if( map.getValue( r + 1, c + 1 ) < cellValue) {
                // add map.getValue( r + 1, c + 1) to adjacentColors[
cellValue][] if not already in array
             }
          }
       }
    }
    for( int i = 0; i < colors.length; i++) {
       // if no adjacentColors[i] == null, give any random color
       // otherwise randomly create 100 colors
       // choose color furthest from colors of adjacentColors[i] array 
    }
}

Sincerely, chuck

-----Original Message-----
From: Hamish [mailto:hamish_nospam@yahoo.com] 
Sent: Wednesday, October 25, 2006 3:01 AM
To: grass5
Cc: hardeep@eml.cc; grass-bugs@intevation.de; c.ehlschlaeger@insightbb.com
Subject: Re: [bug #3112] r.watershed hangs

Hardeep Singh Rai wrote:
> I was doing watershed analysis for spearfish dataset
> (spearfish_grass60data-0.1.tar.gz), and on giving command:
> 
> r.watershed elevation=elevation.dem threshhold=3 basin=raiBasin
> accumulation=raiAccu it starts processing of 5 steps. After about 20
> minutes, it completes all and throw message: "cloing maps", but do
> nothing; prompt never comes. I used it with option -m and without it.
> With both cases, "top" commands show ram or sed running with 99 usage
> of CPU. Even I waited for 12 hours, but it fail to finish. I am using
> GRASS 6.0.1 on Ubuntu.
> 
> Please suggest some solution.

Hamish wrote:
> It's a known bug (#3112). It gets stuck trying to make a huge set of
> colormap rules for the new map. If you cancel (^C) you might find the
> map was created anyway (???).

Chuck wrote:
> The solution is to include a more realistic threshold size.

Yes.

The attached patch shows the problem quite well. As the treshold gets
smaller, the number of areas (cats) grows exponentially. Not only that,
but as the loop within a loop runs, it takes longer and longer to do the
same thing.

The inner loop gets slower I think due to the increased number of color
rules set with G_set_color() that have to be read each time.


#spearfish
g.region rast=elevation.dem
r.watershed elev=elevation.dem basin=tmp_basin thresh=200
d.rast tmp_basin

thresh=200         # 1488 cats
real    0m38.558s
user    0m37.770s
sys     0m0.110s

thresh=100         # 2858 cats
real    0m41.296s
user    0m41.050s
sys     0m0.120s

thresh=50          # 5482 cats
real    0m57.495s
user    0m55.870s
sys     0m0.180s

thresh=37          # 7258 cats
real    1m20.985s
user    1m20.340s
sys     0m0.200s

thresh=25          # 10648 cats
real    3m5.185s
user    3m4.020s
sys     0m0.280s




there are a lot of loops here, but I think the main problem is with so
many calls to G_{get,set}_color():


raster/r.watershed/ram/close_maps2.c
..
        G_init_colors(&colors);
        G_make_random_colors(&colors, 1, max);
// new code next line
        if( max < 10000) {
        G_set_color((CELL)0, 0, 0, 0, &colors);
        r = 1;
        incr = 0;
        while(incr >= 0) {
            for (gr=130+incr; gr<=255; gr += 20) {
                for (rd=90+incr; rd<=255; rd += 30) {
                    for (bl=90+incr; bl<=255; bl += 40) {
                        flag = 1;
                        while (flag) { 
                            G_get_color(r,&red,&green,&blue, &colors);
                            if((blue*.11 + red*.30 + green*.59) < 100) {

                                G_set_color(r, rd, gr, bl, &colors);
                                flag = 0;
                            }
                            if (++r > max) {
                                gr = rd = bl = 300;
                                flag = 0;
                                incr = -1;
                            }
                        }
                    }
                }
            }
            if (incr >= 0) {
                incr += 15;
                if (incr > 120)
                    incr = 7;
            }
        }// new code next line
        }



Hamish


Thu, Oct 26 2006 07:33:45    Comments added by hbowman  
From: "Charles Ehlschlaeger"
Subject: RE: [bug #3112] r.watershed hangs
Date: Wed, 25 Oct 2006 16:35:16 -0500


Dear Hamish,

Thanks for posting the patch of r.watershed. That snippet of code ensures
that colors allocated to watersheds are as different as possible. Back in
the early '90s, we were so worried about RAM use that we wrote convoluted
code that prevented making new arrays whenever possible.


The simpliest fix would be to insert a conditional after
G_make_random_colors(&colors, 1, max) as shown in the patch below (two lines
added). From a visualization perspective, 1000 basins are too many basins to
matter anyway, so 1000 might be good. If you look at the experiment you
performed, you can see 10000 basins took 3 minutes to run.

The best fix would be to write an algorithm that chooses category color
based on adjacent cell values. Here is a psuedo code solution:

random_colors( Raster map, Color[] colors) {
    long adjacentColors[][] = new long[ colors.length][];
    for( int r = 0; r < map.rows(); r++) {
       for( int c = 0; c < map.cols(); c++) {
          long cellValue = map.getValue( r, c);
          if( c < map.cols() - 1) {
             if( map.getValue( r, c + 1) < cellValue) {
                // add map.getValue( r, c + 1) to adjacentColors[
cellValue][] if not already in array
             }
          }
          if( r < map.rows() - 1) {
             if( map.getValue( r + 1, c ) < cellValue) {
                // add map.getValue( r + 1, c) to adjacentColors[
cellValue][] if not already in array
             }
          }
          if( (r < map.rows() - 1) && c < map.cols() - 1) {
             if( map.getValue( r + 1, c + 1 ) < cellValue) {
                // add map.getValue( r + 1, c + 1) to adjacentColors[
cellValue][] if not already in array
             }
          }
       }
    }
    for( int i = 0; i < colors.length; i++) {
       // if no adjacentColors[i] == null, give any random color
       // otherwise randomly create 100 colors
       // choose color furthest from colors of adjacentColors[i] array 
    }
}

Sincerely, chuck


raster/r.watershed/ram/close_maps2.c
..
        G_init_colors(&colors);
        G_make_random_colors(&colors, 1, max);
// new code next line
        if( max < 10000) {
        G_set_color((CELL)0, 0, 0, 0, &colors);
        r = 1;
[...]
            if (incr >= 0) {
                incr += 15;
                if (incr > 120)
                    incr = 7;
            }
        }// new code next line
        }
Thu, Oct 26 2006 07:59:56    Mail sent by hamish_nospam@yahoo.com  
Return-Path <hamish_nospam@yahoo.com>
Delivered-To grass-bugs@lists.intevation.de
Date Thu, 26 Oct 2006 18:59:37 +1300
From Hamish <hamish_nospam@yahoo.com>
To "Charles Ehlschlaeger" <c.ehlschlaeger@insightbb.com>
Cc grass-dev@grass.itc.it, hardeep@eml.cc, grass-bugs@intevation.de
Subject Re: [bug #3112] r.watershed hangs
Message-Id <20061026185937.6ba81de6.hamish_nospam@yahoo.com>
In-Reply-To <5ag1pm$5chkh4@asav13.insightbb.com>
References <20061025210126.4a3f0da8.hamish_nospam@yahoo.com> <5ag1pm$5chkh4@asav13.insightbb.com>
X-Mailer Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
X-Face M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
Mime-Version 1.0
Content-Type text/plain; charset=US-ASCII
Content-Transfer-Encoding 7bit
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-2.3 tagged_above=-999 required=3 tests=[BAYES_00=-5, FORGED_YAHOO_RCVD=2.7]
X-Spam-Level
Charles Ehlschlaeger wrote:
> Thanks for posting the patch of r.watershed. That snippet of code
> ensures that colors allocated to watersheds are as different as
> possible. Back in the early '90s, we were so worried about RAM use
> that we wrote convoluted code that prevented making new arrays
> whenever possible.

IMO this is why GRASS has (in general) scaled so well for today's
massive datasets. Raster overheads are very low.


> The simpliest fix would be to insert a conditional after
> G_make_random_colors(&colors, 1, max) as shown in the patch below (two
> lines added).

thanks, applied to the 6.2 & 6.3(HEAD) branches.

Also took the opportunity to slop on lots of metadata in the output
maps (6.3cvs only).


> From a visualization perspective, 1000 basins are too
> many basins to matter anyway, so 1000 might be good. If you look at
> the experiment you performed, you can see 10000 basins took 3 minutes
> to run.

I went with 10000.


> The best fix would be to write an algorithm that chooses category
> color based on adjacent cell values. Here is a psuedo code solution:
>
> random_colors( Raster map, Color[] colors) {
>     long adjacentColors[][] = new long[ colors.length][];
>     for( int r = 0; r < map.rows(); r++) {
>        for( int c = 0; c < map.cols(); c++) {
>           long cellValue = map.getValue( r, c);
>           if( c < map.cols() - 1) {
>              if( map.getValue( r, c + 1) < cellValue) {
>                 // add map.getValue( r, c + 1) to adjacentColors[
> cellValue][] if not already in array
>              }
>           }
>           if( r < map.rows() - 1) {
>              if( map.getValue( r + 1, c ) < cellValue) {
>                 // add map.getValue( r + 1, c) to adjacentColors[
> cellValue][] if not already in array
>              }
>           }
>           if( (r < map.rows() - 1) && c < map.cols() - 1) {
>              if( map.getValue( r + 1, c + 1 ) < cellValue) {
>                 // add map.getValue( r + 1, c + 1) to adjacentColors[
> cellValue][] if not already in array
>              }
>           }
>        }
>     }
>     for( int i = 0; i < colors.length; i++) {
>        // if no adjacentColors[i] == null, give any random color
>        // otherwise randomly create 100 colors
>        // choose color furthest from colors of adjacentColors[i] array
>        
>     }
> }


The random case doesn't do too poorly. (256^3 color possibilities vs.
the four-color theorem)

> // choose color furthest from colors of adjacentColors[i] array

color distance in 3D RGB space isn't something I understand how to do
well. (I've wondered for a while how to find the "opposite" RGB color:
think of it as a vector ray from the origin inside a 3D sphere?)

Anyway, I'll have to leave the "better way" for someone who understands
color-space a bit better than I do.


thanks,
Hamish


Thu, Nov 9 2006 07:14:41    Status changed to resolved by hbowman  
Thu, Nov 9 2006 07:14:41    Comments added by hbowman  
fixed in CVS, closing bug


Hamish
Comment | Reply | Take | Open

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request