Details Ticket 4164


Comment | Reply | Take | Resolve


Serial Number 4164
Subject v.clean: uses 1GB RAM, 1 GB swap for a vector with only 628 boundaries
Area grass6
Queue grass
Requestors tutey@o2.pl
Owner none
Status open
Last User Contact Fri Mar 24 20:07:32 2006 (2 yr ago)
Current Priority 30
Final Priority 70
Due No date assigned
Last Action Wed Jul 26 18:21:29 2006 (2 yr ago)
Created Fri Mar 10 16:13:57 2006 (2 yr ago)

Transaction History Ticket 4164


Fri, Mar 10 2006 16:13:57    Request created by guest  
Subject: v.clean: uses 1GB RAM, 1 GB swap for a vector with only 628 boundaries
Platform: GNU/Linux/x86
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: 2006-02-20

v.buffer used all my memory (1 GB RAM and 1GB swap) when building a clean buffer
for my vector file, and I had to kill it to continue my work.

For a test, I forced v.buffer to skip cleaning, using option debug=buffer, and
tried to clean this "dirty" output myself:

v.clean input=rogow_parcels_06_water_buff100 output=rogow_parcels_06_water_buff100_cl
type=boundary tool=break

Again, it ate all my memory. The rogow_parcels_06_water_buff100 is very small:
Number of boundaries:   628

v.clean reaches the memory limit at about:

Intersections: 78194 (line 150883)

There must be some problem with memory handling in the vector code.

I'm putting the location with (only) my problematic vector file, if somebody
is interested in fixing the problem.

http://www.biol.uni.wroc.pl/sieczka/udostepnione/grass/huha.tar.bz2 (364 KB)
I'm not looking for a workaround - I accomplished my task using r.buffer, r.to.vect.
Maciek
Fri, Mar 10 2006 17:14:49    Mail sent by hmitaso@unity.ncsu.edu  
Return-Path <hmitaso@unity.ncsu.edu>
Delivered-To grass-bugs@lists.intevation.de
Message-ID <4411A5EF.5080407@unity.ncsu.edu>
Date Fri, 10 Mar 2006 11:14:39 -0500
From Helena Mitasova <hmitaso@unity.ncsu.edu>
User-Agent Mozilla Thunderbird 1.0.2 (X11/20050317)
X-Accept-Language en-us, en
MIME-Version 1.0
To Request Tracker <grass-bugs@intevation.de>
Cc grass5@grass.itc.it
Subject Re: [GRASS5] [bug #4164] (grass) v.clean: uses 1GB RAM, 1 GB swap for a vector with only 628 boundaries
References <20060310151357.6AF801006C9@lists.intevation.de>
In-Reply-To <20060310151357.6AF801006C9@lists.intevation.de>
Content-Type text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding 7bit
X-PMX-Version 4.7.1.128075, Antispam-Engine: 2.1.0.0, Antispam-Data: 2006.03.10.065104
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Request Tracker wrote:
> this bug's URL: http://intevation.de/rt/webrt?serial_num=4164
> -------------------------------------------------------------------------
> 
> Subject: v.clean: uses 1GB RAM, 1 GB swap for a vector with only 628 boundaries
> 
> Platform: GNU/Linux/x86
> grass obtained from: CVS
> grass binary for platform: Compiled from Sources
> GRASS Version: 2006-02-20
> 
> v.buffer used all my memory (1 GB RAM and 1GB swap) when building a clean buffer
for my vector file, and I had to kill it to continue my work.
> 
> For a test, I forced v.buffer to skip cleaning, using option debug=buffer,
and  tried to clean this "dirty" output myself:
> 
> v.clean input=rogow_parcels_06_water_buff100 output=rogow_parcels_06_water_buff100_cl
type=boundary tool=break
> 
> Again, it ate all my memory. The rogow_parcels_06_water_buff100 is very small:
> 
> Number of boundaries:   628
> 
> v.clean reaches the memory limit at about:
> 
> Intersections: 78194 (line 150883)
> 
> There must be some problem with memory handling in the vector code.
> 
> I'm putting the location with (only) my problematic vector file, if somebody
is interested in fixing the problem.
> 
> http://www.biol.uni.wroc.pl/sieczka/udostepnione/grass/huha.tar.bz2 (364 KB)
> 
> I'm not looking for a workaround - I accomplished my task using r.buffer, r.to.vect.
> 
> Maciek
> 
> 
> -------------------------------------------- Managed by Request Tracker
> 
> _______________________________________________
> grass5 mailing list
> grass5@grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5

Although this may not be related to your problem, v.buffer may be 
calling V_build which has a memory management problem/feature that it 
eats up all your memory. But it should not happen with 628 boundaries
- we have that problem with about 600,000+ points.
Maybe Radim can explain whether this may be the same. If you digg 
through the emails related to v.in.ascii and handling of large files 
there was some discussion between Hamish and Radim on how to solve it.

Helena


Thu, Mar 16 2006 17:48:31    Status changed to resolved by msieczka  
Thu, Mar 16 2006 17:48:30    Mail sent by msieczka  
Seems fixed in today's CVS (2006.03.16). Maybe the fix was:
http://grass.itc.it/pipermail/grass-commit/2006-March/021025.html

Nothing else related I can find - but on the other hand fix' author didn't
reffered to this bug. So I'm not sure what the cure was actually.

?

Anyway, closing it.

Maciek
Fri, Mar 17 2006 10:23:10    Mail sent by radim.blazek@gmail.com  
Return-Path <radim.blazek@gmail.com>
Delivered-To grass-bugs@lists.intevation.de
DomainKey-Signature a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=e2y87Im3il0GEc/gzJ13lHP0Oi/bJO+kQjB5xOANZHHoCSqD3gQCPUr5q5+InpiP7Pg3MhdyzjU3bxNMbw5WyYhnrfJflne1J0nyRSeWBbx0rGhAwe1+hH2J2b5zxDLWSz5NDNOayXKz31wZlFK3ZSgX0Y8vYy+fwJ98Hal5cwc=
Message-ID <340505ef0603170119y4239e603t65ca8dbe099dcf9a@mail.gmail.com>
Date Fri, 17 Mar 2006 10:19:56 +0100
From "Radim Blazek" <radim.blazek@gmail.com>
To "Maciek Sieczka via RT" <grass-bugs@intevation.de>
Subject Re: [GRASS5] [bug #4164] (grass) v.clean: uses 1GB RAM, 1 GB swap for a vector with only 628 boundaries
Cc grass5@grass.itc.it, hmitaso@unity.ncsu.edu
In-Reply-To <20060316164831.30B3C1005A3@lists.intevation.de>
MIME-Version 1.0
Content-Type text/plain; charset=ISO-8859-1
Content-Transfer-Encoding quoted-printable
Content-Disposition inline
References <20060316164831.30B3C1005A3@lists.intevation.de>
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
On 3/16/06, Maciek Sieczka via RT <grass-bugs@intevation.de> wrote:
> Seems fixed in today's CVS (2006.03.16). Maybe the fix was:
> http://grass.itc.it/pipermail/grass-commit/2006-March/021025.html
>
> Nothing else related I can find - but on the other hand fix' author didn'=
t
> reffered to this bug. So I'm not sure what the cure was actually.

It seems to be strange. Did you get correct result from v.buffer???!!!
It maybe that I just introduced another bug.

Radim

> ?
>
> Anyway, closing it.
>
> Maciek
>
>
> -------------------------------------------- Managed by Request Tracker
>
> _______________________________________________
> grass5 mailing list
> grass5@grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5
>


Fri, Mar 17 2006 10:23:10    Status changed to open by _rt_system  
Tue, Mar 21 2006 21:03:00    Mail sent by msieczka  
Radim wrote:

> It seems to be strange. Did you get correct result from v.buffer???!!!
> It maybe that I just introduced another bug.

Radim,

Sorry for late reply. I've been away for some time.

Regarding the issue: indeed, v.buffer's output is not a clean buffer. I
reproduced my orignal v.buffer command:

v.buffer input=x_extr output=x_extr_buff100 type=boundary layer=1 buffer=100
scale=1.0 tolerance=0.01 debug=clean

and in spite specifying debug=clean the output is full of tiny polygons made
of crossing circles and arcs. What's good, their topology looks allright after
inspection in Qgis Grass digit. Even better, that v.buffer used very little
memory - several times less than before; which is great! It didn't even touch
swap!

Summarising: the memory consumption is sane now, yippie!, but buffer is not
created.

Do you need my dataset for your work on this?

Maciek
Wed, Mar 22 2006 09:25:06    Mail sent by radim.blazek@gmail.com  
Return-Path <radim.blazek@gmail.com>
Delivered-To grass-bugs@lists.intevation.de
DomainKey-Signature a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=AFS06mP8fzfmbofNlSeYcGnX8oVugvRdpb7qrdDhJed5kvLk8RJK1euUGQMdDra2IOsXn8UK6/9ObGTBlq/DoAWNNVX7d3j/yHQxECckT6VUupw1Ch0srtmU9HJnhvreJ77/4UUz7gqcLHqdjq6CftQLb5VIGMOcKaUDkyWdmqY=
Message-ID <340505ef0603220025i4baf8390t985fe1e1b3f8e2d@mail.gmail.com>
Date Wed, 22 Mar 2006 09:25:02 +0100
From "Radim Blazek" <radim.blazek@gmail.com>
To "Maciek Sieczka via RT" <grass-bugs@intevation.de>
Subject Re: [bug #4164] (grass) v.clean: uses 1GB RAM, 1 GB swap for a vector with only 628 boundaries
Cc grass5@grass.itc.it
In-Reply-To <20060321200300.15E66100160@lists.intevation.de>
MIME-Version 1.0
Content-Type text/plain; charset=ISO-8859-1
Content-Transfer-Encoding quoted-printable
Content-Disposition inline
References <20060321200300.15E66100160@lists.intevation.de>
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
On 3/21/06, Maciek Sieczka via RT <grass-bugs@intevation.de> wrote:
> Radim wrote:
> Sorry for late reply. I've been away for some time.
>
> Regarding the issue: indeed, v.buffer's output is not a clean buffer. I
> reproduced my orignal v.buffer command:
>
> v.buffer input=3Dx_extr output=3Dx_extr_buff100 type=3Dboundary layer=3D1=
buffer=3D100
> scale=3D1.0 tolerance=3D0.01 debug=3Dclean
>
> and in spite specifying debug=3Dclean the output is full of tiny polygons=
made
> of crossing circles and arcs. What's good, their topology looks allright =
after
> inspection in Qgis Grass digit.

Thats correct, debug means debug, if you want final output don't use debug.

Radim

> Even better, that v.buffer used very little

> memory - several times less than before; which is great! It didn't even t=
ouch
>
> swap!
>
>
>
> Summarising: the memory consumption is sane now, yippie!, but buffer is n=
ot
>
> created.
>
>
>
> Do you need my dataset for your work on this?
>
>
>
> Maciek
>
> -------------------------------------------- Managed by Request Tracker
>


Thu, Mar 23 2006 08:43:39    Mail sent by radim.blazek@gmail.com  
Return-Path <radim.blazek@gmail.com>
Delivered-To grass-bugs@lists.intevation.de
DomainKey-Signature a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=YqvdgevCdKgHXTEv1oSqv6NvFefNAg6WU9rHE+qdlv6aWzhYc/X+ANvNNHdxhAibsZDqNnjdKotdxFiK0ccr4cV7CDMERWpwqvolieBW/dEvj8u24Zf2yFe8DaMRhBPg4Okw09PWkFPi0C78nOf61rRBbUpTG8CXPLa8WZybVaI=
Message-ID <340505ef0603222343t7426c080rca155e5422d3c19c@mail.gmail.com>
Date Thu, 23 Mar 2006 08:43:36 +0100
From "Radim Blazek" <radim.blazek@gmail.com>
To "Maciek Sieczka" <werchowyna@epf.pl>
Subject Re: [GRASS5] Re: [bug #4164] (grass) v.clean: uses 1GB RAM, 1 GB swap for a vector with only 628 boundaries
Cc grass-bugs@intevation.de, grass5@grass.itc.it
In-Reply-To <20060322224654.4b1d4f2b.werchowyna@epf.pl>
MIME-Version 1.0
Content-Type text/plain; charset=ISO-8859-1
Content-Transfer-Encoding quoted-printable
Content-Disposition inline
References <20060321200300.15E66100160@lists.intevation.de> <340505ef0603220025i4baf8390t985fe1e1b3f8e2d@mail.gmail.com> <20060322224654.4b1d4f2b.werchowyna@epf.pl>
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
On 3/22/06, Maciek Sieczka <werchowyna@epf.pl> wrote:
> Now, not using "debug=3Dclean": the v.buffer creates _mostly_ good result
> when compared to r.buffer at resolution =3D1. But there are at least 2
> places with errors visible in v.buffer behaviour. Please see the
> attached screendump, you will spot them easily.
>
> The YELLOW lines are the v.buffer and r.buffer input.
>
> The RED backgroung is r.buffer output (GOOD).
>
> The BLACK lines are v.buffer output, with at least 2 ERRORS visible,
> indicated by GREEN circles.
>
> Are those possible to be fixed Radim?

Not now by me. If possible publish your input data so that it can
be used for debugging in future.

How do look those 2 places if run with debug=3Dclean?

Radim


> Maciek
>
>
> --------------------
> W polskim Internecie s? setki milion?w stron. My przekazujemy Tobie tylko=
najlepsze z nich!
> http://katalog.panoramainternetu.pl/
>
>
>


Fri, Mar 24 2006 20:07:32    Mail sent by msieczka  
On Thu, 23 Mar 2006 08:43:36 +0100
"Radim Blazek" <radim.blazek@gmail.com> wrote:

> On 3/22/06, Maciek Sieczka <werchowyna@epf.pl> wrote:
> > Now, not using "debug=clean": the v.buffer creates _mostly_ good
> > result when compared to r.buffer at resolution =1. But there are at
> > least 2 places with errors visible in v.buffer behaviour. Please
> > see the attached screendump, you will spot them easily.
> >
> > The YELLOW lines are the v.buffer and r.buffer input.
> >
> > The RED backgroung is r.buffer output (GOOD).
> >
> > The BLACK lines are v.buffer output, with at least 2 ERRORS visible,
> > indicated by GREEN circles.
> >
> > Are those possible to be fixed Radim?
> >

> Not now by me. If possible publish your input data so that it can
> be used for debugging in future.


Here (3,3 MB):
http://www.biol.uni.wroc.pl/sieczka/udostepnione/grass/huha2.tar.bz2


There is an input vector boundaries

x_extr

and the 100m buffer with a visible fault

x_extr_buff100

I deleted my original input and created it once more to prepare this
sample dataset - and somehow now there is only 1 error in the
v.buffer output instead of 2 as previously... Hmm. Maybe my fault.

> How do look those 2 places if run with debug=clean?

Topologicaly clean, but the fault in buffer is still clearly visible.

Maciek
Wed, Jul 26 2006 18:21:29    User changed to tutey@o2.pl by msieczka  
Comment | Reply | Take | Resolve

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request