Sun, Dec 4 2005
22:54:51
|
|
Request created by guest
|
|
Subject: r.to.vect: severe memory leaks, I'm helpless
Platform: GNU/Linux/i386
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: 2005-11-16
r.to.vect -z feature=point input=dem_5 output=dem_5_pt
The input is a 5m floating point DEM, 6305x7930 cells.
I need to convert it to 3D points and reproject them into another coordinate
system to re-interpolate my DEM there. However, r.to.vect consumes all my RAM
(1GB) and swap (1GB) and hangs the system at the "Registering lines:" stage with
about 5 000 000 points already processed out of almost 50 000 000 total.
Trying without "-z" flag is even worse of course - not only many times slower,
but the dbf driver eats whole memory much erlier - r.to.vect doesn't even make
it to "Registering lines:"!
Please do something about it. It's been several months now since well known problems
with memory leaks in vectors and dbf driver remain. This issue renders Grass
useless for processing large vector datasets. In my case it is a definite show
stoper. And I wonder: even if I make it to convert my DEM into vector points
with another software and reproject it with ogr2ogr, would Grass alone manage
to:
1. import this 50 000 000 point vector shapefile with v.in.ogr?
2. interpolate a 5m DEM out of it with v.surf.rst?
Maciek |
|
Mon, Dec 5 2005
13:38:35
|
|
Mail sent by radim.blazek@gmail.com
|
|
Return-Path |
<radim.blazek@gmail.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
DomainKey-Signature |
a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=N0LxWvz+hB24FWUpn5ETABjQNmuXvG6Bp+5K+C4CpNr93sGPO/GQjMsh66E9jEh1mD05bZuLKV9UtBW4Y8NLXdyYScg/l6p+1oxRZlodpYG7VIGXIJcoZDc41bUkgvFyeu6aGXsVPTsnHLtveoXjYJm54MI+8nc066NSRL4OSEA=
|
Message-ID |
<340505ef0512050438w365af4b2mab884be87277c61@mail.gmail.com>
|
Date |
Mon, 5 Dec 2005 13:38:28 +0100
|
From |
Radim Blazek <radim.blazek@gmail.com>
|
To |
Request Tracker <grass-bugs@intevation.de>
|
Subject |
Re: [GRASS5] [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
Cc |
grass5@grass.itc.it
|
In-Reply-To |
<20051204215451.2F0D01005BC@lists.intevation.de>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=ISO-8859-1
|
Content-Transfer-Encoding |
quoted-printable
|
Content-Disposition |
inline
|
References |
<20051204215451.2F0D01005BC@lists.intevation.de>
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Hi,
please read old mails on this problem. I dont have time to explain it
again and again. AFAIK there are no big memory leaks.
Radim
On 12/4/05, Request Tracker <grass-bugs@intevation.de> wrote:
> this bug's URL: http://intevation.de/rt/webrt?serial_num=3D3877
> -------------------------------------------------------------------------
>
> Subject: r.to.vect: severe memory leaks, I'm helpless
>
> Platform: GNU/Linux/i386
> grass obtained from: CVS
> grass binary for platform: Compiled from Sources
> GRASS Version: 2005-11-16
>
> r.to.vect -z feature=3Dpoint input=3Ddem_5 output=3Ddem_5_pt
>
> The input is a 5m floating point DEM, 6305x7930 cells.
>
> I need to convert it to 3D points and reproject them into another coordin=
ate system to re-interpolate my DEM there. However, r.to.vect consumes all =
my RAM (1GB) and swap (1GB) and hangs the system at the "Registering lines:=
" stage with about 5 000 000 points already processed out of almost 50 000 =
000 total.
>
> Trying without "-z" flag is even worse of course - not only many times sl=
ower, but the dbf driver eats whole memory much erlier - r.to.vect doesn't =
even make it to "Registering lines:"!
>
> Please do something about it. It's been several months now since well kno=
wn problems with memory leaks in vectors and dbf driver remain. This issue =
renders Grass useless for processing large vector datasets. In my case it i=
s a definite show stoper. And I wonder: even if I make it to convert my DEM=
into vector points with another software and reproject it with ogr2ogr, wo=
uld Grass alone manage to:
> 1. import this 50 000 000 point vector shapefile with v.in.ogr?
> 2. interpolate a 5m DEM out of it with v.surf.rst?
>
> Maciek
>
> -------------------------------------------- Managed by Request Tracker
>
> _______________________________________________
> grass5 mailing list
> grass5@grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5
>
|
|
Mon, Dec 5 2005
18:38:07
|
|
Mail sent by werchowyna@epf.pl
|
|
Return-Path |
<werchowyna@epf.pl>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Subject |
Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
From |
Maciek Sieczka <werchowyna@epf.pl>
|
To |
radim.blazek@gmail.com, grass5@grass.itc.it, grass-bugs@intevation.de
|
In-Reply-To |
<20051205123835.AE456101EFD@lists.intevation.de>
|
References |
<20051205123835.AE456101EFD@lists.intevation.de>
|
Content-Type |
text/plain
|
Date |
Mon, 05 Dec 2005 18:38:21 +0100
|
Message-Id |
<1133804301.8199.67.camel@localhost.localdomain>
|
Mime-Version |
1.0
|
X-Mailer |
Evolution 2.2.1.1
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
On pon, 2005-12-05 at 13:38 +0100, Radim Blazek via RT wrote:
> please read old mails on this problem. I dont have time to explain it
> again and again. AFAIK there are no big memory leaks.
Is it aknowledged by Grass developers that a machine freeze at 5 mln
vector points file is a BUG (no matter what the reason is)?
If it is aknowledged, can we expect it to be fixed? When - soon, month
time, year time? Or is it going to be a "feature" and left as is?
Maciek
--------------------
W polskim Internecie są setki milionów stron. My przekazujemy Tobie tylko najlepsze
z nich!
http://katalog.epf.pl/
|
|
Mon, Dec 5 2005
19:11:44
|
|
Mail sent by Roger.Bivand@nhh.no
|
|
Return-Path |
<Roger.Bivand@nhh.no>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
X-IronPort-AV |
i="3.99,217,1131318000"; d="scan'208"; a="1804111:sNHT102600936"
|
Date |
Mon, 5 Dec 2005 19:11:36 +0100 (CET)
|
From |
Roger Bivand <Roger.Bivand@nhh.no>
|
X-X-Sender |
rsb@reclus.nhh.no
|
Reply-To |
Roger.Bivand@nhh.no
|
To |
Maciek Sieczka <werchowyna@epf.pl>
|
Cc |
radim.blazek@gmail.com, <grass5@grass.itc.it>, <grass-bugs@intevation.de>
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
In-Reply-To |
<1133804301.8199.67.camel@localhost.localdomain>
|
Message-ID |
<Pine.LNX.4.44.0512051851500.20745-100000@reclus.nhh.no>
|
MIME-Version |
1.0
|
Content-Type |
TEXT/PLAIN; charset=ISO-8859-1
|
Content-Transfer-Encoding |
QUOTED-PRINTABLE
|
X-OriginalArrivalTime |
05 Dec 2005 18:11:30.0896 (UTC) FILETIME=[515FAD00:01C5F9C7]
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
On Mon, 5 Dec 2005, Maciek Sieczka wrote:
> On pon, 2005-12-05 at 13:38 +0100, Radim Blazek via RT wrote:
> > please read old mails on this problem. I dont have time to explain it
> > again and again. AFAIK there are no big memory leaks.
>=20
> Is it aknowledged by Grass developers that a machine freeze at 5 mln
> vector points file is a BUG (no matter what the reason is)?
>=20
> If it is aknowledged, can we expect it to be fixed? When - soon, month
> time, year time? Or is it going to be a "feature" and left as is?
I think this is unfair. There has been progress in GRASS 6 on this, and=20
the vector architecture is much stronger than it was in GRASS 5 for=20
moderate and large data sets, but not for XXL.=20
Have you considered using GRASS 5, which has sites, a very much simpler=20
data model for points? Have you considered tiling your data - reading=20
portions of your data and patching the resulting spline surfaces? Once you=
=20
have the surface, you can transfer it to GRASS 6, because as yet the=20
raster storage data model is effectively unchanged.=20
This is not a bug, it is a mis-match of data models and intentions. While=
=20
accepting that freezing (meaning causing total OS failure, or rather=20
occupation of all machine resources? - I don't think that a non-root user=
=20
on a sensible OS can freeze the system so that a hard shutdown (pull=20
power) is required) is unfortunate, it is usually caused by 100% CPU use=20
and swapping caused by memory being fully occupied. In well-written=20
software, like GRASS 6 vector or R, say, there is a balance between how=20
things are written, perceived needs, and user perceptions. The GRASS 6=20
vector data model handles areas and lines pretty well, but because it is=20
trying harder on these, is not well suited to XXL points data sets. My=20
understanding is that the authors of the *.rst programs themselves also=20
use GRASS 5, among other things because its sites data model is very=20
simple.
Best wishes,
Roger
>=20
> Maciek
>=20
>=20
> --------------------
> W polskim Internecie s=B1 setki milion=F3w stron. My przekazujemy Tobie t=
ylko najlepsze z nich!
> http://katalog.epf.pl/
>=20
> _______________________________________________
> grass5 mailing list
> grass5@grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5
>=20
--=20
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand@nhh.no
|
|
Mon, Dec 5 2005
20:40:50
|
|
Mail sent by werchowyna@epf.pl
|
|
Return-Path |
<werchowyna@epf.pl>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
From |
Maciek Sieczka <werchowyna@epf.pl>
|
To |
Roger.Bivand@nhh.no
|
Cc |
radim.blazek@gmail.com, grass5@grass.itc.it, grass-bugs@intevation.de
|
In-Reply-To |
<Pine.LNX.4.44.0512051851500.20745-100000@reclus.nhh.no>
|
References |
<Pine.LNX.4.44.0512051851500.20745-100000@reclus.nhh.no>
|
Content-Type |
text/plain
|
Date |
Mon, 05 Dec 2005 20:40:55 +0100
|
Message-Id |
<1133811655.8199.167.camel@localhost.localdomain>
|
Mime-Version |
1.0
|
X-Mailer |
Evolution 2.2.1.1
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
On pon, 2005-12-05 at 19:11 +0100, Roger Bivand wrote:
> On Mon, 5 Dec 2005, Maciek Sieczka wrote:
>
> > On pon, 2005-12-05 at 13:38 +0100, Radim Blazek via RT wrote:
> > > please read old mails on this problem. I dont have time to explain it
> > > again and again. AFAIK there are no big memory leaks.
> >
> > Is it aknowledged by Grass developers that a machine freeze at 5 mln
> > vector points file is a BUG (no matter what the reason is)?
> >
> > If it is aknowledged, can we expect it to be fixed? When - soon, month
> > time, year time? Or is it going to be a "feature" and left as is?
>
> I think this is unfair.
No. This is a question. I've got a task to accomplish which I can't in
Grass6 currently. I'm asking what are the chances my problem will be
fixed soon/ever. I have about 3 months or so for my task. I need to know
where I'm standing instead of "I don't have time to explain". And how do
I even know if those "old mails" reflect the current state?
It's Radim's answer that is unfair, not my reply. Or maybe I'm doing
something rude reporting bugs and I get what I deserve?
> There has been progress in GRASS 6 on this, and
> the vector architecture is much stronger than it was in GRASS 5
Am I saying it isn't progressing? I'm saying it still not good enough.
But I'm happy with any single improvement that takes place. Sorry if I
don't express it enough. Sure it is easier to point out errors than good
things. But anyway, this are a devel list and bugtracker - a place for
discussing problems mainly.
> for moderate and large data sets, but not for XXL.
What's XXL? I need to reproject a detailed 5m DEM of one national park
only. To do it properly, it has to be transformed into vector points,
these will be reprojected, then a DEM in new projection will be
reinterpolated. Unfortunatelly reprojecting a DEM as raster yields
distrotions AFAIK.
> Have you considered using GRASS 5, which has sites, a very much simpler
> data model for points?
I had had bad experience with vector point in Grass 6 before, so
actually it was the first thing to try sites in 5.4. Although I managed
to transform my 50 mln cells DEM into sites and reproject those,
s.surf.rst crashed on such dataset. Since it was an 8GB P4 with plenty
of swap, I didn't even try it at home on my 1GB RAM machine - I wonder
if I could get at least that far.
And I didn't report the bug in s.surf.rst because Grass 5 is no longer
mantained by the core dev crew.
Then I tried 6.1, wondering how far I cold go and hoping that when I
enconter problems, I'm more likely to be helped, ie. the bug would be
fixed. Or that at least my experience and the bug report will be somehow
appreciated. That was silly I see.
> Have you considered tiling your data - reading
> portions of your data and patching the resulting spline surfaces?
I would like to avoid it. Is it a good idea to mosaick DEM? Won't there
be artifacts at the connetcions?
> Once you
> have the surface, you can transfer it to GRASS 6, because as yet the
> raster storage data model is effectively unchanged.
>
> This is not a bug,
That's a very tollerant approach toward bug definition.
> it is a mis-match of data models and intentions.
Do you mean that the fact Grass is not able to handle even 5 mln vector
points (1 tenth of my whole possible dataset) is something normal?
What's 5 mln points? 2236x2236 points, 10x500 km GPS tracks at 1m
interval. Something a serious GIS vector model should handle perfect.
> While
> accepting that freezing (meaning causing total OS failure, or rather
> occupation of all machine resources?
The latter.
> - I don't think that a non-root user
> on a sensible OS can freeze the system so that a hard shutdown (pull
> power) is required) is unfortunate, it is usually caused by 100% CPU use
> and swapping caused by memory being fully occupied.
That was the case. like I described it - both 1GB ram and 1GB swap where
used, total hang, even mouse pointer freezed, had to reset.
> In well-written
> software, like GRASS 6 vector
That's your perception. From my point of view Grass vector model is not
well written yet. Of course, one might say "send us a patch", and
"limited man power". Unfortunatelly I'm not able to send a patch. All I
can do for Grass is to test the code, report bugs, help in the
bugtracker a bit, help other users when I have some time (it is cool to
show off a little, isn't it). Which I do. Not as much as I would like
to, but anyway, I do contribute a bit. And even if I didn't, I do
deserve a decent reply.
> or R, say, there is a balance between how
> things are written, perceived needs, and user perceptions. The GRASS 6
> vector data model handles areas and lines pretty well, but because it is
> trying harder on these, is not well suited to XXL points data sets.
Again, is 5 mln points an XXL? If it was the raster engine in Grass to
fail on 5 mln cells, would you still say it is XXL? Or that it was a
serious bug in the raster engine?
> My
> understanding is that the authors of the *.rst programs themselves also
> use GRASS 5, among other things because its sites data model is very
> simple.
Like I said, I managed to transform to sites and reprojected them on an
8GB beast but s.surf.rst crashed anyway. Can't recall the error message,
assumed it was pointless anyway since Grass 5 is no longer mantained
AAMOF. But I could retry to reproduce the error if it would mnake sense
(ie. if any chances it would be fixed in 5.4).
>
> Best wishes,
Thanks for your interest.
Best regards,
Maciek
--------------------
W polskim Internecie są setki milionów stron. My przekazujemy Tobie tylko najlepsze
z nich!
http://katalog.epf.pl/
|
|
Mon, Dec 5 2005
21:52:25
|
|
Mail sent by Roger.Bivand@nhh.no
|
|
Return-Path |
<Roger.Bivand@nhh.no>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
X-IronPort-AV |
i="3.99,218,1131318000"; d="scan'208"; a="1804881:sNHT146266684"
|
Date |
Mon, 5 Dec 2005 21:52:16 +0100 (CET)
|
From |
Roger Bivand <Roger.Bivand@nhh.no>
|
X-X-Sender |
rsb@reclus.nhh.no
|
Reply-To |
Roger.Bivand@nhh.no
|
To |
Maciek Sieczka <werchowyna@epf.pl>
|
Cc |
radim.blazek@gmail.com, <grass5@grass.itc.it>, <grass-bugs@intevation.de>
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
In-Reply-To |
<1133811655.8199.167.camel@localhost.localdomain>
|
Message-ID |
<Pine.LNX.4.44.0512052043350.20745-100000@reclus.nhh.no>
|
MIME-Version |
1.0
|
Content-Type |
TEXT/PLAIN; charset=ISO-8859-1
|
Content-Transfer-Encoding |
QUOTED-PRINTABLE
|
X-OriginalArrivalTime |
05 Dec 2005 20:52:09.0990 (UTC) FILETIME=[C2B8B260:01C5F9DD]
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
On Mon, 5 Dec 2005, Maciek Sieczka wrote:
> On pon, 2005-12-05 at 19:11 +0100, Roger Bivand wrote:
> > On Mon, 5 Dec 2005, Maciek Sieczka wrote:
> >=20
> > > On pon, 2005-12-05 at 13:38 +0100, Radim Blazek via RT wrote:
> > > > please read old mails on this problem. I dont have time to explain =
it
> > > > again and again. AFAIK there are no big memory leaks.
> > >=20
> > > Is it aknowledged by Grass developers that a machine freeze at 5 mln
> > > vector points file is a BUG (no matter what the reason is)?
> > >=20
> > > If it is aknowledged, can we expect it to be fixed? When - soon, mont=
h
> > > time, year time? Or is it going to be a "feature" and left as is?
> >=20
> > I think this is unfair.
>=20
> No. This is a question. I've got a task to accomplish which I can't in
> Grass6 currently. I'm asking what are the chances my problem will be
> fixed soon/ever. I have about 3 months or so for my task. I need to know
> where I'm standing instead of "I don't have time to explain". And how do
> I even know if those "old mails" reflect the current state?
>=20
> It's Radim's answer that is unfair, not my reply. Or maybe I'm doing
> something rude reporting bugs and I get what I deserve?
No, you have (your perception of) a real problem, and are trying to find a=
=20
feasible solution. Radim's work on the vector data model has helped with=20
many problems, but not this one. Getting at a problem like this is, as you=
=20
know well, very layered. You have an input DEM, which you need to warp=20
with high precision in the z-dimension to a different spatial reference=20
system.=20
Assuming that the input DEM is an exact representation of the
sub-vegetation surface as it was when it was surveyed, warping will
introduce some errors and interpolating will introduce others. You have
chosen to transform the raster points (abot 50M) to the target spatial
reference system and interpolate, I guess because you tried warping and
found the error unacceptable. You have three months (but there is snow in
the ground now), so field surveying to establish a baseline for error in
one or several trial plots is still feasible, maybe?
But I'm not aware of surveyed or interpolated DEMs that are without=20
measurement error themselves - David Unwin has a nice statement in one of=
=20
his books about the sobering effect of comparing field survey elevations=20
from leveling and DEM values. So what we are looking for is a way of=20
getting from the input DEM to the output DEM without introducing=20
systematic error (like the artefacts at patch/tile boundaries) and without=
=20
adding too much to the error already present.
Given that GRASS 6 vector points for even 10% of the data set are a=20
problem, is it possible to establish the relative performance of warping=20
versus interpolation on - say - a number of 1% sample plots? Does r.proj=20
give similar outcomes to gdalwarp? I guess you've looked at all of this,=20
and I apologize for thinking aloud. I just feel that getting to the main=20
question of making sure output map errors are not systematic is quite=20
difficult, and not obvious.=20
Since the area of interest has fairly large changes of elevation over=20
short distances in some parts, it might even be possible to thin out=20
uninformative points (say those within some threshold of their=20
neighbours) where the relief is not detailed, keeping points that are=20
"needed" for interpolation.=20
>=20
> > There has been progress in GRASS 6 on this, and=20
> > the vector architecture is much stronger than it was in GRASS 5
>=20
> Am I saying it isn't progressing? I'm saying it still not good enough.
> But I'm happy with any single improvement that takes place. Sorry if I
> don't express it enough. Sure it is easier to point out errors than good
> things. But anyway, this are a devel list and bugtracker - a place for
> discussing problems mainly.
>=20
> > for moderate and large data sets, but not for XXL.
>=20
> What's XXL? I need to reproject a detailed 5m DEM of one national park
> only. To do it properly, it has to be transformed into vector points,
> these will be reprojected, then a DEM in new projection will be
> reinterpolated. Unfortunatelly reprojecting a DEM as raster yields
> distrotions AFAIK.
>=20
> > Have you considered using GRASS 5, which has sites, a very much simpler=
=20
> > data model for points?
>=20
> I had had bad experience with vector point in Grass 6 before, so
> actually it was the first thing to try sites in 5.4. Although I managed
> to transform my 50 mln cells DEM into sites and reproject those,
> s.surf.rst crashed on such dataset. Since it was an 8GB P4 with plenty
> of swap, I didn't even try it at home on my 1GB RAM machine - I wonder
> if I could get at least that far.
>=20
> And I didn't report the bug in s.surf.rst because Grass 5 is no longer
> mantained by the core dev crew.
Where crash means segmentation fault or complete occupation of machine=20
resources? Again, I'm unsure whether all the points are essential to reach=
=20
a result without larger and systematic errors.=20
>=20
> Then I tried 6.1, wondering how far I cold go and hoping that when I
> enconter problems, I'm more likely to be helped, ie. the bug would be
> fixed. Or that at least my experience and the bug report will be somehow
> appreciated. That was silly I see.
Not silly, but still a difference between what you expected the=20
combination of software and hardware to carry out and what other users and=
=20
developers have seen as being their priority.=20
>=20
> > Have you considered tiling your data - reading=20
> > portions of your data and patching the resulting spline surfaces?
>=20
> I would like to avoid it. Is it a good idea to mosaick DEM? Won't there
> be artifacts at the connetcions?
Yes, but are they larger (wide overlaps and average the values) than the=20
errors already in the data? If yes, we are stuck, if no, there is a way=20
forward, and recall that *.rst and other interpolators for this kind of=20
data use a (very) small moving window over the data anyway, so tiling and=
=20
patching is happening anyway. The key thing is the scale of the errors and=
=20
whether they are systematic.
>=20
> > Once you=20
> > have the surface, you can transfer it to GRASS 6, because as yet the=20
> > raster storage data model is effectively unchanged.=20
> >=20
> > This is not a bug,
>=20
> That's a very tollerant approach toward bug definition.
>=20
A bug is when software does not do what it is designed to do, leaving=20
quite a margin for interpretation in what users/developers think it is=20
supposed to do. Things like deleting files when not asked to are bugs, as=
=20
are overwriting objects in memory or seg-faults from freeing unallocated=20
pointers. Those are more "objective", even though they can be very hard to=
=20
find (valgrind helps), but differences in understanding of purpose are not=
=20
bugs in my view.
> > it is a mis-match of data models and intentions.
>=20
> Do you mean that the fact Grass is not able to handle even 5 mln vector
> points (1 tenth of my whole possible dataset) is something normal?
> What's 5 mln points? 2236x2236 points, 10x500 km GPS tracks at 1m
> interval. Something a serious GIS vector model should handle perfect.
>=20
Each of these data points is carrying information, so the question is how=
=20
much you need to handle to deal with the problem - and this varies very=20
much indeed. I have no idea which commercial GIS could interpolate your=20
data or warp them perfectly, but as you've seen, perfect isn't a word I=20
associate with the natural world, seems to fit virtual reality better! If=
=20
your 5m grid data are really very accurate (and the accuracy is invariant=
=20
across the whole area), then going by patches or warping are options in=20
GRASS, but as you've demonstrated, neither interpolating in GRASS 5 (what=
=20
did gdb say when s.surf.rst failed) nor handling the 50M points in GRASS=20
6 in one mouthful seem to work. Any solution is going to be messy, even if=
=20
s.surf.rst had run in GRASS 5, how should we know that the same tension=20
parameter should be applied across the whole map? Is it at all feasible to=
=20
divide the map up into zones of similar roughness (more rough meaning=20
more careful choice of *.rst parameters)?=20
> > While=20
> > accepting that freezing (meaning causing total OS failure, or rather=20
> > occupation of all machine resources?
>=20
> The latter.
>=20
> > - I don't think that a non-root user=20
> > on a sensible OS can freeze the system so that a hard shutdown (pull=20
> > power) is required) is unfortunate, it is usually caused by 100% CPU us=
e=20
> > and swapping caused by memory being fully occupied.
>=20
> That was the case. like I described it - both 1GB ram and 1GB swap where
> used, total hang, even mouse pointer freezed, had to reset.
You can run GRASS from the command line without a GUI - even though=20
response time may be very slow, having one terminal (say Ctrl-Alt-1) for=20
you and GRASS, and another for you logged in with the correct process=20
number for what you are running keyed in already to kill -9 should be=20
accessible. Mice die, but CLI still runs, like Duracell rabbits.=20
>=20
> > In well-written=20
> > software, like GRASS 6 vector
>=20
> That's your perception. From my point of view Grass vector model is not
> well written yet. Of course, one might say "send us a patch", and
> "limited man power". Unfortunatelly I'm not able to send a patch. All I
> can do for Grass is to test the code, report bugs, help in the
> bugtracker a bit, help other users when I have some time (it is cool to
> show off a little, isn't it). Which I do. Not as much as I would like
> to, but anyway, I do contribute a bit. And even if I didn't, I do
> deserve a decent reply.
I'm trying (both senses, I'm sure), but this is a serious problem (the=20
error propagation is what needs controlling), so finding out how the=20
errors behave for different approaches seems justified (and you have=20
certainly thought of this, your earlier many constructive and positive=20
contributions to the list show that you are serious about what you do).=20
Please forgive the length of my reply, but I still think that you may be=20
able to use the software as it is to get an acceptable result, but that=20
doing it all at once is not the only option.
>=20
>=20
> > or R, say, there is a balance between how=20
> > things are written, perceived needs, and user perceptions. The GRASS 6=
=20
> > vector data model handles areas and lines pretty well, but because it i=
s=20
> > trying harder on these, is not well suited to XXL points data sets.
>=20
> Again, is 5 mln points an XXL? If it was the raster engine in Grass to
> fail on 5 mln cells, would you still say it is XXL? Or that it was a
> serious bug in the raster engine?
>=20
It is quite a lot of data, certainly far more that the people who wrote=20
much of the software thought of handling.=20
>=20
> > My=20
> > understanding is that the authors of the *.rst programs themselves also=
=20
> > use GRASS 5, among other things because its sites data model is very=20
> > simple.
>=20
> Like I said, I managed to transform to sites and reprojected them on an
> 8GB beast but s.surf.rst crashed anyway. Can't recall the error message,
> assumed it was pointless anyway since Grass 5 is no longer mantained
> AAMOF. But I could retry to reproduce the error if it would mnake sense
> (ie. if any chances it would be fixed in 5.4).
>=20
Because the *.surf.rst programs are closely related, and are associated=20
with published research, I think their authors are the best people to=20
comment on this. I recall that you were in touch with them about faults in=
=20
September.
Roger
> >=20
> > Best wishes,
>=20
> Thanks for your interest.
>=20
> Best regards,
> Maciek
>=20
>=20
> --------------------
> W polskim Internecie s=B1 setki milion=F3w stron. My przekazujemy Tobie t=
ylko najlepsze z nich!
> http://katalog.epf.pl/
>=20
--=20
Roger Bivand
Economic Geography Section, Department of Economics, Norwegian School of
Economics and Business Administration, Helleveien 30, N-5045 Bergen,
Norway. voice: +47 55 95 93 55; fax +47 55 95 95 43
e-mail: Roger.Bivand@nhh.no
|
|
Tue, Dec 6 2005
01:17:05
|
|
Mail sent by twiens@interbaun.com
|
|
Return-Path |
<twiens@interbaun.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Mon, 5 Dec 2005 17:17:03 -0700
|
From |
Trevor Wiens <twiens@interbaun.com>
|
To |
Roger.Bivand@nhh.no
|
Cc |
Maciek Sieczka <werchowyna@epf.pl>, radim.blazek@gmail.com, grass5@grass.itc.it, grass-bugs@intevation.de
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
Message-ID |
<20051205171703.5472a52b@localhost.localdomain>
|
In-Reply-To |
<Pine.LNX.4.44.0512052043350.20745-100000@reclus.nhh.no>
|
References |
<1133811655.8199.167.camel@localhost.localdomain> <Pine.LNX.4.44.0512052043350.20745-100000@reclus.nhh.no>
|
X-Mailer |
Sylpheed-Claws 1.9.14 (GTK+ 2.8.6; i486-pc-linux-gnu)
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Roger and Radim,
I don't really want to piss anyone off but this is my opinion.
First, I searched the archives for the old archives and was unable to
find any references to r.to.vect halting the system.
Second, if this is a problem that has come up repeatedly, this is an
indication that work is needed in this area. Probably updating the
documentation outlining this problem would be a good idea so someone
doesn't have to reply with more than "Look here" or "RTFM".
Third, as a former database programmer, I would say that when a
program fails, it should fail early, loudly (with abundant information
to help solve the problem), and nicely (not halting the system). AFAICT
Maciek was using r.to.vect as the program was intended, converting a
raster to a vector. If there are is some sort of artificial file size
issues, that should be clearly stated in the documentation, or better
yet the program should check is first and fail without wasting the users
time or computing resources. Based on these reasonable user and
developer based criteria, this is a bug. That said I don't know what the
issues are surrounding this process, but I took a brief look at the
vector specification in the developers documentation this afternoon out
of curiosity and I can't see how this conversion would need to be
written for point files where file size or even computing resources
should be an issue.
I appreciate the work that Radim has put into the vector portion of
GRASS. I'm sure he is busy just like everyone else, thus point 2 above.
Roger thanks for constructive suggestions. It is always a pleasure to
read your posts on any list; always helpful and constructive.
T
--
Trevor Wiens
twiens@interbaun.com
The significant problems that we face cannot be solved at the same
level of thinking we were at when we created them.
(Albert Einstein)
|
|
Tue, Dec 6 2005
07:03:32
|
|
Mail sent by dca.gis@gmail.com
|
|
Return-Path |
<dca.gis@gmail.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
DomainKey-Signature |
a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=g9TZGdYuTh6UvLTDhsdYnCGXSBXY6p3EXgXx5q7x7VlewY/8NODyMhSpM0xYVNTxVV9Xff7WeySY5yvrsrFKXIy1wCl6I5jzThbwfwDljvvK1TI2qrwsIcB+V1dAsc6PRxYm+r0kgtm0N7LVZJZ4GoJoP/JZMrP3P0mAIdIAyzU=
|
Message-ID |
<1a486f560512052203r55acc02j8d086c649817179d@mail.gmail.com>
|
Date |
Tue, 6 Dec 2005 01:03:27 -0500
|
From |
Daniel Calvelo <dca.gis@gmail.com>
|
To |
Maciek Sieczka <werchowyna@epf.pl>
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
Cc |
grass5@grass.itc.it, grass-bugs@intevation.de
|
In-Reply-To |
<1133804301.8199.67.camel@localhost.localdomain>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=ISO-8859-1
|
Content-Transfer-Encoding |
quoted-printable
|
Content-Disposition |
inline
|
References |
<20051205123835.AE456101EFD@lists.intevation.de> <1133804301.8199.67.camel@localhost.localdomain>
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
I really sympathize with your current predicament, Maciek. Just to
clear things a little, I think that what Radim meant is that there are
no memory *leaks*. It's just that the memory *requirements* of your
problem are well beyond grass's architecture current capabilities. You
just don't have enough memory to perform the operation, the machine
goes into swapping out everything, including your X server, and never
goes out of it before you kill it.
The only workaround I can think of is
1) LOTS of swap and
2) LOTS of patience.
This is linux, right? You can try to add swapfiles (to more than the
8GB you already tried) and then, knowing that disk access is in the
order of 10^3 slower than RAM, maybe some timings on small datasets
might allow you to calculate the required time for it to complete.
Then launch it at night. Or week-end. I've been through this kind of
hassle for other applications, and that approach worked for me before.
Especially the "patience" part :)
But anyway, we should indeed put these memory-full-related problems on
the TODO list for 7.0.
Best wishes.
Daniel.
On 12/5/05, Maciek Sieczka <werchowyna@epf.pl> wrote:
> On pon, 2005-12-05 at 13:38 +0100, Radim Blazek via RT wrote:
> > please read old mails on this problem. I dont have time to explain it
> > again and again. AFAIK there are no big memory leaks.
>
> Is it aknowledged by Grass developers that a machine freeze at 5 mln
> vector points file is a BUG (no matter what the reason is)?
>
> If it is aknowledged, can we expect it to be fixed? When - soon, month
> time, year time? Or is it going to be a "feature" and left as is?
>
> Maciek
>
>
> --------------------
> W polskim Internecie s=B1 setki milion=F3w stron. My przekazujemy Tobie t=
ylko najlepsze z nich!
> http://katalog.epf.pl/
>
> _______________________________________________
> grass5 mailing list
> grass5@grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5
>
--
-- Daniel Calvelo Aros
|
|
Tue, Dec 6 2005
09:29:47
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Tue, 6 Dec 2005 21:29:03 +1300
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
Maciek Sieczka <werchowyna@epf.pl>
|
Cc |
Roger.Bivand@nhh.no, radim.blazek@gmail.com, grass5@grass.itc.it, grass-bugs@intevation.de, wiens@interbaun.com
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
Message-Id |
<20051206212903.41fd3391.hamish_nospam@yahoo.com>
|
In-Reply-To |
<1133811655.8199.167.camel@localhost.localdomain>
|
References |
<Pine.LNX.4.44.0512051851500.20745-100000@reclus.nhh.no> <1133811655.8199.167.camel@localhost.localdomain>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
|
X-Spam-Level |
|
> > for moderate and large data sets, but not for XXL.
>
> What's XXL?
t-shirt speak for extra-extra-large.
> I need to reproject a detailed 5m DEM of one national park
> only. To do it properly, it has to be transformed into vector points,
> these will be reprojected, then a DEM in new projection will be
> reinterpolated. Unfortunatelly reprojecting a DEM as raster yields
> distrotions AFAIK.
give it a try, then compare. It might not be that bad in practice. For a
non-categorical map, try 'r.proj method=cubic' for a better result.
Randomly select test points (v.random -> v.what.rast; v.proj ->
v.what.rast) for a simple quantitative comparison or error.
> > If it is aknowledged, can we expect it to be fixed? When - soon,
> > month time, year time? Or is it going to be a "feature" and left as
> > is?
the official Free software answer to this FAQ: "It will be done when it
is ready, sooner if you help." (c.1992 ref Linux 1.0)
> And I didn't report the bug in s.surf.rst because Grass 5 is no longer
> mantained by the core dev crew.
severe bugs will be given a cursory look, data corruption bugs will be
explored, and if reported to the bug tracker (in the correct category;
5.4 is still there) any normal [unfixed] bugs listed will at minimum
give others seeing the same bug a sense of solidarity and closure. Minor
bugs will be likely ignored, unless trivial and/or a patch is supplied.
i.e. GRASS 5.4 is still maintained, just not actively.
> > This is not a bug,
> That's a very tollerant approach toward bug definition.
This is a known limitation in the data model. There is no error in the
code, it just wasn't designed to deal with that large of a dataset.
The GRASS 6 vector model was designed before the popularization of LIDAR
and multibeam sonar. Anything over 3 million points or so will use up
all your memory. Your OS should see this and kill the offending
application when this happens. All solutions and work-arounds suggested
to date have not been satisfactory, so the issue remains open until we
can figure out something better. Who knows when that will be? As far as
I understand from Radim's explainations, there appears to be no quick
fix.
An outstanding issue which can be resolved is a port of s.cellstats to
GRASS 6. This would help ameliorate the problem for some people..
Or a preprossor utility...
> > - I don't think that a non-root user
> > on a sensible OS can freeze the system so that a hard shutdown (pull
> > power) is required) is unfortunate, it is usually caused by 100% CPU
> > use and swapping caused by memory being fully occupied.
>
> That was the case. like I described it - both 1GB ram and 1GB swap
> where used, total hang, even mouse pointer freezed, had to reset.
The point being that your OS should have killed the process that was
causing the problem. GRASS just asks the OS for more memory. If the OS
keeps giving a program more memory after it has already run out and is
gasping for air, we can't help that.
> That's your perception. From my point of view Grass vector model is
> not well written yet.
It is beautifully written to do what it was intended to do. The problem
is that you (and others) want it to do something that it was not
intended to do. It can still call itself a good GIS vector engine in
good conscience, but it is not all things.
Also, I would suggest that this sort of comment is not the best way to
motivate a designer to make changes for you. Remember that nothing kills
the volunteer spirit faster than demands and insulting the creator's
baby.
If this problem affects enough people (and developers) it will
eventually affect someone who can a) fix it or b) pay someone else to
fix it; and then it will be fixed. This problem seems to affect a number
of people, so there is hope. Can't promise you more than that.
Trevor:
> First, I searched the archives for the old archives and was unable to
> find any references to r.to.vect halting the system.
No, you won't find it looking for r.to.vect; the same issue was mostly
discussed with respect to creating vector points with v.in.ascii. Try as
search terms: "LIDAR" "memory" "v.in.ascii" "valgrind"; for emails from
Radim, Helena, and myself.
> Second, if this is a problem that has come up repeatedly, this is an
> indication that work is needed in this area.
I think we all agree that is true. But we don't know of a good solution
yet.
> Probably updating the documentation outlining this problem would be a
> good idea
Yes, it would be good to mention in the v.in.ascii and r.to.vect help
pages that creating several million data points may use large amounts of
memory, which may cause problems.
Note that once the import issue is solved, other modules start to have
problems with these huge datasets too.
> Third, as a former database programmer, I would say that when a
> program fails, it should fail early, loudly (with abundant information
> to help solve the problem), and nicely (not halting the system).
> AFAICT Maciek was using r.to.vect as the program was intended,
> converting a raster to a vector. If there are is some sort of
> artificial file size issues, that should be clearly stated in the
> documentation, or better yet the program should check is first and
> fail without wasting the users time or computing resources. Based on
> these reasonable user and developer based criteria, this is a bug.
> That said I don't know what the issues are surrounding this process,
> but I took a brief look at the vector specification in the developers
> documentation this afternoon out of curiosity and I can't see how this
> conversion would need to be written for point files where file size or
> even computing resources should be an issue.
Band-aid approach (curing the symptom, not the cause):
Once number of points is known, a calculation of memory use could be
done (~300 bytes per vector point?, best create a test point & sizeof()
rather than hardcode "300"). G_malloc() and G_free() could be
called as a test, which will call G_fatal_error() if the dataset is too
huge to complete. I don't think this reduces the need for a solution,
just makes the failure friendlier.
Another angle of attack is to get that 300 bytes per point number lower.
10 million points should be possible given enough swap space, the
current model, and today's hardware; 50 million is really pushing it.
>From the valgrind analysis the growth seems to be linear:
http://bambi.otago.ac.nz/hamish/grass/memleak/v.in.ascii/
hint for adding more swap space to linux on the fly:
http://grass.itc.it/pipermail/grassuser/2002-February/006070.html
Hamish
|
|
Tue, Dec 6 2005
10:32:56
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Tue, 6 Dec 2005 22:32:17 +1300
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
werchowyna@epf.pl
|
Cc |
Roger.Bivand@nhh.no, radim.blazek@gmail.com, grass5@grass.itc.it, grass-bugs@intevation.de, wiens@interbaun.com
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
Message-Id |
<20051206223217.4a83c099.hamish_nospam@yahoo.com>
|
In-Reply-To |
<20051206212903.41fd3391.hamish_nospam@yahoo.com>
|
References |
<Pine.LNX.4.44.0512051851500.20745-100000@reclus.nhh.no> <1133811655.8199.167.camel@localhost.localdomain> <20051206212903.41fd3391.hamish_nospam@yahoo.com>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.0 tagged_above=-999.0 required=3.0 tests=BAYES_00, FORGED_YAHOO_RCVD
|
X-Spam-Level |
|
I wrote:
> If this problem affects enough people (and developers) it will
> eventually affect someone who can a) fix it or b) pay someone else to
> fix it; and then it will be fixed. This problem seems to affect a number
> of people, so there is hope. Can't promise you more than that.
well what do you know, a first step just 8 day ago:
http://grass.itc.it/pipermail/grass-commit/2005-November/019366.html
Hamish
|
|
Tue, Dec 6 2005
11:30:41
|
|
Mail sent by radim.blazek@gmail.com
|
|
Return-Path |
<radim.blazek@gmail.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
DomainKey-Signature |
a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=Mm9pGZhNHqE/ztgx7goLL9/rhZhU4AC6c0SsKntQYDAR0T3L7QddyRM9+2JpkplPfVWv3QMIgTnpzt61KOPHGtX40WYns0svwK/qiZrJl0+3ZNUKhuIYPQN0xAj4qCCwtpVJwVUw4qdJHsuQDO0pIA/bTOIR0K644W4hnCkl7Ns=
|
Message-ID |
<340505ef0512060230y479d471bkc4da5423d3006dc1@mail.gmail.com>
|
Date |
Tue, 6 Dec 2005 11:30:36 +0100
|
From |
Radim Blazek <radim.blazek@gmail.com>
|
To |
Maciek Sieczka <werchowyna@epf.pl>
|
Subject |
Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
Cc |
grass5@grass.itc.it, grass-bugs@intevation.de
|
In-Reply-To |
<1133804301.8199.67.camel@localhost.localdomain>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=ISO-8859-1
|
Content-Transfer-Encoding |
quoted-printable
|
Content-Disposition |
inline
|
References |
<20051205123835.AE456101EFD@lists.intevation.de> <1133804301.8199.67.camel@localhost.localdomain>
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
On 12/5/05, Maciek Sieczka <werchowyna@epf.pl> wrote:
> On pon, 2005-12-05 at 13:38 +0100, Radim Blazek via RT wrote:
> > please read old mails on this problem. I dont have time to explain it
> > again and again. AFAIK there are no big memory leaks.
>
> Is it aknowledged by Grass developers that a machine freeze at 5 mln
> vector points file is a BUG (no matter what the reason is)?
It is not bug, it is feature, bad feature.
> If it is aknowledged, can we expect it to be fixed? When - soon, month
> time, year time? Or is it going to be a "feature" and left as is?
I suggested solution. Unfortunately those who need to work with larger
datasets do not have will to fix that and I don't have time enough
to work on something I don't currently need.
Radim
|
|
Tue, Dec 6 2005
12:31:00
|
|
Mail sent by werchowyna@epf.pl
|
|
Return-Path |
<werchowyna@epf.pl>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
From |
Maciek Sieczka <werchowyna@epf.pl>
|
To |
Hamish <hamish_nospam@yahoo.com>
|
Cc |
Roger.Bivand@nhh.no, grass5@grass.itc.it, grass-bugs@intevation.de, wiens@interbaun.com
|
In-Reply-To |
<20051206212903.41fd3391.hamish_nospam@yahoo.com>
|
References |
<Pine.LNX.4.44.0512051851500.20745-100000@reclus.nhh.no> <1133811655.8199.167.camel@localhost.localdomain> <20051206212903.41fd3391.hamish_nospam@yahoo.com>
|
Content-Type |
text/plain
|
Date |
Tue, 06 Dec 2005 12:31:02 +0100
|
Message-Id |
<1133868662.7956.105.camel@localhost.localdomain>
|
Mime-Version |
1.0
|
X-Mailer |
Evolution 2.2.1.1
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Thanks to all Folks who replied! I appreciate your comments a lot! Sorry
if I insulted anyone. Sure nobody likes to be treated in an arogant and
dismissive way.
On wto, 2005-12-06 at 21:29 +1300, Hamish wrote:
> > > for moderate and large data sets, but not for XXL.
> >
> > What's XXL?
>
> t-shirt speak for extra-extra-large.
I was asking what Roger meant by "XXL datasets", might be I was not
explicit enough :).
We do have t-shirts in Poland. However, it is true we prefer hunting our
dinosaurs naked :).
> > I need to reproject a detailed 5m DEM of one national park
> > only. To do it properly, it has to be transformed into vector points,
> > these will be reprojected, then a DEM in new projection will be
> > reinterpolated. Unfortunatelly reprojecting a DEM as raster yields
> > distrotions AFAIK.
>
> give it a try, then compare. It might not be that bad in practice. For a
> non-categorical map, try 'r.proj method=cubic' for a better result.
> Randomly select test points (v.random -> v.what.rast; v.proj ->
> v.what.rast) for a simple quantitative comparison or error.
Whatever resampling method there will be artifacts when reprojecting an
fp raster. For nearest nigbour my fp DEM will get a systematiic error
detectable on eg. aspect maps, for bilinear and more adavnced this
systematic error will decrease but original elavations will get
distorted. I don't want either.
> > > If it is aknowledged, can we expect it to be fixed? When - soon,
> > > month time, year time? Or is it going to be a "feature" and left as
> > > is?
>
> the official Free software answer to this FAQ: "It will be done when it
> is ready, sooner if you help." (c.1992 ref Linux 1.0)
So I'm helping the way I can. Testing, repoting. Sorry if not enough. We
are all busy.
> > And I didn't report the bug in s.surf.rst because Grass 5 is no longer
> > mantained by the core dev crew.
> severe bugs will be given a cursory look, data corruption bugs will be
> explored, and if reported to the bug tracker (in the correct category;
> 5.4 is still there) any normal [unfixed] bugs listed will at minimum
> give others seeing the same bug a sense of solidarity and closure. Minor
> bugs will be likely ignored, unless trivial and/or a patch is supplied.
> i.e. GRASS 5.4 is still maintained, just not actively.
Ok, when I have time to reproduce the problem in 5.4, I will report.
> > > This is not a bug,
> > That's a very tollerant approach toward bug definition.
>
> This is a known limitation in the data model. There is no error in the
> code, it just wasn't designed to deal with that large of a dataset.
> The GRASS 6 vector model was designed before the popularization of LIDAR
> and multibeam sonar. Anything over 3 million points or so will use up
> all your memory. Your OS should see this and kill the offending
> application when this happens. All solutions and work-arounds suggested
> to date have not been satisfactory, so the issue remains open until we
> can figure out something better. Who knows when that will be? As far as
> I understand from Radim's explainations, there appears to be no quick
> fix.
You are talking from the developer's point of view, myself from the
user's. We will never agree here.
> An outstanding issue which can be resolved is a port of s.cellstats to
> GRASS 6. This would help ameliorate the problem for some people..
> Or a preprossor utility...
Grass 6 was supposed to be sites free. If sites functionality gets to
Grass6 the point vector engine will be in danger of being less used,
tested and fixed. This we don't want. IMHO functional sites import, and
maybe export, should be all we need here. Vecto engine should be fixed
instaed to handle any dataset. Besides, although my problem refers to
vector points, I guess it would also appear for other geometry types
when milions of objects are involved - or am I wrong here?
> > > - I don't think that a non-root user
> > > on a sensible OS can freeze the system so that a hard shutdown (pull
> > > power) is required) is unfortunate, it is usually caused by 100% CPU
> > > use and swapping caused by memory being fully occupied.
> >
> > That was the case. like I described it - both 1GB ram and 1GB swap
> > where used, total hang, even mouse pointer freezed, had to reset.
>
> The point being that your OS should have killed the process that was
> causing the problem. GRASS just asks the OS for more memory. If the OS
> keeps giving a program more memory after it has already run out and is
> gasping for air, we can't help that.
> > That's your perception. From my point of view Grass vector model is
> > not well written yet.
>
> It is beautifully written to do what it was intended to do. The problem
> is that you (and others) want it to do something that it was not
> intended to do. It can still call itself a good GIS vector engine in
> good conscience, but it is not all things.
>
> Also, I would suggest that this sort of comment is not the best way to
> motivate a designer to make changes for you. Remember that nothing kills
> the volunteer spirit faster than demands and insulting the creator's
> baby.
Nothing kills the volunteer user spirit for testing and reporting as his
effort not being taken into account. Most Grass developers know it and
appreciate bug reports, even if sometimes silly or wrong. Look at it his
way: Lots of people use Grass. All users must have problems. How many
report bugs in a reasonable way presently, ecluding developers? 10? Out
of hundreds or thousands using Grass? Can Grass afford loosing these 10?
> If this problem affects enough people (and developers) it will
> eventually affect someone who can a) fix it or b) pay someone else to
> fix it; and then it will be fixed. This problem seems to affect a number
> of people, so there is hope. Can't promise you more than that.
Thanks.
> Trevor:
> > First, I searched the archives for the old archives and was unable to
> > find any references to r.to.vect halting the system.
>
> No, you won't find it looking for r.to.vect; the same issue was mostly
> discussed with respect to creating vector points with v.in.ascii. Try as
> search terms: "LIDAR" "memory" "v.in.ascii" "valgrind"; for emails from
> Radim, Helena, and myself.
None of those emails provide an answer if and when the problem may be
fixed. I can't find it at least. Maybe it is there, but I don't see it.
> > Second, if this is a problem that has come up repeatedly, this is an
> > indication that work is needed in this area.
>
> I think we all agree that is true. But we don't know of a good solution
> yet.
Thanks,
Maciek
--------------------
W polskim Internecie są setki milionów stron. My przekazujemy Tobie tylko najlepsze
z nich!
http://katalog.epf.pl/
|
|
Tue, Dec 6 2005
12:40:55
|
|
Mail sent by werchowyna@epf.pl
|
|
Return-Path |
<werchowyna@epf.pl>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
From |
Maciek Sieczka <werchowyna@epf.pl>
|
To |
Hamish <hamish_nospam@yahoo.com>
|
Cc |
Roger.Bivand@nhh.no, radim.blazek@gmail.com, grass5@grass.itc.it, grass-bugs@intevation.de, wiens@interbaun.com
|
In-Reply-To |
<20051206223217.4a83c099.hamish_nospam@yahoo.com>
|
References |
<Pine.LNX.4.44.0512051851500.20745-100000@reclus.nhh.no> <1133811655.8199.167.camel@localhost.localdomain> <20051206212903.41fd3391.hamish_nospam@yahoo.com> <20051206223217.4a83c099.hamish_nospam@yahoo.com>
|
Content-Type |
text/plain
|
Date |
Tue, 06 Dec 2005 12:41:10 +0100
|
Message-Id |
<1133869270.7956.111.camel@localhost.localdomain>
|
Mime-Version |
1.0
|
X-Mailer |
Evolution 2.2.1.1
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
On wto, 2005-12-06 at 22:32 +1300, Hamish wrote:
> well what do you know, a first step just 8 day ago:
>
> http://grass.itc.it/pipermail/grass-commit/2005-November/019366.html
Cool. Can anybody say how many points without topology I can import
using this modified v.in.ascii?
Maciek
--------------------
W polskim Internecie są setki milionów stron. My przekazujemy Tobie tylko najlepsze
z nich!
http://katalog.epf.pl/
|
|
Tue, Dec 6 2005
15:14:45
|
|
Subject changed to r.to.vect, v.in.ascii use too much memory for millions of points by hbowman
|
|
Tue, Dec 6 2005
17:17:37
|
|
Mail sent by adanner@cs.duke.edu
|
|
Return-Path |
<adanner@cs.duke.edu>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Subject |
Re: [GRASS5] Re: [bug #3877] (grass) r.to.vect: severe memory leaks, I'm helpless
|
From |
Andrew Danner <adanner@cs.duke.edu>
|
To |
Maciek Sieczka <werchowyna@epf.pl>
|
Cc |
Hamish <hamish_nospam@yahoo.com>, Roger.Bivand@nhh.no, radim.blazek@gmail.com, grass5@grass.itc.it, grass-bugs@intevation.de, wiens@interbaun.com
|
In-Reply-To |
<1133869270.7956.111.camel@localhost.localdomain>
|
References |
<Pine.LNX.4.44.0512051851500.20745-100000@reclus.nhh.no> <1133811655.8199.167.camel@localhost.localdomain> <20051206212903.41fd3391.hamish_nospam@yahoo.com> <20051206223217.4a83c099.hamish_nospam@yahoo.com> <1133869270.7956.111.camel@localhost.localdomain>
|
Content-Type |
text/plain
|
Date |
Tue, 06 Dec 2005 11:16:51 -0500
|
Message-Id |
<1133885811.8143.8.camel@localhost.localdomain>
|
Mime-Version |
1.0
|
X-Mailer |
Evolution 2.2.1.1
|
Content-Transfer-Encoding |
7bit
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
I haven't tried this particular patch, but I tried something very
similar and got over 20 million points with very low memory usage, so it
should scale beyond that. In 5.4, my largest set was over 500 million.
Even though v.in.ascii will be able to import large sets, other vector
modules (v.surf.rst, for example) would need to be modified to handle
the case when topology isn't built. It can be done, but as Radim
suggested, you end up with a sites model and a vector model, which Grass
6+ is trying to avoid.
I have some familiarity with working with large sets with v.surf.rst
and v.in.ascii and I can try to look into possible "fixes", though I'm
reluctant to use the word "fix". You can turn off the topology and get
rst and ascii import working for you particular application, but you
break the vector model as a result.
-Andy
On Tue, 2005-12-06 at 12:41 +0100, Maciek Sieczka wrote:
> On wto, 2005-12-06 at 22:32 +1300, Hamish wrote:
> > well what do you know, a first step just 8 day ago:
> >
> > http://grass.itc.it/pipermail/grass-commit/2005-November/019366.html
>
> Cool. Can anybody say how many points without topology I can import
> using this modified v.in.ascii?
>
> Maciek
>
>
> --------------------
> W polskim Internecie s setki milionw stron. My przekazujemy Tobie tylko najlepsze
z nich!
> http://katalog.epf.pl/
>
> _______________________________________________
> grass5 mailing list
> grass5@grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass5
|
|
Tue, Jul 4 2006
15:27:29
|
|
Mail sent by mneteler
|
|
New patch submitted, see
https://intevation.de/rt/webrt?serial_num=3354&display=History
Does it solve the problem?
Markus |
|
Tue, Jul 4 2006
23:27:54
|
|
Mail sent by werchowyna@epf.pl
|
|
Return-Path |
<werchowyna@epf.pl>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Tue, 4 Jul 2006 23:28:00 +0200
|
From |
Maciek Sieczka <werchowyna@epf.pl>
|
To |
Markus Neteler via RT <grass-bugs@intevation.de>
|
Cc |
neteler@itc.it
|
Subject |
Re: [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Message-Id |
<20060704232800.26390f94.werchowyna@epf.pl>
|
In-Reply-To |
<20060704132729.3A7521005A4@lists.intevation.de>
|
References |
<20060704132729.3A7521005A4@lists.intevation.de>
|
X-Mailer |
Sylpheed version 2.1.1 (GTK+ 2.8.6; i486-pc-linux-gnu)
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-3.427 tagged_above=-999 required=4 tests=[AWL=1.522, BAYES_00=-5, RCVD_BY_IP=0.051]
|
X-Spam-Level |
|
On Tue, 4 Jul 2006 15:27:29 +0200 (CEST)
Markus Neteler via RT <grass-bugs@intevation.de> wrote:
> New patch submitted, see
>
> https://intevation.de/rt/webrt?serial_num=3354&display=History
>
> Does it solve the problem?
Will check it out as time allows. Thanks for letting me know!
Maciek
------------------------------------------------------------------------
CIEP?E KRAJE - CIEP?E MORZA. Szukasz atrakcyjnego wypoczynku w przyst?pnej cenie,
zapoznaj si? z nasz? ofert?.
ZAPRASZAMY
www.skarpatravel.pl
|
|
Wed, Jul 5 2006
20:39:04
|
|
Mail sent by msieczka
|
|
Markus wrote:
> New patch submitted, see
>
> https://intevation.de/rt/webrt?serial_num=3354&display=History
>
> Does it solve the problem?
So I checked with current CVS and the same problem still applies to r.to.vect.
"r.to.vect -z feature=point input=dem_5 output=dem_5_pt" eats up all 1GB RAM
+
1GB SWAP at about 5 000 000 points.
The above mentioned Andrew Danner's fix for v.in.ascii is great stuff but
r.to.vect problem remains (in my bug report I was complaining about only
r.to.vect, few days later Hamish changed the subject, as v.in.ascii issue
popped up during discussion).
Is it possible that r.to.vect suffers from a similar problem as v.in.ascii
did, so a similar fix would do? Andrew?
Maciek
|
|
Wed, Jul 5 2006
21:09:07
|
|
Mail sent by adanner@cs.duke.edu
|
|
Return-Path |
<adanner@cs.duke.edu>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Subject |
Re: [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
From |
Andrew Danner <adanner@cs.duke.edu>
|
To |
Maciek Sieczka via RT <grass-bugs@intevation.de>
|
Cc |
grass-dev@grass.itc.it, neteler@itc.it
|
In-Reply-To |
<20060705183904.4D5031005B9@lists.intevation.de>
|
References |
<20060705183904.4D5031005B9@lists.intevation.de>
|
Content-Type |
text/plain
|
Date |
Wed, 05 Jul 2006 15:09:02 -0400
|
Message-Id |
<1152126542.5695.28.camel@localhost>
|
Mime-Version |
1.0
|
X-Mailer |
Evolution 2.6.1
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-4.259 tagged_above=-999 required=4 tests=[AWL=0.741, BAYES_00=-5]
|
X-Spam-Level |
|
Maciek,
My initial guess is that r.to.vect suffers from a similar bug/feature
that plagued v.in.ascii awhile ago and that is the building of vector
topology. As it stands now, r.to.vect calls Vect_build after processing
all the features and this is the same function call that ate all the
memory in v.in.ascii. The solution for v.in.ascii was to add another
flag "-b" to skip topology building in points mode. The rest of the
r.to.vect code looks pretty clean and I don't immediately expect memory
leaks. Radim has said many times that there are not leaks in the
Vect_build code, but the memory requirements are high for the topology
building. Without looking into the Vect_build code, I tend to believe
Radim, so if you want to extract 5 000 000 points from a raster you will
need to skip the topology. Note that many vector modules are not able to
use vector layers without topology (v.surf.rst being the primary
execption), so the "-b" flag is more of a workaround than a long term
solution.
I haven't had a chance to look into the Vect_build code and see if
there is a way to reduce memory usage. Is there any white paper or
technical specs on how the new vector library is organized and what the
vector topology looks like?
-Andy
On Wed, 2006-07-05 at 20:39 +0200, Maciek Sieczka via RT wrote:
> Markus wrote:
>
> > New patch submitted, see
> >
> > https://intevation.de/rt/webrt?serial_num=3354&display=History
> >
> > Does it solve the problem?
>
> So I checked with current CVS and the same problem still applies to r.to.vect.
>
> "r.to.vect -z feature=point input=dem_5 output=dem_5_pt" eats up all 1GB RAM
+
> 1GB SWAP at about 5 000 000 points.
>
> The above mentioned Andrew Danner's fix for v.in.ascii is great stuff but
> r.to.vect problem remains (in my bug report I was complaining about only
> r.to.vect, few days later Hamish changed the subject, as v.in.ascii issue
> popped up during discussion).
>
> Is it possible that r.to.vect suffers from a similar problem as v.in.ascii
> did, so a similar fix would do? Andrew?
>
> Maciek
>
>
> -------------------------------------------- Managed by Request Tracker
|
|
Thu, Jul 6 2006
00:00:39
|
|
Mail sent by neteler@itc.it
|
|
Return-Path |
<neteler@itc.it>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Thu, 6 Jul 2006 00:00:33 +0200
|
From |
Markus Neteler <neteler@itc.it>
|
To |
Andrew Danner <adanner@cs.duke.edu>
|
Cc |
Maciek Sieczka via RT <grass-bugs@intevation.de>, grass-dev@grass.itc.it
|
Subject |
Re: [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Message-ID |
<20060705220033.GE24748@bartok.itc.it>
|
Mail-Followup-To |
Andrew Danner <adanner@cs.duke.edu>, Maciek Sieczka via RT <grass-bugs@intevation.de>, grass-dev@grass.itc.it
|
References |
<20060705183904.4D5031005B9@lists.intevation.de> <1152126542.5695.28.camel@localhost>
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Disposition |
inline
|
In-Reply-To |
<1152126542.5695.28.camel@localhost>
|
X-PGP-Key |
http://www.gdf-hannover.de/neteler/markus_gpgkey.asc
|
X-PGP-Fingerprint |
D4D5 2F80 120E AD60 E2F6 2297 21B3 D02B E1E7 E789
|
User-Agent |
Mutt/1.5.11
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-3.43 tagged_above=-999 required=4 tests=[AWL=1.304, BAYES_00=-5, FORGED_RCVD_HELO=0.266]
|
X-Spam-Level |
|
Andrew,
On Wed, Jul 05, 2006 at 03:09:02PM -0400, Andrew Danner wrote:
...
> I haven't had a chance to look into the Vect_build code and see if
> there is a way to reduce memory usage. Is there any white paper or
> technical specs on how the new vector library is organized and what the
> vector topology looks like?
there is a document here (part of the programmer's manual):
http://mpa.itc.it/markus/grass61progman/Vector_Library.html
That what I could extract from Radim and sketch up :-)
It's generated from the (aprtially) doxygenized source code.
Markus
> -Andy
>
>
> On Wed, 2006-07-05 at 20:39 +0200, Maciek Sieczka via RT wrote:
> > Markus wrote:
> >
> > > New patch submitted, see
> > >
> > > https://intevation.de/rt/webrt?serial_num=3354&display=History
> > >
> > > Does it solve the problem?
> >
> > So I checked with current CVS and the same problem still applies to r.to.vect.
> >
> > "r.to.vect -z feature=point input=dem_5 output=dem_5_pt" eats up all 1GB
RAM +
> > 1GB SWAP at about 5 000 000 points.
> >
> > The above mentioned Andrew Danner's fix for v.in.ascii is great stuff but
> > r.to.vect problem remains (in my bug report I was complaining about only
> > r.to.vect, few days later Hamish changed the subject, as v.in.ascii issue
> > popped up during discussion).
> >
> > Is it possible that r.to.vect suffers from a similar problem as v.in.ascii
> > did, so a similar fix would do? Andrew?
> >
> > Maciek
> >
> >
> > -------------------------------------------- Managed by Request Tracker
>
--
Markus Neteler <neteler itc it> http://mpa.itc.it/markus/
ITC-irst - Centro per la Ricerca Scientifica e Tecnologica
MPBA - Predictive Models for Biol. & Environ. Data Analysis
Via Sommarive, 18 - 38050 Povo (Trento), Italy
|
|
Thu, Jul 6 2006
00:05:29
|
|
Mail sent by werchowyna@epf.pl
|
|
Return-Path |
<werchowyna@epf.pl>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Thu, 6 Jul 2006 00:05:32 +0200
|
From |
Maciek Sieczka <werchowyna@epf.pl>
|
To |
Andrew Danner <adanner@cs.duke.edu>
|
Cc |
grass-bugs@intevation.de, neteler@itc.it, grass-dev@grass.itc.it
|
Subject |
Re: [GRASS-dev] Re: [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Message-Id |
<20060706000532.553858f1.werchowyna@epf.pl>
|
In-Reply-To |
<1152126542.5695.28.camel@localhost>
|
References |
<20060705183904.4D5031005B9@lists.intevation.de> <1152126542.5695.28.camel@localhost>
|
X-Mailer |
Sylpheed version 2.1.1 (GTK+ 2.8.6; i486-pc-linux-gnu)
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-3.437 tagged_above=-999 required=4 tests=[AWL=1.512, BAYES_00=-5, RCVD_BY_IP=0.051]
|
X-Spam-Level |
|
Andrew,
I first thought that your v.in.ascii fix enabled v.in.ascii to load
huge datasets *not* skipping the topology. After your clarification now
I see it was my mistake. Sorry if confusing.
Although I realize how important it is for many of us to be able to
load huge point datasets in any possible way, for now, like using this
no-topology hack, I hope there will one day be a real solution for
Grass to be able to process such big datasets in a normal, topological
way. Because propably the no-topology hack will be not suitable for
anything else besides points and propably we can't expect every single
vector module to be extended to support both non-topological and
topological vectors - also because there are GIS operations which
simply require a topological data model. The few 10^6 number of
features limit is a serious limitation in current Grass vector engine.
I wouldn't consider the bug solved, even regarding v.in.ascii alone.
But I do really appreciate all your effort towards making out as much
as possible of v.in.ascii for the moment. Thank you.
Maciek
--------------------
W polskim Internecie s? setki milion?w stron. My przekazujemy Tobie tylko najlepsze
z nich!
http://katalog.panoramainternetu.pl/
|
|
Thu, Jul 6 2006
15:20:05
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Fri, 7 Jul 2006 01:19:42 +1200
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
Maciek Sieczka via RT <grass-bugs@intevation.de>
|
Cc |
grass-dev@grass.itc.it, adanner@cs.duke.edu
|
Subject |
Re: [GRASS-dev] [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Message-Id |
<20060707011942.00c7db9b.hamish_nospam@yahoo.com>
|
In-Reply-To |
<20060705183904.4D5031005B9@lists.intevation.de>
|
References |
<20060705183904.4D5031005B9@lists.intevation.de>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-1.519 tagged_above=-999 required=4 tests=[AWL=1.386, BAYES_00=-5, FORGED_YAHOO_RCVD=2.095]
|
X-Spam-Level |
|
Maciek Sieczka wrote:
> So I checked with current CVS and the same problem still applies to
> r.to.vect.
>
> "r.to.vect -z feature=point input=dem_5 output=dem_5_pt" eats up all
> 1GB RAM + 1GB SWAP at about 5 000 000 points.
>
> The above mentioned Andrew Danner's fix for v.in.ascii is great stuff
> but r.to.vect problem remains (in my bug report I was complaining
> about only r.to.vect, few days later Hamish changed the subject, as
> v.in.ascii issue popped up during discussion).
>
> Is it possible that r.to.vect suffers from a similar problem as
> v.in.ascii did, so a similar fix would do? Andrew?
does it happen during the "building lines" (or "registering lines"?)
step?
(watch the memory use using 'top' in another xterm, use "M" to sort by
memory use)
if so, it's the same problem as v.in.ascii building topology.
I added a -b flag to r.to.vect (in CVS) to skip building topology for
this reason. Only tested with raster cells->vector points in mind.
(r.in.xyz -> r.to.vect -> v.surf.rst)
Hamish
|
|
Thu, Jul 6 2006
16:45:16
|
|
Mail sent by werchowyna@epf.pl
|
|
Return-Path |
<werchowyna@epf.pl>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Thu, 6 Jul 2006 16:45:22 +0200
|
From |
Maciek Sieczka <werchowyna@epf.pl>
|
To |
Hamish <hamish_nospam@yahoo.com>
|
Cc |
grass-bugs@intevation.de, grass-dev@grass.itc.it, adanner@cs.duke.edu
|
Subject |
Re: [GRASS-dev] [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Message-Id |
<20060706164522.004be5dc.werchowyna@epf.pl>
|
In-Reply-To |
<20060707011942.00c7db9b.hamish_nospam@yahoo.com>
|
References |
<20060705183904.4D5031005B9@lists.intevation.de> <20060707011942.00c7db9b.hamish_nospam@yahoo.com>
|
X-Mailer |
Sylpheed version 2.1.1 (GTK+ 2.8.6; i486-pc-linux-gnu)
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-3.456 tagged_above=-999 required=4 tests=[AWL=1.493, BAYES_00=-5, RCVD_BY_IP=0.051]
|
X-Spam-Level |
|
On Fri, 7 Jul 2006 01:19:42 +1200
Hamish <hamish_nospam@yahoo.com> wrote:
> Maciek Sieczka wrote:
> > So I checked with current CVS and the same problem still applies to
> > r.to.vect.
> >
> > "r.to.vect -z feature=point input=dem_5 output=dem_5_pt" eats up all
> > 1GB RAM + 1GB SWAP at about 5 000 000 points.
> >
> > The above mentioned Andrew Danner's fix for v.in.ascii is great
> > stuff but r.to.vect problem remains (in my bug report I was
> > complaining about only r.to.vect, few days later Hamish changed the
> > subject, as v.in.ascii issue popped up during discussion).
> >
> > Is it possible that r.to.vect suffers from a similar problem as
> > v.in.ascii did, so a similar fix would do? Andrew?
>
>
> does it happen during the "building lines" (or "registering lines"?)
> step?
Yes.
> if so, it's the same problem as v.in.ascii building topology.
> I added a -b flag to r.to.vect (in CVS) to skip building topology for
> this reason. Only tested with raster cells->vector points in mind.
> (r.in.xyz -> r.to.vect -> v.surf.rst)
I'm not sure if this is right the way to go. If we proceed this way then
v.proj, v.in.*, v.perturb, v.to.points and other would require the
same. Do we want it? Double standards will be confusing, expecially for
newcommers. Shouldn't the vector engine be fixed instead not to use all
memory? Every no-topology hack will reduce the chance for a real
solution.
(On the other hand, sure I will bless your "r.to.vect -b" having no
other solution handy. But really this is not a sustainable approach.)
Maciek
--------------------
W polskim Internecie s? setki milion?w stron. My przekazujemy Tobie tylko najlepsze
z nich!
http://katalog.panoramainternetu.pl/
|
|
Fri, Jul 7 2006
10:20:34
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Fri, 7 Jul 2006 20:19:58 +1200
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
Maciek Sieczka <werchowyna@epf.pl>
|
Cc |
grass-bugs@intevation.de, grass-dev@grass.itc.it, adanner@cs.duke.edu
|
Subject |
Re: [GRASS-dev] [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Message-Id |
<20060707201958.1da471d9.hamish_nospam@yahoo.com>
|
In-Reply-To |
<20060706164522.004be5dc.werchowyna@epf.pl>
|
References |
<20060705183904.4D5031005B9@lists.intevation.de> <20060707011942.00c7db9b.hamish_nospam@yahoo.com> <20060706164522.004be5dc.werchowyna@epf.pl>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-1.534 tagged_above=-999 required=4 tests=[AWL=1.371, BAYES_00=-5, FORGED_YAHOO_RCVD=2.095]
|
X-Spam-Level |
|
Maciek wrote:
> I'm not sure if this is right the way to go. If we proceed this way
> then v.proj, v.in.*, v.perturb, v.to.points and other would require
> the same. Do we want it? Double standards will be confusing,
> expecially for newcommers. Shouldn't the vector engine be fixed
> instead not to use all memory? Every no-topology hack will reduce the
> chance for a real solution.
>
> (On the other hand, sure I will bless your "r.to.vect -b" having no
> other solution handy. But really this is not a sustainable approach.)
In principal I agree, in practice I am willing to compromise.
The -b flag is a temporary work-around until we have a better solution.
A pure solution is nice, but may take time and we have deadlines to
meet.
Or stated another way, I know enough of the vector code to add a -b flag
but not enough to rewrite the engine to fix the underlying problem. So I
add a -b flag and agree that a better solution is needed.
I was never very clear on this, but have an idea that topology is
meaningless for data which is only points (no tree; only bounding box
matters?). If so, the (correct) solution becomes much easier.
Hamish
|
|
Fri, Jul 7 2006
13:48:20
|
|
Mail sent by neteler@itc.it
|
|
Return-Path |
<neteler@itc.it>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Message-ID |
<44AE4A00.1000702@itc.it>
|
Date |
Fri, 07 Jul 2006 13:48:16 +0200
|
From |
Markus Neteler <neteler@itc.it>
|
User-Agent |
Thunderbird 1.5.0.4 (X11/20060516)
|
MIME-Version |
1.0
|
To |
grass-dev@grass.itc.it
|
Cc |
grass-bugs@intevation.de
|
Subject |
Re: [GRASS-dev] [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
References |
<20060705183904.4D5031005B9@lists.intevation.de> <20060707011942.00c7db9b.hamish_nospam@yahoo.com> <20060706164522.004be5dc.werchowyna@epf.pl> <20060707201958.1da471d9.hamish_nospam@yahoo.com>
|
In-Reply-To |
<20060707201958.1da471d9.hamish_nospam@yahoo.com>
|
X-Enigmail-Version |
0.94.0.0
|
OpenPGP |
url=http://mpa.itc.it/markus/markus_gpgkey.asc
|
Content-Type |
text/plain; charset=ISO-8859-1
|
Content-Transfer-Encoding |
7bit
|
X-OriginalArrivalTime |
07 Jul 2006 11:48:15.0861 (UTC) FILETIME=[3BAA4A50:01C6A1BB]
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-3.511 tagged_above=-999 required=4 tests=[AWL=1.223, BAYES_00=-5, FORGED_RCVD_HELO=0.266]
|
X-Spam-Level |
|
Hamish wrote on 07/07/2006 10:19 AM:
> I was never very clear on this, but have an idea that topology is
> meaningless for data which is only points (no tree; only bounding box
> matters?). If so, the (correct) solution becomes much easier.
>
... nor me.
*If* topology is meaningless for point data, then we could add a test
in Vect_built() to
- check if only points are present in the map,
- if so, skip the topology creation.
A likewise test would be needed in the Vect_open() routine. Here the
question is if we can check beforehand that the map only contains
points and then ignore the topology (skip Vect_open_topo() in Vectlib?)
Markus
|
|
Fri, Jul 7 2006
16:27:21
|
|
Mail sent by hmitaso@unity.ncsu.edu
|
|
Return-Path |
<hmitaso@unity.ncsu.edu>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
In-Reply-To |
<44AE4A00.1000702@itc.it>
|
References |
<20060705183904.4D5031005B9@lists.intevation.de> <20060707011942.00c7db9b.hamish_nospam@yahoo.com> <20060706164522.004be5dc.werchowyna@epf.pl> <20060707201958.1da471d9.hamish_nospam@yahoo.com> <44AE4A00.1000702@itc.it>
|
Mime-Version |
1.0 (Apple Message framework v746.2)
|
Content-Type |
text/plain; charset=US-ASCII; delsp=yes; format=flowed
|
Message-Id |
<B0883BBD-3DCD-4843-9B92-0311F398BAF5@unity.ncsu.edu>
|
Cc |
grass-dev@grass.itc.it, grass-bugs@intevation.de
|
Content-Transfer-Encoding |
7bit
|
From |
Helena Mitasova <hmitaso@unity.ncsu.edu>
|
Subject |
Re: [GRASS-dev] [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Date |
Fri, 7 Jul 2006 10:27:15 -0400
|
To |
Markus Neteler <neteler@itc.it>
|
X-Mailer |
Apple Mail (2.746.2)
|
X-Virus-Scanned |
Symantec AntiVirus Scan Engine
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-3.944 tagged_above=-999 required=4 tests=[AWL=1.056, BAYES_00=-5]
|
X-Spam-Level |
|
Helena Mitasova
Dept. of Marine, Earth and Atm. Sciences
1125 Jordan Hall, NCSU Box 8208,
Raleigh NC 27695
http://skagit.meas.ncsu.edu/~helena/
On Jul 7, 2006, at 7:48 AM, Markus Neteler wrote:
> Hamish wrote on 07/07/2006 10:19 AM:
>> I was never very clear on this, but have an idea that topology is
>> meaningless for data which is only points (no tree; only bounding box
>> matters?). If so, the (correct) solution becomes much easier.
>>
> ... nor me.
> *If* topology is meaningless for point data, then we could add a test
> in Vect_built() to
> - check if only points are present in the map,
> - if so, skip the topology creation.
this is not generaly a good solution - I will get back to this when I
have more time -
it is good to read Radims document about what to do with the vector
format
first before further engaging in this discussion - Maciek please read
it -
that will give you a better idea what is going on.
Helena
>
> A likewise test would be needed in the Vect_open() routine. Here the
> question is if we can check beforehand that the map only contains
> points and then ignore the topology (skip Vect_open_topo() in
> Vectlib?)
>
> Markus
>
>
> _______________________________________________
> grass-dev mailing list
> grass-dev@grass.itc.it
> http://grass.itc.it/mailman/listinfo/grass-dev
|
|
Fri, Jul 7 2006
18:02:07
|
|
Mail sent by neteler@itc.it
|
|
Return-Path |
<neteler@itc.it>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Fri, 7 Jul 2006 18:02:05 +0200
|
From |
Markus Neteler <neteler@itc.it>
|
To |
grass-dev@grass.itc.it
|
Cc |
grass-bugs@intevation.de
|
Subject |
Re: [GRASS-dev] [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Message-ID |
<20060707160204.GJ15770@bartok.itc.it>
|
Mail-Followup-To |
grass-dev@grass.itc.it, grass-bugs@intevation.de
|
References |
<20060705183904.4D5031005B9@lists.intevation.de> <20060707011942.00c7db9b.hamish_nospam@yahoo.com> <20060706164522.004be5dc.werchowyna@epf.pl> <20060707201958.1da471d9.hamish_nospam@yahoo.com> <44AE4A00.1000702@itc.it> <B0883BBD-3DCD-4843-9B92-0311F398BAF5@unity.ncsu.edu> <44AE78BC.4090406@itc.it> <44AE83BA.2000300@unity.ncsu.edu>
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Disposition |
inline
|
In-Reply-To |
<44AE83BA.2000300@unity.ncsu.edu>
|
X-PGP-Key |
http://www.gdf-hannover.de/neteler/markus_gpgkey.asc
|
X-PGP-Fingerprint |
D4D5 2F80 120E AD60 E2F6 2297 21B3 D02B E1E7 E789
|
User-Agent |
Mutt/1.5.11
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-3.533 tagged_above=-999 required=4 tests=[AWL=1.201, BAYES_00=-5, FORGED_RCVD_HELO=0.266]
|
X-Spam-Level |
|
On Fri, Jul 07, 2006 at 11:54:34AM -0400, Helena Mitasova wrote:
> Markus Neteler wrote:
> >Helena Mitasova wrote on 07/07/2006 04:27 PM:
> >
> >>On Jul 7, 2006, at 7:48 AM, Markus Neteler wrote:
> >>
> >>>Hamish wrote on 07/07/2006 10:19 AM:
> >>>
> >>>>I was never very clear on this, but have an idea that topology is
> >>>>meaningless for data which is only points (no tree; only bounding box
> >>>>matters?). If so, the (correct) solution becomes much easier.
> >>>>
> >>>>
> >>>... nor me.
> >>>*If* topology is meaningless for point data, then we could add a test
> >>>in Vect_built() to
> >>>- check if only points are present in the map,
> >>>- if so, skip the topology creation.
> >>>
> >>this is not generaly a good solution - I will get back to this when I
> >>have more time -
> >>it is good to read Radims document about what to do with the vector
> >>format
> >>first before further engaging in this discussion - Maciek please read
> >>it -
> >>that will give you a better idea what is going on.
> >>
> >
> >This is certainly a good idea. May I suggest that someone picks all the
> >pieces from the various (Radim et al.) emails and creates a Wiki page out
> >of that?
> >
> Markus - the better way would be to add the document that he has written
> about the next step
> to do with vector support as a reference into
> http://mpa.itc.it/markus/grass61progman/Vector_Library.html
> he has identified scalability as a main issue for vector support and
> suggests some solutions
> (I believe it is the building of spatial index that is needed for
> topology building but potentially for
> other things that needs to be modified - but I really don't want to go
> into this without reading it again).
> As for the emails - most of it is just repeating the same thing over and
> over (I am starting
> to be like Radim), although I have posted Radim's suggestion on how to
> modify
> v.info and v.to.rast that has not been implemented yet and that might be
> useful (maybe add it to Radim's document)
Agreed - add it to Radim's document. At least a document.
Currently the info is scattered around and hard to find.
Radim's document is there:
doc/vector/TODO
Markus
|
|
Mon, Jul 10 2006
09:03:53
|
|
Mail sent by hamish_nospam@yahoo.com
|
|
Return-Path |
<hamish_nospam@yahoo.com>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Date |
Mon, 10 Jul 2006 19:03:47 +1200
|
From |
Hamish <hamish_nospam@yahoo.com>
|
To |
Markus Neteler <neteler@itc.it>
|
Cc |
grass-dev@grass.itc.it, grass-bugs@intevation.de
|
Subject |
Re: [GRASS-dev] [bug #3877] (grass) r.to.vect, v.in.ascii use too much memory for millions of points
|
Message-Id |
<20060710190347.1dbba617.hamish_nospam@yahoo.com>
|
In-Reply-To |
<20060707160204.GJ15770@bartok.itc.it>
|
References |
<20060705183904.4D5031005B9@lists.intevation.de> <20060707011942.00c7db9b.hamish_nospam@yahoo.com> <20060706164522.004be5dc.werchowyna@epf.pl> <20060707201958.1da471d9.hamish_nospam@yahoo.com> <44AE4A00.1000702@itc.it> <B0883BBD-3DCD-4843-9B92-0311F398BAF5@unity.ncsu.edu> <44AE78BC.4090406@itc.it> <44AE83BA.2000300@unity.ncsu.edu> <20060707160204.GJ15770@bartok.itc.it>
|
X-Mailer |
Sylpheed version 1.0.4 (GTK+ 1.2.10; i386-pc-linux-gnu)
|
X-Face |
M<EoB)"*Z~u!,vFhXmw}R_KbdBta*P_=T|rbBL'e1/CQ9;/1g\BU3&!=y8ria$2Uk!HT&BB 8i?|X_+7~1jsy}F~g$2va%3fV`*=L(*cem[@3\yg,G,@rg6/QMJ
|
Mime-Version |
1.0
|
Content-Type |
text/plain; charset=US-ASCII
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
by amavisd-new at intevation.de
|
X-Spam-Status |
No, hits=-1.579 tagged_above=-999 required=4 tests=[AWL=1.326, BAYES_00=-5, FORGED_YAHOO_RCVD=2.095]
|
X-Spam-Level |
|
Markus:
> Agreed - add it to Radim's document. At least a document.
> Currently the info is scattered around and hard to find.
>
> Radim's document is there:
> doc/vector/TODO
Are you speaking of a list links to historic grass5 emails at the end of
the TODO file? I think that is a great idea.
Hamish
|
|
Wed, Jul 26 2006
18:10:18
|
|
User changed to tutey@o2.pl by msieczka
|
|
Tue, Sep 26 2006
18:36:02
|
|
Comments added by guest
|
|
Good Luck! http://xoomer.alice.it/pik0/poker-rooms/
|
|
Tue, Sep 26 2006
22:40:51
|
|
Comments added by guest
|
|
Cool design http://xoomer.alice.it/pik0/razz-poker/
|
|
Wed, Sep 27 2006
01:25:52
|
|
Comments added by guest
|
|
Great work on website. <a href="http://xoomer.alice.it/pik0/rules-of-poker/">rules
of poker</a> [url=http://xoomer.alice.it/pik0/rules-of-poker/]rules of poker[/url]
http://xoomer.alice.it/pik0/rules-of-poker/ |
|
Thu, Nov 9 2006
06:27:23
|
|
Comments added by hbowman
|
|
for the search engine: This bug contains SPAM
This bug is worked around useing the "-b" don't build topology flag.
Hamish
|
|
Thu, Nov 9 2006
15:26:52
|
|
Comments added by msieczka
|
|
hbowman wrote (Thu, Nov 9 2006 06:27:23):
> This bug is worked around useing the "-b" don't build topology flag.
This workaround only helps in case of points. And even in case of points most
GRASS modules can't use topology-less vectors. So this is like a 10% of 1/3 of
a workaround ;).
Maciek
|
|