Wed, Jan 12 2005
05:57:59
|
|
Request created by guest
|
|
Subject: v.in.ascii for larger data sets with attributes does not work
Platform: GNU/Linux/i386
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: 5.7 CVS updated 20040111
v.in.ascii has a memory problem for dbf option for data sets that include
500000 points or more.
As an example -
it reads 50000 points for all options in about 10 seconds
it reads 200000 points in minutes using cca 100MB memory,
it cannot read 500000 points: it fills 1.5GB memory and starts filling swap too.
I was unable to get it finish importing any file over 500K points
after many hours and the machine gets frozen.
The problem occurs also with -z option if there are 4 columns in the file,
the third one is ID (or category) - such files are created by s.in.ascii in 5.4
Examples of runs that won't finish
v.in.ascii -z testlid560K.ascii out=testlid560Kxyz z=4 fs=space
v.in.ascii testlid560K.ascii out=testlid560K fs=space
For a 280000 point data set it is slow but it does the job:
This takes 30 minutes:
v.in.ascii -z atmlid99280K.ascii out=atmlid99280Kxyz z=4 fs=space
Maximum input row length: 45
Maximum number of columns: 4
Minimum number of columns: 4
column: 1 type: double
column: 2 type: double
column: 3 type: integer
column: 4 type: double
Building topology ...
282294 primitives registered
...etc
The same file without attributes takes 40 seconds to read
v.in.ascii -t atmlid99280K.ascii out=atmlid99280Kxy fs=space
Maximum input row length: 45
Maximum number of columns: 4
Minimum number of columns: 4
Building topology ...
282294 primitives registered
...etc
The test data are available at http://mpa.itc.it/grasstutor/data_menu2nd.phtml
Jockey's ridge LIDAR data, but a sequence of test data can be easily created
by applying r.random to any DEM(e.g. spearfish) in GRASS5.4.
Helena |
|
Fri, Apr 29 2005
03:43:34
|
|
Status changed to resolved by hbowman
|
|
Fri, Apr 29 2005
03:43:34
|
|
Mail sent by hbowman
|
|
fixed in CVS.
see
http://grass.itc.it/pipermail/grass5/2005-March/017834.html
I think some small memory leaks remain but I can now load in the 1.15 million
point LIDAR test data set ok using about 500mb RAM.
Hamish
|
|
Fri, Apr 29 2005
05:02:29
|
|
Mail sent by hmitaso@unity.ncsu.edu
|
|
Return-Path |
<hmitaso@unity.ncsu.edu>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
Message-ID |
<4271A550.7020005@unity.ncsu.edu>
|
Date |
Thu, 28 Apr 2005 23:09:04 -0400
|
From |
Helena <hmitaso@unity.ncsu.edu>
|
Organization |
NCSU
|
User-Agent |
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624
|
X-Accept-Language |
en-us, en
|
MIME-Version |
1.0
|
To |
Harmish Bowman via RT <grass-bugs@intevation.de>
|
Subject |
Re: [bug #2903] (grass) v.in.ascii for larger data sets with attributes does not work
|
References |
<20050429014334.350E51005BC@lists.intevation.de>
|
In-Reply-To |
<20050429014334.350E51005BC@lists.intevation.de>
|
Content-Type |
text/plain; charset=us-ascii; format=flowed
|
Content-Transfer-Encoding |
7bit
|
X-Virus-Scanned |
Symantec AntiVirus Scan Engine
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Harmish Bowman via RT wrote:
> fixed in CVS.
>
> see
> http://grass.itc.it/pipermail/grass5/2005-March/017834.html
>
>
> I think some small memory leaks remain but I can now load in the 1.15 million
> point LIDAR test data set ok using about 500mb RAM.
>
>
>
> Hamish
>
>
> -------------------------------------------- Managed by Request Tracker
Yes, I tested it right away when Radim fixed it - it works - please remove the
report
Thank you for helping finding the bug
Helena
|
|
Fri, Apr 29 2005
05:02:29
|
|
Status changed to open by _rt_system
|
|
Fri, Sep 2 2005
18:37:37
|
|
Area changed to grass6 by mneteler
|
|
Tue, Jul 4 2006
15:05:30
|
|
Status changed to resolved by mneteler
|
|