Details Ticket 2903


Comment | Reply | Take | Open


Serial Number 2903
Subject v.in.ascii for larger data sets with attributes does not work
Area grass6
Queue grass
Requestors hmitaso@unity.ncsu.edu
Owner none
Status resolved
Last User Contact Fri Apr 29 03:43:34 2005 (3 yr ago)
Current Priority 30
Final Priority 70
Due No date assigned
Last Action Tue Jul 4 15:05:30 2006 (2 yr ago)
Created Wed Jan 12 05:57:59 2005 (4 yr ago)

Transaction History Ticket 2903


Wed, Jan 12 2005 05:57:59    Request created by guest  
Subject: v.in.ascii for larger data sets with attributes does not work

Platform: GNU/Linux/i386
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: 5.7 CVS updated 20040111

v.in.ascii has a memory problem for dbf option for data sets that include
500000 points or more.
As an example -
it reads 50000 points for all options in about 10 seconds
it reads 200000 points in minutes  using cca 100MB memory,
it cannot read 500000 points: it fills 1.5GB memory and starts filling swap too.
I was unable to get it finish importing any file over 500K points
after many hours and the machine gets frozen.

The problem occurs also with -z option if there are 4 columns in the file,
the third one is ID (or category) - such files are created by s.in.ascii in 5.4
Examples of runs that won't finish
v.in.ascii -z testlid560K.ascii out=testlid560Kxyz z=4 fs=space
v.in.ascii testlid560K.ascii out=testlid560K fs=space

For a 280000 point data set it is slow but it does the job:
This takes 30 minutes:
v.in.ascii -z atmlid99280K.ascii out=atmlid99280Kxyz z=4 fs=space
Maximum input row length: 45
Maximum number of columns: 4
Minimum number of columns: 4
column: 1  type: double
column: 2  type: double
column: 3  type: integer
column: 4  type: double
Building topology ...
282294 primitives registered
...etc

The same file without attributes takes 40 seconds to read
v.in.ascii -t atmlid99280K.ascii out=atmlid99280Kxy fs=space
Maximum input row length: 45
Maximum number of columns: 4
Minimum number of columns: 4
Building topology ...
282294 primitives registered
...etc

The test data are available at http://mpa.itc.it/grasstutor/data_menu2nd.phtml
Jockey's ridge LIDAR data, but a sequence of test data can be easily created
by applying r.random to any DEM(e.g. spearfish) in GRASS5.4.

Helena
Fri, Apr 29 2005 03:43:34    Status changed to resolved by hbowman  
Fri, Apr 29 2005 03:43:34    Mail sent by hbowman  
fixed in CVS. 

see
  http://grass.itc.it/pipermail/grass5/2005-March/017834.html


I think some small memory leaks remain but I can now load in the 1.15 million
point LIDAR test data set ok using about 500mb RAM.



Hamish
Fri, Apr 29 2005 05:02:29    Mail sent by hmitaso@unity.ncsu.edu  
Return-Path <hmitaso@unity.ncsu.edu>
Delivered-To grass-bugs@lists.intevation.de
Message-ID <4271A550.7020005@unity.ncsu.edu>
Date Thu, 28 Apr 2005 23:09:04 -0400
From Helena <hmitaso@unity.ncsu.edu>
Organization NCSU
User-Agent Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4) Gecko/20030624
X-Accept-Language en-us, en
MIME-Version 1.0
To Harmish Bowman via RT <grass-bugs@intevation.de>
Subject Re: [bug #2903] (grass) v.in.ascii for larger data sets with attributes does not work
References <20050429014334.350E51005BC@lists.intevation.de>
In-Reply-To <20050429014334.350E51005BC@lists.intevation.de>
Content-Type text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding 7bit
X-Virus-Scanned Symantec AntiVirus Scan Engine
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Harmish Bowman via RT wrote:
> fixed in CVS. 
> 
> see
>   http://grass.itc.it/pipermail/grass5/2005-March/017834.html
> 
> 
> I think some small memory leaks remain but I can now load in the 1.15 million
> point LIDAR test data set ok using about 500mb RAM.
> 
> 
> 
> Hamish
> 
> 
> -------------------------------------------- Managed by Request Tracker

Yes, I tested it right away when Radim fixed it - it works - please remove the
report
Thank you for helping finding the bug

Helena


Fri, Apr 29 2005 05:02:29    Status changed to open by _rt_system  
Fri, Sep 2 2005 18:37:37    Area changed to grass6 by mneteler  
Tue, Jul 4 2006 15:05:30    Status changed to resolved by mneteler  
Comment | Reply | Take | Open

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request