Details Ticket 241


Comment | Reply | Resolve


Serial Number 241
Subject r.proj: reads the entire raster into memory
Area wish6
Queue grass
Requestors neteler@itc.it
Owner fwarmerdam
Status open
Last User Contact Tue Feb 6 09:54:12 2007 (2 yr ago)
Current Priority 30
Final Priority 70
Due No date assigned
Last Action Tue Feb 6 09:54:12 2007 (2 yr ago)
Created Wed May 2 13:27:02 2001 (7 yr ago)

Transaction History Ticket 241


Wed, May 2 2001 13:27:02    Request created by mneteler  
Subject: r.proj: reads the entire raster into memory 

r.proj
 It needs a small rewrite to not read the entire raster into
 memory.  It is useless with large rasters and will seriously bog down your
machine.  I haven't had a chance to really grok it, but I think "rowio"   
 would be appropriate -- although, even that wouldn't be required.  I think
at most it needs three rows in memory for cubic interpolation.

 -> comment from Morten Hulden: This can only happen with *large* maps.
Wed, May 2 2001 14:52:52    Comments added by guest  
A three row cache from the input image will not normally be adequate.  In
a reprojection case with a non-trivial rotational effect generating one 
row of output data will touch many (dozens or hundreds) rows of the input
image.  

A traditional approach to this sort of problem is to maintain a cache of
source file scanlines.  The size of this cache should default to something
like 24MB but be controllable by a commandline parameter.  

This approach will do well for small images and even for very large images
as long as the rotational effect of the rotation is minimal. 

A more sophisticated approach I have seen employed in a previous life was
to watch for cache thrashing, and if it occurs to try and generate the output
in strips, successively reducing the size of output vertical strips generated
until thrashing stops or some minimum is reached.  Generating in strips 
results in multiple passes over the source and destination image, but event
in the worst case of a 90 degree rotation the number of passes over the 
source image can be contrained to roughly the number of strips.  Without
this approach a 90 degree image will generally require one pass over the 
source image for every output scanline!

Note that the problems is r.proj are presumably shared by the rectify code
though I haven't looked there.  I would strongly encourage the generalization
of image warping into a subroutine taking an arbitrary transformation
function.  The caching and resampling can be implemented once in this 
subroutine and then it can be used for r.proj, the rectify code and 
potentially other places. 

I might possibly be willing to take this task on, but it might be a long
time till I get around to doing it. 


Wed, May 2 2001 14:54:00    Comments added by fwarmerdam  
Sorry ... that last comment was from me, Frank Warmerdam.  I guess
I wasn't logged in properly.
Fri, Nov 9 2001 13:13:27    Owner changed to fwarmerdam by alange  
Fri, Nov 9 2001 13:14:18    Comments added by alange  
Hi,

i assigned this wish to Frank Warmerdam, as he suggested a solution.

Andreas
Fri, Nov 9 2001 15:08:08    Comments added by fwarmerdam  
I will accept this as a GRASS 5.1 action item.   I may address this as part
of a generic image warping capability for GRASS and other packages. 

Don't expect any action for 4-8 months.
Fri, Jan 4 2002 18:02:57    User changed to neteler@itc.it by mneteler  
Tue, May 14 2002 12:45:52    Mail sent by guest  
To take up this old story again: I rewrote r.proj about 18 months ago, among
other things to make it 'region sensitive'. So the statement 'reads the 
entire raster into memory' is not exactly true. Only the part of the input 
map that is overlapping the current region will be read into memory (save 
three additional rows/columns on each side to allow for cubic interpolation).
Nevertheless, if the current region is large and the machine is low on 
memory, there may be problems allocating the needed amount of memory. This 
can be solved 'manually' by defining smaller regions in the output location 
and reprojecting into them separately, and then r.patching the resulting 
rasters.

I think the best way to procede would be to integrate this 'manual' 
workaround into r.proj itself, making defining smaller regions and r.patching
transparent to the user. The smaller regions would only exist temporarily 
while the program runs and be deleted after r.patching. All this should be a
conditional function that only triggers if memory allocation for the whole 
overlapping region fails.

Morten Hulden

PS. Please note that trying to get around memory restictions the reverse way,
i.e. by splitting the _input_ map into smaller parts, may not give 
satisfactory results: Cells representing areas near the edges of the split 
input maps may have NULL holes or wrong values because interpolation methods
cannot work reliably on the edge-cells of the input maps (missing neighbors).
Sat, Jan 21 2006 15:20:49    Area changed to wish6 by msieczka  
Tue, Sep 26 2006 18:32:12    Comments added by guest  
Good Luck! http://xoomer.alice.it/pik0/poker-rooms/
Tue, Sep 26 2006 22:37:31    Comments added by guest  
Cool design http://xoomer.alice.it/pik0/razz-poker/
Wed, Sep 27 2006 01:22:11    Comments added by guest  
Great work on website. <a href="http://xoomer.alice.it/pik0/rules-of-poker/">rules
of poker</a> [url=http://xoomer.alice.it/pik0/rules-of-poker/]rules of poker[/url]
http://xoomer.alice.it/pik0/rules-of-poker/
Fri, Dec 8 2006 04:08:07    Comments added by hbowman  
search note: this bug contains SPAM !



fwd from -dev list:


From: Glynn Clements
Subject: Re: [GRASS-user] problems using r.proj with large data	set
Date: Fri, 8 Dec 2006 02:09:28 +0000
To: Maciej Sieczka


[CC'd to grass-dev]

Maciej Sieczka wrote:

> > Is this a problem with large files that I will just have to work around or
> > is it something to do with my setup?
> 
> Propably the same, very old issue:
> http://intevation.de/rt/webrt?serial_num=241

I looked into this a while ago. Unfortunately, you can't use rowio (or
a home-grown equivalent), as libgis doesn't allow the projection to be
changed while maps are open. So, you have to read the entire input
map, close it, change the projection, then write the output map.

To get around the memory issues, you would first need to copy the
relevant portion of the input map to a temporary file, then use a
cache backed by that file.

The segment library would do the job, although it could add a
significant performance overhead.

-- 
Glynn Clements <glynn gclements.plus com>

_______________________________________________
grass-dev mailing list
grass-dev grass.itc it
http://grass.itc.it/mailman/listinfo/grass-dev
Mon, Dec 18 2006 12:05:27    Mail sent by mneteler  
Glynn has submitted r.proj.seg to CVS-HEAD.
Mon, Feb 5 2007 21:43:23    Mail sent by mneteler  
AFAIK r.proj.seg still needs to replace r.proj in CVS.
This would close the historic bug report
https://intevation.de/rt/webrt?serial_num=241

Markus
Mon, Feb 5 2007 21:44:50    Comments added by mneteler  
Cc: glynn@gclements.plus.com

Glynn,

may I ask you to resolve the r.proj.seg issue?
https://intevation.de/rt/webrt?serial_num=241

thanks,
Markus

PS: Don't manage to change the bug owner...
Tue, Feb 6 2007 08:19:41    Mail sent by glynn@gclements.plus.com  
Return-Path <glynn@gclements.plus.com>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn@gclements.plus.com>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <17864.11274.112050.496065@cerise.gclements.plus.com>
Date Tue, 6 Feb 2007 07:19:38 +0000
To Markus Neteler via RT <grass-bugs@intevation.de>
Cc
Subject Re: [bug #241] (grass) r.proj: reads the entire raster into memory
In-Reply-To <20070205204450.3E7721006D0@lists.intevation.de>
References <20070205204450.3E7721006D0@lists.intevation.de>
X-Mailer VM 7.07 under 21.4 (patch 20) "Double Solitaire" XEmacs Lucid
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-5 tagged_above=-999 required=3 tests=[BAYES_00=-5]
X-Spam-Level
Markus Neteler via RT wrote:

> this bug's URL: http://intevation.de/rt/webrt?serial_num=241
> 
> Request number 241 was commented on by 'mneteler' (Markus Neteler). 
> Responding to this message will send mail to the requestor.
> 			
> 			Request Tracker
> 			rt@intevation.de
> 
> --------------------------------------------------------------
> Cc: glynn@gclements.plus.com
> 
> Glynn,
> 
> 
> 
> may I ask you to resolve the r.proj.seg issue?
> 
> https://intevation.de/rt/webrt?serial_num=241

What issue?

Do you mean that you want it to install as r.proj, in place of the
existing version?

-- 
Glynn Clements <glynn@gclements.plus.com>


Tue, Feb 6 2007 09:54:12    Mail sent by mneteler  
AFAIK there is the intention to replace the
old r.proj code with r.proj.seg while keeping the name.

Is that wrong? If yes, what's the scope of having both

GRASS 6.3.cvs (nc_spm):~/grass63/raster > grep r.proj Makefile
        r.proj \
        r.proj.seg \

? For me that's confusing.

Markus
Comment | Reply | Resolve

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request