Details Ticket 2599


Comment | Reply | Take | Resolve


Serial Number 2599
Subject r.patch file size limit exceeded
Area bug
Queue grass
Requestors gwurman@asu.edu
Owner none
Status open
Last User Contact Wed Jul 20 00:06:53 2005 (3 yr ago)
Current Priority 30
Final Priority 70
Due No date assigned
Last Action Wed Jul 20 00:06:53 2005 (3 yr ago)
Created Fri Sep 3 01:32:52 2004 (4 yr ago)

Transaction History Ticket 2599


Fri, Sep 3 2004 01:32:52    Request created by guest  
Subject: r.patch file size limit exceeded

Platform: Solaris2.X/Sparc
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: 5.3-cvs

I am attempting to run r.patch to create a LARGE (57330 rows x 47340 cols) raster
map and at row 45622 it craps out and says 

WARNING: map [Orthophoto] - unable to write row 45622
File size limit exceeded

I tried this on a Sparc running solaris and on an Intel Xeon running linux, and
it craps out on the same line, so I don't think it's hardware...

Is this a soft limit that can be reset in code, or is this deeper?

Thanks,
Gilead Wurman
ASU Active Tectonics
Fri, Sep 3 2004 07:09:29    Mail sent by glynn.clements@virgin.net  
Return-Path <glynn.clements@virgin.net>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn.clements@virgin.net>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <16695.62184.683027.678133@cerise.nosuchdomain.co.uk>
Date Fri, 3 Sep 2004 05:28:24 +0100
To Request Tracker <grass-bugs@intevation.de>
Cc grass5@grass.itc.it
Subject Re: [GRASS5] [bug #2599] (grass) r.patch file size limit exceeded
In-Reply-To <20040902233252.1A68210014C@lists.intevation.de>
References <20040902233252.1A68210014C@lists.intevation.de>
X-Mailer VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
X-Spam-Status No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
X-Spam-Level
Request Tracker wrote:

> this bug's URL: http://intevation.de/rt/webrt?serial_num=2599
> -------------------------------------------------------------------------
> 
> Subject: r.patch file size limit exceeded
> 
> Platform: Solaris2.X/Sparc
> grass obtained from: CVS
> grass binary for platform: Compiled from Sources
> GRASS Version: 5.3-cvs
> 
> I am attempting to run r.patch to create a LARGE (57330 rows x 47340
> cols) raster map and at row 45622 it craps out and says
> 
> WARNING: map [Orthophoto] - unable to write row 45622
> File size limit exceeded
> 
> I tried this on a Sparc running solaris and on an Intel Xeon running
> linux, and it craps out on the same line, so I don't think it's
> hardware...
> 
> Is this a soft limit that can be reset in code, or is this deeper?

Raster files are limited by the number of bits in the platform's off_t
type. However, if you are using a version from prior to mid-August,
it's also limited by the number of bits in the platform's int type
(older versions stored offsets using long, but some offset
calculations were performed using int).

On platforms with a 64-bit long type, then the current 5.3 CVS version
should support raster files larger than 2Gb.

On platforms where long is only 32 bits (e.g. x86), you will need to
compile GRASS (or, at least, certain parts of libgis) with
"-D_FILE_OFFSET_BITS=64", so that off_t is defined as "long long int". 
Again, this requires a version more recent than mid-August.

However, ideally only the following files (in src/libes/gis) would be
compiled with that switch:

	opencell.c
	closecell.c
	get_row.c
	put_row.c
	format.c

Those are the files which are involved in reading and writing raster
data; they have all been updated to use off_t for file offsets, and
will work correctly where an off_t is larger than an int or long.

Much of the rest of GRASS uses long for file offsets, and in some
cases performs offset calculations using ints. Such code will
typically fail in unpredictable ways if it tries to handle files
larger than 2Gb.

Developer notes:

Making GRASS reliably handle files larger than 2Gb would involve
analysing all uses of the lseek, fseek and ftell functions, which
amounts to 164 files (56 use lseek, 95 use fseek, and 45 use ftell).

The lseek case is relatively straightforward; offsets must be stored
and calculated as off_t rather than int or long (note: if x and y are
ints then you have to use "(off_t) x * y" and not "(off_t) (x * y)",
as the latter won't promote to off_t until after the value has been
truncated to fit an int).

The use of fseek/ftell is harder, as these are ANSI C (rather than
POSIX) functions, and explicitly use long for offsets. Some platforms
(e.g. GNU libc 2.x) provide fseeko/ftello functions, which use off_t
for offsets instead of long. However, some platforms don't have these
functions, so they can't just be changed unconditionally.

However, a significant number of such files only use fseek to rewind
to the beginning of the file, i.e. fseek(fp, 0, SEEK_SET). Such usage
doesn't need to be changed (although it could be changed to just use
rewind(fp) instead).

-- 
Glynn Clements <glynn.clements@virgin.net>


Fri, Sep 3 2004 21:23:25    Mail sent by guest  
I'm using a 26 August CVS and 64 bit architecture.  As you predict, the file
craps out at just over 2GB.  This actually confuses me a bit, because 2e9 is
either 16 bit or at most 32 bit signed.  Shouldn't 64 bit be of order 10^19?
Might it be possible to use unsigned long to double the capacity?  This would
make the file size limit about 4GB which, at least for the moment, would be
enough for my purposes.

Thanks for your help,
Gilead Wurman
ASU Active Tectonics
Wed, Jul 20 2005 00:06:53    Mail sent by mneteler  
Hi,

can you please try GRASS 6.1-CVS?

Now there is:
  configure --help | grep LFS
    --enable-largefile     enable support for large files (LFS)

We are interested if it works everywhere.

Markus
Comment | Reply | Take | Resolve

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request