Return-Path |
<glynn.clements@virgin.net>
|
Delivered-To |
grass-bugs@lists.intevation.de
|
From |
Glynn Clements <glynn.clements@virgin.net>
|
MIME-Version |
1.0
|
Content-Type |
text/plain; charset=us-ascii
|
Content-Transfer-Encoding |
7bit
|
Message-ID |
<16695.62184.683027.678133@cerise.nosuchdomain.co.uk>
|
Date |
Fri, 3 Sep 2004 05:28:24 +0100
|
To |
Request Tracker <grass-bugs@intevation.de>
|
Cc |
grass5@grass.itc.it
|
Subject |
Re: [GRASS5] [bug #2599] (grass) r.patch file size limit exceeded
|
In-Reply-To |
<20040902233252.1A68210014C@lists.intevation.de>
|
References |
<20040902233252.1A68210014C@lists.intevation.de>
|
X-Mailer |
VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
|
X-Spam-Status |
No, hits=-4.9 tagged_above=-999.0 required=3.0 tests=BAYES_00
|
X-Spam-Level |
|
Request Tracker wrote:
> this bug's URL: http://intevation.de/rt/webrt?serial_num=2599
> -------------------------------------------------------------------------
>
> Subject: r.patch file size limit exceeded
>
> Platform: Solaris2.X/Sparc
> grass obtained from: CVS
> grass binary for platform: Compiled from Sources
> GRASS Version: 5.3-cvs
>
> I am attempting to run r.patch to create a LARGE (57330 rows x 47340
> cols) raster map and at row 45622 it craps out and says
>
> WARNING: map [Orthophoto] - unable to write row 45622
> File size limit exceeded
>
> I tried this on a Sparc running solaris and on an Intel Xeon running
> linux, and it craps out on the same line, so I don't think it's
> hardware...
>
> Is this a soft limit that can be reset in code, or is this deeper?
Raster files are limited by the number of bits in the platform's off_t
type. However, if you are using a version from prior to mid-August,
it's also limited by the number of bits in the platform's int type
(older versions stored offsets using long, but some offset
calculations were performed using int).
On platforms with a 64-bit long type, then the current 5.3 CVS version
should support raster files larger than 2Gb.
On platforms where long is only 32 bits (e.g. x86), you will need to
compile GRASS (or, at least, certain parts of libgis) with
"-D_FILE_OFFSET_BITS=64", so that off_t is defined as "long long int".
Again, this requires a version more recent than mid-August.
However, ideally only the following files (in src/libes/gis) would be
compiled with that switch:
opencell.c
closecell.c
get_row.c
put_row.c
format.c
Those are the files which are involved in reading and writing raster
data; they have all been updated to use off_t for file offsets, and
will work correctly where an off_t is larger than an int or long.
Much of the rest of GRASS uses long for file offsets, and in some
cases performs offset calculations using ints. Such code will
typically fail in unpredictable ways if it tries to handle files
larger than 2Gb.
Developer notes:
Making GRASS reliably handle files larger than 2Gb would involve
analysing all uses of the lseek, fseek and ftell functions, which
amounts to 164 files (56 use lseek, 95 use fseek, and 45 use ftell).
The lseek case is relatively straightforward; offsets must be stored
and calculated as off_t rather than int or long (note: if x and y are
ints then you have to use "(off_t) x * y" and not "(off_t) (x * y)",
as the latter won't promote to off_t until after the value has been
truncated to fit an int).
The use of fseek/ftell is harder, as these are ANSI C (rather than
POSIX) functions, and explicitly use long for offsets. Some platforms
(e.g. GNU libc 2.x) provide fseeko/ftello functions, which use off_t
for offsets instead of long. However, some platforms don't have these
functions, so they can't just be changed unconditionally.
However, a significant number of such files only use fseek to rewind
to the beginning of the file, i.e. fseek(fp, 0, SEEK_SET). Such usage
doesn't need to be changed (although it could be changed to just use
rewind(fp) instead).
--
Glynn Clements <glynn.clements@virgin.net>
|
|
I'm using a 26 August CVS and 64 bit architecture. As you predict, the file
craps out at just over 2GB. This actually confuses me a bit, because 2e9 is
either 16 bit or at most 32 bit signed. Shouldn't 64 bit be of order 10^19?
Might it be possible to use unsigned long to double the capacity? This would
make the file size limit about 4GB which, at least for the moment, would be
enough for my purposes.
Thanks for your help,
Gilead Wurman
ASU Active Tectonics |
|