Details Ticket 1140


Comment | Reply | Resolve


Serial Number 1140
Subject Non-portable snprintf() used in several programs
Area bug
Queue grass
Requestors paul-grass@stjohnspoint.co.uk
Owner bdouglas
Status open
Last User Contact Fri Jul 14 18:42:46 2006 (2 yr ago)
Current Priority 30
Final Priority 70
Due No date assigned
Last Action Fri Jul 14 18:42:46 2006 (2 yr ago)
Created Thu Jun 27 11:42:42 2002 (6 yr ago)

Transaction History Ticket 1140


Thu, Jun 27 2002 11:42:42    Request created by guest  
Subject: Non-portable snprintf() used in several programs

Platform: Irix
grass obtained from: CVS
grass binary for platform: Compiled from Sources
GRASS Version: CVSreleasebranch_26_april_2002_5_0_0 grass checked out 20020618
I have already mentioned this problem on the grass5 mailing list, but was just
reading the descriptions of types of bugs and realised this is a release-critical
one, since it wasn't present in earlier versions and has only appeared in the
last six months or so since I started using GRASS. Therefore I am obliging by
reporting it. There are probably more important things GRASS developers could
be working on fixing but I think attention to detail and overall code quality
and portability is very important.

The following programs contain references to the snprintf() function and therefore
fail to compile on the IRIX 6.2 system I am using.

Compilation error in module: src/libes/dbmi/clients (ignored)
Compilation error in module: src/display/d.db (ignored)
Compilation error in module: src/display/d.what.db (ignored)
Compilation error in module: src/display/d.what.vect (ignored)
Compilation error in module: src/mapdev/v.db.reclass (ignored)
Compilation error in module: src/mapdev/v.in.mif (ignored)
Compilation error in module: src/mapdev/v.in.shape (ignored)
Compilation error in module: src/mapdev/v.out.shape (ignored)
Compilation error in module: src/mapdev/v.to.db (ignored)
Compilation error in module: src/raster/r.in.shape (ignored)
Compilation error in module: src/sites/s.in.dbf (ignored)
Compilation error in module: src.contrib/GMSL/NVIZ2.2 (ignored)

(This list may not be exhaustive; I didn't attempt to compile every module.)
It was also mentioned in the mailing list before that even on systems that do
provide snprintf(), the return value when an error occurs is not standardised,
so that is another reason for not relying on the system library version.

Paul Kelly
Tue, Oct 8 2002 10:07:38    Mail sent by mneteler  
The full list is:

find . -type f -name "*.c" -exec grep -l snprintf {} \;
./libes/dbmi/drivers/odbc/cursor.c
./libes/dbmi/drivers/odbc/db.c
./libes/dbmi/drivers/odbc/describe.c
./libes/dbmi/drivers/odbc/error.c
./libes/dbmi/drivers/odbc/execute.c
./libes/dbmi/drivers/odbc/select.c
./libes/dbmi/drivers/odbc/table.c
./libes/dbmi/lib/dbmscap.c
./libes/dbmi/lib/reclass.c
./libes/gis/extracted-funcs/_make_toplevel.c
./libes/gis/user_config.c
./mapdev/v.in.mif/main.c
./mapdev/v.in.mif/write_lines.c
./mapdev/v.in.shape/lines.c
./mapdev/v.in.shape/vmap_import.c
./mapdev/v.in.shape/test/arctest1.c
./mapdev/v.to.db/update.c
./display/d.db/main.c
./display/d.what.db/attr.c
./display/d.what.db/main.c
./display/d.what.vect/cmd/attr.c
./raster/r.in.shape/main.c
./sites/s.in.dbf/dump.c


Is the problem fixed for you?

 Markus
Tue, Oct 8 2002 16:49:35    Mail sent by paul-grass@stjohnspoint.co.uk  
Return-Path <paul-grass@stjohnspoint.co.uk>
Delivered-To grass-bugs@lists.intevation.de
Date Tue, 8 Oct 2002 15:49:22 +0100 (BST)
From Paul Kelly <paul-grass@stjohnspoint.co.uk>
X-X-Sender paulk@agrippa.ukshells.co.uk
To Markus Neteler via RT <grass-bugs@intevation.de>
Subject Re: [bug #1140] (grass) Transaction (mneteler)
In-Reply-To <20021008080738.0139C13AB4@lists.intevation.de>
Message-ID <Pine.LNX.4.44.0210081511210.13091-100000@agrippa.ukshells.co.uk>
MIME-Version 1.0
Content-Type TEXT/PLAIN; charset=US-ASCII
X-Spam-Status No, hits=-3.8 required=5.0 tests=IN_REP_TO,SPAM_PHRASE_00_01,USER_AGENT_PINE version=2.41
X-Spam-Level
Hello Markus
>
> Is the problem fixed for you?
>
>  Markus
>
Well, I have a workaround for it, which is to compile a 3rd-party snprintf
library (obtained from http://www.ijs.si/software/snprintf/ ) and add it
to the end of the GISLIB line in src/CMD/generic/make.mid

But I don't really regard the bug as cleared. There were some interesting
messages on the mailing list and I thought somebody was going to do
something about it, e.g.
http://grass.itc.it/pipermail/grass5/2002-May/002936.html
http://grass.itc.it/pipermail/grass5/2002-April/002733.html

So it is not urgent but I think you should keep it there with lower
priority and someone will sort it out eventually.

Paul


Sun, May 8 2005 08:12:03    Taken by bdouglas  
Tue, Jul 4 2006 14:38:13    Mail sent by mneteler  
Hi,

the snprintf() is used as of today in:

./raster3d/r3.in.ascii/main.c
./db/drivers/dbf/dbfexe.c
./raster/r.support/front/front.c
./raster/r.support/front/check.c
./raster/r.support/front/run.c
./raster/r.support/modhead/check_un.c
./raster/r.support/modhead/modhead.c
./raster/r.support/modhead/ask_format.c
./lib/init/clean_temp.c
./lib/db/dbmi_client/select.c
./lib/gis/user_config.c
./lib/vector/dglib/examples/components.c

I suggest to implement G_snprintf() and use that
function everywhere. Then we can fix in a single place
if needed.

Markus
Tue, Jul 4 2006 15:09:02    Mail sent by paul-grass@stjohnspoint.co.uk  
Return-Path <paul-grass@stjohnspoint.co.uk>
Delivered-To grass-bugs@lists.intevation.de
Date Tue, 4 Jul 2006 14:08:48 +0100 (BST)
From Paul Kelly <paul-grass@stjohnspoint.co.uk>
X-X-Sender paulk@agrippa.ukshells.co.uk
To Markus Neteler via RT <grass-bugs@intevation.de>
Cc grass-dev@grass.itc.it
Subject Re: [bug #1140] (grass) Non-portable snprintf() used in several programs
In-Reply-To <20060704123813.F3AE41005CA@lists.intevation.de>
Message-ID <Pine.LNX.4.62.0607041401560.25038@agrippa.ukshells.co.uk>
References <20060704123813.F3AE41005CA@lists.intevation.de>
MIME-Version 1.0
Content-Type TEXT/PLAIN; charset=US-ASCII; format=flowed
X-SA-Do-Not-Run Yes
X-SA-Exim-Connect-IP 217.10.143.90
X-SA-Exim-Mail-From paul-grass@stjohnspoint.co.uk
X-SA-Exim-Scanned No (on mail.ukshells.net); SAEximRunCond expanded to false
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-3.741 tagged_above=-999 required=4 tests=[AWL=1.259, BAYES_00=-5]
X-Spam-Level
Hello Markus!
I contend that most of those recent introductions of snprintf() probably 
don't need it. E.g. the first one (raster3d/r3.in.acii/main.c) can 
calculate the length of the string to be malloc'ed using strlen() so can 
guarantee to leave enough space. The second one (db/drivers/dbf/dbfexe.c) 
can use a fixed field width specifier for the %d format string to fix the 
length also - again predictable.

So I will work through those and see if I can change them not to use 
snprintf(). Started it now. I feel if there's something where the length 
of the resulting string really cannot be predicted, then we should use 
G_asprintf() because it's there. It does seem to be a matter of personal 
preference and philosophy though! But I think as Glynn has said before, 
most of the places snprintf() is used the return value is not checked. So 
if an overflow was prevented, and the string was not the expected value, 
the program would just ignore it!

So to summarise - G_snprintf() would be useful to have to easily fix 
the places that snprintf() was erroneously introduced - so you could put 
it in, but I think we should discourage its use in favour of calculating 
how long the string will be and allocating enough memory.

Paul

On Tue, 4 Jul 2006, Markus Neteler via RT wrote:

> Hi,
>
>
>
> the snprintf() is used as of today in:
>
>
>
> ./raster3d/r3.in.ascii/main.c
>
> ./db/drivers/dbf/dbfexe.c
>
> ./raster/r.support/front/front.c
>
> ./raster/r.support/front/check.c
>
> ./raster/r.support/front/run.c
>
> ./raster/r.support/modhead/check_un.c
>
> ./raster/r.support/modhead/modhead.c
>
> ./raster/r.support/modhead/ask_format.c
>
> ./lib/init/clean_temp.c
>
> ./lib/db/dbmi_client/select.c
>
> ./lib/gis/user_config.c
>
> ./lib/vector/dglib/examples/components.c
>
>
>
> I suggest to implement G_snprintf() and use that
>
> function everywhere. Then we can fix in a single place
>
> if needed.
>
>
>
> Markus
>
>
>
> -------------------------------------------- Managed by Request Tracker
>


Tue, Jul 4 2006 15:24:01    Mail sent by neteler@itc.it  
Return-Path <neteler@itc.it>
Delivered-To grass-bugs@lists.intevation.de
Date Tue, 4 Jul 2006 15:23:57 +0200
From Markus Neteler <neteler@itc.it>
To Paul Kelly <paul-grass@stjohnspoint.co.uk>
Cc Markus Neteler via RT <grass-bugs@intevation.de>, grass-dev@grass.itc.it
Subject Re: [GRASS-dev] Re: [bug #1140] (grass) Non-portable snprintf() used in several programs
Message-ID <20060704132357.GE21351@bartok.itc.it>
Mail-Followup-To Paul Kelly <paul-grass@stjohnspoint.co.uk>, Markus Neteler via RT <grass-bugs@intevation.de>, grass-dev@grass.itc.it
References <20060704123813.F3AE41005CA@lists.intevation.de> <Pine.LNX.4.62.0607041401560.25038@agrippa.ukshells.co.uk>
Mime-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Disposition inline
In-Reply-To <Pine.LNX.4.62.0607041401560.25038@agrippa.ukshells.co.uk>
X-PGP-Key http://www.gdf-hannover.de/neteler/markus_gpgkey.asc
X-PGP-Fingerprint D4D5 2F80 120E AD60 E2F6 2297 21B3 D02B E1E7 E789
User-Agent Mutt/1.5.11
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-3.361 tagged_above=-999 required=4 tests=[AWL=1.373, BAYES_00=-5, FORGED_RCVD_HELO=0.266]
X-Spam-Level
Hello Paul,

On Tue, Jul 04, 2006 at 02:08:48PM +0100, Paul Kelly wrote:
> Hello Markus!
> I contend that most of those recent introductions of snprintf() probably 
> don't need it. E.g. the first one (raster3d/r3.in.acii/main.c) can 
> calculate the length of the string to be malloc'ed using strlen() so can 
> guarantee to leave enough space. The second one (db/drivers/dbf/dbfexe.c) 
> can use a fixed field width specifier for the %d format string to fix the 
> length also - again predictable.
> 
> So I will work through those and see if I can change them not to use 
> snprintf(). Started it now. I feel if there's something where the length 
> of the resulting string really cannot be predicted, then we should use 
> G_asprintf() because it's there. It does seem to be a matter of personal 
> preference and philosophy though! But I think as Glynn has said before, 
> most of the places snprintf() is used the return value is not checked. So 
> if an overflow was prevented, and the string was not the expected value, 
> the program would just ignore it!
> 
> So to summarise - G_snprintf() would be useful to have to easily fix 
> the places that snprintf() was erroneously introduced - so you could put 
> it in, but I think we should discourage its use in favour of calculating 
> how long the string will be and allocating enough memory.

thanks for working on it.

I have submitted now the G_snprintf() wrapper function to
lib/gis/snprintf.c
with the discouragement included on top.

You may use it now if snprintf() cannot be avoided.

Markus


Tue, Jul 4 2006 21:50:18    Mail sent by glynn@gclements.plus.com  
Return-Path <glynn@gclements.plus.com>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn@gclements.plus.com>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <17578.50807.818514.870528@cerise.gclements.plus.com>
Date Tue, 4 Jul 2006 20:50:15 +0100
To Paul Kelly <paul-grass@stjohnspoint.co.uk>
Cc Markus Neteler via RT <grass-bugs@intevation.de>, grass-dev@grass.itc.it
Subject Re: [GRASS-dev] Re: [bug #1140] (grass) Non-portable snprintf() used in several programs
In-Reply-To <Pine.LNX.4.62.0607041401560.25038@agrippa.ukshells.co.uk>
References <20060704123813.F3AE41005CA@lists.intevation.de> <Pine.LNX.4.62.0607041401560.25038@agrippa.ukshells.co.uk>
X-Mailer VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-3.261 tagged_above=-999 required=4 tests=[AWL=1.473, BAYES_00=-5, FORGED_RCVD_HELO=0.266]
X-Spam-Level
Paul Kelly wrote:

> I contend that most of those recent introductions of snprintf() probably 
> don't need it. E.g. the first one (raster3d/r3.in.acii/main.c) can 
> calculate the length of the string to be malloc'ed using strlen() so can 
> guarantee to leave enough space. The second one (db/drivers/dbf/dbfexe.c) 
> can use a fixed field width specifier for the %d format string to fix the 
> length also - again predictable.

Field width specifiers only control the minimum width of a field (i.e. 
the amount of padding or the number of leading zeroes); a field can
still be larger than the specified width.

For %d, it's reasonable to assume a 32-bit integer, which means that
it can never be more than 11 characters wide (10 digits, with a
leading minus if negative).

-- 
Glynn Clements <glynn@gclements.plus.com>


Tue, Jul 4 2006 22:07:27    Mail sent by paul-grass@stjohnspoint.co.uk  
Return-Path <paul-grass@stjohnspoint.co.uk>
Delivered-To grass-bugs@lists.intevation.de
Message-ID <44AACA6D.8030200@stjohnspoint.co.uk>
Date Tue, 04 Jul 2006 21:07:09 +0100
From Paul Kelly <paul-grass@stjohnspoint.co.uk>
User-Agent Mozilla Thunderbird 1.0.6 (Windows/20050716)
X-Accept-Language en-us, en
MIME-Version 1.0
To Glynn Clements <glynn@gclements.plus.com>
Cc Markus Neteler via RT <grass-bugs@intevation.de>, grass-dev@grass.itc.it
Subject Re: [GRASS-dev] Re: [bug #1140] (grass) Non-portable snprintf() used in several programs
References <20060704123813.F3AE41005CA@lists.intevation.de> <Pine.LNX.4.62.0607041401560.25038@agrippa.ukshells.co.uk> <17578.50807.818514.870528@cerise.gclements.plus.com>
In-Reply-To <17578.50807.818514.870528@cerise.gclements.plus.com>
Content-Type text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding 7bit
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-3.333 tagged_above=-999 required=4 tests=[AWL=1.667, BAYES_00=-5]
X-Spam-Level
Glynn Clements wrote:
> Paul Kelly wrote:
> 
> 
>>I contend that most of those recent introductions of snprintf() probably 
>>don't need it. E.g. the first one (raster3d/r3.in.acii/main.c) can 
>>calculate the length of the string to be malloc'ed using strlen() so can 
>>guarantee to leave enough space. The second one (db/drivers/dbf/dbfexe.c) 
>>can use a fixed field width specifier for the %d format string to fix the 
>>length also - again predictable.
> 
> 
> Field width specifiers only control the minimum width of a field (i.e. 
> the amount of padding or the number of leading zeroes); a field can
> still be larger than the specified width.

Ah yes I realised that when I actually went to implement my proposed change.
> For %d, it's reasonable to assume a 32-bit integer, which means that
> it can never be more than 11 characters wide (10 digits, with a
> leading minus if negative).

Well in this case the existing code was allocating 32 bytes of space and 
then using snprintf so I think it should be safe to change the snprintf 
a simple sprintf in this case (db/drivers/dbf/dbfexe.c).


Thu, Jul 6 2006 05:40:49    Mail sent by glynn@gclements.plus.com  
Return-Path <glynn@gclements.plus.com>
Delivered-To grass-bugs@lists.intevation.de
From Glynn Clements <glynn@gclements.plus.com>
MIME-Version 1.0
Content-Type text/plain; charset=us-ascii
Content-Transfer-Encoding 7bit
Message-ID <17580.34325.326800.333470@cerise.gclements.plus.com>
Date Thu, 6 Jul 2006 04:40:05 +0100
To Paul Kelly <paul-grass@stjohnspoint.co.uk>
Cc Markus Neteler via RT <grass-bugs@intevation.de>, grass-dev@grass.itc.it
Subject Re: [GRASS-dev] Re: [bug #1140] (grass) Non-portable snprintf() used in several programs
In-Reply-To <Pine.LNX.4.62.0607051206030.25038@agrippa.ukshells.co.uk>
References <20060704123813.F3AE41005CA@lists.intevation.de> <Pine.LNX.4.62.0607051206030.25038@agrippa.ukshells.co.uk>
X-Mailer VM 7.07 under 21.4 (patch 15) "Security Through Obscurity" XEmacs Lucid
X-Virus-Scanned by amavisd-new at intevation.de
X-Spam-Status No, hits=-3.307 tagged_above=-999 required=4 tests=[AWL=1.427, BAYES_00=-5, FORGED_RCVD_HELO=0.266]
X-Spam-Level
Paul Kelly wrote:

> > the snprintf() is used as of today in:
> >
> > ./raster3d/r3.in.ascii/main.c
> > ./db/drivers/dbf/dbfexe.c
> > ./raster/r.support/front/front.c
> > ./raster/r.support/front/check.c
> > ./raster/r.support/front/run.c
> > ./raster/r.support/modhead/check_un.c
> > ./raster/r.support/modhead/modhead.c
> > ./raster/r.support/modhead/ask_format.c
> > ./lib/init/clean_temp.c
> > ./lib/db/dbmi_client/select.c
> > ./lib/gis/user_config.c
> > ./lib/vector/dglib/examples/components.c
> 
> Please see attached patch to replace use of snprintf in all those 
> modules. I am posting it to the list first, hopefully for some comments, 
> before committing.

In cases where the strings being inserted are themselves supposed to
have a maximum length (e.g. map names), I would just check that the
strings don't exceed their maximum length and use a fixed-size buffer.

> Index: lib/init/clean_temp.c
> ===================================================================
> RCS file: /grassrepository/grass6/lib/init/clean_temp.c,v
> retrieving revision 2.5
> diff -u -r2.5 clean_temp.c
> --- lib/init/clean_temp.c	24 Jun 2006 19:33:03 -0000	2.5
> +++ lib/init/clean_temp.c	5 Jul 2006 11:02:49 -0000
> @@ -19,7 +19,6 @@
>   *
>   *   2006: Rewritten for GRASS 6 by roberto Flor, ITC-irst
>   *
> - * TODO (?): Implement snprintf() as G_snprintf() for portability
>   **************************************************************/
>  
>  #include <limits.h>
> @@ -39,7 +38,6 @@
>  
>  void clean_dir(const char *pathname,uid_t uid,pid_t pid,time_t now,int max_age)
>  {
> -    char buf[BUF_MAX];
>      DIR *curdir;
>      struct dirent *cur_entry;
>      struct stat info;
> @@ -53,11 +51,16 @@
>      /* loop over current dir */
>      while ((cur_entry = readdir(curdir)))
>      {
> -	if ((G_strcasecmp(cur_entry->d_name,".") == 0 )|| (G_strcasecmp(cur_entry->d_name,"..")==0))
> +	static char *buf = NULL;
> +       
> +	if ((G_strcasecmp(cur_entry->d_name,".") == 0 )|| (G_strcasecmp(cur_entry->d_name,"..")==0))
>  		continue; /* Skip dir and parent dir entries */
> -		
> -	if ( (pathlen=snprintf(buf,BUF_MAX,"%s/%s",pathname,cur_entry->d_name)) >=
BUF_MAX) 
> -		G_fatal_error("clean_temp: exceeded maximum pathname length %d, got %d,
should'nt happen",BUF_MAX,pathlen);
> +	
> +	if(buf)
> +		G_free(buf);
> +       
> +	buf = G_malloc(strlen(pathname) + strlen(cur_entry->d_name) + 2);
> +	sprintf(buf, "%s/%s", pathname, cur_entry->d_name);

In this situation, I would start with a reasonably-sized buffer (e.g. 
4Kb), then use G_realloc() to enlarge it if necessary.

Both malloc() and free() can be quite computationally expensive, and
repeated use can cause heap fragmentation, particularly if the size
tends to increase with time (as the previously free()d block is too
small to be used by the subsequent malloc()).

The same applies to any kind of loop: re-using a buffer is more
efficient (in terms of both CPU and memory usage) than allocating a
new buffer every time.

-- 
Glynn Clements <glynn@gclements.plus.com>


Fri, Jul 14 2006 18:42:46    Mail sent by mneteler  
Hi,

I have changed

./raster3d/r3.in.ascii/main.c
./raster/r.support/front/front.c
./raster/r.support/front/check.c
./raster/r.support/front/run.c
./raster/r.support/modhead/check_un.c
./raster/r.support/modhead/modhead.c
./raster/r.support/modhead/ask_format.c
./lib/init/clean_temp.c
./lib/db/dbmi_client/select.c

from snprintf() to G_snprintf() to address portability problems
in a single place.

Markus
Comment | Reply | Resolve

You are currently authenticated as guest.
[Show Configuration] [Login as another user]

Users Guide - Mail Commands - Homepage of RequestTracker 1.0.7 - list any request