A New Type Conversion Mechanism for Boost.Python
By David Abrahams.
Introduction
This document describes a redesign of the mechanism for automatically
converting objects between C++ and Python. The current implementation
uses two functions for any type T:
U from_python(PyObject*, type<T>);
void to_python(V);
where U is convertible to T and T is convertible to V. These functions
are at the heart of C++/Python interoperability in Boost.Python, so
why would we want to change them? There are many reasons:
Bugs
Firstly, the current mechanism relies on a common C++ compiler
bug. This is not just embarrassing: as compilers get to be more
conformant, the library stops working. The issue, in detail, is the
use of inline friend functions in templates to generate
conversions. It is a very powerful, and legal technique as long as
it's used correctly:
template <class Derived>
struct add_some_functions
{
friend return-type some_function1(..., Derived cv-*-&-opt, ...);
friend return-type some_function2(..., Derived cv-*-&-opt, ...);
};
template <class T>
struct some_template : add_some_functions<some_template<T> >
{
};
The add_some_functions template generates free functions
which operate on Derived, or on related types. Strictly
speaking the related types are not just cv-qualified Derived
values, pointers and/or references. Section 3.4.2 in the standard
describes exactly which types you must use as parameters to these
functions if you want the functions to be found
(there is also a less-technical description in section 11.5.1 of
C++PL3 [1]). Suffice it to say that
with the current design, the from_python and
to_python functions are not supposed to be callable under any
conditions!
Compilation and Linking Time
The conversion functions generated for each wrapped class using the
above technique are not function templates, but regular functions. The
upshot is that they must all be generated regardless of whether
they are actually used. Generating all of those functions can slow
down module compilation, and resolving the references can slow down
linking.
Efficiency
The conversion functions are primarily used in (member) function
wrappers to convert the arguments and return values. Being functions,
converters have no interface which allows us to ask "will the
conversion succeed?" without calling the function. Since the
return value of the function must be the object to be passed as an
argument, Boost.Python currently uses C++ exception-handling to detect
an unsuccessful conversion. It's not a particularly good use of
exception-handling, since the failure is not handled very far from
where it occurred. More importantly, it means that C++ exceptions are
thrown during overload resolution as we seek an overload that matches
the arguments passed. Depending on the implementation, this approach
can result in significant slowdowns.
It is also unclear that the current library generates a minimal
amount of code for any type conversion. Many of the conversion
functions are nontrivial, and partly because of compiler limitations,
they are declared inline. Also, we could have done a better
job separating the type-specific conversion code from the code which
is type-independent.
Cross-module Support
The current strategy requires every module to contain the definition
of conversions it uses. In general, a new module can never supply
conversion code which is used by another module. Ralf Grosse-Kunstleve
designed a clever system which imports conversions directly from one
library into another using some explicit declarations, but it has some
disadvantages also:
- The system Ullrich Koethe designed for implicit conversion between
wrapped classes related through inheritance does not currently work if
the classes are defined in separate modules.
- The writer of the importing module is required to know the name of
the module supplying the imported conversions.
- There can be only one way to extract any given C++ type from a
Python object in a given module.
The first item might be addressed by moving Boost.Python into a shared
library, but the other two cannot. Ralf turned the limitation in item
two into a feature: the required module is loaded implicitly when a
conversion it defines is invoked. We will probably want to provide
that functionality anyway, but it's not clear that we should require
the declaration of all such conversions. The final item is a more
serious limitation. If, for example, new numeric types are defined in
separate modules, and these types can all be converted to
doubles, we have to choose just one conversion method.
Ease-of-use
One persistent source of confusion for users of Boost.Python has been
the fact that conversions for a class are not be visible at
compile-time until the declaration of that class has been seen. When
the user tries to expose a (member) function operating on or returning
an instance of the class in question, compilation fails...even though
the user goes on to expose the class in the same translation unit!
The new system lifts all compile-time checks for the existence of
particular type conversions and replaces them with runtime checks, in
true Pythonic style. While this might seem cavalier, the compile-time
checks are actually not much use in the current system if many classes
are wrapped in separate modules, since the checks are based only on
the user's declaration that the conversions exist.
The New Design
Motivation
The new design was heavily influenced by a desire to generate as
little code as possible in extension modules. Some of Boost.Python's
clients are enormous projects where link time is proportional to the
amount of object code, and there are many Python extension modules. As
such, we try to keep type-specific conversion code out of modules
other than the one the converters are defined in, and rely as much as
possible on centralized control through a shared library.
The Basics
The library contains a registry which maps runtime type
identifiers (actually an extension of std::type_info which
preserves references and constness) to entries containing type
converters. An entry can contain only one converter from C++ to Python
(wrapper), but many converters from Python to C++
(unwrappers). What should happen if
multiple modules try to register wrappers for the same type?. Wrappers
and unwrappers are known as body objects, and are accessed
by the user and the library (in its function-wrapping code) through
corresponding handle (wrap<T> and
unwrap<T>) objects. The handle objects are
extremely lightweight, and delegate all of their operations to
the corresponding body.
When a handle object is constructed, it accesses the
registry to find a corresponding body that can convert the
handle's constructor argument. Actually the registry record for any
type
Tused in a module is looked up only once and stored in a
static registration<T> object for efficiency. For
example, if the handle is an unwrap<Foo&> object,
the entry for Foo& is looked up in the
registry, and each unwrapper it contains is queried
to determine if it can convert the
PyObject* with which the unwrap was constructed. If
a body object which can perform the conversion is found, a pointer to
it is stored in the handle. A body object may at any point store
additional data in the handle to speed up the conversion process.
Now that the handle has been constructed, the user can ask it whether
the conversion can be performed. All handles can be tested as though
they were convertible to bool; a true value
indicates success. If the user forges ahead and tries to do the
conversion without checking when no conversion is possible, an
exception will be thrown as usual. The conversion itself is performed
by the body object.
Handling complex conversions
Some conversions may require a dynamic allocation. For example,
when a Python tuple is converted to a std::vector<double>
const&, we need some storage into which to construct the
vector so that a reference to it can be formed. Furthermore, multiple
conversions of the same type may need to be "active"
simultaneously, so we can't keep a single copy of the storage
anywhere. We could keep the storage in the body object, and
have the body clone itself in case the storage is used, but in that
case the storage in the body which lives in the registry is never
used. If the storage was actually an object of the target type (the
safest way in C++), we'd have to find a way to construct one for the
body in the registry, since it may not have a default constructor.
The most obvious way out of this quagmire is to allocate the object using a
new-expression, and store a pointer to it in the handle. Since
the body object knows everything about the data it needs to
allocate (if any), it is also given responsibility for destroying that
data. When the handle is destroyed it asks the body
object to tear down any data it may have stored there. In many ways,
you can think of the body as a "dynamically-determined
vtable" for the handle.
Eliminating Redundancy
If you look at the current Boost.Python code, you'll see that there
are an enormous number of conversion functions generated for each
wrapped class. For a given class T, functions are generated
to extract the following types from_python:
T*
T const*
T const* const&
T* const&
T&
T const&
T
std::auto_ptr<T>&
std::auto_ptr<T>
std::auto_ptr<T> const&
boost::shared_ptr<T>&
boost::shared_ptr<T>
boost::shared_ptr<T> const&
Most of these are implemented in terms of just a few conversions, and
if you're lucky, they will be inlined and cause no extra
overhead. In the new system, however, a significant amount of data
will be associated with each type that needs to be converted. We
certainly don't want to register a separate unwrapper object for all
of the above types.
Fortunately, much of the redundancy can be eliminated. For example,
if we generate an unwrapper for T&, we don't need an
unwrapper for T const& or T. Accordingly, the user's
request to wrap/unwrap a given type is translated at compile-time into
a request which helps to eliminate redundancy. The rules used to
unwrap a type are:
- Treat built-in types specially: when unwrapping a value or
constant reference to one of these, use a value for the target
type. It will bind to a const reference if neccessary, and more
importantly, avoids having to dynamically allocate room for
an lvalue of types which can be cheaply copied.
-
Reduce everything else to a reference to an un-cv-qualified type
where possible. Since cv-qualification is lost on Python
anyway, there's no point in trying to convert to a
const&. What about conversions
to values like the tuple->vector example above? It seems to me
that we don't want to make a vector<double>&
(non-const) converter available for that case. We may need to
rethink this slightly.
To handle the problem described above in item 2, we modify the
procedure slightly. To unwrap any non-scalar T, we seek an
unwrapper for add_reference<T>::type. Unwrappers for
T const& always return T&, and are
registered under both T & and
T const&.
For compilers not supporting partial specialization, unwrappers for
T const& must return T const&
(since constness can't be stripped), but a separate unwrapper object
need to be registered for T & and
T const& anyway, for the same reasons.
We may want to make it possible to compile as
though partial specialization were unavailable even on compilers where
it is available, in case modules could be compiled by different
compilers with compatible ABIs (e.g. Intel C++ and MSVC6).
Efficient Argument Conversion
Since type conversions are primarily used in function wrappers, an
optimization is provided for the case where a group of conversions are
used together. Each handle class has a corresponding
"_more" class which does the same job, but has a
trivial destructor. Instead of asking each "_more"
handle to destroy its own body, it is linked into an endogenous list
managed by the first (ordinary) handle. The wrap and
unwrap destructors are responsible for traversing that list
and asking each body class to tear down its
handle. This mechanism is also used to determine if all of
the argument/return-value conversions can succeed with a single
function call in the function wrapping code. We
might need to handle return values in a separate step for Python
callbacks, since the availablility of a conversion won't be known
until the result object is retrieved.
References
[1]B. Stroustrup, The C++ Programming Language
Special Edition Addison-Wesley, ISBN 0-201-70073-5.
Revised
13 November, 2002
© Copyright David Abrahams, 2001