Sarray Object Does Not Support Item Assignment Abroad

On By In 1

is a foreign function library for Python. It provides C compatible data types, and allows calling functions in DLLs or shared libraries. It can be used to wrap these libraries in pure Python.

16.16.1. ctypes tutorial¶

Note: The code samples in this tutorial use to make sure that they actually work. Since some code samples behave differently under Linux, Windows, or Mac OS X, they contain doctest directives in comments.

Note: Some code samples reference the ctypes type. On platforms where it is an alias to . So, you should not be confused if is printed if you would expect — they are actually the same type. Loading dynamic link libraries¶

exports the cdll, and on Windows windll and oledll objects, for loading dynamic link libraries.

You load libraries by accessing them as attributes of these objects. cdll loads libraries which export functions using the standard calling convention, while windll libraries call functions using the calling convention. oledll also uses the calling convention, and assumes the functions return a Windows error code. The error code is used to automatically raise an exception when the function call fails.

Changed in version 3.3: Windows errors used to raise , which is now an alias of .

Here are some examples for Windows. Note that is the MS standard C library containing most standard C functions, and uses the cdecl calling convention:

Windows appends the usual file suffix automatically.


Accessing the standard C library through will use an outdated version of the library that may be incompatible with the one being used by Python. Where possible, use native Python functionality, or else import and use the module.

On Linux, it is required to specify the filename including the extension to load a library, so attribute access can not be used to load libraries. Either the method of the dll loaders should be used, or you should load the library by creating an instance of CDLL by calling the constructor:

>>> fromctypesimport*>>> print(windll.kernel32)<WinDLL 'kernel32', handle ... at ...>>>> print(cdll.msvcrt)<CDLL 'msvcrt', handle ... at ...>>>> libc=cdll.msvcrt>>>
>>> cdll.LoadLibrary("")<CDLL '', handle ... at ...>>>> libc=CDLL("")>>> libc<CDLL '', handle ... at ...>>>> Accessing functions from loaded dlls¶

Functions are accessed as attributes of dll objects:

Note that win32 system dlls like and often export ANSI as well as UNICODE versions of a function. The UNICODE version is exported with an appended to the name, while the ANSI version is exported with an appended to the name. The win32 function, which returns a module handle for a given module name, has the following C prototype, and a macro is used to expose one of them as depending on whether UNICODE is defined or not:

windll does not try to select one of them by magic, you must access the version you need by specifying or explicitly, and then call it with bytes or string objects respectively.

Sometimes, dlls export functions with names which aren’t valid Python identifiers, like . In this case you have to use to retrieve the function:

On Windows, some dlls export functions not by name but by ordinal. These functions can be accessed by indexing the dll object with the ordinal number:

>>> fromctypesimport*>>> libc.printf<_FuncPtr object at 0x...>>>> print(windll.kernel32.GetModuleHandleA)<_FuncPtr object at 0x...>>>> print(windll.kernel32.MyOwnFunction)Traceback (most recent call last): File "<stdin>", line 1, in <module> File "", line 239, in __getattr__func=_StdcallFuncPtr(name,self)AttributeError: function 'MyOwnFunction' not found>>>
>>> getattr(cdll.msvcrt,"??2@YAPAXI@Z")<_FuncPtr object at 0x...>>>>
>>> cdll.kernel32[1]<_FuncPtr object at 0x...>>>> cdll.kernel32[0]Traceback (most recent call last): File "<stdin>", line 1, in <module> File "", line 310, in __getitem__func=_StdcallFuncPtr(name,self)AttributeError: function ordinal 0 not found>>> Calling functions¶

You can call these functions like any other Python callable. This example uses the function, which returns system time in seconds since the Unix epoch, and the function, which returns a win32 module handle.

This example calls both functions with a NULL pointer ( should be used as the NULL pointer):


may raise a after calling the function, if it detects that an invalid number of arguments were passed. This behavior should not be relied upon. It is deprecated in 3.6.2, and will be removed in 3.7.

is raised when you call an function with the calling convention, or vice versa:

To find out the correct calling convention you have to look into the C header file or the documentation for the function you want to call.

On Windows, uses win32 structured exception handling to prevent crashes from general protection faults when functions are called with invalid argument values:

There are, however, enough ways to crash Python with , so you should be careful anyway. The module can be helpful in debugging crashes (e.g. from segmentation faults produced by erroneous C library calls).

, integers, bytes objects and (unicode) strings are the only native Python objects that can directly be used as parameters in these function calls. is passed as a C pointer, bytes objects and strings are passed as pointer to the memory block that contains their data ( or ). Python integers are passed as the platforms default C type, their value is masked to fit into the C type.

Before we move on calling functions with other parameter types, we have to learn more about data types.

>>> print(libc.time(None))1150640792>>> print(hex(windll.kernel32.GetModuleHandleA(None)))0x1d000000>>>
>>> cdll.kernel32.GetModuleHandleA(None)Traceback (most recent call last): File "<stdin>", line 1, in <module>ValueError: Procedure probably called with not enough arguments (4 bytes missing)>>>>>> windll.msvcrt.printf(b"spam")Traceback (most recent call last): File "<stdin>", line 1, in <module>ValueError: Procedure probably called with too many arguments (4 bytes in excess)>>>
>>> windll.kernel32.GetModuleHandleA(32)Traceback (most recent call last): File "<stdin>", line 1, in <module>OSError: exception: access violation reading 0x00000020>>> Fundamental data types¶

defines a number of primitive C compatible data types:

  1. The constructor accepts any object with a truth value.

All these types can be created by calling them with an optional initializer of the correct type and value:

Since these types are mutable, their value can also be changed afterwards:

Assigning a new value to instances of the pointer types , , and changes the memory location they point to, not the contents of the memory block (of course not, because Python bytes objects are immutable):

You should be careful, however, not to pass them to functions expecting pointers to mutable memory. If you need mutable memory blocks, ctypes has a function which creates these in various ways. The current memory block contents can be accessed (or changed) with the property; if you want to access it as NUL terminated string, use the property:

The function replaces the function (which is still available as an alias), as well as the function from earlier ctypes releases. To create a mutable memory block containing unicode characters of the C type use the function.

>>> c_int()c_long(0)>>> c_wchar_p("Hello, World")c_wchar_p(140018365411392)>>> c_ushort(-3)c_ushort(65533)>>>
>>> i=c_int(42)>>> print(i)c_long(42)>>> print(i.value)42>>> i.value=-99>>> print(i.value)-99>>>
>>> s="Hello, World">>> c_s=c_wchar_p(s)>>> print(c_s)c_wchar_p(139966785747344)>>> print(c_s.value)Hello World>>> c_s.value="Hi, there">>> print(c_s)# the memory location has changedc_wchar_p(139966783348904)>>> print(c_s.value)Hi, there>>> print(s)# first object is unchangedHello, World>>>
>>> fromctypesimport*>>> p=create_string_buffer(3)# create a 3 byte buffer, initialized to NUL bytes>>> print(sizeof(p),repr(p.raw))3 b'\x00\x00\x00'>>> p=create_string_buffer(b"Hello")# create a buffer containing a NUL terminated string>>> print(sizeof(p),repr(p.raw))6 b'Hello\x00'>>> print(repr(p.value))b'Hello'>>> p=create_string_buffer(b"Hello",10)# create a 10 byte buffer>>> print(sizeof(p),repr(p.raw))10 b'Hello\x00\x00\x00\x00\x00'>>> p.value=b"Hi">>> print(sizeof(p),repr(p.raw))10 b'Hi\x00lo\x00\x00\x00\x00\x00'>>> Calling functions, continued¶

Note that printf prints to the real standard output channel, not to , so these examples will only work at the console prompt, not from within IDLE or PythonWin:

As has been mentioned before, all Python types except integers, strings, and bytes objects have to be wrapped in their corresponding type, so that they can be converted to the required C data type:

>>> printf=libc.printf>>> printf(b"Hello, %s\n",b"World!")Hello, World!14>>> printf(b"Hello, %S\n","World!")Hello, World!14>>> printf(b"%d bottles of beer\n",42)42 bottles of beer19>>> printf(b"%f bottles of beer\n",42.5)Traceback (most recent call last): File "<stdin>", line 1, in <module>ArgumentError: argument 2: exceptions.TypeError: Don't know how to convert parameter 2>>>
>>> printf(b"An int %d, a double %f\n",1234,c_double(3.14))An int 1234, a double 3.14000031>>> Calling functions with your own custom data types¶

You can also customize argument conversion to allow instances of your own classes be used as function arguments. looks for an attribute and uses this as the function argument. Of course, it must be one of integer, string, or bytes:

If you don’t want to store the instance’s data in the instance variable, you could define a which makes the attribute available on request.

>>> classBottles:... def__init__(self,number):... self._as_parameter_=number...>>> bottles=Bottles(42)>>> printf(b"%d bottles of beer\n",bottles)42 bottles of beer19>>> Specifying the required argument types (function prototypes)¶

It is possible to specify the required argument types of functions exported from DLLs by setting the attribute.

must be a sequence of C data types (the function is probably not a good example here, because it takes a variable number and different types of parameters depending on the format string, on the other hand this is quite handy to experiment with this feature):

Specifying a format protects against incompatible argument types (just as a prototype for a C function), and tries to convert the arguments to valid types:

If you have defined your own classes which you pass to function calls, you have to implement a class method for them to be able to use them in the sequence. The class method receives the Python object passed to the function call, it should do a typecheck or whatever is needed to make sure this object is acceptable, and then return the object itself, its attribute, or whatever you want to pass as the C function argument in this case. Again, the result should be an integer, string, bytes, a instance, or an object with an attribute.

>>> printf.argtypes=[c_char_p,c_char_p,c_int,c_double]>>> printf(b"String '%s', Int %d, Double %f\n",b"Hi",10,2.2)String 'Hi', Int 10, Double 2.20000037>>>
>>> printf(b"%d%d%d",1,2,3)Traceback (most recent call last): File "<stdin>", line 1, in <module>ArgumentError: argument 2: exceptions.TypeError: wrong type>>> printf(b"%s%d%f\n",b"X",2,3)X 2 3.00000013>>> Return types¶

By default functions are assumed to return the C type. Other return types can be specified by setting the attribute of the function object.

Here is a more advanced example, it uses the function, which expects a string pointer and a char, and returns a pointer to a string:

If you want to avoid the calls above, you can set the attribute, and the second argument will be converted from a single character Python bytes object into a C char:

You can also use a callable Python object (a function or a class for example) as the attribute, if the foreign function returns an integer. The callable will be called with the integer the C function returns, and the result of this call will be used as the result of your function call. This is useful to check for error return values and automatically raise an exception:

is a function which will call Windows api to get the string representation of an error code, and returns an exception. takes an optional error code parameter, if no one is used, it calls to retrieve it.

Please note that a much more powerful error checking mechanism is available through the attribute; see the reference manual for details.

>>> strchr=libc.strchr>>> strchr(b"abcdef",ord("d"))8059983>>> strchr.restype=c_char_p# c_char_p is a pointer to a string>>> strchr(b"abcdef",ord("d"))b'def'>>> print(strchr(b"abcdef",ord("x")))None>>>
>>> strchr.restype=c_char_p>>> strchr.argtypes=[c_char_p,c_char]>>> strchr(b"abcdef",b"d")'def'>>> strchr(b"abcdef",b"def")Traceback (most recent call last): File "<stdin>", line 1, in <module>ArgumentError: argument 2: exceptions.TypeError: one character string expected>>> print(strchr(b"abcdef",b"x"))None>>> strchr(b"abcdef",b"d")'def'>>>
>>> GetModuleHandle=windll.kernel32.GetModuleHandleA>>> defValidHandle(value):... ifvalue==0:... raiseWinError()... returnvalue...>>>>>> GetModuleHandle.restype=ValidHandle>>> GetModuleHandle(None)486539264>>> GetModuleHandle("something silly")Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in ValidHandleOSError: [Errno 126] The specified module could not be found.>>> Passing pointers (or: passing parameters by reference)¶

Sometimes a C api function expects a pointer to a data type as parameter, probably to write into the corresponding location, or if the data is too large to be passed by value. This is also known as passing parameters by reference.

exports the function which is used to pass parameters by reference. The same effect can be achieved with the function, although does a lot more work since it constructs a real pointer object, so it is faster to use if you don’t need the pointer object in Python itself:

>>> i=c_int()>>> f=c_float()>>> s=create_string_buffer(b'\000'*32)>>> print(i.value,f.value,repr(s.value))0 0.0 b''>>> libc.sscanf(b"1 3.14 Hello",b"%d%f%s",... byref(i),byref(f),s)3>>> print(i.value,f.value,repr(s.value))1 3.1400001049 b'Hello'>>> Structures and unions¶

Structures and unions must derive from the and base classes which are defined in the module. Each subclass must define a attribute. must be a list of 2-tuples, containing a field name and a field type.

The field type must be a type like , or any other derived type: structure, union, array, pointer.

Here is a simple example of a POINT structure, which contains two integers named x and y, and also shows how to initialize a structure in the constructor:

You can, however, build much more complicated structures. A structure can itself contain other structures by using a structure as a field type.

Here is a RECT structure which contains two POINTs named upperleft and lowerright:

Nested structures can also be initialized in the constructor in several ways:

Field descriptors can be retrieved from the class, they are useful for debugging because they can provide useful information:


does not support passing unions or structures with bit-fields to functions by value. While this may work on 32-bit x86, it’s not guaranteed by the library to work in the general case. Unions and structures with bit-fields should always be passed to functions by pointer.

>>> fromctypesimport*>>> classPOINT(Structure):... _fields_=[("x",c_int),... ("y",c_int)]...>>> point=POINT(10,20)>>> print(point.x,point.y)10 20>>> point=POINT(y=5)>>> print(point.x,point.y)0 5>>> POINT(1,2,3)Traceback (most recent call last): File "<stdin>", line 1, in <module>ValueError: too many initializers>>>
>>> classRECT(Structure):... _fields_=[("upperleft",POINT),... ("lowerright",POINT)]...>>> rc=RECT(point)>>> print(rc.upperleft.x,rc.upperleft.y)0 5>>> print(rc.lowerright.x,rc.lowerright.y)0 0>>>
>>> r=RECT(POINT(1,2),POINT(3,4))>>> r=RECT((1,2),(3,4))
>>> print(POINT.x)<Field type=c_long, ofs=0, size=4>>>> print(POINT.y)<Field type=c_long, ofs=4, size=4>>>> Structure/union alignment and byte order¶

By default, Structure and Union fields are aligned in the same way the C compiler does it. It is possible to override this behavior be specifying a class attribute in the subclass definition. This must be set to a positive integer and specifies the maximum alignment for the fields. This is what also does in MSVC.

uses the native byte order for Structures and Unions. To build structures with non-native byte order, you can use one of the , , , and base classes. These classes cannot contain pointer fields. Bit fields in structures and unions¶

It is possible to create structures and unions containing bit fields. Bit fields are only possible for integer fields, the bit width is specified as the third item in the tuples:

>>> classInt(Structure):... _fields_=[("first_16",c_int,16),... ("second_16",c_int,16)]...>>> print(Int.first_16)<Field type=c_long, ofs=0:0, bits=16>>>> print(Int.second_16)<Field type=c_long, ofs=0:16, bits=16>>>> Arrays¶

Arrays are sequences, containing a fixed number of instances of the same type.

The recommended way to create array types is by multiplying a data type with a positive integer:

Here is an example of a somewhat artificial data type, a structure containing 4 POINTs among other stuff:

Instances are created in the usual way, by calling the class:

The above code print a series of lines, because the array contents is initialized to zeros.

Initializers of the correct type can also be specified:

>>> fromctypesimport*>>> classPOINT(Structure):... _fields_=("x",c_int),("y",c_int)...>>> classMyStruct(Structure):... _fields_=[("a",c_int),... ("b",c_float),... ("point_array",POINT*4)]>>>>>> print(len(MyStruct().point_array))4>>>
>>> fromctypesimport*>>> TenIntegers=c_int*10>>> ii=TenIntegers(1,2,3,4,5,6,7,8,9,10)>>> print(ii)<c_long_Array_10 object at 0x...>>>> foriinii:print(i,end=" ")...1 2 3 4 5 6 7 8 9 10>>> Pointers¶

Pointer instances are created by calling the function on a type:

Pointer instances have a attribute which returns the object to which the pointer points, the object above:

Note that does not have OOR (original object return), it constructs a new, equivalent object each time you retrieve an attribute:

Assigning another instance to the pointer’s contents attribute would cause the pointer to point to the memory location where this is stored:

Pointer instances can also be indexed with integers:

Assigning to an integer index changes the pointed to value:

It is also possible to use indexes different from 0, but you must know what you’re doing, just as in C: You can access or change arbitrary memory locations. Generally you only use this feature if you receive a pointer from a C function, and you know that the pointer actually points to an array instead of a single item.

Behind the scenes, the function does more than simply create pointer instances, it has to create pointer types first. This is done with the function, which accepts any type, and returns a new type:

Calling the pointer type without an argument creates a pointer. pointers have a boolean value:

checks for when dereferencing pointers (but dereferencing invalid non- pointers would crash Python):

>>> fromctypesimport*>>> i=c_int(42)>>> pi=pointer(i)>>>
>>> pi.contentsc_long(42)>>>
>>> pi.contentsisiFalse>>> pi.contentsispi.contentsFalse>>>
>>> i=c_int(99)>>> pi.contents=i>>> pi.contentsc_long(99)>>>
>>> pi[0]99>>>
>>> print(i)c_long(99)>>> pi[0]=22>>> print(i)c_long(22)>>>
>>> PI=POINTER(c_int)>>> PI<class 'ctypes.LP_c_long'>>>> PI(42)Traceback (most recent call last): File "<stdin>", line 1, in <module>TypeError: expected c_long instead of int>>> PI(c_int(42))<ctypes.LP_c_long object at 0x...>>>>
>>> null_ptr=POINTER(c_int)()>>> print(bool(null_ptr))False>>>
>>> null_ptr[0]Traceback (most recent call last):....ValueError: NULL pointer access>>>>>> null_ptr[0]=1234Traceback (most recent call last):....ValueError: NULL pointer access>>> Type conversions¶

Usually, ctypes does strict type checking. This means, if you have in the list of a function or as the type of a member field in a structure definition, only instances of exactly the same type are accepted. There are some exceptions to this rule, where ctypes accepts other objects. For example, you can pass compatible array instances instead of pointer types. So, for , ctypes accepts an array of c_int:

In addition, if a function argument is explicitly declared to be a pointer type (such as ) in , an object of the pointed type ( in this case) can be passed to the function. ctypes will apply the required conversion in this case automatically.

To set a POINTER type field to , you can assign :

Sometimes you have instances of incompatible types. In C, you can cast one type into another type. provides a function which can be used in the same way. The structure defined above accepts pointers or arrays for its field, but not instances of other types:

For these cases, the function is handy.

The function can be used to cast a ctypes instance into a pointer to a different ctypes data type. takes two parameters, a ctypes object that is or can be converted to a pointer of some kind, and a ctypes pointer type. It returns an instance of the second argument, which references the same memory block as the first argument:

So, can be used to assign to the field of the structure:

>>> classBar(Structure):... _fields_=[("count",c_int),("values",POINTER(c_int))]...>>> bar=Bar()>>> bar.values=(c_int*3)(1,2,3)>>> bar.count=3>>> foriinrange(bar.count):... print(bar.values[i])...123>>>
>>> bar.values=None>>>
>>> bar.values=(c_byte*4)()Traceback (most recent call last): File "<stdin>", line 1, in <module>TypeError: incompatible types, c_byte_Array_4 instance instead of LP_c_long instance>>>
>>> a=(c_byte*4)()>>> cast(a,POINTER(c_int))<ctypes.LP_c_long object at ...>>>>
>>> bar=Bar()>>> bar.values=cast((c_byte*4)(),POINTER(c_int))>>> print(bar.values[0])0>>> Incomplete Types¶

Incomplete Types are structures, unions or arrays whose members are not yet specified. In C, they are specified by forward declarations, which are defined later:

The straightforward translation into ctypes code would be this, but it does not work:

>>> classcell(Structure):... _fields_=

The function defines a linkage between two classes. When the linkage defines a one-to-many or many-to-many relationship, it’s represented as a Python collection when objects are loaded and manipulated. This section presents additional information about collection configuration and techniques.

Working with Large Collections¶

The default behavior of is to fully load the collection of items in, as according to the loading strategy of the relationship. Additionally, the by default only knows how to delete objects which are actually present within the session. When a parent instance is marked for deletion and flushed, the loads its full list of child items in so that they may either be deleted as well, or have their foreign key value set to null; this is to avoid constraint violations. For large collections of child items, there are several strategies to bypass full loading of child items both at load time as well as deletion time.

Dynamic Relationship Loaders¶

A key feature to enable management of a large collection is the so-called “dynamic” relationship. This is an optional form of which returns a object in place of a collection when accessed. criterion may be applied as well as limits and offsets, either explicitly or via array slices:

The dynamic relationship supports limited write operations, via the and methods:

Since the read side of the dynamic relationship always queries the database, changes to the underlying collection will not be visible until the data has been flushed. However, as long as “autoflush” is enabled on the in use, this will occur automatically each time the collection is about to emit a query.

To place a dynamic relationship on a backref, use the function in conjunction with :

Note that eager/lazy loading options cannot be used in conjunction dynamic relationships at this time.


The “dynamic” loader applies to collections only. It is not valid to use “dynamic” loaders with many-to-one, one-to-one, or uselist=False relationships. Newer versions of SQLAlchemy emit warnings or exceptions in these cases.

classUser(Base):__tablename__='user'posts=relationship(Post,lazy="dynamic")jack=session.query(User).get(id)# filter Jack's blog postsposts=jack.posts.filter(Post.headline=='this is a post')# apply array slicesposts=jack.posts[5:20]
oldpost=jack.posts.filter(Post.headline=='old post').one()jack.posts.remove(oldpost)jack.posts.append(Post('new post'))

Setting Noload, RaiseLoad¶

A “noload” relationship never loads from the database, even when accessed. It is configured using :

Above, the collection is fully writeable, and changes to it will be persisted to the database as well as locally available for reading at the time they are added. However when instances of are freshly loaded from the database, the collection stays empty. The noload strategy is also available on a query option basis using the loader option.

Alternatively, a “raise”-loaded relationship will raise an where the attribute would normally emit a lazy load:

Above, attribute access on the collection will raise an exception if it was not previously eagerloaded. This includes read access but for collections will also affect write access, as collections can’t be mutated without first loading them. The rationale for this is to ensure that an application is not emitting any unexpected lazy loads within a certain context. Rather than having to read through SQL logs to determine that all necessary attributes were eager loaded, the “raise” strategy will cause unloaded attributes to raise immediately if accessed. The raise strategy is also available on a query option basis using the loader option.

New in version 1.1: added the “raise” loader strategy.


Using Passive Deletes¶

Use to disable child object loading on a DELETE operation, in conjunction with “ON DELETE (CASCADE|SET NULL)” on your database to automatically cascade deletes to child objects:


To use “ON DELETE CASCADE”, the underlying database engine must support foreign keys.

When is applied, the relationship will not be loaded into memory when an instance of is marked for deletion. The will take effect for instances of which are currently present in the session; however for instances of which are not loaded, SQLAlchemy assumes that “ON DELETE CASCADE” rules will ensure that those rows are deleted by the database.

classMyClass(Base):__tablename__='mytable'id=Column(Integer,primary_key=True)children=relationship("MyOtherClass",cascade="all, delete-orphan",passive_deletes=True)classMyOtherClass(Base):__tablename__='myothertable'id=Column(Integer,primary_key=True)parent_id=Column(Integer,ForeignKey('',ondelete='CASCADE'))

Customizing Collection Access¶

Mapping a one-to-many or many-to-many relationship results in a collection of values accessible through an attribute on the parent instance. By default, this collection is a :

Collections are not limited to lists. Sets, mutable sequences and almost any other Python object that can act as a container can be used in place of the default list, by specifying the option on :

Dictionary Collections¶

A little extra detail is needed when using a dictionary as a collection. This because objects are always loaded from the database as lists, and a key-generation strategy must be available to populate the dictionary correctly. The function is by far the most common way to achieve a simple dictionary collection. It produces a dictionary class that will apply a particular attribute of the mapped class as a key. Below we map an class containing a dictionary of items keyed to the attribute:

is then a dictionary:

will ensure that the attribute of each complies with the key in the dictionary. Such as, when assigning to , the dictionary key we supply must match that of the actual object:

The attribute which uses as a key does not need to be mapped at all! Using a regular Python allows virtually any detail or combination of details about the object to be used as the key, as below when we establish it as a tuple of and the first ten letters of the field:

Above we added a backref. Assigning to this reverse relationship, the is added to the dictionary and the key is generated for us automatically:

Other built-in dictionary types include , which is almost like except given the object directly:

as well as which is passed any callable function. Note that it’s usually easier to use along with a as mentioned earlier:

Dictionary mappings are often combined with the “Association Proxy” extension to produce streamlined dictionary views. See Proxying to Dictionary Based Collections and Composite Association Proxies for examples.


A dictionary-based collection type with attribute-based keying.

Returns a factory with a keying based on the ‘attr_name’ attribute of entities in the collection, where is the string name of the attribute.

The key value must be immutable for the lifetime of the object. You can not, for example, map on foreign key values if those key values will change during the session, i.e. from None to a database-assigned integer after a session flush.


A dictionary-based collection type with column-based keying.

Returns a factory with a keying function generated from mapping_spec, which may be a Column or a sequence of Columns.

The key value must be immutable for the lifetime of the object. You can not, for example, map on foreign key values if those key values will change during the session, i.e. from None to a database-assigned integer after a session flush.


A dictionary-based collection type with arbitrary keying.

Returns a factory with a keying function generated from keyfunc, a callable that takes an entity and returns a key value.

The key value must be immutable for the lifetime of the object. You can not, for example, map on foreign key values if those key values will change during the session, i.e. from None to a database-assigned integer after a session flush.

fromsqlalchemyimportColumn,Integer,String,ForeignKeyfromsqlalchemy.ormimportrelationshipfromsqlalchemy.orm.collectionsimportattribute_mapped_collectionfromsqlalchemy.ext.declarativeimportdeclarative_baseBase=declarative_base()classItem(Base):__tablename__='item'id=Column(Integer,primary_key=True)notes=relationship("Note",collection_class=attribute_mapped_collection('keyword'),cascade="all, delete-orphan")classNote(Base):__tablename__='note'id=Column(Integer,primary_key=True)item_id=Column(Integer,ForeignKey(''),nullable=False)keyword=Column(String)text=Column(String)def__init__(self,keyword,text):self.keyword=keywordself.text=text
>>> item=Item()>>> item.notes['a']=Note('a','atext')>>> item.notes.items(){'a': <__main__.Note object at 0x2eaaf0>}
classItem(Base):__tablename__='item'id=Column(Integer,primary_key=True)notes=relationship("Note",collection_class=attribute_mapped_collection('note_key'),backref="item",cascade="all, delete-orphan")classNote(Base):__tablename__='note'id=Column(Integer,primary_key=True)item_id=Column(Integer,ForeignKey(''),nullable=False)keyword=Column(String)text=Column(String)@propertydefnote_key(self):return(self.keyword,self.text[0:10])def__init__(self,keyword,text):self.keyword=keywordself.text=text
>>> item=Item()>>> n1=Note("a","atext")>>> n1.item=item>>> item.notes{('a', 'atext'): <__main__.Note object at 0x2eaaf0>}
fromsqlalchemy.orm.collectionsimportcolumn_mapped_collectionclassItem(Base):__tablename__='item'id=Column(Integer,primary_key=True)notes=relationship("Note",collection_class=column_mapped_collection(Note.__table__.c.keyword),cascade="all, delete-orphan")
fromsqlalchemy.orm.collectionsimportmapped_collectionclassItem(Base):__tablename__='item'id=Column(Integer,primary_key=True)notes=relationship("Note",collection_class=mapped_collection(lambdanote:note.text[0:10]),cascade="all, delete-orphan")
classParent(Base):__tablename__='parent'parent_id=Column(Integer,primary_key=True)# use a setchildren=relationship(Child,collection_class=set)parent=Parent()child=Child()parent.children.add(child)assertchildinparent.children

Custom Collection Implementations¶

You can use your own types for collections as well. In simple cases, inherting from or , adding custom behavior, is all that’s needed. In other cases, special decorators are needed to tell SQLAlchemy more detail about how the collection operates.

Do I need a custom collection implementation?

In most cases not at all! The most common use cases for a “custom” collection is one that validates or marshals incoming values into a new form, such as a string that becomes a class instance, or one which goes a step beyond and represents the data internally in some fashion, presenting a “view” of that data on the outside of a different form.

For the first use case, the decorator is by far the simplest way to intercept incoming values in all cases for the purposes of validation and simple marshaling. See Simple Validators for an example of this.

For the second use case, the Association Proxy extension is a well-tested, widely used system that provides a read/write “view” of a collection in terms of some attribute present on the target object. As the target attribute can be a that returns virtually anything, a wide array of “alternative” views of a collection can be constructed with just a few functions. This approach leaves the underlying mapped collection unaffected and avoids the need to carefully tailor collection behavior on a method-by-method basis.

Customized collections are useful when the collection needs to have special behaviors upon access or mutation operations that can’t otherwise be modeled externally to the collection. They can of course be combined with the above two approaches.

Collections in SQLAlchemy are transparently instrumented. Instrumentation means that normal operations on the collection are tracked and result in changes being written to the database at flush time. Additionally, collection operations can fire events which indicate some secondary operation must take place. Examples of a secondary operation include saving the child item in the parent’s (i.e. the cascade), as well as synchronizing the state of a bi-directional relationship (i.e. a ).

The collections package understands the basic interface of lists, sets and dicts and will automatically apply instrumentation to those built-in types and their subclasses. Object-derived types that implement a basic collection interface are detected and instrumented via duck-typing:

, , and are known list-like methods, and will be instrumented automatically. is not a mutator method and won’t be instrumented, and won’t be either.

Duck-typing (i.e. guesswork) isn’t rock-solid, of course, so you can be explicit about the interface you are implementing by providing an class attribute:

This class looks list-like because of , but forces it to set-like. is known to be part of the set interface and will be instrumented.

But this class won’t work quite yet: a little glue is needed to adapt it for use by SQLAlchemy. The ORM needs to know which methods to use to append, remove and iterate over members of the collection. When using a type like or , the appropriate methods are well-known and used automatically when present. This set-like class does not provide the expected method, so we must supply an explicit mapping for the ORM via a decorator.

Annotating Custom Collections via Decorators¶

Decorators can be used to tag the individual methods the ORM needs to manage collections. Use them when your class doesn’t quite meet the regular interface for its container type, or when you otherwise would like to use a different method to get the job done.

And that’s all that’s needed to complete the example. SQLAlchemy will add instances via the method. and are the default methods for sets and will be used for removing and iteration. Default methods can be changed as well:

There is no requirement to be list-, or set-like at all. Collection classes can be any shape, so long as they have the append, remove and iterate interface marked for SQLAlchemy’s use. Append and remove methods will be called with a mapped entity as the single argument, and iterator methods are called with no arguments and must return an iterator.


Decorators for entity collection classes.

The decorators fall into two groups: annotations and interception recipes.

The annotating decorators (appender, remover, iterator, linker, converter, internally_instrumented) indicate the method’s purpose and take no arguments. They are not written with parens:

The recipe decorators all require parens, even those that take no arguments:

static ()¶

Mark the method as adding an entity to the collection.

Adds “add to collection” handling to the method. The decorator argument indicates which method argument holds the SQLAlchemy-relevant value. Arguments can be specified positionally (i.e. integer) or by name:

static ()¶

Tag the method as the collection appender.

The appender method is called with one positional argument: the value to append. The method will be automatically decorated with ‘adds(1)’ if not already decorated:

If the value to append is not allowed in the collection, you may raise an exception. Something to remember is that the appender will be called for each object mapped by a database query. If the database contains rows that violate your collection semantics, you will need to get creative to fix the problem, as access via the collection will not work.

If the appender method is internally instrumented, you must also receive the keyword argument ‘_sa_initiator’ and ensure its promulgation to collection events.

@collection.appenderdefadd(self,append):...# or, equivalently@collection.appender@collection.adds(1)defadd(self,append):...# for mapping type, an 'append' may kick out a previous value# that occupies that slot. consider d['a'] = 'foo'- any previous# value in d['a'] is discarded.@collection.appender@collection.replaces(1)defadd(self,entity):key=some_key_func(entity)previous=Noneifkeyinself:previous=self[key]self[key]=entityreturnprevious
static ()¶

Tag the method as the collection converter.

This optional method will be called when a collection is being replaced entirely, as in:

The converter method will receive the object being assigned and should return an iterable of values suitable for use by the method. A converter must not assign values or mutate the collection, its sole job is to adapt the value the user provides into an iterable of values for the ORM’s use.

The default converter implementation will use duck-typing to do the conversion. A dict-like collection will be convert into an iterable of dictionary values, and other types will simply be iterated:

If the duck-typing of the object does not match the type of this collection, a TypeError is raised.

Supply an implementation of this method if you want to expand the range of possible types that can be assigned in bulk or perform validation on the values about to be assigned.

static ()¶

Tag the method as instrumented.

This tag will prevent any decoration from being applied to the method. Use this if you are orchestrating your own calls to in one of the basic SQLAlchemy interface methods, or to prevent an automatic ABC method decoration from wrapping your implementation:

# normally an 'extend' method on a list-like class would be# automatically intercepted and re-implemented in terms of# SQLAlchemy events and append(). your implementation will# never be called, unless:@collection.internally_instrumenteddefextend(self,items):...
static ()¶

Tag the method as the collection remover.

The iterator method is called with no arguments. It is expected to return an iterator over all collection members:

static ()¶

deprecated; synonym for .

static ()¶

Tag the method as a “linked to attribute” event handler.

This optional event handler will be called when the collection class is linked to or unlinked from the InstrumentedAttribute. It is invoked immediately after the ‘_sa_adapter’ property is set on the instance. A single argument is passed: the collection adapter that has been linked, or None if unlinking.

static ()¶

Tag the method as the collection remover.

The remover method is called with one positional argument: the value to remove. The method will be automatically decorated with if not already decorated:

If the value to remove is not present in the collection, you may raise an exception or return None to ignore the error.

If the remove method is internally instrumented, you must also receive the keyword argument ‘_sa_initiator’ and ensure its promulgation to collection events.

@collection.removerdefzap(self,entity):...# or, equivalently@collection.remover@collection.removes_return()defzap(self,):...
static ()¶

Mark the method as removing an entity in the collection.

Adds “remove from collection” handling to the method. The decorator argument indicates which method argument holds the SQLAlchemy-relevant value to be removed. Arguments can be specified positionally (i.e. integer) or by name:

For methods where the value to remove is not known at call-time, use collection.removes_return.

static ()¶

Mark the method as removing an entity in the collection.

Adds “remove from collection” handling to the method. The return value of the method, if any, is considered the value to remove. The method arguments are not inspected:

For methods where the value to remove is known at call-time, use collection.remove.

static ()¶

Mark the method as replacing an entity in the collection.

Adds “add to collection” and “remove from collection” handling to the method. The decorator argument indicates which method argument holds the SQLAlchemy-relevant value to be added, and return value, if any will be considered the value to remove.

Arguments can be specified positionally (i.e. integer) or by name:

fromsqlalchemy.orm.collectionsimportcollectionclassMyList(list):@collection.removerdefzark(self,item):# do something special...@collection.iteratordefhey_use_this_instead_for_iteration(self):# ...

Custom Dictionary-Based Collections¶

The class can be used as a base class for your custom types or as a mix-in to quickly add collection support to other classes. It uses a keying function to delegate to and :

When subclassing , user-defined versions of or should be decorated with , if they call down to those same methods on . This because the methods on are already instrumented - calling them from within an already instrumented call can cause events to be fired off repeatedly, or inappropriately, leading to internal state corruption in rare cases:

The ORM understands the interface just like lists and sets, and will automatically instrument all dict-like methods if you choose to subclass or provide dict-like collection behavior in a duck-typed class. You must decorate appender and remover methods, however- there are no compatible methods in the basic dictionary interface for SQLAlchemy to use by default. Iteration will go through unless otherwise decorated.


Due to a bug in MappedCollection prior to version 0.7.6, this workaround usually needs to be called before a custom subclass of which uses can be used:

This will ensure that the has been properly initialized with custom and methods before used in a custom subclass.

class (keyfunc


A basic dictionary-based collection class.

Extends dict with the minimal bag semantics that collection classes require. and are implemented in terms of a keying function: any callable that takes an object and returns an object for use as a dictionary key.


Create a new collection with keying provided by keyfunc.

keyfunc may be any callable that takes an object and returns an object for use as a dictionary key.

The keyfunc will be called every time the ORM needs to add a member by value-only (such as when loading instances from the database) or remove a member. The usual cautions about dictionary keying apply- should return the same output for the life of the collection. Keying based on mutable properties can result in unreachable instances “lost” in the collection.

() → None. Remove all items from D.¶
(k[, d]) → v, remove specified key and return the corresponding value.¶

If key is not found, d is returned if given, otherwise KeyError is raised

() → (k, v), remove and return some (key, value) pair as a¶

2-tuple; but raise KeyError if D is empty.

(value, _sa_initiator=None

Remove an item by value, consulting the keyfunc for the key.

(value, _sa_initiator=None

Add an item by value, consulting the keyfunc for the key.

(k[, d]) → D.get(k,d), also set D[k]=d if k not in D¶
([E, ]**F) → None. Update D from dict/iterable E and F.¶

If E present and has a .keys() method, does: for k in E: D[k] = E[k] If E present and lacks .keys() method, does: for (k, v) in E: D[k] = v In either case, this is followed by: for k in F: D[k] = F[k]

fromsqlalchemy.utilimportOrderedDictfromsqlalchemy.orm.collectionsimportMappedCollectionclassNodeMap(OrderedDict,MappedCollection):"""Holds 'Node' objects, keyed by the 'name' attribute with insert order maintained."""def__init__(self,*args,**kw):MappedCollection.__init__(self,,*args,**kw)
fromsqlalchemy.orm.collectionsimportMappedCollection,\ collectionclassMyMappedCollection(MappedCollection):"""Use @internally_instrumented when your methods call down to already-instrumented methods. """@collection.internally_instrumenteddef__setitem__(self,key,value,_sa_initiator=None):# do something with key, valuesuper(MyMappedCollection,self).__setitem__(key,value,_sa_initiator)@collection.internally_instrumenteddef__delitem__(self,key,_sa_initiator=None):# do something with keysuper(MyMappedCollection,self).__delitem__(key,_sa_initiator)

Instrumentation and Custom Types¶

Many custom types and existing library classes can be used as a entity collection type as-is without further ado. However, it is important to note that the instrumentation process will modify the type, adding decorators around methods automatically.



Leave a Reply

Your email address will not be published. Required fields are marked *