dill module documentation

dill module

dill: a utility for serialization of python objects

Based on code written by Oren Tirosh and Armin Ronacher. Extended to a (near) full set of the builtin types (in types module), and coded to the pickle interface, by <mmckerns@caltech.edu>. Initial port to python3 by Jonathan Dobson, continued by mmckerns. Test against “all” python types (Std. Lib. CH 1-15 @ 2.7) by mmckerns. Test against CH16+ Std. Lib. … TBD.

CONTENTS_FMODE = 1

Pickles the file contents, creating a new file if on load the file does not exist. The position = min(pickled position, EOF) and mode is chosen as such that “best” preserves behavior of the original file.

FILE_FMODE = 2

Pickles the entire file (handle and contents), preserving mode and position.

HANDLE_FMODE = 0

Pickles the file handle, preserving mode. The position of the unpickled object is as for a new file handle.

exception PickleError

Bases: Exception

__module__ = '_pickle'

__weakref__

list of weak references to the object (if defined)

exception PickleWarning

Bases: Warning, _pickle.PickleError

__annotations__ = {}

__module__ = 'dill._dill'

__weakref__

list of weak references to the object (if defined)

class Pickler(*args, **kwds)

Bases: pickle._Pickler

python’s Pickler extended to interpreter sessions

This takes a binary file for writing a pickle data stream.

The optional protocol argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default protocol is 4. It was introduced in Python 3.4, and is incompatible with previous versions.

Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.

The file argument must have a write() method that accepts a single bytes argument. It can thus be a file object opened for binary writing, an io.BytesIO instance, or any other custom object that meets this interface.

If fix_imports is True and protocol is less than 3, pickle will try to map the new Python 3 names to the old module names used in Python 2, so that the pickle data stream is readable with Python 2.

If buffer_callback is None (the default), buffer views are serialized into file as part of the pickle stream.

If buffer_callback is not None, then it can be called any number of times with a buffer view. If the callback returns a false value (such as None), the given buffer is out-of-band; otherwise the buffer is serialized in-band, i.e. inside the pickle stream.

It is an error if buffer_callback is not None and protocol is None or smaller than 5.

__init__(*args, **kwds)

This takes a binary file for writing a pickle data stream.

The optional protocol argument tells the pickler to use the given protocol; supported protocols are 0, 1, 2, 3, 4 and 5. The default protocol is 4. It was introduced in Python 3.4, and is incompatible with previous versions.

Specifying a negative protocol version selects the highest protocol version supported. The higher the protocol used, the more recent the version of Python needed to read the pickle produced.

The file argument must have a write() method that accepts a single bytes argument. It can thus be a file object opened for binary writing, an io.BytesIO instance, or any other custom object that meets this interface.

If fix_imports is True and protocol is less than 3, pickle will try to map the new Python 3 names to the old module names used in Python 2, so that the pickle data stream is readable with Python 2.

If buffer_callback is None (the default), buffer views are serialized into file as part of the pickle stream.

If buffer_callback is not None, then it can be called any number of times with a buffer view. If the callback returns a false value (such as None), the given buffer is out-of-band; otherwise the buffer is serialized in-band, i.e. inside the pickle stream.

It is an error if buffer_callback is not None and protocol is None or smaller than 5.

__module__ = 'dill._dill'

_session = False

dispatch = {<class 'NoneType'>: <function _Pickler.save_none>, <class 'bool'>: <function _Pickler.save_bool>, <class 'int'>: <function _Pickler.save_long>, <class 'float'>: <function _Pickler.save_float>, <class 'bytes'>: <function _Pickler.save_bytes>, <class 'bytearray'>: <function _Pickler.save_bytearray>, <class 'pickle.PickleBuffer'>: <function _Pickler.save_picklebuffer>, <class 'str'>: <function _Pickler.save_str>, <class 'tuple'>: <function _Pickler.save_tuple>, <class 'list'>: <function _Pickler.save_list>, <class 'dict'>: <function save_module_dict>, <class 'set'>: <function _Pickler.save_set>, <class 'frozenset'>: <function _Pickler.save_frozenset>, <class 'function'>: <function save_function>, <class 'type'>: <function save_type>, <class 'code'>: <function save_code>, <class 'dict_keys'>: <function save_dict_view.<locals>.save_dict_view_for_function.<locals>._save_dict_view>, <class 'dict_values'>: <function save_dict_view.<locals>.save_dict_view_for_function.<locals>._save_dict_view>, <class 'dict_items'>: <function save_dict_view.<locals>.save_dict_view_for_function.<locals>._save_dict_view>, <class 'odict_keys'>: <function save_dict_view.<locals>.save_dict_view_for_function.<locals>._save_dict_view>, <class 'odict_values'>: <function save_dict_view.<locals>.save_dict_view_for_function.<locals>._save_dict_view>, <class 'odict_items'>: <function save_dict_view.<locals>.save_dict_view_for_function.<locals>._save_dict_view>, <class '_thread.lock'>: <function save_lock>, <class '_thread.RLock'>: <function save_rlock>, <class '_io.TextIOWrapper'>: <function save_file>, <class '_io.BufferedWriter'>: <function save_file>, <class '_io.BufferedReader'>: <function save_file>, <class '_io.BufferedRandom'>: <function save_file>, <class '_io.FileIO'>: <function save_file>, <class '_pyio.TextIOWrapper'>: <function save_file>, <class '_pyio.BufferedWriter'>: <function save_file>, <class '_pyio.BufferedReader'>: <function save_file>, <class '_pyio.BufferedRandom'>: <function save_file>, <class 'functools._lru_cache_wrapper'>: <function save_lru_cache>, <class 'super'>: <function save_super>, <class 'classmethod_descriptor'>: <function save_wrapper_descriptor>, <class 'wrapper_descriptor'>: <function save_wrapper_descriptor>, <class 'method_descriptor'>: <function save_wrapper_descriptor>, <class 'getset_descriptor'>: <function save_wrapper_descriptor>, <class 'member_descriptor'>: <function save_wrapper_descriptor>, <class 'method-wrapper'>: <function save_instancemethod>, <class 'cell'>: <function save_cell>, <class 'mappingproxy'>: <function save_dictproxy>, <class 'slice'>: <function save_slice>, <class 'NotImplementedType'>: <function save_singleton>, <class 'ellipsis'>: <function save_singleton>, <class 'range'>: <function save_singleton>, <class 'weakref.ReferenceType'>: <function save_weakref>, <class 'weakref.CallableProxyType'>: <function save_weakproxy>, <class 'weakref.ProxyType'>: <function save_weakproxy>, <class 'module'>: <function save_module>, <class 'property'>: <function save_property>, <class 'classmethod'>: <function save_classmethod>, <class 'staticmethod'>: <function save_classmethod>}

dump(obj)

Write a pickled representation of obj to the open file.

settings = {'byref': False, 'fmode': 0, 'ignore': False, 'protocol': 4, 'recurse': False}

exception PicklingError

Bases: _pickle.PickleError

__annotations__ = {}

__module__ = '_pickle'

exception PicklingWarning

Bases: dill._dill.PickleWarning, _pickle.PicklingError

__annotations__ = {}

__module__ = 'dill._dill'

class Unpickler(*args, **kwds)

Bases: _pickle.Unpickler

python’s Unpickler extended to interpreter sessions and more types

__dict__ = mappingproxy({'__module__': 'dill._dill', '__doc__': "python's Unpickler extended to interpreter sessions and more types", 'settings': {'protocol': 4, 'byref': False, 'fmode': 0, 'recurse': False, 'ignore': False}, '_session': False, 'find_class': <function Unpickler.find_class>, '__init__': <function Unpickler.__init__>, 'load': <function Unpickler.load>, '__dict__': <attribute '__dict__' of 'Unpickler' objects>, '__weakref__': <attribute '__weakref__' of 'Unpickler' objects>, '__annotations__': {}})

__init__(*args, **kwds)

__module__ = 'dill._dill'

__weakref__

list of weak references to the object (if defined)

_session = False

find_class(module, name)

Return an object from a specified module.

If necessary, the module will be imported. Subclasses may override this method (e.g. to restrict unpickling of arbitrary classes and functions).

This method is called whenever a class or a function object is needed. Both arguments passed are str objects.

load()

Load a pickle.

Read a pickled object representation from the open file object given in the constructor, and return the reconstituted object hierarchy specified therein.

settings = {'byref': False, 'fmode': 0, 'ignore': False, 'protocol': 4, 'recurse': False}

exception UnpicklingError

Bases: _pickle.PickleError

__annotations__ = {}

__module__ = '_pickle'

exception UnpicklingWarning

Bases: dill._dill.PickleWarning, _pickle.UnpicklingError

__annotations__ = {}

__module__ = 'dill._dill'

check(obj, *args, **kwds)

Check pickling of an object across another process.

python is the path to the python interpreter (defaults to sys.executable)

Set verbose=True to print the unpickled object in the other process.

Additional keyword arguments are as dumps() and loads().

copy(obj, *args, **kwds)

Use pickling to ‘copy’ an object (i.e. loads(dumps(obj))).

See dumps() and loads() for keyword arguments.

dump(obj, file, protocol=None, byref=None, fmode=None, recurse=None, **kwds)

Pickle an object to a file.

See dumps() for keyword arguments.

dump_session(filename='/tmp/session.pkl', main=None, byref=False, **kwds)

pickle the current state of __main__ to a file

dumps(obj, protocol=None, byref=None, fmode=None, recurse=None, **kwds)

Pickle an object to a string.

protocol is the pickler protocol, as defined for Python pickle.

If byref=True, then dill behaves a lot more like pickle as certain objects (like modules) are pickled by reference as opposed to attempting to pickle the object itself.

If recurse=True, then objects referred to in the global dictionary are recursively traced and pickled, instead of the default behavior of attempting to store the entire global dictionary. This is needed for functions defined via exec().

fmode (HANDLE_FMODE, CONTENTS_FMODE, or FILE_FMODE) indicates how file handles will be pickled. For example, when pickling a data file handle for transfer to a remote compute service, FILE_FMODE will include the file contents in the pickle and cursor position so that a remote method can operate transparently on an object with an open file handle.

Default values for keyword arguments can be set in dill.settings.

load(file, ignore=None, **kwds)

Unpickle an object from a file.

See loads() for keyword arguments.

load_session(filename='/tmp/session.pkl', main=None, **kwds)

update the __main__ module with the state from the session file

loads(str, ignore=None, **kwds)

Unpickle an object from a string.

If ignore=False then objects whose class is defined in the module __main__ are updated to reference the existing class in __main__, otherwise they are left to refer to the reconstructed type, which may be different.

Default values for keyword arguments can be set in dill.settings.

pickle(t, func)

expose dispatch table for user-created extensions

pickles(obj, exact=False, safe=False, **kwds)

Quick check if object pickles with dill.

If exact=True then an equality test is done to check if the reconstructed object matches the original object.

If safe=True then any exception will raised in copy signal that the object is not picklable, otherwise only pickling errors will be trapped.

Additional keyword arguments are as dumps() and loads().

register(t)

register type to Pickler’s dispatch table

detect module

Methods for detecting objects leading to pickling failures.

baditems(obj, exact=False, safe=False)

get items in object that fail to pickle

badobjects(obj, depth=0, exact=False, safe=False)

get objects that fail to pickle

badtypes(obj, depth=0, exact=False, safe=False)

get types for objects that fail to pickle

code(func)

get the code object for the given function or method

NOTE: use dill.source.getsource(CODEOBJ) to get the source code

errors(obj, depth=0, exact=False, safe=False)

get errors for objects that fail to pickle

freevars(func)

get objects defined in enclosing code that are referred to by func

returns a dict of {name:object}

getmodule(object, _filename=None, force=False)

get the module of the object

globalvars(func, recurse=True, builtin=False)

get objects defined in global scope that are referred to by func

return a dict of {name:object}

nestedcode(func, recurse=True)

get the code objects for any nested functions (e.g. in a closure)

nestedglobals(func, recurse=True)

get the names of any globals found within func

outermost(func)

get outermost enclosing object (i.e. the outer function in a closure)

NOTE: this is the object-equivalent of getsource(func, enclosing=True)

referredglobals(func, recurse=True, builtin=False)

get the names of objects in the global scope referred to by func

referrednested(func, recurse=True)

get functions defined inside of func (e.g. inner functions in a closure)

NOTE: results may differ if the function has been executed or not. If len(nestedcode(func)) > len(referrednested(func)), try calling func(). If possible, python builds code objects, but delays building functions until func() is called.

trace(boolean)

print a trace through the stack when pickling; useful for debugging

varnames(func)

get names of variables defined by func

returns a tuple (local vars, local vars referrenced by nested functions)

objtypes module

all Python Standard Library object types (currently: CH 1-15 @ 2.7) and some other common object types (i.e. numpy.ndarray)

to load more objects and types, use dill.load_types()

pointers module

at(address, module=None)

get object located at the given memory address (inverse of id(obj))

children(obj, objtype, depth=1, ignore=())

Find the chain of referrers for obj. Chain will start with obj.

objtype: an object type or tuple of types to search for depth: search depth (e.g. depth=2 is ‘grandchildren’) ignore: an object or tuple of objects to ignore in the search

NOTE: a common thing to ignore is all globals, ‘ignore=(globals(),)’

NOTE: repeated calls may yield different results, as python stores the last value in the special variable ‘_’; thus, it is often good to execute something to replace ‘_’ (e.g. >>> 1+1).

parent(obj, objtype, ignore=())

>>> listiter = iter([4,5,6,7])
>>> obj = parent(listiter, list)
>>> obj == [4,5,6,7]  # actually 'is', but don't have handle any longer
True

NOTE: objtype can be a single type (e.g. int or list) or a tuple of types.

WARNING: if obj is a sequence (e.g. list), may produce unexpected results. Parent finds one parent (e.g. the last member of the sequence).

parents(obj, objtype, depth=1, ignore=())

Find the chain of referents for obj. Chain will end with obj.

objtype: an object type or tuple of types to search for depth: search depth (e.g. depth=2 is ‘grandparents’) ignore: an object or tuple of objects to ignore in the search

reference(obj)

get memory address of proxy’s reference object

settings module

global settings for Pickler

source module

Extensions to python’s ‘inspect’ module, which can be used to retrieve information from live python objects. The methods defined in this module are augmented to facilitate access to source code of interactively defined functions and classes, as well as provide access to source code for objects defined in a file.

_importable(obj, alias='', source=None, enclosing=False, force=True, builtin=True, lstrip=True)

get an import string (or the source code) for the given object

This function will attempt to discover the name of the object, or the repr of the object, or the source code for the object. To attempt to force discovery of the source code, use source=True, to attempt to force the use of an import, use source=False; otherwise an import will be sought for objects not defined in __main__. The intent is to build a string that can be imported from a python file. obj is the object to inspect. If alias is provided, then rename the object with the given alias.

If source=True, use these options:

If enclosing=True, then also return any enclosing code. If force=True, catch (TypeError,IOError) and try to use import hooks. If lstrip=True, ensure there is no indentation in the first line of code.

If source=False, use these options:

If enclosing=True, get the import for the outermost enclosing callable. If force=True, then don’t test the import string before returning it. If builtin=True, then force an import for builtins where possible.

_namespace(obj); return namespace hierarchy (as a list of names)

for the given object. For an instance, find the class hierarchy.

For example:

>>> from functools import partial
>>> p = partial(int, base=2)
>>> _namespace(p)
['functools', 'partial']

_wrap(f)

encapsulate a function and it’s __import__

dumpsource(object, alias='', new=False, enclose=True)

‘dump to source’, where the code includes a pickled object.

If new=True and object is a class instance, then create a new instance using the unpacked class source code. If enclose, then create the object inside a function enclosure (thus minimizing any global namespace pollution).

findsource(object)

Return the entire source file and starting line number for an object. For interactively-defined objects, the ‘file’ is the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a list of all the lines in the file and the line number indexes a line in that list. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

getimport(obj, alias='', verify=True, builtin=False, enclosing=False)

get the likely import string for the given object

obj is the object to inspect If verify=True, then test the import string before returning it. If builtin=True, then force an import for builtins where possible. If enclosing=True, get the import for the outermost enclosing callable. If alias is provided, then rename the object on import.

getname(obj, force=False, fqn=False)

get the name of the object. for lambdas, get the name of the pointer

getsource(object, alias='', lstrip=False, enclosing=False, force=False, builtin=False)

Return the text of the source code for an object. The source code for interactively-defined objects are extracted from the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a single string. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

If alias is provided, then add a line of code that renames the object. If lstrip=True, ensure there is no indentation in the first line of code. If enclosing=True, then also return any enclosing code. If force=True, catch (TypeError,IOError) and try to use import hooks. If builtin=True, force an import for any builtins

getsourcelines(object, lstrip=False, enclosing=False)

Return a list of source lines and starting line number for an object. Interactively-defined objects refer to lines in the interpreter’s history.

The argument may be a module, class, method, function, traceback, frame, or code object. The source code is returned as a list of the lines corresponding to the object and the line number indicates where in the original source file the first line of code was found. An IOError is raised if the source code cannot be retrieved, while a TypeError is raised for objects where the source code is unavailable (e.g. builtins).

If lstrip=True, ensure there is no indentation in the first line of code. If enclosing=True, then also return any enclosing code.

importable(obj, alias='', source=None, builtin=True)

get an importable string (i.e. source code or the import string) for the given object, including any required objects from the enclosing and global scope

This function will attempt to discover the name of the object, or the repr of the object, or the source code for the object. To attempt to force discovery of the source code, use source=True, to attempt to force the use of an import, use source=False; otherwise an import will be sought for objects not defined in __main__. The intent is to build a string that can be imported from a python file.

obj is the object to inspect. If alias is provided, then rename the object with the given alias. If builtin=True, then force an import for builtins where possible.

indent(code, spaces=4)

indent a block of code with whitespace (default is 4 spaces)

isdynamic(obj)

check if object was built in the interpreter

isfrommain(obj)

check if object was built in __main__

outdent(code, spaces=None, all=True)

outdent a block of code (default is to strip all leading whitespace)

temp module

Methods for serialized objects (or source code) stored in temporary files and file-like objects.

capture(stream='stdout')

builds a context that temporarily replaces the given stream name

>>> with capture('stdout') as out:
...   print ("foo!")
...
>>> print (out.getvalue())
foo!

dump(object, **kwds)

dill.dump of object to a NamedTemporaryFile. Loads with “dill.temp.load”. Returns the filehandle.

>>> dumpfile = dill.temp.dump([1, 2, 3, 4, 5])
>>> dill.temp.load(dumpfile)
[1, 2, 3, 4, 5]

Optional kwds:

If ‘suffix’ is specified, the file name will end with that suffix, otherwise there will be no suffix.

If ‘prefix’ is specified, the file name will begin with that prefix, otherwise a default prefix is used.

If ‘dir’ is specified, the file will be created in that directory, otherwise a default directory is used.

If ‘text’ is specified and true, the file is opened in text mode. Else (the default) the file is opened in binary mode. On some operating systems, this makes no difference.

NOTE: Keep the return value for as long as you want your file to exist !

dumpIO(object, **kwds)

dill.dump of object to a buffer. Loads with “dill.temp.loadIO”. Returns the buffer object.

>>> dumpfile = dill.temp.dumpIO([1, 2, 3, 4, 5])
>>> dill.temp.loadIO(dumpfile)
[1, 2, 3, 4, 5]

dumpIO_source(object, **kwds)

write object source to a buffer (instead of dill.dump) Loads by with dill.temp.loadIO_source. Returns the buffer object.

>>> f = lambda x:x**2
>>> pyfile = dill.temp.dumpIO_source(f, alias='_f')
>>> _f = dill.temp.loadIO_source(pyfile)
>>> _f(4)
16

Optional kwds:

If ‘alias’ is specified, the object will be renamed to the given string.

dump_source(object, **kwds)

write object source to a NamedTemporaryFile (instead of dill.dump) Loads with “import” or “dill.temp.load_source”. Returns the filehandle.

>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, alias='_f')
>>> _f = dill.temp.load_source(pyfile)
>>> _f(4)
16

>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, dir='.')
>>> modulename = os.path.basename(pyfile.name).split('.py')[0]
>>> exec('from %s import f as _f' % modulename)
>>> _f(4)
16

Optional kwds:

If ‘alias’ is specified, the object will be renamed to the given string.

If ‘prefix’ is specified, the file name will begin with that prefix, otherwise a default prefix is used.

If ‘dir’ is specified, the file will be created in that directory, otherwise a default directory is used.

If ‘text’ is specified and true, the file is opened in text mode. Else (the default) the file is opened in binary mode. On some operating systems, this makes no difference.

NOTE: Keep the return value for as long as you want your file to exist !

load(file, **kwds)

load an object that was stored with dill.temp.dump

file: filehandle mode: mode to open the file, one of: {‘r’, ‘rb’}

>>> dumpfile = dill.temp.dump([1, 2, 3, 4, 5])
>>> dill.temp.load(dumpfile)
[1, 2, 3, 4, 5]

loadIO(buffer, **kwds)

load an object that was stored with dill.temp.dumpIO

buffer: buffer object

>>> dumpfile = dill.temp.dumpIO([1, 2, 3, 4, 5])
>>> dill.temp.loadIO(dumpfile)
[1, 2, 3, 4, 5]

loadIO_source(buffer, **kwds)

load an object that was stored with dill.temp.dumpIO_source

buffer: buffer object alias: string name of stored object

>>> f = lambda x:x**2
>>> pyfile = dill.temp.dumpIO_source(f, alias='_f')
>>> _f = dill.temp.loadIO_source(pyfile)
>>> _f(4)
16

load_source(file, **kwds)

load an object that was stored with dill.temp.dump_source

file: filehandle alias: string name of stored object mode: mode to open the file, one of: {‘r’, ‘rb’}

>>> f = lambda x: x**2
>>> pyfile = dill.temp.dump_source(f, alias='_f')
>>> _f = dill.temp.load_source(pyfile)
>>> _f(4)
16