lambda default parameters

Making lambdas behave nicely by using default parameters

>>> [c() for c in [(lambda: i) for i in range(10)]]
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

>>> [c() for c in [(lambda a=i: a) for i in range(10)]]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

is not None, is (not None)

>>> False is not None

True

>>> False is (not None)

False

__dict__ and vars()

vars() is the lesser-known and arguably more Pythonic version of the __dict__ attribute (available on all classes that do not use slots). From the docs:

vars([object])

Without an argument, act like locals().

With a module, class or class instance object as argument (or anything else that has a __dict__ attribute), return that attribute.

As core Python contributor Raymond Hettinger points out on Twitter:

"Most folks prefer len(s) to s.__len__(), prefer next(it) to it.next(), but forget to use vars(s) rather than s.__dict__"

However, I think there's room to argue that s.__dict__ may be a little clearer than vars(s). After all, (almost) everything in Python is an object, and very nearly all objects are backed by dicts. Furthermore, s.__dict__ is faster:

>>> from timeit import timeit
>>> timeit('object.__dict__;object.__dict__')
0.2776460647583008
>>> timeit('vars(object);vars(object)')
1.276643991470337
>>>

Four million iterations later, we see that grabbing the __dict__ is over four and a half times faster. This should actually come as no surprise: accessing built-in functions (like len()/vars()/etc.) is always slower than object attribute access.

Detecting new- and old-style classes

Be it typos, old libraries, or just backwards-compatibility, old-style classes are Python relics that have a tendency to creep into your shiny new programs. Given that new-style classes are almost certainly what you want, it can't hurt to know how to detect the difference.

>>> # Using Python 2
...
>>> class old_style():       # doesn't inherit from object
...    pass
...
>>> class new_style(object): # does inherit from object
...    pass
...
>>> old_object = old_style()
>>> new_object = new_style()
>>> type(old_object) == old_object.__class__
False
>>> type(new_object) == new_object.__class__
True

For the inquisitive, new-style classes unified the concept of class and type in Python 2.2. For a complete description, check out Guido's essay on the topic, circa 2002.

Merry Guido-mas, and may you only find new-style classes under your AST.

Easily import a dynamically created module

Python modules are typically Python source files that are used like simple namespaces for variables. Take this simple module:

mod.py:

x = "Hi, I'm a module variable."

Provided mod.py is on the import path (which usually includes the current working directory), you can run Python and see:

>>> import mod
>>> print mod.x
"Hi, I'm a module variable."

Of course, this is Python and Python modules are objects of type 'module'. You can dynamically create an object of this type easily enough:

>>> from types import ModuleType
>>> mymodule = ModuleType('mymodule')
>>> mymodule.x = "Hi, I'm in a dynamically-created module."
>>> print mymodule.x
"Hi, I'm in a dynamically-created module."

Pretty much what you'd expect. ModuleType is the same thing you'd get back from import sys; type(sys). That's actually what the types module uses.

But, you can't import mymodule because it's not a real, source-based module on the import path. If you want other Pythonauts to be able to use standard import syntax to import your dynamic module, a little more trickery is involved.

dynmod.py:

import sys
from types import ModuleType

not_present = "Hi, I was masked by the dynamic module."

dm = ModuleType(__name__)
dm.x = 5

sys.modules[__name__] = dm

If you're doing this you probably already know, but __name__ is just a string containing the name of the current module, based on the filename ('dynmod'). Using __name__ just keeps things a bit more consistent.

Now, look what happens when you import dynmod:

>>> import dynmod
>>> dynmod.x
5
>>> dynmod.not_present
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'module' object has no attribute 'not_present'
>>> dir(dynmod)
['__doc__', '__name__', 'x']

All the normal import syntax should work, including from dynmod import x, even though the module isn't strictly what's in the source file.

The dynamic behavior is only executed once, on initial import. When imported, Python modules create a name->module_object entry in the sys.modules dictionary. From then on, if the module name is found in sys.modules, the associated module object is used. That's how normal modules work, and that's how we want our dynamic module to work, too.

The key here is that the sys.modules entry seems to be created before the module code is run, allowing the module code to modify its own entry. Be careful with this, because Python's import system has some complex locking behavior that can make certain manipulations not work:

dynmod_broken.py:

# Breaks some import locks or something
import sys
from types import ModuleType

not_present = "masked"

sys.modules[__name__]   = ModuleType(__name__)
sys.modules[__name__].x = 5

(In fact, according to my testing, this left every single variable in the namespace set to None. Pretty PDW, if you ask me.)

If that's not interesting enough for you, then stay tuned for the follow-up post on more advanced dynamic modules where we show you how to use modules' simple and familiar nature to do your darkest biddings.

(Note: if you're looking for the right way to simply mask certain variables, consider using __all__. Also, you could probably do more fancy things with Import Hooks, but this is tested to work on Python 2.7 and 3.1, so maybe keep it simple, if simple's all you need.)

when is a no-op not a no-op?

When is a = a not a no-op?  When a is a class variable accessed through self.a.  May be useful when you want a dynamic initializer for your instances on your class.  For example, a property like time_created.

>>> class A(object):

...    w = 4

...    def __init__(self):

...       self.w = self.w

...

>>> A()

<__main__.A object at 0x00DAE770>

>>> A().__dict__

{'w': 4}

>>> a = A()

>>> a.__dict__

{'w': 4}

>>> a.w = 5

>>> A.w

4

>>> a.w

5

metaclass depth > call depth

How far deep can a metaclass tree go?
type(A) = B, type(B) = C, type(C) = D...?
Can it go further than the function call depth?

def tower(n=100, sofar=type):

   class next(sofar):

      __metaclass__ = sofar

   return tower(n-1, next)

>>> tower(997)

<class '__main__.sofar'>

>>> tower(998)

#...

  File "<stdin>", line 3, in tower

RuntimeError: maximum recursion depth exceeded while calling a Python object