So a list and a tuple walk into a sum()

As a direct side effect of glom's 19.1.0 release, the authors here at PDW got to re-experience one of the more surprising behaviors of three of Python's most basic constructs:

• list()
• tuple()
• sum()

Most experienced developers know the quickest way to combine a short list of short lists:

list_of_lists = [[1], [2], [3, 4]]
sum(list_of_lists, [])
# [1, 2, 3, 4]

Ah, nice and flat, much better.

But what happens when we throw a tuple into the mix:

list_of_seqs = [[1], [2], (3, 4)]
sum(list_of_seqs, [])
# TypeError: can only concatenate list (not "tuple") to list

This is kind of surprising! Especially when you consider this:

seq = [1, 2]
seq += (3, 4)
# [1, 2, 3, 4]

Why should sum() fail when addition succeeds?! We'll get to that.

new_list = [1, 2] + (3, 4)
# TypeError: can only concatenate list (not "tuple") to list

There's that error again!

The trick here is that Python has two addition operators. The simple "+" or "add" operator, used by sum(), and the more nuanced "+=" or "iadd" operator, add's inplace variant.

But why is ok for one addition to error and the other to succeed?

Symmetry. And maybe commutativity if you remember that math class.

"+" in Python is symmetric: A + B and B + A should always yield the same result. To do otherwise would be more surprising than any of the surprises above. list and tuple cannot be added with this operator because in a mixed-type situation, the return type would change based on ordering.

Meanwhile, "+=" is asymmetric. The left side of the statement determines the type of the return completely. A += B keeps A's type. A straightforward, Pythonic reason if there ever was one.

Going back to the start of our story, by building on operator.iadd, glom's new flatten() function avoids sum()'s error-raising behavior and works wonders on all manner of nesting iterable.

The Zen of Empty Lists

"There should be one-- and preferably only one --obvious way to do it". One of the many philosophies that has earned Python its acclaim.

But while the Zen of Python limits on the number of obvious ways, the Zen of Python says nothing about the boundless freedom of unobvious ways.

Let's empty a list named bucket.

The most obvious way is to simply not. 99 times out of 100, you want to assign a new empty list rather than mutating the old.

bucket = []

But let's say you really wanted to empty it, well the clearest way is clear:

bucket.clear()

But even the docs say this is equivalent to:

del bucket[:]

I guess that's crossing the obvious line. Of course, it may be more obvious than Norvig's "dumbell" operator:

bucket[:]=[]

Actually the slice assignment can take any iterable, so our list can lift plates of many shapes:

bucket[:]={}

If you don't want your list getting ripped and/or cut, maybe keep it warm with Norvig's ski hat:

bucket *=0

The ski hat is of particular interest because it's using a very obvious list feature, much more commonly used than list.clear(). Nobody would bat an eye at:

bucket = [0, 1, 2, 3] * 2
# bucket = [0, 1, 2, 3, 0, 1, 2, 3]

If you multiply any list by 0, you make a new list of length 0. Bizarrely, this is actually true of any multiplier less than 0, too.

bucket *= -3
# bucket = []

Safe to say we are deep in the territory of the unobvious. Is there a syntax we might meditate on to take us further?

None on the left

A natural default, None is probably the most commonly assigned value in Python. But what happens if you move it to the left side of that equation?

In Python 2:

>>> None = 2
  File "<stdin>", line 1
SyntaxError: cannot assign to None

This is similar to what happens when you assign to a literal:

>>> 1 = 2
  File "<stdin>", line 1
SyntaxError: can't assign to literal

In Python 3 this walk on the wild side will get you a slightly different error:

>>> None = 1
  File "<stdin>", line 1
SyntaxError: can't assign to keyword

None has graduated from useful snowflake to full-blown keyword!

a return to yield

I remember when, almost a decade ago, I was first discovering generators. It was a heady time, and I saw applications everywhere.

def fib_gen():
    x, y = 1, 1
    while x < 100:
       x, y = y, x + y
       yield x
    return

I also remember the first time I tried to mix a return value into my generator.

def fib_gen():
    x, y = 1, 1
    while x < 100:
       x, y = y, x + y
       yield x
    return True

Imagine my surprise, as I'm sure countless others experienced as well:

SyntaxError: 'return' with argument inside generator

A rare compile-time error! Only the decorative, bare return is allowed in generators, where they serve to raise StopIteration.

Now, imagine my surprise, so many years later when I import that same code in Python 3.


...

Nothing! No error. So what happened?

Turns out that the coroutine and asyncio machinery of Python 3 has repurposed this old impossibility.

If we manually iterate to skip over our yield:

fib_iter = fib_gen()                                                                                                                                                                                                    
for i in range(11):                                                                                                                 
    next(fib_iter)                                                                                                                  
next(fib_iter)

We see what's really happening with our return:

Traceback (most recent call last):
  File "fib_gen.py", line 13, in <module>
    next(fib_iter)
StopIteration: True

That's right, returns in generators now raise StopIteration with a single argument of the return value.

Most of the time you won't see this. StopIterations are automatically consumed and handled correctly by for loops, list comprehensions, and sequence constructors (like list). But it's yet another reason to be extra careful when writing your own generators, specific to Python 3.

Bit by bit: CPU architecture

There are a variety of reasons you might want to know how many bits the architecture of the CPU running your Python program has. Maybe you're about to use some statically-compiled C, or maybe you're just taking a survey.

Either way, you've got to know. One historical way way is:

import sys
IS_64BIT = sys.maxint > 2 ** 32

Except that sys.maxint is specific to Python 2. Being the crossover point where ints transparently become longs, sys.maxint doesn't apply in Python 3, where ints and longs have been merged into just one type: int (even though the C calls it PyLongObject). And Python 3's introduction of sys.maxsize doesn't help much if you're trying to support Python <2.7, where it doesn't exist.

So instead we can use the struct module:

import struct
IS_64BIT = struct.calcsize("P") > 4

This is a little less clear, but being backwards and forwards compatible, and given struct is still part of the standard library, it's a pretty good approach, and is the one taken in boltons.ecoutils.

But let's say you really wanted to get it down to a single line, and even standard library imports were out of the question, for some reason. You could do something like this:

IS_64BIT = tuple.__itemsize__ > 4

While not extensively documented, a lot of built-in types have __itemsize__ and __basicsize__ attributes, which describes the memory requirement of the underlying structure. For tuples, each item requires a pointer. Pointer size * 8 = bits in the architecture. 4 * 8 = 32-bit architecture, and 8 * 8 = 64-bit architecture.

Even though documentation isn't great, the __itemsize__ approach works back to at least Python 2.6 and forward to Python 3.7. Memory profilers like pympler use __itemsize__ and it might work for you, too!

String optimization in Python

Strings are terribly important in programming. A program without some form of string input, manipulation, and output is a rarity.

Of course this means that speed and sanity surrounding string features is important. One important feature of Python is string immutability. This opens up dozens of features, such as using strings as dictionary keys, but there are some downsides.

Immutable strings means that any string manipulation, such as splitting or appending, is making a copy of that string. This can become a performance problem, especially in a world where zero-copy is one of the favorite general optimization techniques. If you've done enough string mutation, you're probably aware of the following techniques:

• ''.join(iterable_of_strings) (instead of repeated +=)
• bytearray()
• StringIO()

But in some cases Python uses the immutability to avoid making copies:

>>> a = 'a' * 1024 * 1024  # a 1 megabyte string
>>> z = '' + a
>>> z is a
True

Here, because adding an empty string does not change the value, z is the same exact string object as a. And it doesn't matter how many times you append an empty string:

>>> z = '' + '' + '' + a
>>> z is a
True

It even works when a is the only item in a list:

>>> z = ''.join([a])
>>> z is a
True

But it falls apart when you put an empty string in the list with a:

>>> z = ''.join(['', a])
>>> z is a
False

And unfortunately even the first example seems to make a copy on PyPy:

>>>> a = 'a' * 1024 * 1024  # a 1 megabyte string again
>>>> z = '' + a
>>>> z is a 

False 

Although something more advanced may be going on under the covers, as is often the case with PyPy.

I'm almost done stringing you along, but as a corollary reminder:

Never rely on "is" checks with ints, floats, and strings. "==" and other value checks are what you need. As a general rule, "is" is for objects, None, and sometimes True/False.

Keep on stringifying!

Mahmoud
http://sedimental.org/
https://github.com/mahmoud
https://twitter.com/mhashemi

Undecoratable

Decorators are one of Python's bigger success stories, and many programmers' first experience with higher-order programming. Most practiced and prolific Python programmers will find themselves making good use of them regularly.

But every feature has its limits, and here's a new one to try on for size:

>>> @x().y()
  File "<stdin>", line 1
    @x().y()
        ^
SyntaxError: invalid syntax

That's right, decoration is not an arbitrary Python expression. It doesn't matter what x and y were, or even if they were defined. You can't follow a function call with a dot. @x() works fine, @x.y() would work fine, too. But @x().y(), that's only for mad Pythonists who would take things TOO FAR.

Decorator invocations, defined at the top of the Python grammar, can only be followed by class definitions and function definitions.

Well, now we know, and now we can all say we've been there

-- Mahmoud
http://sedimental.org/
https://github.com/mahmoud
https://twitter.com/mhashemi

List Comprehension Code Golf

Ah code golf, pastime of our navelgazing alter egos. Being designed for readability and maintainability, Python doesn't always show well in this sort of sport, but occasionally we get thrown a bone. For instance, for nonzero even numbers less than 10:

>>> [x for x in range(10) if x and not x % 2]
[2, 4, 6, 8]

is equivalent to

>>> [x for x in range(10) if x if not x % 2]
[2, 4, 6, 8]

A whole character saved! Yes, a close reading of PEP 202 will show that one of the canonical examples of list comprehensions uses this pattern for... some reason.

Either way, now you know. Sally forth and do what must be done with all code golf tricks: Never, Ever Use Them For Production Code.

-- Mahmoud
http://sedimental.org/
https://github.com/mahmoud
https://twitter.com/mhashemi

Loopy references

This may not come as a surprise to experienced Python programmers, but objects in Python can reference or "contain" themselves and/or their parent objects.

A trivial example, the self-containing list:

>>> a = []
>>> a.append(a)
>>> a
[[...]]

Kind of a cool repr, at least. How else could you flatly represent this infinitely nested object? What happens when you take two of these and smash them together with an equals comparator?

>>> b = []
>>> b.append(b)
>>> a == b
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: maximum recursion depth exceeded in cmp

Like two mirrors pointed at one another, they compare back and forth forever, or at least for...

>>> sys.getrecursionlimit()
1000 

... loops.

decoding base64.decode

base64 is Python's module for providing various standard encoding functionalities, especially the eponymous base64. Little quirk about the decoding in Python 2.7, it ignores invalid characters!



>>> import base64
>>> test_str = base64.b64encode('123456789')
>>> base64.b64decode(test_str[:-4] + '....' + test_str[-4:])
'123456789'
>>> base64.b64decode(test_str[:-4] + '...' + test_str[-4:])
'123456789'
>>> base64.b64decode(test_str[:-4] + '.' + test_str[-4:])
'123456789'
>>> test_str[:-4]
'MTIzNDU2'
>>> base64.b64decode(test_str[:-4])
'123456'

Python 3 adds a flag, validate, but it's off by default. Oh well!

Detecting new- and old-style classes

Be it typos, old libraries, or just backwards-compatibility, old-style classes are Python relics that have a tendency to creep into your shiny new programs. Given that new-style classes are almost certainly what you want, it can't hurt to know how to detect the difference.

>>> # Using Python 2
...
>>> class old_style():       # doesn't inherit from object
...    pass
...
>>> class new_style(object): # does inherit from object
...    pass
...
>>> old_object = old_style()
>>> new_object = new_style()
>>> type(old_object) == old_object.__class__
False
>>> type(new_object) == new_object.__class__
True

For the inquisitive, new-style classes unified the concept of class and type in Python 2.2. For a complete description, check out Guido's essay on the topic, circa 2002.

Merry Guido-mas, and may you only find new-style classes under your AST.

Easily import a dynamically created module

Python modules are typically Python source files that are used like simple namespaces for variables. Take this simple module:

mod.py:

x = "Hi, I'm a module variable."

Provided mod.py is on the import path (which usually includes the current working directory), you can run Python and see:

>>> import mod
>>> print mod.x
"Hi, I'm a module variable."

Of course, this is Python and Python modules are objects of type 'module'. You can dynamically create an object of this type easily enough:

>>> from types import ModuleType
>>> mymodule = ModuleType('mymodule')
>>> mymodule.x = "Hi, I'm in a dynamically-created module."
>>> print mymodule.x
"Hi, I'm in a dynamically-created module."

Pretty much what you'd expect. ModuleType is the same thing you'd get back from import sys; type(sys). That's actually what the types module uses.

But, you can't import mymodule because it's not a real, source-based module on the import path. If you want other Pythonauts to be able to use standard import syntax to import your dynamic module, a little more trickery is involved.

dynmod.py:

import sys
from types import ModuleType

not_present = "Hi, I was masked by the dynamic module."

dm = ModuleType(__name__)
dm.x = 5

sys.modules[__name__] = dm

If you're doing this you probably already know, but __name__ is just a string containing the name of the current module, based on the filename ('dynmod'). Using __name__ just keeps things a bit more consistent.

Now, look what happens when you import dynmod:

>>> import dynmod
>>> dynmod.x
5
>>> dynmod.not_present
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'module' object has no attribute 'not_present'
>>> dir(dynmod)
['__doc__', '__name__', 'x']

All the normal import syntax should work, including from dynmod import x, even though the module isn't strictly what's in the source file.

The dynamic behavior is only executed once, on initial import. When imported, Python modules create a name->module_object entry in the sys.modules dictionary. From then on, if the module name is found in sys.modules, the associated module object is used. That's how normal modules work, and that's how we want our dynamic module to work, too.

The key here is that the sys.modules entry seems to be created before the module code is run, allowing the module code to modify its own entry. Be careful with this, because Python's import system has some complex locking behavior that can make certain manipulations not work:

dynmod_broken.py:

# Breaks some import locks or something
import sys
from types import ModuleType

not_present = "masked"

sys.modules[__name__]   = ModuleType(__name__)
sys.modules[__name__].x = 5

(In fact, according to my testing, this left every single variable in the namespace set to None. Pretty PDW, if you ask me.)

If that's not interesting enough for you, then stay tuned for the follow-up post on more advanced dynamic modules where we show you how to use modules' simple and familiar nature to do your darkest biddings.

(Note: if you're looking for the right way to simply mask certain variables, consider using __all__. Also, you could probably do more fancy things with Import Hooks, but this is tested to work on Python 2.7 and 3.1, so maybe keep it simple, if simple's all you need.)

Range woes

If you've ever iterated, you probably know about the built-in range() and xrange() functions. They usually do the job, but neither is perfect. xrange() can't deal with numbers larger than the system max integer, while range() chokes on very long ranges:



If you simply must have numbers larger than sys.maxint, then you need to use range() while limiting the length with a start argument.