String optimization in Python

Strings are terribly important in programming. A program without some form of string input, manipulation, and output is a rarity.

Of course this means that speed and sanity surrounding string features is important. One important feature of Python is string immutability. This opens up dozens of features, such as using strings as dictionary keys, but there are some downsides.

Immutable strings means that any string manipulation, such as splitting or appending, is making a copy of that string. This can become a performance problem, especially in a world where zero-copy is one of the favorite general optimization techniques. If you've done enough string mutation, you're probably aware of the following techniques:

• ''.join(iterable_of_strings) (instead of repeated +=)
• bytearray()
• StringIO()

But in some cases Python uses the immutability to avoid making copies:

>>> a = 'a' * 1024 * 1024  # a 1 megabyte string
>>> z = '' + a
>>> z is a
True

Here, because adding an empty string does not change the value, z is the same exact string object as a. And it doesn't matter how many times you append an empty string:

>>> z = '' + '' + '' + a
>>> z is a
True

It even works when a is the only item in a list:

>>> z = ''.join([a])
>>> z is a
True

But it falls apart when you put an empty string in the list with a:

>>> z = ''.join(['', a])
>>> z is a
False

And unfortunately even the first example seems to make a copy on PyPy:

>>>> a = 'a' * 1024 * 1024  # a 1 megabyte string again
>>>> z = '' + a
>>>> z is a 

False 

Although something more advanced may be going on under the covers, as is often the case with PyPy.

I'm almost done stringing you along, but as a corollary reminder:

Never rely on "is" checks with ints, floats, and strings. "==" and other value checks are what you need. As a general rule, "is" is for objects, None, and sometimes True/False.

Keep on stringifying!

Mahmoud
http://sedimental.org/
https://github.com/mahmoud
https://twitter.com/mhashemi

__dict__ and vars()

vars() is the lesser-known and arguably more Pythonic version of the __dict__ attribute (available on all classes that do not use slots). From the docs:

vars([object])

Without an argument, act like locals().

With a module, class or class instance object as argument (or anything else that has a __dict__ attribute), return that attribute.

As core Python contributor Raymond Hettinger points out on Twitter:

"Most folks prefer len(s) to s.__len__(), prefer next(it) to it.next(), but forget to use vars(s) rather than s.__dict__"

However, I think there's room to argue that s.__dict__ may be a little clearer than vars(s). After all, (almost) everything in Python is an object, and very nearly all objects are backed by dicts. Furthermore, s.__dict__ is faster:

>>> from timeit import timeit
>>> timeit('object.__dict__;object.__dict__')
0.2776460647583008
>>> timeit('vars(object);vars(object)')
1.276643991470337
>>>

Four million iterations later, we see that grabbing the __dict__ is over four and a half times faster. This should actually come as no surprise: accessing built-in functions (like len()/vars()/etc.) is always slower than object attribute access.