Thursday, May 5, 2016

String optimization in Python

Strings are terribly important in programming. A program without some form of string input, manipulation, and output is a rarity.

Of course this means that speed and sanity surrounding string features is important. One important feature of Python is string immutability. This opens up dozens of features, such as using strings as dictionary keys, but there are some downsides.

Immutable strings means that any string manipulation, such as splitting or appending, is making a copy of that string. This can become a performance problem, especially in a world where zero-copy is one of the favorite general optimization techniques. If you've done enough string mutation, you're probably aware of the following techniques:
But in some cases Python uses the immutability to avoid making copies:
>>> a = 'a' * 1024 * 1024  # a 1 megabyte string
>>> z = '' + a
>>> z is a
True
Here, because adding an empty string does not change the value, z is the same exact string object as a. And it doesn't matter how many times you append an empty string:
>>> z = '' + '' + '' + a
>>> z is a
True
It even works when a is the only item in a list:
>>> z = ''.join([a])
>>> z is a
True
But it falls apart when you put an empty string in the list with a:
>>> z = ''.join(['', a])
>>> z is a
False
And unfortunately even the first example seems to make a copy on PyPy:
>>>> a = 'a' * 1024 * 1024  # a 1 megabyte string again
>>>> z = '' + a
>>>> z is a 
False 
Although something more advanced may be going on under the covers, as is often the case with PyPy.

I'm almost done stringing you along, but as a corollary reminder:
Never rely on "is" checks with ints, floats, and strings. "==" and other value checks are what you need. As a general rule, "is" is for objects, None, and sometimes True/False.

Keep on stringifying!

Mahmoud
http://sedimental.org/
https://github.com/mahmoud
https://twitter.com/mhashemi

43 comments:

  1. I think the cpython behaviour is an optimisation possible because of reference counting. Cpython can tell when adding strings if it's the only reference, and can reuse the memory rather than copying in cases like above. Pypy doesn't use reference counting, so can't do the same trick, aiui

    ReplyDelete
    Replies
    1. While it's true that PyPy does not use CPython-style reference counting, I don't think that implies that PyPy can't have this optimization, per se. PyPy still has immutable strings, as that's a property of the Python language, not the runtime.

      Delete
  2. This could be related to interned strings: you can force any string to be unique and in a global table using the intern() function.

    ReplyDelete
    Replies
    1. You are correct about intern() and that may be the culprit with shorter strings, but you'll notice I created a 1MB string to start with. Very much hope our relatively lightweight CPython doesn't magically intern that :)

      Delete
  3. I should thank you for the undertakings you have made in making this article. I am confiding in a similar best work from you later on as well.. Enterprise SEO Services

    ReplyDelete
  4. currently trending technologies are phyton , azure . learn azure through azure training

    ReplyDelete
  5. I learned World's Trending Technology from certified experts for free of cost. I got a job in decent Top MNC Company with handsome 14 LPA salary, I have learned the World's Trending Technology from python training in btm layout experts who know advanced concepts which can help to solve any type of Real-time issues in the field of Python. Really worth trying Freelance SEO expert in Bangalore

    ReplyDelete
  6. Thank you so much for this useful article. Visit OGEN Infosystem for Web Designing and SEO Services in Delhi, India.
    SEO Service in Delhi

    ReplyDelete
  7. thank you for sharing this blog, it is very useful information for python learning.
    python course bangalore

    ReplyDelete
  8. Excellent blog information shared was very informative and valuable looking forward for next blog thank you.
    Data Analytics Course Online 360DigiTMG

    ReplyDelete
  9. Awesome article with top quality information and I appreciate the writer's choice for choosing this excellent topic found valuable thank you.
    Data Science Training in Hyderabad

    ReplyDelete
  10. Thankyou for posting this informative blog, i come to know something new with this. Great Job! Keep it up.

    1000 free youtube subscribers

    ReplyDelete
  11. I am sure that this is going to help a lot of individuals. Keep up the good work. It is highly convincing and I enjoyed going through the entire blog.

    business analytics course

    ReplyDelete
  12. Amazing blog.Thanks for sharing such excellent information with us. keep sharing...
    machine learning course in aurangabad

    ReplyDelete
  13. Wonderful blog. I delighted in perusing your articles. This is genuinely an incredible perused for me. I have bookmarked it and I am anticipating perusing new articles. Keep doing awesome!
    best machine learning course in aurangabad

    ReplyDelete

  14. I like what you guys tend to be up too. This kind of clever work and reporting! Keep up the very good works guys I’ve added you guys to our blogroll.

    Try to check my blog: 부산달리기
    (jk)

    ReplyDelete
  15. it is valuable informative. nice blogs. thanks for sharing these information with all of us.Kinemaster Gold

    ReplyDelete
  16. Nice blog, it is valuable informative. thanks for sharing these information with all of us. whatsapp mod

    ReplyDelete
  17. I’m excited to uncover this page. I need to to thank you for ones time for this particularly fantastic read !! I definitely really liked every part of it and i also have you saved to fav to look at new information in your site. cyber security course in noida

    ReplyDelete
  18. This post is so interactive and informative.keep update more information...
    AWS Training in Anna Nagar
    AWS Training in Chennai

    ReplyDelete
  19. I truly like you're composing style, incredible data, thankyou for posting.
    cyber security training malaysia

    ReplyDelete
  20. This post is so interactive and informative.keep update more information...
    hadoop training in tambaram
    Big data training in chennai

    ReplyDelete
  21. This is the first time I visit here. I found such a large number of engaging stuff in your blog, particularly its conversation. From the huge amounts of remarks on your articles, I surmise I am by all accounts not the only one having all the recreation here! Keep doing awesome. I have been important to compose something like this on my site and you have given me a thought.

    ReplyDelete
  22. 360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

    ReplyDelete
  23. Autodesk Revit Crack stretches a full key to the whole construction scheme facet and provision triggers. The designers, constructing corporations to clarify, condition more¬ well-versed selections beforehand. They give techniques a bit more skillfully. It's novel variations of user¬ demanded exploits, positive Global Boundaries https://freeprosoftz.com/autodesk-revit-crack-key/

    ReplyDelete
  24. With iTop VPN Crack, clients might conceal their characters and explore namelessly. Utilizing top VPN break, clients might shield their specifically distinguishing data from outer associations. Organizations might impart data to different organizations, restricting their ability to apply fundamental attributes.Itop vpn crack download pc

    ReplyDelete
  25. Boost your professional reputation with a surefire way to pick up some impressive new skills in data science by registering for the Data science courses near me. Learn to collect, clean, and analyze data with tools like Hadoop and Spark. Learn to develop algorithms and build models in machine learning to optimize product performance and gross profit for your organization. Become an expert in techniques like Data Mining, Data Cleansing, and Data Exploring that help refine data, making it possible to present it in an understandable format.

    Best Data Science Training institute in Bangalore

    ReplyDelete
  26. Fine page, in which did u come happening a distant memory the assessment concerning this posting?i have right of access the majority of the articles with respect to your web website now, and I as a matter of fact in addition to your style. much thanks to you a million and absorb save happening the vivacious deed. Reloader Ultima Versioner

    ReplyDelete