Thursday, December 22, 2011

Easily import a dynamically created module

Python modules are typically Python source files that are used like simple namespaces for variables. Take this simple module:

mod.py:

x = "Hi, I'm a module variable."

Provided mod.py is on the import path (which usually includes the current working directory), you can run Python and see:

>>> import mod
>>> print mod.x
"Hi, I'm a module variable."


Of course, this is Python and Python modules are objects of type 'module'. You can dynamically create an object of this type easily enough:

>>> from types import ModuleType
>>> mymodule = ModuleType('mymodule')
>>> mymodule.x = "Hi, I'm in a dynamically-created module."
>>> print mymodule.x
"Hi, I'm in a dynamically-created module."

Pretty much what you'd expect. ModuleType is the same thing you'd get back from import sys; type(sys). That's actually what the types module uses.

But, you can't import mymodule because it's not a real, source-based module on the import path. If you want other Pythonauts to be able to use standard import syntax to import your dynamic module, a little more trickery is involved.

dynmod.py:

import sys
from types import ModuleType

not_present = "Hi, I was masked by the dynamic module."

dm = ModuleType(__name__)
dm.x = 5

sys.modules[__name__] = dm


If you're doing this you probably already know, but __name__ is just a string containing the name of the current module, based on the filename ('dynmod'). Using __name__ just keeps things a bit more consistent.

Now, look what happens when you import dynmod:

>>> import dynmod
>>> dynmod.x
5
>>> dynmod.not_present
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'module' object has no attribute 'not_present'
>>> dir(dynmod)
['__doc__', '__name__', 'x']


All the normal import syntax should work, including from dynmod import x, even though the module isn't strictly what's in the source file.

The dynamic behavior is only executed once, on initial import. When imported, Python modules create a name->module_object entry in the sys.modules dictionary. From then on, if the module name is found in sys.modules, the associated module object is used. That's how normal modules work, and that's how we want our dynamic module to work, too.

The key here is that the sys.modules entry seems to be created before the module code is run, allowing the module code to modify its own entry. Be careful with this, because Python's import system has some complex locking behavior that can make certain manipulations not work:

dynmod_broken.py:

# Breaks some import locks or something
import sys
from types import ModuleType

not_present = "masked"

sys.modules[__name__] = ModuleType(__name__)
sys.modules[__name__].x = 5

(In fact, according to my testing, this left every single variable in the namespace set to None. Pretty PDW, if you ask me.)

If that's not interesting enough for you, then stay tuned for the follow-up post on more advanced dynamic modules where we show you how to use modules' simple and familiar nature to do your darkest biddings.

(Note: if you're looking for the right way to simply mask certain variables, consider using __all__. Also, you could probably do more fancy things with Import Hooks, but this is tested to work on Python 2.7 and 3.1, so maybe keep it simple, if simple's all you need.)

20 comments:

  1. Such a very useful article. Very interesting to read this article. I would like to thank you for the efforts you had made for writing this awesome article.
    Data Science Course in Pune
    Data Science Training in Pune

    ReplyDelete
  2. Cool you write, the information is very good and interesting, I'll give you a link to my site.
    Data Science Course in Bangalore

    ReplyDelete
  3. Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates. Just saying thanks will not just be sufficient, for the fantasti c lucidity in your writing. I will instantly grab your rss feed to stay informed of any updates.
    Data Science Training in Bangalore

    ReplyDelete
  4. I am impressed by the information that you have on this blog. It shows how well you understand this subject.
    Data Science Certification in Bangalore

    ReplyDelete
  5. Very useful information, the post shared was very nice.
    Python Online Training

    ReplyDelete
  6. i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
    data science training in bangalore

    ReplyDelete
  7. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
    Data Science Training in Bangalore

    ReplyDelete
  8. I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, hope you will provide more information on these topics in your next articles.
    data analytics training in bangalore

    ReplyDelete
  9. I Want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging endeavors.
    data science course in bangalore with placement

    ReplyDelete
  10. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    data analytics course in bangalore

    ReplyDelete
  11. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data Science Course in Bangalore

    ReplyDelete
  12. It's good to visit your blog again, it's been months for me. Well, this article that I have been waiting for so long. I will need this post to complete my college homework, and it has the exact same topic with your article. Thanks, have a good game.
    Business Analytics Course

    ReplyDelete
  13. It fully emerged to crown Singapore's southern shores and has undoubtedly put it on the world residential monument map. Still, I scored more points than I have in one season for GS. I think it would be hard to find someone with the same consistency that I have had over the years, so I'm happy.
    Data Analytics Course in Bangalore

    ReplyDelete
  14. They are produced by high level developers who will stand out for the creation of their polo dress. You will find Ron Lauren polo shirts in an exclusive range which includes private lessons for men and women.
    Digital Marketing Course in Bangalore

    ReplyDelete
  15. It fully emerged to crown Singapore's southern shores and has undoubtedly put it on the world residential monument map. Still, I scored more points than I have in one season for GS. I think it would be hard to find someone with the same consistency that I have had over the years, so I'm happy.
    Best Data Science Courses in Bangalore

    ReplyDelete


  16. Hey I'm at work surfing around your blog from my new iphone 3gs! Just wanted to say I love reading through your blog and look forward to all your posts! Carry on the superb work!

    Review my page please: 안마
    (jk)

    ReplyDelete
  17. I read this post,very well explain about python.Keep sharing this information.

    Python Classes in Pune

    ReplyDelete
  18. I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
    Data Science Course Syllabus

    ReplyDelete