Thursday, December 22, 2011

Easily import a dynamically created module

Python modules are typically Python source files that are used like simple namespaces for variables. Take this simple module:

mod.py:

x = "Hi, I'm a module variable."

Provided mod.py is on the import path (which usually includes the current working directory), you can run Python and see:

>>> import mod
>>> print mod.x
"Hi, I'm a module variable."


Of course, this is Python and Python modules are objects of type 'module'. You can dynamically create an object of this type easily enough:

>>> from types import ModuleType
>>> mymodule = ModuleType('mymodule')
>>> mymodule.x = "Hi, I'm in a dynamically-created module."
>>> print mymodule.x
"Hi, I'm in a dynamically-created module."

Pretty much what you'd expect. ModuleType is the same thing you'd get back from import sys; type(sys). That's actually what the types module uses.

But, you can't import mymodule because it's not a real, source-based module on the import path. If you want other Pythonauts to be able to use standard import syntax to import your dynamic module, a little more trickery is involved.

dynmod.py:

import sys
from types import ModuleType

not_present = "Hi, I was masked by the dynamic module."

dm = ModuleType(__name__)
dm.x = 5

sys.modules[__name__] = dm


If you're doing this you probably already know, but __name__ is just a string containing the name of the current module, based on the filename ('dynmod'). Using __name__ just keeps things a bit more consistent.

Now, look what happens when you import dynmod:

>>> import dynmod
>>> dynmod.x
5
>>> dynmod.not_present
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'module' object has no attribute 'not_present'
>>> dir(dynmod)
['__doc__', '__name__', 'x']


All the normal import syntax should work, including from dynmod import x, even though the module isn't strictly what's in the source file.

The dynamic behavior is only executed once, on initial import. When imported, Python modules create a name->module_object entry in the sys.modules dictionary. From then on, if the module name is found in sys.modules, the associated module object is used. That's how normal modules work, and that's how we want our dynamic module to work, too.

The key here is that the sys.modules entry seems to be created before the module code is run, allowing the module code to modify its own entry. Be careful with this, because Python's import system has some complex locking behavior that can make certain manipulations not work:

dynmod_broken.py:

# Breaks some import locks or something
import sys
from types import ModuleType

not_present = "masked"

sys.modules[__name__] = ModuleType(__name__)
sys.modules[__name__].x = 5

(In fact, according to my testing, this left every single variable in the namespace set to None. Pretty PDW, if you ask me.)

If that's not interesting enough for you, then stay tuned for the follow-up post on more advanced dynamic modules where we show you how to use modules' simple and familiar nature to do your darkest biddings.

(Note: if you're looking for the right way to simply mask certain variables, consider using __all__. Also, you could probably do more fancy things with Import Hooks, but this is tested to work on Python 2.7 and 3.1, so maybe keep it simple, if simple's all you need.)

29 comments:

  1. Such a very useful article. Very interesting to read this article. I would like to thank you for the efforts you had made for writing this awesome article.
    Data Science Course in Pune
    Data Science Training in Pune

    ReplyDelete
  2. Cool you write, the information is very good and interesting, I'll give you a link to my site.
    Data Science Course in Bangalore

    ReplyDelete
  3. Easily, the article is actually the best topic on this registry related issue. I fit in with your conclusions and will eagerly look forward to your next updates. Just saying thanks will not just be sufficient, for the fantasti c lucidity in your writing. I will instantly grab your rss feed to stay informed of any updates.
    Data Science Training in Bangalore

    ReplyDelete
  4. I am impressed by the information that you have on this blog. It shows how well you understand this subject.
    Data Science Certification in Bangalore

    ReplyDelete
  5. Very useful information, the post shared was very nice.
    Python Online Training

    ReplyDelete
  6. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!
    data analytics course in bangalore

    ReplyDelete
  7. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data Science Course in Bangalore

    ReplyDelete
  8. It's good to visit your blog again, it's been months for me. Well, this article that I have been waiting for so long. I will need this post to complete my college homework, and it has the exact same topic with your article. Thanks, have a good game.
    Business Analytics Course

    ReplyDelete
  9. It fully emerged to crown Singapore's southern shores and has undoubtedly put it on the world residential monument map. Still, I scored more points than I have in one season for GS. I think it would be hard to find someone with the same consistency that I have had over the years, so I'm happy.
    Best Data Science Courses in Bangalore

    ReplyDelete
  10. I read this post,very well explain about python.Keep sharing this information.

    Python Classes in Pune

    ReplyDelete
  11. I am glad to discover this page. I have to thank you for the time I spent on this especially great reading !! I really liked each part and also bookmarked you for new information on your site.
    Data Science Course Syllabus

    ReplyDelete
  12. You have completed certain reliable points there. I did some research on the subject and found that almost everyone will agree with your blog.

    Best Data Science Courses in Bangalore

    ReplyDelete
  13. I am a new user of this site, so here I saw several articles and posts published on this site, I am more interested in some of them, will provide more information on these topics in future articles.
    Data Analytics Course in Bangalore

    ReplyDelete
  14. You have completed certain reliable points there. I did some research on the subject and found that almost everyone will agree with your blog.

    Digital Marketing Course in Bangalore

    ReplyDelete
  15. I value the blog article. Thanks Again. Really Great.온라인경마

    ReplyDelete
  16. Infycle Technologies offers couples for care and technology in addition to Python Training in Chennai, 100% of the internship class will be prepared. After completing the training, the participants will be sent to the upper MNCs interviews. Call 750633333 to get more information and get a free display.

    ReplyDelete
  17. I'm always looking online for articles that can help me. I think you also made some good comments on the functions. Keep up the good work!

    Data Scientist Training and Placement Bangalore

    ReplyDelete
  18. It's like you've got the point right, but forgot to include your readers. Maybe you should think about it from different angles.

    Data Science Training in Erode

    ReplyDelete
  19. Very efficiently written information. It will be beneficial to anybody who utilizes it, including me. Keep up the good work. For sure i will check out more posts. This site seems to get a good amount of visitors. Ielts

    ReplyDelete
  20. Thanks for getting the effort to write this amazing post. Visit Best Python training course in Delhi

    ReplyDelete
  21. My spouse and I stumbled over here by a different web address and thought I might check things out. I like what I see so i am just following you. Look forward to checking out your web page again. https://python.engineering/image-processing-without-opencv-python/

    ReplyDelete
  22. It's like you've got the point right, but forgot to include your readers. Maybe you should think about it from different angles.

    Data Scientist Course in Erode

    ReplyDelete
  23. Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one.
    Continue posting. A debt of gratitude is in order for sharing.
    data analytics course in warangal

    ReplyDelete
  24. Great tips and very easy to understand. This will definitely be very useful for me when I get a chance to start my blog. data science course in mysore

    ReplyDelete
  25. A decent blog consistently concocts new and energizing data and keeping in mind that perusing I have feel that this blog is truly have each one of those quality that qualify a blog to be a one.digital marketing course

    ReplyDelete
  26. This comment has been removed by the author.

    ReplyDelete
  27. Great post i must say and thanks for the information. Visit at https://www.uncodemy.com/course/python/

    ReplyDelete