Nidelven IT - All about Python, Zope & Plone - and Open Source!

Here you'll find issues related to our services. Mostly about Python, Zope and Plone, as well as hosting-related issues.

"Keeping IT real"






Older entries



Atom - Subscribe - Categories

An updated version of the Issue Dealer, building Zope 2.13.22

Morten has been blogging again over at http://blogologue.com/blog_entry?id=1423796724X07 about building the latest and greatest Zope + a self-built product.

[Permalink] [By morphex] [Python-only or Python related (Atom feed)] [13 Feb 04:41 GMT+2]

A SoundCloud(R) export tool in Python

Morten has been fiddling with Python again, new blog post here: http://blogologue.com/blog_entry?id=1420246529X05

[Permalink] [By morphex] [Python-only or Python related (Atom feed)] [03 Jan 02:19 GMT+2]

SASS, Compass & Zen Grid and a maybe new Python Framework?

Morten has been testing out some new technologies, check out his blog post:

http://blogologue.com/blog_entry?id=1417710581X85

[Permalink] [By morphex] [Python-only or Python related (Atom feed)] [05 Dec 09:46 GMT+2]

Jep, Jython and CPython compared, all in a nice little script

So, in relation to the work with Jep ( http://www.nidelven-it.no/weblogs/hosting/blog_entry?id=1368... ) - I've been building a script that can build and setup Jep, Jython and CPython in a directory, and then have a compare.sh script that runs the same code on the 3 different systems.

The installation script is here:

https://raw.github.com/morphex/PythonCompare/master/install....

Here's the output from the generated compare.sh file:

morphex@copyleft-laptop:~/projects/self/jython_jepp_installer6$ ./compare.sh
Starting comparison of Jython, Jepp and CPython..
Starting with Jep..
0.39471411705
0.341079950333
0.333596944809
0.34021282196
0.322955131531
0.322613954544
0.322247982025
0.323844909668
0.319895029068
0.324920892715
Ran in milliseconds: 3878.0
Now Jython..
1.25800013542
0.579999923706
0.436999797821
0.361000061035
0.43799996376
0.287999868393
0.289000034332
0.910000085831
0.289000034332
0.290999889374
Ran in milliseconds: 7576
Now CPython..
0.321110010147
0.312225103378
0.312016010284
0.309517145157
0.30745100975
0.313014984131
0.312664985657
0.312491893768
0.311804056168
0.312137126923
Ran in milliseconds: 3165

As you can see, the Jep and CPython scripts are fairly stable in terms of execution speed while Jython varies a bit but gets faster (probably due to the HotSpot technology). If you want to change the test to something else, you can try "./jdk1.7.0_21/bin/java -jar usr/lib/jython-standalone-2.5.3.jar -m compileall -l ." and mytest.py will be recompiled, along with the other .py files.

Now, feel free to use the install.py script as you like, I think it's a good example of how to setup the whole thing in a specific directory.. there was some hair-pulling to get the entire build process setup so that linking and dependencies were setup the right way.

Fun to work with a mix if Python, Java and Bash scripting for a change. I think being able to use Python to script for example prototypes or even production code is a big win in terms of productivity.. weak typing has its place in rapid application development. :)

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [20 May 13:59 GMT+2]

Python in Java

So, these last couple of days I've been playing around with Jep ( http://jepp.sourceforge.net ), the project that embeds Python and makes it available in Java.. it's been a while since I've worked with Java, dependencies and so on, so been pulling hair a bit to get things working.

But, got it working, and one things I've become more aware of lately is how things run, in terms of speed. So I decided to compare Jep, Jython ( http://www.jython.org ) and (C)Python ( http://www.python.org ).

Here's Jep:

>>> from test import pystone
>>> for x in range(10):
... pystone.main()
...
Pystone(1.1) time for 50000 passes = 1
This machine benchmarks at 50000 pystones/second
Pystone(1.1) time for 50000 passes = 1.01
This machine benchmarks at 49505 pystones/second
Pystone(1.1) time for 50000 passes = 1.01
This machine benchmarks at 49505 pystones/second
Pystone(1.1) time for 50000 passes = 1.01
This machine benchmarks at 49505 pystones/second
Pystone(1.1) time for 50000 passes = 1
This machine benchmarks at 50000 pystones/second
Pystone(1.1) time for 50000 passes = 0.99
This machine benchmarks at 50505.1 pystones/second
Pystone(1.1) time for 50000 passes = 1.01
This machine benchmarks at 49505 pystones/second
Pystone(1.1) time for 50000 passes = 1
This machine benchmarks at 50000 pystones/second
Pystone(1.1) time for 50000 passes = 0.99
This machine benchmarks at 50505.1 pystones/second
Pystone(1.1) time for 50000 passes = 1.01
This machine benchmarks at 49505 pystones/second
>>>

It is fairly consistent when it comes to the number of pystones.

Then trying Jython:

>>> for x in range(10):
... pystone.main()
...
Pystone(1.1) time for 50000 passes = 1.816
This machine benchmarks at 27533 pystones/second
Pystone(1.1) time for 50000 passes = 1.128
This machine benchmarks at 44326.2 pystones/second
Pystone(1.1) time for 50000 passes = 0.569
This machine benchmarks at 87873.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.495
This machine benchmarks at 101010 pystones/second
Pystone(1.1) time for 50000 passes = 0.574
This machine benchmarks at 87108 pystones/second
Pystone(1.1) time for 50000 passes = 0.519
This machine benchmarks at 96339.1 pystones/second
Pystone(1.1) time for 50000 passes = 0.497
This machine benchmarks at 100604 pystones/second
Pystone(1.1) time for 50000 passes = 0.501
This machine benchmarks at 99800.4 pystones/second
Pystone(1.1) time for 50000 passes = 0.575
This machine benchmarks at 86956.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.515
This machine benchmarks at 97087.3 pystones/second
>>>

Which shows a great variation, but improvement, in how many pystones can be calculated per second. I assume this is due to the Java HotSpot technology. How well the HotSpot system works with real-life code and data is a different matter.

Finally, we have (C)Python:

>>> for x in range(10):
... pystone.main()
...
Pystone(1.1) time for 50000 passes = 0.86
This machine benchmarks at 58139.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.86
This machine benchmarks at 58139.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.86
This machine benchmarks at 58139.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.85
This machine benchmarks at 58823.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.86
This machine benchmarks at 58139.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.87
This machine benchmarks at 57471.3 pystones/second
Pystone(1.1) time for 50000 passes = 0.87
This machine benchmarks at 57471.3 pystones/second
Pystone(1.1) time for 50000 passes = 0.85
This machine benchmarks at 58823.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.86
This machine benchmarks at 58139.5 pystones/second
Pystone(1.1) time for 50000 passes = 0.86
This machine benchmarks at 58139.5 pystones/second
>>>

Which shows a fairly consistent number of pystones, around 58000 per second.

Now, the reason I started looking at Jython and then Jep, is that I'm looking at some Java-based content and development frameworks. I'm not sure if there is a big difference in how you can use Jep and Jython, but these are alternatives that can be used when developing in Java systems. In most cases data needs to be sent back to the Java-based application.

Being able to script things fast in Python is a benefit in some situations for rapid prototyping and so on. As is my experience with Python, the speed in Python can be good enough with the right programming techniques and systems. If the applications run too slowly, then switching between Jep and Jython can be a good idea to see which one actually runs the fastest.

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [15 May 18:07 GMT+2]

100000 downloads, a significant contribution to the Python community

So, yesterday the counter for downloaded PyPi packages created by me tipped 100000 downloads, which was cool. It has been going steadily for days and months now, and it was nice to see that that magic number was surpassed.

You can see the packages here:

http://blogologue.com/morten%20w.%20petersen%20-%20pypi%20pa...

Most of those packages are tied to the Zope and Plone systems, but at least one package is Python-only, the email_backport package:

"The email_backport package is a wrap-up of the email > 4.x module
found in Python 2.5 and above."

I created that to be able to use mailing features across a large range of Plone versions, and it has worked fairly well.

The more I work with coding and programming, the lazier I get, should really get better at looking for right places to put code in other Python projects or as separate Python software packages, but the most important and pressing thing is to get something working and working well with the code it is supposed to run in and on..

With that thought, it would be nice if we in the Python community get some documentation and training on how to split things into packages, and maybe have some coordinators that can point people in the right direction based on the requirements that have to be fulfilled. There is a lot of redundant code and having quality over quantity is probably better in terms of quality of the code and that focus is on a smaller codebase, which would result in less bugs and maybe more documentation and so on for the projects as well.

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [19 Apr 14:21 GMT+2]

A walk up the pyramid

So, this Sunday I decided I could play a little around with new technology that I haven't tried much. I've worked with Zope and Plone for years, and it is important stay somewhat in touch with other Python web frameworks - to seize opportunities there as well.

I've seen some buzz about the Pyramid project and especially that it is very promising technically, and lightweight (when something says light-weight in software I think OK, somebody knows what they want and are doing).

So, first thing was to install a fresh Python, as the standard Python setup didn't include easy_install. Installed Python, installed ez_setup.py and then did a 'easy_install -u pyramid'. So far so good. :)

OK, so onto testing this thing out, I followed the first example in this page:

http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch...

and modified it so that the server was running on port 1234, because other ports were taken.

Accessing

http://blogologue.com:1234/hello/man

Gave the result

Hello man!

and yes, that's how easy it was to get up and running. :)

For a while now I've been contemplating building a simple URL shortening service (yes late to the party I know), so I decided I could try to create such a service using Pyramid.

I ended up with this code:

from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
from pyramid import httpexceptions

global urls
urls = []

def redirect(request):
    raise httpexceptions.HTTPFound(urls[int(request.matchdict['index'])])

def index(request):
    try:
        add_url = request.params['b']
        referer = request.referer
        global urls
        urls.append(referer)
        return Response("""Shortened URL: http://blogologue.com:1234/r%s""" %\
                        (len(urls)-1))
    except KeyError:
        return Response("""<html><head><title>Redirect service</title></head>   
    <body><h1>Redirect services</h1>                                            
    <p>Hello there, welcome :)</p>                                              
                                                                                
    <p>To get started using this URL shorting service,                          
    please drag the following link to your bookmark                             
    bar, and click on the link on whatever page you'd                           
    like bookmarked and accessible via this URL shortening                      
    service.</p>                                                                
                                                                                
    <p><a href="javascript:document.location='http://blogologue.com:1234/?b'"   
         >Shorten URL</a></p>                                                   
    </body>                                                                     
    </html>""")

if __name__ == '__main__':
    config = Configurator()
    config.add_route('index', '/')
    config.add_view(index, route_name='index')
    config.add_route('redirect', '/r{index}')
    config.add_view(redirect, route_name='redirect')
    app = config.make_wsgi_app()
    server = make_server('0.0.0.0', 1234, app)
    server.serve_forever()

Which is running on http://blogologue.com:1234 now. Give it a try, in total I think I spent 1 hour setting up Pyramid and getting this URL shortening service running. That is very interesting and promising, think I'll have to find some project to get more familiar with Pyramid.

[Update..] See in the comments section for an updated code example that works harder to get an actual URL.

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [20 Jan 10:15 GMT+2]

Learning things the hackish way

I've been studying more Spanish the last couple of months, and take it as I guess a part of the job, to increase our website reach and open up opportunities in one of the major world languages.

So, I've been using various places, products and services to learn Spanish. One of the most recent services I've started to use is Livemocha, and signed up for one of their new year offers for cheap access.

Well, all is good. But then I accessed the website and remembered that, ah, they *require* version 11.3 of the Flash Player, not 11.2 which is the highest version that will be available on Linux.

OK, so after some quick thinking, I thought I could edit the flash player binary plug-in using emacs, because the version string is available in the browser about:plugins page. So searching for the last part of the version string (version was 11.2 r202), r202 through emacs and replacing the 2 with 3 where I could see the 11.2 or 11,2 was, did the trick. I was able to fool the flash plugin and Livemocha that I was indeed running 11.3 and videos etc. worked fine. I read somewhere that Adobe were abandoning Flash but can see now that they have grand plans, just for fewer platforms. It's interesting to see how they are basically shafting a lot of the customers that were depending on Flash as a true cross-platform, browser-based development and runtime environment.

http://www.w3schools.com/browsers/browsers_os.asp shows that Linux is at 4-5%, alone for Linux the numbers are significant, including Android they are staggering.

Anyway, enough about that, I think it's safe to assume they won't be changing Flash or adding new features on minor versions (so 11.<whatever> should be safe). Just fixing bugs, improving performance perhaps.

But, this is a Python post, so to the point. I searched for binary diff for Linux on Google but couldn't find anything useful.

Thinking this was a simple, byte-by-byte comparison I decided I could write my own script, as shown here:

"""
#!/usr/bin/env python
import sys

try:
    script, first, second = sys.argv
except ValueError:
    print 'diff.py firstfile secondfile'

first_data, second_data = open(first, 'r').read(), open(second, 'r').read()

first_size = len(first_data)
second_size = len(second_data)

if first_size != second_size:
    print 'Files differ in size', first_size, second_size

for index in range(first_size):
    first_ = first_data[index]
    second_ = second_data[index]
    if first_ != second_:
        print 'differing in position %s, first has %s, second has %s' % (index, ord(first_), ord(second_))

print 'finished'
"""

When running this script, it gives:

morphex@laptop:/usr/lib/flashplugin-installer$ ./diff.py libflashplayer.so libflashplayer.so.bak
differing in position 21225, first has 51, second has 50
differing in position 15618949, first has 51, second has 50
differing in position 15957965, first has 51, second has 50
differing in position 16008569, first has 51, second has 50
differing in position 16008652, first has 51, second has 50
finished

a list of the differences between the two files. I dropped handling files of different size, as I guess it is a point that things in the file need to be in the same place. Replacing one byte one place in the file with two for example seems like a bad idea, the OS or program using the binary file could depend on correct positions throughout the file.

You can see that the script reads up both files into memory, it would probably be better to read byte-by-byte but file reading, blocking etc. were a bit too much to go into for a simple comparison script, so another argument for skipping files of different size.

So there you go, hope you found this interesting and fun. :)

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [10 Jan 19:32 GMT+2]

Functional testing - there's no excuse now

I've developed things for a long time, but never really got bit by the testing part of development. But now, after having some bugs arise at an inopportune moment (software running on several major version of other software), I decided that it was time to get serious about quality control and thus testing.

I looked around here and there and eventually found the zope.testbrowser package which promised to be able to work with any website, so OK. And it does. I know this blog entry is also published on planet.python.org so bear with me, even if it says zope.testbrowser it doesn't depend on a big part of the zope framework, just little bits of it.

This was on an old account, so I had to unpack and build Python 2.7 first, and then download and run ez_setup.py ( from http://peak.telecommunity.com/dist/ez_setup.py ).

OK, so I did that and then ran easy_install to install zope.testbrowser in the current directory:

PYTHONPATH=. ./usr/bin/easy_install --install-dir=. zope.testbrowser

And then created a tester.py in the current directory like this:

from zope.testbrowser.browser import Browser
b=Browser('http://blogologue.com')

and then ran Python with

PYTHONPATH=. ./usr/bin/python -i tester.py

To have it include the current directory in sys.path. Running Python this way runs the script and then lets you play with the b variable.

An interactive session can be like this:

# Get the first (and only form)
>>> b.getForm()
<zope.testbrowser.browser.Form object at 0x8599a6c>
# _ is the last returned value - you know this
>>> form=_
# Get the search_string input element
>>> form.getControl(name='search_string')
<Control name='search_string' type='text'>
>>> search=_
# Look at what we can get
>>> dir(search)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__implemented__', '__init__', '__module__', '__new__', '__providedBy__', '__provides__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_browser_counter', '_enable_setattr_errors', 'add_file', 'browser', 'clear', 'disabled', 'mech_control', 'mech_form', 'multiple', 'name', 'type', 'value']
# Pretend we're entering python as a user in a browser
>>> search.value = 'python'
# And submit the form
>>> form.submit()
>>> dir(b)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__implemented__', '__init__', '__module__', '__new__', '__providedBy__', '__provides__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_changed', '_clickSubmit', '_contents', '_counter', '_enable_setattr_errors', '_findByLabel', '_findByName', '_get_all_controls', '_start_timer', '_stop_timer', 'addHeader', 'contents', 'cookies', 'follow', 'getControl', 'getForm', 'getLink', 'goBack', 'handleErrors', 'headers', 'isHtml', 'lastRequestPystones', 'lastRequestSeconds', 'mech_browser', 'open', 'post', 'raiseHttpErrors', 'reload', 'timer', 'title', 'url']
# Here we're checking that some of the contents of
# the returned page is correct, ironically for this
# example search doesn't work properly, but that's
# what you can expect when you don't run a lot of
# tests. :)
>>> 'Search results for <em>python</em>.' in b.contents
True
>>>

That's about as simple as it can get IMO. :)

Reading through the zope.testbrowser.interfaces gives a good explanation of different methods and properties that are available, so just go read that if this is intesting.

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [10 Oct 20:24 GMT+2]

Discovering sets. . What? That fast?

There has been some dialogue on the posts on this blog, especially when it comes to Python, ways of coding and speed.

Speed has never been a big deal for me when I've worked with Python, it has been fast enough so the trade-off between programmer time costs and hardware costs has been an easy one.

However, as I look around to learn more about Python I've blogged about it and gotten amongst other, a comment about Python and sets.

Sets looked nice to me, to do unions, intersections and other things that are sometimes necessary.

However, I did not expect to find what I did find when running some speed tests..

If you run this code in a module called time_sets.py:

list_ = [1,2,3,4,5,6,7, 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j']
set_ = set(list_)
tuple_ = tuple(list_)

import timeit
repeat = range(10000000)

def in_sequence(sequence, item, repeat=repeat):
    for x in repeat:
        item in sequence

if __name__ == '__main__':
    print 3 in set_, 'j' in set_

    runner = timeit.Timer(setup='import time_sets', stmt='time_sets.in_sequence(time_sets.list_, 3)')
    print runner.timeit(number=1)
    runner = timeit.Timer(setup='import time_sets', stmt='time_sets.in_sequence(time_sets.tuple_, 3)')
    print runner.timeit(number=1)
    runner = timeit.Timer(setup='import time_sets', stmt='time_sets.in_sequence(time_sets.set_, 3)')
    print runner.timeit(number=1)

    runner = timeit.Timer(setup='import time_sets', stmt='time_sets.in_sequence(time_sets.list_, "j")')
    print runner.timeit(number=1)
    runner = timeit.Timer(setup='import time_sets', stmt='time_sets.in_sequence(time_sets.tuple_, "j")')
    print runner.timeit(number=1)
    runner = timeit.Timer(setup='import time_sets', stmt='time_sets.in_sequence(time_sets.set_, "j")')
    print runner.timeit(number=1)

Results:

True True
1.43136191368
1.42777395248
0.926907062531
10.305590868
10.6528999805
0.954577922821

To test speed of "if item is in sequence" operations, you'll see it took about 10% of the time with sets compared to lists and tuples. I expected tuples to be faster because they are immutable, but that lists and tuples were over 10 times slower than sets when testing for whether an item was in sequence I never expected. And it looks like to me that the difference in speed will be greater the bigger the sequence is.

Whoa.



[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [06 Aug 12:27 GMT+2]

Printing and recursion, a printer for list, tuple and dictionary

Over the time I've followed on PyPi on Twitter, there has been a number of entries regarding printing of lists. Now I know there is a pprint module and that it works well, but I decided to see if I could create a one-function printer of lists (and dictionaries as well) with the help of recursion.

Some 30 minutes later I ended up with this:

import types

def printer(data, prefix='', prefix_extend='  ',
            sequence_types = (types.ListType, types.TupleType),
            dictionary_type = types.DictionaryType):
    if type(data) in sequence_types:
        for item in data:
            if not type(item) in sequence_types:
                if not type(item) == dictionary_type:
                    print prefix, item
                else:
                    printer(item, prefix=prefix+prefix_extend,
                            prefix_extend=prefix_extend)
            else:
                printer(item, prefix=prefix+prefix_extend,
                        prefix_extend=prefix_extend)
    if type(data) == dictionary_type:
        for key, value in data.items():
            if not type(value) == dictionary_type:
                if not type(value) in sequence_types:
                    print prefix, key, ':', value
                else:
                    printer(value, prefix=prefix+prefix_extend,
                            prefix_extend=prefix_extend)
            else:
                print prefix+prefix_extend, key, ':'
                printer(value, prefix=prefix+prefix_extend+prefix_extend,
                        prefix_extend=prefix_extend)

data = [1,2,3, [4,5, [6,7] ], {'a':1, 'b':{'c':3, 'd':{1:1}}}]
printer(data)

And the output is what you'd expect:

 1
 2
 3
   4
   5
     6
     7
   a : 1
     b :
       c : 3
         d :
           1 : 1

Now, if you're working with deeply nested sequences and dictionaries, it might be an idea to go more object oriented.. but that depends. d:]

Any better ones?



[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [29 Jul 14:33 GMT+2]

List comprehensions vs filter and for loop [update 2]

I've been looking more into the use of list comprehensions lately, as I was looking to learn a bit more about Python. I think list comprehensions "look good" but wondered a bit about why it was necessary to add more variation to the Python syntax (thinking less is more).

One of the arguments for list comprehensions is that is executes faster than a for loop or a run through the filter function for example.

Well, OK. So I sat down and wrote this script, called list_comprehensions.py (needs this name to work properly).

import timeit

def simple_range_lc():
    return [x for x in range(100) if not x%2]

def simple_range_filter():
    return filter(lambda x: not x%2, range(100))

def simple_range_for():
    result = []
    for x in range(100):
        if not x%2:
            result.append(x)
    return result

if __name__ == '__main__':
    runner = timeit.Timer(setup='import list_comprehensions',
        stmt='list_comprehensions.simple_range_lc()')
    print 'list comprehensions', runner.timeit()
    runner = timeit.Timer(setup='import list_comprehensions',
        stmt='list_comprehensions.simple_range_filter()')
    print 'filter', runner.timeit()
    runner = timeit.Timer(setup='import list_comprehensions',
        stmt='list_comprehensions.simple_range_for()')
    print 'for loop', runner.timeit()

Running this script on my laptop with nothing else big running yields these results:

list comprehensions 20.1266300678
filter 31.8105700016
for loop 26.9202849865

and the percentage difference is pretty consistent. This tells me that filter uses 50% more time than list comprehensions and a for loop 25%.

I was surprised that the filter function used 50% more time, but that's probably because it is doing a function call.

So, some food for thought, and a good reason to start using list comprehensions more.

[Later..]

DaveE suggested running a test with generators as well, with this snippet:

def simple_gen():
    def gen():
        for x in range(100):
            if not x%2:
                yield x
    return list(gen())

The timing results now were:

list comprehensions 19.3058991432
filter 31.7537689209
for loop 25.6983029842
generator 25.2884697914

Interestingly the generator is a wee bit faster than a pure for loop, I'd like to see some suggestions on why this is. :)

[Even later..]

Nathan Wright wrote in and suggested that the list append method lookup could be sped up, and it does speed things up just a little bit, see the simple_range_for2 function below. Stephen Simmons quite correctly pointed out that not all the functions used a function to test if (not x%2) was true. This was discussed in the blog post as well, that being able to "embed code" instead of doing a function call was a benefit of list comprehensions (and for loops as well).

See the updated code below and the results now that all iterators are using a function call.

import timeit

def simple_range_lc():
    test = lambda x: not x%2
    return [x for x in range(100) if test(x)]

def simple_range_filter():
    return filter(lambda x: not x%2, range(100))

def simple_range_for():
    result = []
    test = lambda x: not x%2
    for x in range(100):
        if test(x):
            result.append(x)
    return result

def simple_range_gen():
    def gen():
        test = lambda x: not x%2
        for x in range(100):
            if test(x):
                yield x
    return list(gen())

def simple_range_for2():
    result = []
    result_append = result.append
    test = lambda x: not x%2
    for x in range(100):
        if test(x):
            result_append(x)
    return result

if __name__ == '__main__':
    runner = timeit.Timer(setup='import list_comprehensions',
        stmt='list_comprehensions.simple_range_lc()')
    print 'list comprehensions', runner.timeit()
    runner = timeit.Timer(setup='import list_comprehensions',
        stmt='list_comprehensions.simple_range_filter()')
    runner = timeit.Timer(setup='import list_comprehensions',
        stmt='list_comprehensions.simple_range_for()')
    print 'for loop', runner.timeit()
    runner = timeit.Timer(setup='import list_comprehensions',
        stmt='list_comprehensions.simple_range_for2()')
    print 'for loop 2', runner.timeit()
    print 'filter', runner.timeit()
    runner = timeit.Timer(setup='import list_comprehensions',
        stmt='list_comprehensions.simple_range_gen()')
    print 'generator', runner.timeit()

And the timing results this time with all tests being function calls:

list comprehensions 38.9694008827
for loop 42.6255960464
for loop 2 39.0485188961
filter 38.9864330292
generator 45.6232538223

Which shows that things are pretty much the same when all sequence handlers are using a function call to test whether to include the item in the new sequence. Skipping the attribute lookup in simple_range_for2 gives a few % speed up. But this is interesting, it seems to me that list comprehensions should be used where possible, as they make it possible to "embed" expressions instead of using function calls.



[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [15 Jun 15:59 GMT+2]

Are those eggs on my face? Proper development-release cycle

So, I've been working to release some packages today. In this case they are some eggs for Plone, which enable easier editing and creation of content types in Plone (see Products.MegamanicEditContentTypes).

I use PyPi for things we release to the community, and other customer sites that would like to use the same eggs get their buildout (zc.buildout) updated to fetch these packages.

Well, things kind of blew up today as I was updating the customer site which was using a more recent version of Plone than the one I was developing on, some renaming changes broke some of the templates so it ended up with a little bit of stress to get things working.

We have SVN where we track changes to the code so that's covered, but the release and distribution parts needs to be improved..

I see there is

http://www.python.org/dev/peps/pep-0386/

for version numbering, and that's fine.. I guess the right thing to do is to always have code eggs in an alpha status (1.0a for example) until the code has been tested on at least one different site than what it was developed on and then after a period of time, maybe 1 week of testing and bugfixing (after alpha feature freeze), create a beta (1.0b) and again after 1 week of beta testing release a release candidate (1.0rc) and after another week, create a final release (1.0). And test the code properly on the different major platform versions.

I think it's important for us at least to have a somewhat rigid testing and release system so that users of our products can depend on the quality (freedom from show-stopping bugs) of the eggs.

How do you manage these things?

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [18 May 11:08 GMT+2]

Python style & flexibility, tabs or spaces

I've been programming Python for a good while, and in the beginning I was very occupied with the style in which things are written.

I always advocated using_names_like_this for everything, I guess because it is easy to read and understand, and it also takes into account acronyms which should be uppercase, as these acronyms are also separated from other words, for_example_html_text.

However, over the years, mostly these last couple of years, naming conventions aren't that important any more. Python is a very flexible language, and one can for example create a function that returns an instance of a class, the class being decided by the invoked function.. With this flexibility, separating functions from classes is somewhat redundant IMO, and one might just as well do everything in lower-case separated by an underscore.

However, I'm not riding these arguments anymore, there are a number of different Python systems such as Zope, Django, Repoze, Archetypes, Plone etc. that use a myriad of different ways of naming things.

I think the most important thing is, and has always been, the proper use of tabs and spaces in Python files, as this is the thing that can get you and has real impact on the "flow" of the program.

I think there are a couple of times in my Python career that I've encountered bugs due to mixed and erroneous spacing, and at least one of those bugs was pretty tough to get, but it's been a good while since that one. I don't think much about spacing any more, Emacs does the right thing whether I'm in text or Python mode.

One snag that always seems to get me is the use of commas in complex schema definitions in Archetypes for example, but that's thankfully always an error that stops the entire system so it's easy to catch.

My argument? Naming conventions aren't that important because there will never be enough willpower and resources to make everything completely consistent in the Python ecosystem.. and if one is very preoccupied with naming conventions, maybe it is time to drink a little less coffee, stress a little less and focus on the important parts of a program. [:)

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [15 May 09:00 GMT+2]

Minimalistic Python, Python as a system administration language

Some months ago there was a leap year day and I didn't have a lot to do right then, so I was following a guy on Twitter that is into command line tricks. So I started writing an as small as possible script that could display leap years and would fit in a Twitter message.

Python has always been called a scripting language I guess, but I've used it for serious programming for many years, dabbled a bit with Perl and other things but Python is the language I'm most comfortable with. Javascript is a language I work with now and then, it has its uses in web pages and apps, although I've been daydreaming at times about writing a Python interpreter in Javascript. [:P

Anyway, after some iterations refining the code I ended up with this:

echo "r=range(-8,2020,4);l=lambda y:((y in r and (y%100 or not y%400)) and str(y));print filter(None,[l(y) for y in r])"|python

[Edit: It could be a lot shorter, but takes into account more advanced rules for leap years]

It's something that should work instantly on most recent Linux distributions (all have Python don't they?) The python command at the end accepts what's sent through a pipe with echo, and with some builtin Python functions, a lambda function, modulo, Python's evaluate object to boolean and list comprehensions features one can build such a script (program) in a limited amount of space. There are only 127 characters that could go into any shell.


[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [08 May 15:17 GMT+2]

New beta release of email_backport

So, email_backport, my effort to make some products available spanning two major versions of Plone has had a new release.

http://pypi.python.org/pypi/email_backport/

There were some issues with imports and things working properly (due to another module in the system being called email) so I griped a bit with it before figuring out that it was a naming conflict.

In the process I ended up cleaning up the email_backport import code, so it should be better than what used to be there. After trying to send an email message I got an AttributeError that the string object was missing a partition method.

Well, I looked at the documentation for the partition method and found that I could alter the message.py code to use .split instead of .partition and now things are working well. Feedback is appreciated.


[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [24 Apr 11:04 GMT+2]

email_backport - modern email package for older Python versions

So I created a backport of the 4.x version (4.0.3) of the email module for Python >= 2.3.. I was working on a Plone site version 3.3.5 and creating an email sending application..

As I read on the email module's documentation page

http://docs.python.org/library/email

The most recent version of the email module was not available for Python 2.4, which the site was using.

So I read up on the documentation and figured that I could create a backport of the package which is available for Plone 3.3.5.

Some moving of files and editing of the email module's __init__.py file was needed, as well as a way to deal with those things in existing code that would use the backported version of the module if available.

The result is here:

http://pypi.python.org/pypi/email_backport/4.0.3b

And as you can see, it's the same version as the most recent 2.x version of the email package, but with a b(eta) because it hasn't been tested a lot (but seems to be working just fine).

[Later..] OK, have gotten some downloads of the package, so there is some interest for it. More downloads than I expected, which is cool.

[Permalink] [By morphex] [Python-only or > 0.5 Python related (Atom feed)] [12 Apr 17:27 GMT+2]

Python-only or Python related

OK, so here's a new category mostly about Python things, that don't belong in the Zope/Plone category.

[Permalink] [By morphex] [Development (Atom feed)] [12 Apr 17:25 GMT+2]