jp's domain

The problem with programming

What is the greatest problem in computer programming today? Is it the impending 'concurrency crisis'? Is it reducing bug counts? Is it shifting over to a programming language that has the full power of macros? Is it type inference in statically-typed languages? Is it dealing with licensing issues?

All of these are genuine issues, and they are things that need to be looked at and are, but I would contend that the major problem in programming today is the waste of millions of programmer hours. What do I mean by this? Well, let's say I'm a Common Lisp programmer, and I need a library which will read Excel files. Such a library might exist in Python or Java, but it doesn't in Common Lisp, and if I really need that functionality, my only real choice is to write that library myself. That's not the worst thing that can happen, but I don't know much about the Excel format on top of the fact that writing such a library is a job that could literally take hundreds of hours. Now remember, my original goal was perhaps to write a general-purpose file analyzer, and not to write an Excel-reading library. What a waste of time!

We see this similar problem throughout the programming world. Perl has its own libraries to read CSV files. As does Python and Ruby. In an ideal world, if it was possible to easily use Perl's CSV libraries from Python, would days of work have been spent working on Python's CSV library? Probably not. More time could have been spent fixing what existed.

It doesn't end there!

This isn't the only problem that this Babel of programming has caused. Ruby's interpreter is widely seen as very mediocre. Due to pretty major speed issues, as well as other concerns, a new Ruby interpreter is being written, YARV, which is slated for a Christmas release this year. Writing interpreters and virtual machines isn't easy. Python has its own, Perl has its own, PHP has its own, and they all have their own little boxes of libraries and communities where it is nearly impossible to use a library between one language and the next. Some languages sidestep the issue by compiling down to native code instead of running off a virtual machine, but the result is the same. Essentially if you want to write a library that is easily usable between many different languages, you need to write it in C. C isn't bad, but it also isn't 1975 anymore, either.

Back to those language implementations. Much uproar has been raised in the Python community this past week over the global interpreter lock on the Python virtual machine. Basically the GIL makes it difficult to write multi-threaded programs in Python, but it does make the VM much more efficient and it allows much easier writing C extensions to be considerably easier. The creator of Python, Guido's response was essentially, "I'd love to have this feature, but it is a hard job, will take a lot of work, and I simply don't have the time to do it myself". Guido also has a different idea on the most effective way to use multiple cores, but that is besides the point. The point is, adding this feature of Python is something that will again take hundreds of man-hours. Ruby, with a similar design, is looking at the same issue in the near-future. What a crappy way of doing things! As programmers, if we have too much duplication in our code, our first response is to rightly want to kick our own ass. The whole idea of programming is to reduce duplication as much as possible, which is why we have variables, and functions, and some languages have macros. "Don't repeat yourself" is a principle most of us work by. I'm not sure if I live by it...eating ice cream twice in the same day isn't so bad. Anyway...

A solution?

As you can see, this is a pretty big problem. Instead of improving on what already exists, we as programmers are spending exorbitant amounts of time, money, and effort duplicating libraries and features found in other programming languages. There's got to be a better way of doing this.

Parrot has long been slated as the solution. Essentially it is a language-neutral VM which should make it easy to run all of these languages on the same platform, with the ability to easily use libraries between them. So there is a light at the end of the tunnel! Well, that's what Parrot will be, eventually. Parrot has been a work in progress since 2001, and it isn't really clear whether it will be ready in the next six months or the next five years. In addition, its focus is dynamic languages, so it isn't clear how well static languages will fare on the VM. With Parrot, much like, Perl 6, there's an air of uncertainty, and even when it hits its first 'production' release, there is no telling how fast uptake will be or what issues will be faced. When mentioning Parrot, it is worth noting that both PyPy and Rubinius have some similar potential but slightly different goals along with that same dash of uncertainty.

But of course there couldn't be an existing software project without Microsoft having something similar. Microsoft's .NET project and its related DLR (Dynamic Language Runtime) also aim to make it easy for different languages to work together. Not only does C# run on the .NET runtime, but there is also the OCaml-inspired F#, IronPython, IronRuby, IronLisp, JavaScript, Boo, and several other languages on the platform. Perhaps most importantly, .NET works now, and it does have major backing. .NET isn't restricted to Windows, either. The Mono project is a Microsoft-supported port of .NET to Linux.

last updated 11 months ago #

Over the past year or so I have spent a whole lot of time evaluating different web frameworks, trying to find what works best for me. Like many programmers, I am a bit of a perfectionist, but I am also a pragmatist. I want my tools to work absolutely right every time, and I want my tools to help me effectively get certain tasks done. However, if you've programmed much at all, you know that perfection isn't something that you are likely to find, as perfection to one person is different to what perfection for another person is. In the end you can only pick what works best for you.

Compounding this decision process is the fact that in Python web development, there are quite a few choices. Now some people have criticized Python for this abundance of options, but I do not believe it is in a bad thing. We have seen how in the Ruby community, since there is realistically only Rails, Rails tends to dictate what is 'good' and what isn't. Rails itself hasn't added a lot of new functionality over the past few years. On the other hand, the numerous frameworks in Python have provided different ways of looking at things and it has ended up creating numerous cycles of improvement on existing components. This ultimately can only be good for the Python web community. If it ever boils down to there being a single framework, things will probably grow stagnant, but the forces of multiple frameworks will only improve all that are involved. But this doesn't help the programmer that is right now, in 2007, looking for a good tool to use. In this post I hope to make an objective analysis of the different frameworks available in the Python community and to hopefully give the reader a better idea of what tool will work for them.

I'm going to break this article down into 5 main sections: Zope and web.py, roll your own framework, Django, Pylons, and TurboGears. In each individual section (I don't do this with the first), I will break down each framework by:

• the general philosophy of the framework
• the URL dispatching system used
• the templating system used
• the object-relational mapper used
• the form handling used
• extra types of functionality
• advantages of using the framework
• disadvantages of using the framework
• the future direction of the framework and concerns I have with it

I attempt to be as fair and objective as possible, as well as to be critical while remaining positive.

Note: The first time I mention a project I provide a link, but because I refer to different projects multiple times, the first time is the only time I will link it. You can scroll back and find the link or just use Google. Sorry, I am lazy. :)

Secondary note: This is by no means a scientific analysis and it includes my own opinions and and mistakes. Hopefully people will speak out and correct them and bring forward their own opinions in the Reddit comments tied to this link.

First, something big and something small

For most users, I think there are 3 main frameworks worth checking out, that is: Django, Pylons, and TurboGears. But there are also a few alternatives, and I would like to briefly mention them. First of all there is Zope. It is what Python web development at one time was instantly associated with. Now you don't hear it discussed nearly as much, and for many people it might seem to be irrelevant. But Zope still has a good-sized community, and it also has great support, so it is perhaps the 'safest' choice. Also, the proliferation of frameworks recently have caused Zope to focus on modularization and simplification, and a lot of Zope components are being split out into eggs of their own. It is possible that Zope components might at some point become something that people use regularly within the other frameworks. Somebody who just wants to build a simple blog or other basic application would probably find Zope overkill, but a bigger company may find it very useful.

In addition to Zope there is web.py. Now web.py is on the extreme other end of the spectrum from Zope, and was originally just a single file released by Aaron Swartz at the end of 2005 after being written to power Reddit.com. Its foremost goal is simplicity and many people like it for that. It appears that lately certain features like a templating language has been added, and it is a great choice if you just want something to hack around with or you just want something that is absolutely miminal. Unfortunately it probably has the smallest amount of users of any of the more recent frameworks, and most people need their framework to do a little more. It also probably wouldn't be the best choice to build company applications with.

Roll your own

However, just because you need a little bit more functionality out of your framework, it doesn't mean you have to use one of the larger ones. Python has a plethora of different components that anyone can plug together to create their own little custom framework to fit their exact needs (infact these components are used throughout the larger frameworks and there is nothing stopping you from pluggin them into Zope or web.py).

Philosophy

Who knows what is best for me or my app besides me?

URL dispatch

Routes (Rails-style URLs), selector (feels a lot like Django's URL system, but a bit simpler), and more fully-featured systems like colubrid (multiple dispatch styles), CherryPy (object-publishing and more), or Spyce

Templates

Genshi (XML-based), Jinja (Django-style templates), the venerable Cheetah, Mako (about as close as you can get to Python in templates; the successor to Myghty), and others

ORM

SQLObject, the popular SQLAlchemy, interesting sytems like Schevo, and useful tools like Migrate and Elixir (an SQLObject-ish declarative layer to be used on top of SQLAlchemy)

Form handling

FormEncode (form validation and data conversion), FormBuild, Toscawidgets, and WebHelpers (Rails-style helper functions)

Extras

And finally for the remaining components like WSGI request/response objects, WSGI middleware, app creation and deployment, there is Paste and its libraries including Paste Script and Paste Deploy as well as smaller projects like Yaro. In addition to all these components you can theoretically use the different parts of Django in your own framework.

Advantages

Basically there is nothing stopping you from just putting something together yourself, and this may satisfy some of the more hacker-minded types out there. Joe Gregorio has written an excellent article describing how he put together his own 'throw away' framework in about 3 hours and about 60 lines of Python code. To me that is a very powerful testament of just how good the different framework components are. And what is even better is most of them have excellent documentation.

Disadvantages

As Joe warns, this approach doesn't buy you everything, "We don't have an instant admin interface, we don't have generic views, automatic form generation, automatic form handling, the Django community, bug tracking, IRC, etc, etc." Community, support, and already-tested code shouldn't be overlooked. And then there are the little things. Is your framework thread-safe? If your application gets used a lot, does your framework integrate nicely with memached like the major frameworks? Rick Copeland has written an interesting piece detailing the problems in writing your own framework.

Future and concerns

Rolling your own framework will never go away. It is a very attractive option and can work well in certain situations. I think if Paste Deploy and its sister libraries pick up more steam, and as modularization of components continues, this will be seen more and more. Hopefully with WSGI as a basis, this will act as a boon to innovation and not a bane. Ideally it shouldn't matter exactly how you create your app, as long as it integrates nicely along with other WSGI applications there isn't an issue. We see this already with some people running the Django admin tool on top of applications written in other frameworks.

Django

Note: 0.96 released a few hours after I got done writing this. It shouldn't affect much here other than when referring to 0.96 in the future tense.

So in alphabetical order, we shall start with Django. Now in the past I've been overly critical of Django and I don't really think that is fair, because my criticism was more over different ideologies in design rather than the quality of the framework itself. Django is a solid project that will provide really everything you need in a web framework except for AJAX (unlike Rails in this respect; Django leaves this up to you). Django is without a doubt one of the more vibrant projects in the Python community.

Philosophy

You can't make everyone happy, but you can focus on providing a complete set of components that is well-documented and integrated.

URL dispatch

Django's dispatching is based around pure regular expressions. You write a set of regular expressions and when your application is triggered, it searches from top to bottom through the regular expressions to see if the URL matches one of them. It then dispatches to a function that you provide. This approach is both extremely flexible and fairly simple, at least conceptually. Most URLs aren't too complex, yet if you have a legacy application it is very easy to integrate Django with it. In addition the mapping of URLs to pure functions allows some neat things like the wrapping of these functions in common bits of funtionality (such as creating an object, listing an object, deleting an object, etc), in what Django calls generic views.

Templates

The entire idea behind Django's templates is to keep them as simple as possible. As has been stated several times, the last thing the developers want to do is just to create another programming language. Django templates provide minimal functionality because they are not designed for application logic, they are designed for formatting. As a result Django's templates work very well in a situation where programmers and designers work separately. Programmers who do both programming and design might feel a little handicapped by these, however, which isn't necessarily a bad thing as they do limit the application logic you can do in the template itself (it shouldn't be there). New functionality is added via tags and filters; tags providing more complex functionality such as looping, filters acting much like UNIX pipes which change the value or appearance of a variable in some way (and yes, like UNIX pipes you can chain several together). Django provies quite a few tags and filters out of the box, and creating simple ones isn't difficult. Django also has a very good inheritance system which makes it easy to create a base template that contains specific parts like header, body, sidebar, and footer that children templates can fill in. If you do find yourself wishing for a bit more power in your templates, theoretically it shouldn't be that difficult to hook up another templating system to Django.

ORM

Django's ORM allows you to define your database objects in pure Python (this will be familiar to those who have used SQLObject before). This is very simple and works pretty well in most cases. I haven't really found anything that the Pythonic-query language cannot handle, and it is enjoyable to work with. Django's ORM is not the most powerful thing out there, but it does provide functionality such as database introspection (if you already have a legacy app with a database), and transactions. There has been talk for more than a year now in building this Django DB API on top of the very powerful SQLAlchemy, but it could be some time before that is completed. In the meantime the current ORM works well in most cases, and provides support for several DB systems.

Form handling

The form handling system in the current release (0.95.1) is decent if a bit cumbersome. Thankfully this will be changing in 0.96 with the newforms system. newforms makes defining your forms feel a lot like defining your database objects (basically easy and Pythonic). Newforms will let you design your form in this manner, and then it will display it, deal with validation errors (you specify what type of validation is required), and either redisplay the form as needed or process the data by creating a new object, editing an existing one, whatever. Newforms also does a good job of separating display logic (widgets) from the form processing itself, and one of the early examples provides a TinyMCE rich Javascript control as a widet used in the place of a regular textbox. newforms makes dealing with forms so much easier, and quite simply frameworks such as Rails have nothing like it. This is one of the most awaited features of Django and for good reason. A super simple form is given below (I am being a little lazy and not looking up anything to verify that it is right, so this isn't 100% accurate):

class TestForm(Form):
    name = CharField(maxlength=20)
    phone = PhoneNumberField(required=False)
    email = EmailField(initial='...@gmail.com')

----

def view(request):
    if request.method == 'GET':
        form = TestForm()
    else:
        if form.is_valid():
            ...do magic processing here...
    return render_to_response('/template.html', {'form':form})

----

<h1>This is my form</h1>
(( form )) (should be double braces, but the formatting is messing that up)

Extras

Individual Django programs are termed 'projects' while pieces of those programs are 'apps'. Apps provide small bits of functionality, for example you might have an app for comments, an app for user registration and authentication, or an app for polls. This creates some pretty neat plug-and-play possibilities, and in fact Django comes bundled with several apps that you are free to use in your own projects. These applications by themselves give Django more functionality out of the box than most frameworks, and are why Django in some ways is seen as similar to a content management system. Perhaps Django's most discussed app is the admin app which provides a basic backend for authenticated users to enter data. It has a decent amount of flexibility, and that flexibility will be be getting even better when the admin app gets based around newforms.

Advantages

Django has by far the largest community out of any of the the major Python frameworks. The IRC channel and mailing lists are fairly friendly and it is easy to get a response to a question you might have. Django also has perhaps the best documentation out of any web framework in existence. This may be a bit of an exaggeration (though I haven't come across a framework with better docs), but it is a definite focus of the developers which is comforting for many users, especially those just getting started with a framework. In addition a book is being written which is and will be available online for free and will be put into print by Apress. Due to this documentation and the aforementioned included applications, you can get quite a bit done with Django in a short amount of time. In addition, one of my favorite things about Django is everything is kept very simple and Pythonic. The configuration files are pure Python, URLs map to simple functions, and you define your database objects in Python as well. If you know Python you should be able to pick up Django pretty quickly.

Disadvantages

The only real disadvantages one might come upon when using Django is if you don't agree with it. Django isn't designed for customization to be particularly easy, so if you don't agree with the URL dispatching, the templating language, the ORM, or anything else, while it is possible to switch them out, Django isn't going to hold your hand. This is kind of compounded by Django's components not being super powerful, so for example you might run into problems using Django's ORM with a large and complex legacy DB. Issues like this aren't things most users will run into, though.

I did find that Django got repetitive on a few counts. First of all I'm not good with regular expressions, so defining URLs led to a lot of cutting and pasting for me. This approach is also pretty error prone (oops, I forgot the ^ at the beginning of my URL!) and it isn't especially easy for new users (which flies in contrast to how Django is in almost every other component). It would be nice if there was a slightly higher-level wrapper available for URLs. I'm also not the biggest fan of the templating language. While it does its job well (that is to not be another half-done programming language), filters and tags aren't nearly as easy to write as the simple functions are that you might write to use in something like Genshi or Mako. But this is a minor problem that you can for the most part look past.

I also ran into little bugs here and there. One bug made uploading large files crash duplicate the file in memory and therefore crash the server, one bug meant that the time zone setting in my settings.py file was off by 3 hours, and a further one was a bizarre bug way down in the bottom of the ORM which led to some unexpected results. Again, these are little things, and they should be fixed over time.

My final issue was the 80% rule. It seemed to me that in many cases Django would almost do what I need but not quite and therefore I'd have to give up using a higher-level component and instead do whatever it was manually. I ran into this with both the admin application and generic views, which are ideally nice features that I would really like to be able to use. Perhaps the best way of looking at both of them is to not expect the moon from them; both have particular goals and purposes, and they aren't made to do everything. Regardless, I would hope that in the future more emphasis is placed on making these higher-level features as flexible as possible (and gladly I do see that in something like the newforms-admin branch).

Future and concerns

The immediate future looks toward the release of newforms in 0.96 as well as later additions like the newforms-admin. The SQLAlchemy branch which layers the Django ORM on top of SQLAlchemy is also worth looking forward to. Another database-centric addition is a migrations tool, which is something a lot of people have requested and something that has been worked on (known as SchemaEvolution in the Django community). For the long-term you can expect the release of Django 1.0 and it is then that the developers have discussed finally 'marketing' Django; considering that they haven't officially done so already, it is remarkable just how popular Django has gotten (really it speaks to the quality of what they've put together). There is also talk of smaller changes like modifying Django's internals to use Unicode throughout and of automatically escaping variables in templates.

My concerns with Django are mainly theoretical in nature. With Django being the most popular project in Python web programming by far, there is a bit of a responsibility to do things which will further the web community as a whole. As I mentioned earlier, it is precisely this competition between frameworks which has led to such great innovation; if Django takes steps which locks users into using the framework and keeps other frameworks from having access to certain tools, the Python community could become too focused on a single framework and things could stagnate (which is what I personally feel has happened in the Ruby world).

Before I start on the theoretical issues, I'd like to deal with practical concerns in the Django community. First of all, the entire project/application model is one of Django's most powerful features, this really needs to be harnessed in some way. Currently it is pretty difficult to find applications created by other users, and there ends up being a lot of duplicated effort. If there was an official repository of Django apps, tags and filters for templating, and middleware, a lot more effort could be focused on fixing what is existing instead of invididuals having to create their own libraries which do the same thing. Any such repository in my opinion should be based around Python eggs for easy installation from the Python Cheese Shop (there is no reason to reinvent the wheel here). This to me is perhaps the most important thing that can happen with Django. There is a massive community out there with a lot of potential code to contribute that is more or less being wasted right now.

In addition, the very structure of the Django project perhaps needs to be analyzed. There are major bugs or features that have lain in an unfinished state for over a year now. The SQLAlchemy branch was announced in August of 2006, there still has been almost no progress on it. SchemaEvolution was a 2006 Summer of Code project, it, too lies in an unfinished state and there seems to be no more progress in making it an official part of Django than there was 6 montshs ago. The bug I mentioned earlier which saw files duplicated in memory (a fairly major issue for anyone doing file uploads in a web application) was something I ran across in August of last year, but it had actually had a bug report (this ticket leads into 2070) opened over a year ago. A fix still hasn't been made available in any Django release. Things like this to put it bluntly are unacceptable because they make Django unusable for certain people. Numerous other Python frameworks have gotten file uploads right, I don't understand why it has been so difficult for Django to do so. This leads into another practical issue I have with Django: the project needs to look at using outside libraries wherever possible. If Paste does file uploads very well already, why not use its implementation? This means the actual Django developers have to worry less about small components like this and more on the overall design of the framework. In addition it means other people are outside of Django are working on improving that component. Less work, less bugs, and for Django essentially a smaller codebase. Now certain outside libraries simply will not fit Django's design, but when they do they really should be utilized, especially with something as fundamental and important as file uploads.

Things just seem to move way too slowly in the Django community. It may be simply due to the size of the project, but there is a slight feeling that the community itself does not have a lot of involvement in the further development of the framework. There's a bit of a disconnect between the developers and the community. I know that I can drop into the Pylons chat room and talk to Ben Bangert the creator of the framework, or talk to Mike Bayer the creator of SQLAlchemy and Mako and say, "Can we look at improving this?", "Can we fix this bug?", "Can you describe how I might plug in this component to the framework?". A lot of times even if they don't agree with my suggestion (this happens a lot; haha), they will explain to me what is going on, or if there is a bug I've literally seen them fix it in the codebase on the spot. I've never had this experience with Django. I've never even seen the lead developers inside the chat room. Now I can understand if they do not have to the time to do so, but I would really like it if there were a community-dev who could be around to deal with issues like this and to for example, hear and get moving on a solution to fix something like the file uploads. Again, Django has a large and enthusiastic community. Harness it! Note: This got posted to Reddit a bit sooner than I was ready. I talked to James Bennett earlier tonight, he is the release manager and also has commit rights to the source code. In many ways he can be seen as the 'community-dev' that I am referring to. He was telling me that recently the community involvement has gotten better. And what do you know, not but 5 minutes after I got talking to him another dev dropped into IRC. Hopefully this developer/community reaction does continue to improve, there really needs to be quicker turnaround on fixing errors. Also it should be noted that Django started out as a proprietary project that became open-source, so it is understandable that this is an ongoing process.

Now back to the theoretical: I would really like to see Django focus on modularization. The easier it is to use Django's components with other frameworks (such as the URL dispatcher, the templating system, the ORM), the more people will end up using them, and more bugs will be spotted and more functionality added (like more tags and filters for the templating system). If this were the case, time spent duplicating functionality to be easily used outside of Django, such as Jinja could instead be used to work on what already exists. This is a win for everyone involved. In addition I'd like to see it made easier to use components from outside Django inside Django itself. For example TurboGears and Pylons both allow you to change a few lines of code to use a different templating system. Integrating that and functionality like this into Django would be very powerful and would alleviate the issues some people (myself included) have with specific Django components.

The support of specific standards and currently existing libraries in the Python web community would also be a net gain for everyone involved. As it stands currently, while an app such as the user application in Django might be very useful to a Pylons user, apps cannot run without a Django project. If there were some way to package the application up as a Python egg (which is the standard unit of packaging in the Python community) and therefore have a very simple install (via setupttools) and deployment (through Paste Deploy), it would be another one of those situations where more features and bugs could be worked out through increased usage. Also instead of using things like a custom type of middleware inside of Django, if Django used WSGI-based middleware as the foundation, we might find that Django users have both access to more tools and user of other frameworks are able to get involved in improving existing ones. In addition, the potential importance of tools like Paste Deploy really need to be analyzed. Its goal is nothing less than to make Python web applications easier to install and run for the end user; to increase adoption of Python on web hosting environments. This is something that is vital to Python as a whole rather than any one framework, and it should be discussed in how Django can support such efforts.

To wrap these suggestions up I must once again say that because Django has such a large audience it also has great responsibility. I hope to see Django move forward continuing to improve upon what exists, continuing to improve upon the already excellent documentation, and most importantly both harnessing the community and not forgetting that there are other projects out there in the Python community with great implementations and ideas; let's try and use them where possible to improve the state of Python programming as a whole. If I sound or have sounded too disparaging of Django in the past, it is because it is something that I have a lot of hope and enthusiasm for, and that I want to see reach its full potential.

All in all, Django is an excellent framework. It is not without some issues, but the focus on documentation and having a well-integrated set of components has led to a framework that has more features than just about anything else out there and is very usable right now.

Pylons

If Zope is on one end of the spectrum from web.py, Pylons is on the other end of the spectrum from Django. The idea of Pylons isn't to provide the user with everything they need, it is to let them chose what they want. Virtually everything can be swapped out, from the URL dispatching mechanism, to the templates, all the way to the ORM - and Pylons helps the user to do so. In some ways it is the roll-it-yourself framework with the hardest things done for you.

Philosophy

What is best for you isn't necessarily best for me. But do you mind if I offer a few suggestions?

URL dispatch

I should preface this by saying that Pylons' method of laying out applications is the classic MVC style. Controllers map to classes and actions map to methods. So my business logic for handling new records would likely be modeled by the class 'RecordController' with actions that mapped to the methods 'search', 'update', whatever. This isn't as simple as Django's mapping to functions, and does not as easily lead to the utilities provided by generic views, but it provides the basic nicities of inheritance; each controller being able to inherit from a master controller which might provide default actions and other functionality.

By default, Pylons uses the Routes URL dispatcher. This system will be very familiar to anyone who has used Ruby on Rails before (it is more or less a direct port). This isn't quite as flexible as pure regular expression matching, but it makes up for it in its basic simplicity and the nice defaults it provides. For example, every Pylons application comes with this route already setup for you:

map.connect(':controller/:action/:id')

This is very nice because it will do virtually all of the basic CRUD style URLs for you without having to specify them. If I have a 'user' controller, for example, with the actions 'new', 'delete', 'edit', I don't have to write a single line of code to map them. It would work the exact same if I added a job controller. Routes however is much more powerful than this, and also supports more advanced features like matching subdomains, matching specific HTTP methods, and even provides a way to generate REST URLs out of the box.

You can ideally switch out the URL dispatching and resolving with something else, and the ability to do so is improving, but this is probably the component that is the most difficult to modify, and that mainly has to do with URL dispatching being so fundamental to the layout of an application.

Templates

Pylons currently by default uses the Myghty templating system. Myghty let's you use regular Python code in your HTML templates. It also has some pretty advanced inheritance features and other functionality. Myghty is modeled after Perl's popular Mason templating system, and as a result has some complicated areas and is not very 'Pythonic' in certain ways. This is why the creator of Myghty, Mike Bayer (the same guy who has done SQLAlchemy, he's a coding machine!), has been working for some time on a successor called Mako. Mako is a good deal simpler and more Pythonic while still retaining the power and functionality from Myghty. It is also one of the fastest templating systems Python has. It wouldn't be inaccurate to consider Mako a better Cheetah.

At some point in the future Mako will be replacing Myghty as the default templating system, but for now Myghty is the default and Pylons' tutorials use Myghty. Using Mako now instead is literally as simple as changing 2 or 3 lines of code. And again, there is nothing stopping you from just as easily usinga different templating library. Pylons is made for the end user to customize it.

ORM

By default Pylons doesn't really come with an ORM. You can enable SQLObject by uncommenting a few lines of code, but most of the tutorials use SQLALchemy, and it is the suggested ORM and likely will be the main ORM Pylons focuses on in the future. SQLObject is roughly equivalent to Django's ORM, but SQLAlchemy is extremely powerful. It gets this power from not doing a simple mapping of tables to classes and records to instances, but in rather allowing the user to map objects to parts of the database themselves, including items like relations. There isn't too much SQLAlchemy cannot do, unfortunately this power comes at a price. SQLAlchemy isn't the simplest database system to understand or use, and it has several ways of doing things. You can use the lower-level functionality and do the mappings yourself, or you can use slightly higher level functionality and let SQLAlchemy do much of the mapping for you (assignmapper). Finally, if you really want something that feels very Pythonic and easy to use like Django's ORM, there is the recent project Elixir.

This proliferation of possibilities is unfortunately a slight problem in the Pylons community. Different tutorials use different ways of doing things and if you want to use SQLAlchemy in your project there's a bit of manual setup involved which is different depending on what exact method you are using. This is pretty difficult on the new users, but once you've decided upon what you want, the Pylons user has a lot of power in maniuplating their database.

Form handling

Pylons currently uses the WebHelpers library to help to generate small pieces of HTML, including form items. These are based on a port of Ruby on Rails' helpers, and they work fairly well as well as being easy for the user to write their own (they are after all only simple Python functions). The following piece of code is an example of generating a text box.

>>> text_area("body", '', size="25x10")
<textarea name="body" id="body" cols="25" rows="10"></textarea>

Pylons uses Ian Bicking's FormEncode to do validation of HTML forms as well as the conversion of text input into their correct Python values. You specify a schema for your form (this feels similar to how Django's newforms works), and them FormEncode knows what to expect. FormEncode doesn't, however, generate your form for you. That is up to either you to do manually or to use a tool like FormBuild. In the future, however, Pylons will be using Toscawidgets, which is Turbogears widgets system for 2.0 that is designed to work with multiple frameworks. These widgets will allow you to put the logic of display and validation of forms and other HTML elements into their own small bundles. It also will deal gracefully with AJAX. A good way of looking at Toscawidgets is as a slightly more powerful version of Django's newforms system (newforms is designed only to handle forms while Toscawidgets has broader objectives).

The main problem with this, however, is that Toscawidgets isn't currently really well documented or heavily used. Therefore the user is stuck for the most part manually building forms, which might be terribly inconvenient depending on your viewpoint.

Extras

WSGI WSGI WSGI. Pylons is focused around WSGI. Where as some frameworks just use WSGI at the bottom of their stack to interface with the server, Pylons uses WSGI throughout, and this is exactly what allows for Pylons' great modularity and flexibility.

The real power of Pylons comes from its foundations on another project by Ian Bicking, Paste. Paste includes small WSGI middlewares and libraries which would help any framework to handle things like file uploads, error handling, request and response objects, and several other things (there's even a pony?). Perhaps more important than these are two other components: Paste Deploy and Paste Script. As I mentioned earlier, Paste Deploy is designed to make it easy for end users to install and run any Python application. Paste Deploy uses basic .ini files for configuration and standard Python eggs and easy-install as the method of installation. Since every Pylons app is an egg, it is simple to both upload one of your projects to the Cheese Shop as well as to download, install them, and then run them on your own machine (if you've never used a Python egg, it can literally be as easy as typing 'easy-install Pylonsapp' at the command line). Pylons doesn't try to create its own application system like Django, it uses what already exists in the Python community and I find that this level of reusability is both brilliant and unmatched in any other framework. Driving all of this is Paste Script. Paste Script provides basic command line commands (what else?) that help you to administer your application. For example, to run an existing Pylons application, you would enter:

paster server --reload development.ini

To create a completely new application, you'd enter:

paster create myproject --template=pylons

In addition Paste Script makes it fairly easy to add your own commands. One thing I'd like you to notice in that last example is the inclusion of the 'template' argument. Several default templates exist such as 'pylons' and 'pylons-minimal', but you can also create your own templates. So if my company has a specific header logo and CSS layout that is needed to be in every single project, I could modify the base 'pylons' template and put my own custom layout in there. Then whenever I needed a new company app, I could simply specify 'customtemplate' as my template instead of the regular 'pylons' one. This also allows me to make whatever modifications I want to the default Pylons setup such as using a different ORM, using a different directory layout, using specific settings, and then being able to easily reuse that template later on. Really the possibilities with this are endless, and Toscawidgets even uses it to generate new widgets for the user to modify.

Advantages

Pylons is really made to be what you want it to be. If you ever find that a certain templating system just doesn't fit your style, or it is buggy, or perhaps that there is simply a better one, just change a few lines of code and you are good to go. Pylons offers you some suggestions but it makes it easy for you to override them. This also makes Pylons very future-proof. If some new component comes out and blows the socks off everyone, integrating it into Pylons will not be difficult. In addition the developers of Pylons are able to focus on the minimal parts of the framework itself rather than larger components like ORMs, the interfaces and basic needs, which allows the codebase to be very slim and for different developers to do what they do best. Like Ben Bangert said the other day in IRC, "I'm not a template developer, you'd have to ask...". While Ben might not be a expert at designing powerful ORM systems, Mike Bayer is, and it allows everyone to focus on what they do best instead of having a project which tries to do everything at once and inevitably ends up doing many things well but not many anything great. In this regard Pylons reminds me a lot of the ideology behind Unix and Unix-based systems.

Add to this the fact that the project leaders are almost always immediately accessible in IRC or via email, and are constantly fixing bugs and making improvements, and you've got a development process which just seems much more accessible than Django or Rails.

One more thing: if you are moving over to Python from using Rails, Pylons will probably be the framework that is most immediately intuitive to you, as it has borrowed many of the same concepts.

Disadvantages

Unfortunately, it is this very modularity and flexibility which can cause problems. It is not Pylons' job to document every ORM that can be used with Pylons. Nor is it that job of Pylons to document templating systems. Neither is it that responsibility of the framework authors to fix bugs or improve the functionality of certain components. They may decide to do so, but there are no guarantees this may happen. As a result if you are using as an example SQLObject with Pylons and you run across a bug, you first must figure out if it is a bug related to Pylons or a bug related to SQLObject, and then you must submit that bug to the correct place and hope they fix it. Luckily this isn't too much of an issue because the authors of the major projects used in Pylons (Paste, SQLAlchemy, Mako), tend to be active members of the Pylons community, but it is always a potential problem.

James Bennett did a good job of describing some of the potential problems in a 'glue' framework in this post from February

Documentation is also somewhat of an issue in the Pylons community. Because again, Pylons cannot document every component, you are on your own hoping that documentation for a component is pretty good (luckly it generally is), and the documentation for one component will almost certainly be in a different style than another, not to mention there is no single location to find it all. In addition, because there are multiple ways of doing things in Pylons, one tutorial might do things one way and another tutorial another. There is less consenus on what the 'right' way of accomplishing things is. And because there is less consensus, if you come across some corner case, there is no guarantee that anyone has pioneered the way before you and made any kind of effort to document the solution.

Topping this off, some cutting edge components like Toscawidgets are currently just out of reach of most people because either they aren't totally ready or they don't have documentation. For Pylons users there is a bit of a paradox: they are both able to use the latest stuff in the Python community, and in some ways limited to the progress of the community at the current time.

For this reason Pylons can be a bit more difficult for new users to get into with its smaller community and less thorough documentation than Django. Pylons doesn't do much hand-holding. There is an assumption that you have some idea of what you are doing. It is always improving, though, so hopefully this well be less of an issue in a few months.

Future and concerns

What exactly composes Pylons tends to be changing quite a bit. In a few months Mako will likely replace Myghty as the default template engine and hopefully Toscawidgets will be ready for production use inside Pylons.

My main concern with Pylons doesn't have to do so much with the framework itself. It is fantastic piece of technology and the ideas behind Pylons (WSGI, Paste Script, and Paste Deploy) are things that I feel can genuinely revolutionize web development.

Unfortunately, as I stated earlier, documentation is not to the level of Django's. Pylons is not nearly as accessible to a newbie to web development. What I'm really hoping to see is for Pylons to include SQLAlchemy as the default ORM and to decide upon a general way of using the ORM, whether that is through assign_mapper or Elixir, or whatever; this default needs to be in the tutorials and what is the primary suggestion to any users. Further, defaults should be chosen for every aspect of web development. Then those defaults and how they work inside of the framework could be heavily documented and focused on to provide a framework which is as user-friendly as Django but provides flexibility that is unparalleled. I have been told before that this is 'not the Pylons way', but there are some suggestions within the Pylons community for something like this. As an extra, it would be great if Pylons could provide functionality similar to what Django does with some type of admin application, something for user registration, an application for comments, etc. This too might be outside the scope of Pylons, but there are efforts underway to do things similar to this, and I've heard it stated before that an eventual paster template for Pylons may include AuthKit for user registration/authentication.

Whether or not Pylons is a long-term 'success' long-term depends upon if developers really buy into the idea of standards and start to develop applications and middleware in a way that makes them reusable by multiple frameworks, and installable through things like eggs. If none of this ever catches on, a lot of the potential of something like Paste Deploy may be wasted. It is more work to make things standard compliant and flexible, but it is ultimately what can really change things.

TurboGears

And at last here we are with TurboGears. TurboGears was started in 2005 by Kevin Dangoor somewhat in reaction to Rails. The entire thought process behind TurboGears was to use existing libraries and components in the Python community to create a 'megaframework' which could rival Rails in functionality and prestige. It was considered a 'best of breed' framework, and it was believed that by using existing components TurboGears would get the benefit of these components being used and improved outside of the TurboGears community. The framework did not initially start out to be flexible, but it was quickly realized that what is 'best' is both subjective and temporary, and TurboGear's renewed focus on flexibility is one of the things which has greatly influenced Pylons and its development.

Philosophy

We'll provide you with the best components available, but we won't make it hard for you to use something else.

URL dispatch

URL dispatching is arguably what defines TurboGears most. It is based around the CherryPy framework, which maps classes and methods directly to URLs and arguments to those methods directly to query args.

So for example, the URL '/user/add/?name=greg' would map to the following bit of code:

class User(object):
    @expose
    def add(self, name='default name'):
        pass

Now I will be the first to say that this isn't a completely accurate representation of how TurboGears maps URLs, but it is a pretty good representation. The great thing about this is it is ridiculously simple. It is Python all the way through. Just classes, methods, and arguments. You don't have to worry about concepts such as controllers, you don't have to write regular expressions, and for the most part you don't have to think too much about the fact that you are writing a web application.

The problem with this beautiful simplicity is that it simply isn't that flexible. Your URLs are defined by your application layout, which is not very realistic, isn't very easy to change, and often does not work well for legacy applications. So while this choice was loved by some, it was hated by others. There have been those that have used Routes with CherryPy, and the latest version of CherryPy, version 3.0, has functionality to allow you to change how URLs are dispatched. Regardless, this design decision still very much is a part of TurboGear's character.

Templates

TurboGears made another interested choice when it came to templating. Instead of chosing a text-oriented system like Django or a Python-oriented system like Myghty, TurboGears used the XML-based Kid. Looping, method calls, and other Pythonic constructs were added via the XML namespace prefix py. Here is a sample piece of code:

<p py:if="5 * 5 == 25">
    Python seems to be handling multiplication okay.
</p>

This gave Kid some interesting characteristics. First of all, every TurboGears template is a well-formed XML document, which is very nice. In addition it allows the use of XML-based tools for processing. On the other hand, this reliance on XML also made Kid somewhat unsuitable for pure text files such as CSV, and added a lot of overhead to the point where Kid templating is fairly slow.

Due to some of these issues with Kid, a successor what developed, named Genshi. It alleviated several of Kid's problems and also added support for XPath and XQuery. Genshi still isn't the fastest templating language out there, but it is a solid one and for those who enjoy XML-based templating it is very powerful. XML isn't the most beloved of technologies, however, so this XML-based templated added another level to either users loving TG or hating it.

ORM

I'd be duplicating too much info if I covered it here, basically TG's ORM situation is very similar to Pylons. TG started out using SQLObject and since then has moved to using SQLAlchemy.

Form handling

You can thank TurboGears for the wonderful newforms library that you can find today in Django; in many ways Django's component is very similar to the TurboGears widget implementation designed more than a year ago. TurboGears' widget system encapusalted the logic and display for small pieces of HTML, CSS, and AJAX and used FormEncode for form validation. Unfortunately the widget system had some issues that were only discovered after heavy use, and they were also tied heavily to both Kid and CherryPy. This is what led to Toscawidgets which is a templating language and framework neutral reimplementation of TG's widgets that will be available in TurboGears 2.0. For now when using TG 1.0, however, the older system still works fairly well and is decently documented.

Extras

TurboGears has done a great job of providing all types of functionality. TG unlike many other frameworks comes bundled with AJAX functionality through the Mochikit JavaScript library. It also includes a widgets browser, admi18n which allows the user to localize their application, ModelDesigner and Catwalk, tools which help with creating and editing database models, Info, a tool which lists installed packages used with TurboGears, and a web-based console. Basically a ton of neat items which help the programmer to get their work done faster and more efficiently.

Advantages

I just listed many of them. TurboGears comes with a lot of administration tools that other frameworks do not have. In addition there has been more than a year of work on tying the current components together and the documentation is fairly good. TurboGears 1.0 is a mature framework. TurboGears also does a good job of allowing you to swap out templating languages and other pieces, in this way not entirely unlike Pylons.

Disadvantages, future, and concerns

I'm cheating a bit here and going to merge these two sections into one, because I feel like in TurboGears they are closely related.

The main problems with TurboGears is it has been realized that there are issues with some of the functionality in 1.0 which is why 2.0 is being created and several key components are being swapped out or upgraded; SQLObject to SQLAlchemy, Kid to Genshi, CherryPy 2 to CherryPy 3, and a few other changes. Essentially, 1.0 is a dead framework or the 'old and busted' components that nobody really wants to use, but it is the only framework available right now. Most of the documentation is based upon 1.0, quite a few of the admin tools such as Catwalk simply won't work with 2.0 without a rewrite, etc. This has caused some real problems in the TG community. People don't really know whether to stick around using and working on improving 1.0 and the documentation, or if they should wait for 2.0. Added to these two options is the third, less complete one to manually swap out components from that will be used 2.0 with those in 1.0, and this is what many have done. This level of confusion has caused the community to slowly decline.

TurboGears was and is in a really difficult spot. It was almost ahead of its time, and since it was based around using the 'best of breed' libraries in the Python community it had to make these major changes. Only shortly after TurboGears really started to get popular, SQLAlchemy became a real alternative to SQLObject, and there were many cries within the community to use it instead. In addition, WSGI which had been somewhat of a secondary concept in Python soon became extremely important with the emergence of Pylons. The TG developers were faced with two options: either rework the framework from the ground up to include these newer components or to stick with what already existed and soon become outdated and an afterthought. The developers did the correct thing as difficult as it was and decided to rework the framework to integrate these newer libraries. Unfortunately it has led to the aforementioned confusion. Heck, even the TurboGears book released in November 2006 is largely outdated.

The focus of 2.0 more than anything else is modularity. CherryPy 3 is much more WSGI-based than its predecessor and just about every component available is being broken down into decoupled packages (such as the Toscawidgets library). TG will also be using Paste Deploy and other Paste-related tools further in its stack. Infact, TurboGears 2.0 has many of the same goals and overall design as Pylons, and therefore there has been some talk of the two projects merging. The notable difference between the two frameworks, however, is TurboGear's reliance on CherryPy as its core, and this is not something easily reconciled with Pylon's design. Ian Bicking has offered a good comparison of the similarities and differences between the two frameworks at this link.

This might not be an accurate view of the project, but I think TurboGears is in a lot of trouble. As I've mentioned, the community has declined a lot over the past year. Users have shifted over to both Pylons and Django. The maling lists tend to be pretty quiet, and the last time I viewed one of them it had quite a bit of spam. Go into the IRC room and ask a question and you'll be lucky to get an answer within the hour if you get one at all. In the end, even if 2.0 ends up being an amazing project (and I can't imagine it being implemented and well-documented in any time shorter than a year), first impressions are key, and many people still have an image of TurboGears as the frankenstein framework which couldn't quite figure out what it wanted to be. How will TG possibly remarket itself?

This is all very unfortunate. TurboGears has gone from being perhaps the most discussed framework at the end of 2005 to one where the community is much smaller than Django's and which even Pylons has eclipsed. It is this combination of bad circumstances which leads me to believe that a merger between Pylons and TG would be the ideal situation if at all possible. If the main thing that separates the two projects is the reliance on Paste by Pylons and the reliance on CherryPy by TG, whatever steps possible should be taken to see if the frameworks can work together. However, it should be noted and has been stated by others that becuase the focus in both Pylons and TurboGears is modularity, the existence of one is not negative to the existence of the other. An admin application written for Pylons should theoretically be able to plug right into TurboGears 2.0 and vice versa. Despite this, it is with some sadness that I must say I have a very difficult time recommending TurboGears to a new web developer, though this could change in the future. There simply isn't enough stability right now. Nobody should have to spend a few months learning the set of tools available in 1.0 and then have to drop them and use the new ones for 2.0.

Wrapping it up

This has taken a lot more time and writing than I had intended. I hope this has been a thorough and fair analysis, and I hope this has perhaps helped you in your choice of frameworks. In the end a developer can only pick what seems best for their situation; nobody can tell you what is best and there is no 'one size fits all' solution. The two main projects that I would recommend are either Django or Pylons, and you would be well-served to try either one. Actually, you really should try them both. Pick a small project or a couple of projects and spend some time learning and using each one. Neither are terribly complicated and both have a friendly community that will help you along. When you get down to it, both have different design goals; one project focusing on being modular and flexible (Pylons), the other being a single set of well-integrated and documented components (Django). Ultimately the existence of these two frameworks and ideologies can only lead to further innovation in Python web development, and it is my hope that both frameworks take the time to learn from each other. And even if TurboGears is in a slump that it never recovers from, it should be remembered in the Python community because it has had a great influence on many projects.

last updated 1 year ago #

There's been a lot of discussion in the Ruby community about the state of VMs. Everyone knows that the current Ruby interpreter has its flaws, so there's been a lot of discussion and development on 3 different projects, YARV, Rubinius, and JRuby.

YARV is the most vanilla of the projects, although the one that most people will end up using. It is the official VM and is C-based and you can pretty much expect out of it what you might expect out of the Python or Perl VMs. JRuby is interesting because it rectifies many of the warts of Ruby. First of all, you get Unicode support, it compiles the Ruby code down to Java bytecodes (therefore you get speed), you can use Java libraries from your Ruby code, you can drop down into Java when performance is really needed, and perhaps most importantly, it should be hit a 1.0 release very soon.

Rubinius is perhaps the most interesting and ambitious of the projects. It follows the Smalltalk-ish idea of writing its VM in itself. So basically you are talking about Ruby running on Ruby. Evan Phoenix, the head of the project, cites some advantages of this in his blog:

There are a lot of different ways to answer this question. So I’ll say this: rubinius aims to be simple at the core so it can be a lot of things to a lot of people. The longer way of saying that is that firstly, rubinius can be customized to make your life easier. While we know ruby is already highly customizable, rubinius aims to go even beyond that.

Secondly, rubinius will fit into places the current interpreter won’t. One good example is that rubinius is all driven by bytecode and will even have hooks to further help companies protect their ruby intellectual property. To companies, that means ruby is much more enticing to use. And the more companies that use ruby, the more people, and the community grows. But don’t misread that and think that I’ve only developed rubinius to make companies happy. That’s just an example of where the “simple core, simple architecture” paradigm allows the whole project to remain agile and customizable by everyone that wants to use it.

Similar ideas led to the creation of the PyPy project for Python. The thought was that by implementing Python in Python, you would get the benefit of a simple and flexible VM that could be easily modified by anyone because instead of being in C, it would be in Python. It was also hoped that through processes like JIT-compilation, PyPy could be as fast as or even faster than CPython. As their homepage still says: "Rumors have it that the secret goal is being faster-than-C which is nonsense, isn't it?".

Well, PyPy is getting nearer and nearer to its 1.0 release, so we can now take a scope of the project and see how well its goals have been accomplished so far. Recently there have been several links on programming.reddit.com with discussion on PyPy.

Some quotes worth highlighting:

So far the PyPy code is not even conceptually simpler than the CPython code, which is overall quite nice and structured, though written in C and cluttered with INCREF and DECREF ( the only annoyance I have it, btw ). We all hope this changes in the future of the PyPy project but I admit I'm somewhat disappointed. This project had more credit than any other newly started OSS project in the world. It was even funded by the EU. The result is a very large amount of somewhat confusing [1], slow Python code that gives hope.

And from another user:

The last time I took a serious look at it, about a year ago, the code base was already insanely complex. Certainly it was much more complex than CPython ever was. Of course, it does a lot more than CPython in a number of ways, but the argument about it being inherently easier to modify for its user community, because it's coded in Python rather than C--well, I don't think that holds any water. If PyPy was closer to being a re-implementation of CPython in Python that would be likely be the case; the reality is that CPython is built on technology that has been well understood for the last 30-40 years, whereas some parts of PyPy are more like rocket science in comparison.

Here are also some interesting benchmarks.

What does this all mean? Well, it shows that PyPy still isn't as fast as CPython, but it is inching closer and closer. Keep in mind, though, that optimization becomes more and more difficult over time. It also shows that the codebase to PyPy didn't really end up being a lot simpler as originally hoped. Now as one of the quotes points out, PyPy isn't just a straight port of CPython, and it does quite a bit, but it still likely won't be any easier for a Python hacker to get in there and modify the VM as it would be for someone to get in and mess with the C-based VM.

Why did I go off discussing PyPy when this post initially had to do with Ruby VMs? There are a few things you must realize when you look at PyPy.

1) They had time 2) They had backing 3) They had experience

The PyPy project started in 2003, and in December 2004 they got adopted as an EU project to receive over 2 million euros over the next 2 years. link They had 5 people working full time on the project during those two years, as well as others working part time, and had some travel expenses paid for. link In addition, the people working on the project really had experience. Armin Rigo had previously worked on the Pysco JIT implementation for Python.

This has to be one of the best-funded and supported open source projects in history. And yet despite this, after 2 years of funding and work, when they hit 1.0 they will not be having a platform that is faster nor simpler than CPython.

What does this mean for Rubinius? It would appear to mean that Rubinius has a huge uphill battle ahead of it, as it is being worked on by a much smaller team, with less experience in things such as JIT, with a small fraction of the funding, and in a language that is somewhat more dynamic than Python. I really do wish all the best to the Rubinius team, but it is an ambitious project and it could be a long time before we ever see something you would run production code on.

Note: Please feel free to correct me if I am wrong in any of my statements. I don't mean to take any credit away from anything that has been accomplished by either of these projects, I just felt like making some observations. I really would like to hear input from anyone involved with either Rubinius or PyPy. And I'm looking forward to the 1.0 release of PyPy later this month. :)

last updated 1 year ago #

Browsing around the wonderful programming.reddit.com last night, I came across a post titled Why Django kicks Ruby on Rails' collective ass. This is an interesting article, mainly because in a sense it is right, but it goes about explaining Django's benefits all wrong. First of all, Ruby on Rails ain't plural, and therefore it cannot have a 'collective ass'. Ok, I kid. But seriously, I started out my web programming 'career' with Rails, and I have since moved over to using Django. I'll try to list some constructive reasons why the 'pragmatic' (buzzword!) developer would chose Django over Rails.

Python

Python is the reason Django is as good as it is. Django follows Python's core concept of keeping things as simple as possible and understandable. This leads to much more maintainable code in the future.

And yes, libraries play a huge part in choosing Django over Rails. Python has a library for just about everything, and is used in so many programming domains. One such instance that led me to using Python over Ruby was my need of a DBF reader. You can't find this anywhere for Ruby, but I found a very solid implementation in Python that I was able to plug right in.

The other article made somewhat of a controversial statement:

Furthermore, since Python has a more active userbase and more frameworks, if Adrian Zolovaty, Jacob Kaplan-Moss and all of the Django developers suddenly dropped dead after I finish writing this, the porting of the code would be much easier than if Rails’ development stopped, where you would be stuck in the middle of nowhere if you ran into an issue.

Now obviously, if DHH did suddenly disappear, coding on Rails would not stop. But this quote isn't as far from the truth as you might think. Ben Bangert, the creator of the Pylons framework, has said that when he went about trying to port the routes URL dispatching system from Rails to Python, he found the source code virtually unreadable, and the implementation unnecessarily complex. Luckily, after talking to its creator, he was able to implement a faster and much simpler version in Python. It is worth noting that the Rails URL system has since been rewritten (I doubt because of this), though such complexity can be found throughout the Rails source.

The key reason why? Ruby programming tends to favor 'magic'. Basically coding practices which help you put together some quick, clever code. This is nice, but much like Perl code it can end up making code a PITA to maintain. In this manner Python's focus on being simple and explicit wins out.

One more thing: packages. Python's package system makes organizing your code so much easier. I cannot express how much better this is than than anything Ruby has. Ruby's module system doesn't even come close.

If you are interested in a more in-depth reasons why one might chose Python over Ruby, feel free to check out my article on that.

Documentation

Yes, documentation. In every single framework I have looked at, whether Rails, Nitro, Turbogears, Pylons, web.py, whatever, nothing matches Django's documentation. I do not exaggerate. The Django developers have done an outstanding job of documenting their framework and it is easy to understand for both the experienced and unexperienced Python/Django user.

Compare the online documentation of Rails to the online documentation of Django. It just isn't even close. It is for this reason that nearly every new Rails developer is told to pickup a copy of Agile Web Development for Rails. While this is a very good book, it will run you more than $20 and it simply can't beat the The Django Book, which is currently being developed with the help of the community, and will be available online for free, and in print at a reasonable price at some point.

It is things like this which led Guido Van Rossum to say that the Django developers 'get open source'.

Suggestion for Django: The documentation is excellent, but it doesn't cover everything. For example, it didn't say anything about there being generic views available both for login and logout. I found that out through digging through the source (which says a lot about how understandable the core code of Django is.

Templating

Now I'll be perfectly honest, coming from Rails, I initially really disliked the Django templating system. It felt like a strafe jacket. It was limited in what it could do, which is completely different than the eRB templating system for Rails which basically lets you put any valid Ruby code into the template itself.

However, having used the Django templating system, there is absolutely no way I'd go back to using something like eRB. As soon as I started using it, I realized that certain pieces of my code wouldn't work. I couldn't jam application code into the template. It truly restricted me to only formatting my content, which is exactly what a templating engine should do. And the Django developers are committed to this, which means it won't just turn into a PHP. This is good.

I quickly found that if I couldn't express something with the template system, I was doing something wrong, and it led me to think a little more about the structure of my app. This is a good thing.

And if you really do need something more than what the default templating system provides, both filters and tags are very flexible and powerful. This for example is one of my favorite examples of chaining filters together:

object.quantity|default_if_none:" "|intcomma

And this is my own custom filter that I wrote:

@register.filter
def ifin(value, arg):
    if value in arg:
        return "<b>%s</b>" % value
    else:
        return value

And how it is used:

field|ifin:record.1

Can you guess what it does?

It is a simple filter which allows me to bold an item if it is inside a list. This simple bit of code helped me to simplify my code by quite a bit and take all of the application level code out of my templates.

Oh, and the inheritance system absolutely rocks. You'd have to use it to understand what I mean, but needless to say I'm a huge fan.

Speed

Now don't let this point be a huge deal, because it isn't. Neither Ruby nor Python are relatively fast compared to say, C, and you can't use them to do everything. However, the entire reason I use Python is because I'd like to avoid C as much as possible. The common statement in the Ruby community of 'you can always drop down into C' scares me. What is nice about Python is it is just fast enough to where you can for the most part entirely avoid using C. With Ruby if you are processing large files or a lot of text (which I have in a few apps), things can really start to drag on. What is also nice about Python is if you really need a speed up, psyco does a very nice job.

This also carries over into frameworks. Most Rails apps that I use online have a slight bit of lag to them. Django on the other hand powers parts of of the Washington Post online. And as the other article pointed out, benchmarks clearly show that Django just blows away the competition.

The Django devs have said on several occasions that they have an eye on performance, and Jacob Kaplan-Moss has said that the templating language is so fast that caching it actually slows it down. Hopefully that will just give you an idea of Django's performance.

Admin interface

As Borat might say, 'Very nice'.

One thing that you must understand about the admin interface, and most people don't, is that it is not a part of the framework. It is not scaffolding. It is simply an application built with the framework that comes bundled with it. You can chose to use it or not. If you do use it, you can literally have the entire back end for your site up in minutes. This thing is flexible and powerful. It won't do everything for you, but if all you need is a back door which allows administrators to do basic work on data, it works like a charm. However, again, it won't do everything, and it was never meant to, which leads us to our next part...

Generic views

Generic views, much like the admin interface, are very misunderstood. They are not to be compared to Rails' scaffolding either. They do not generate any HTML for you. They do, however, do the grunt work of basic CRUD operations and such, and they are very flexible. Infact, the admin application is built ON TOP of generic views. This is a real piece of code in one of my applications:

from django.views.generic.create_update import create_object

@login_required
def create_vendor(request):
    return create_object(request,
        model = Vendor,
        post_save_redirect = '/vendor/list/'
    )

That bit of code right there will do validation on a 'vendor' object of mine, and create it. It'll even do me the favor of passing me error messages which I can use in my template however I'd like.

I cannot express how powerful these are. Because they are simply functions, you can wrap them in another function and through that manipulate how they are used. You could do most everything in most sites in generic views, and the equivalent manual piece of code to what I just showed you is about 3x as big. James Bennett does an excellent job of describing the simplicity, power, and flexibility of generic views here.

One more thing to keep in mind: much like the admin interface, generic views have a limit. Don't try and fit a square peg into a round hole. Most of what you want to do can be done with generic views, but sometimes they can't. It is at that point that you can drop down to the actual framework itself. Don't be afraid to do it, as a matter of fact generic views are built using the framework, and it isn't difficult to use. In this respect, Django is composed of layers. At the very bottom you have the core framework, which is the most low-level but also the most flexible. The next step up is generic views which are built using the framework, and are higher level yet less flexible. And finally at the very top, you've got the admin interface which is built using generic views and while still flexible, is a lot more limited than the other two. Again, don't try and force one part to be what it isn't. Use the right tool for the job.

Suggestion for Django: Generic views are very flexible, but their use cases could be expanded from perhaps 80% to 90% by a few very simple things. Like for the create-object generic view, how about an argument called post-save-callback (should be underscores, formatting won't allow it), which allows me to pass in a function which is triggered when the object is created? Little things like this, while easy to implement and light weight, would add a lot more power to generic views.

As you can see, generic views themselves also can't do everything, which leads me to my next point...

Simplicity and flexibility

Everything about Django is simple. To create a page, you write regular expression in your urls.py file which maps to a view function. The view function only requires a request object as an argument (HTTP request), and once it is done processing it either generates a response (render an HTML page perhaps), or redirects or whatever else. It is that simple. There is nothing complex about Django, and this is one of my favorite parts about it. It is this simplicity which allows me to focus on my application and not the framework.

You might think that simple means limited, right? Wrong. This simplicity leads to great power and flexibility, as seen in the case of wrapping the generic view above. It also allows neat things like middleware. This is one of my favorite parts of Django, let me give you another example.

Django when given POST data, will setup request.POST, and if a file is uploaded, request.FILES. This is nice, but I quickly found out that due to Django's thorough parsing of uploaded files, was too slow for me. So what did I do? I wrote a middleware which setup request.POST and request.FILES how I wanted them. Basically, middleware is like a hook into your application which allows you to do something before each request, after each response, etc. It took 20 lines of code and now Django works as I want. It is like I went in and edited the core of the framework, but thankfully I didn't have to, and now whenever my app needs to upload files, I just drop in my custom file middleware, enable it in settings.py, and that's it.

The little things

Oh, the little things. How about Unicode support? They are working with it in the Rails community, but there still isn't a final nor perfect solution. Django on the other hand, just works, because Python has native Unicode support. You can't beat that.

How about ORMs? What if you don't like Django's ORM? Let's say you really really dig SQLAlchemy? No problem. There is nothing in Django which ties you to an ORM or anything else for that matter. Plug it right in. And if that weren't enough, there is a project which implements Django's ORM on top of SQLAlchemy, so you may soon have the simplicity of Django's ORM along with the power of SQLAlchemy. Try using Rails with a different ORM than ActiveRecord. Yeah...you might as well write your own framework at that point.

What about server support? Django supports Python's standard, called WSGI. This basically means that Django can run on any server that supports the rather simple WSGI standard. This includes FCGI, SCGI, CGI (if you are crazy enough), mod_python, and any number of Python servers. Infact, my own apps are running as Windows services off of the Python servers found in the Paste libraries. These last two items are direct advantages of Django using Python as opposed to Ruby, which does not have as much support for such solid standards nor libraries.

Ah, one more thing which is a criticism of Rails and a praise of Django. One thing that is thrown around a lot in Rails screencasts and such is the 'easy validation' that you get. Well, yes, the validation is indeed easy to use, but it is very limited. What if your HTML form isn't tied to a database? What if it is simply a form which uploads a file along with some settings for the upload? In Rails you'd be out of luck, you'd have to manually do the validation yourself. In Django that isn't the cause, you'd simply use a custom manipulator to populate and deal with the validation of your forms.

Reality, not buzz-words

Rails deserves to be recognized for being the first framework to really get people to understand just how useful a web framework can be and the fact that they don't have to be massive titanic monsters like Zope or a Java alternative. However, I find that I'm not the only person in the Django community which at once time used Rails and has since moved over. Sometimes all of the hype and buzz about Rails can hide some of its deficits. And that's one of the nice things about Django, is it isn't about hyping things up as much as it is about getting things done. Heck, the framework hasn't even yet hit 1.0 (yet it is still stable), and the Django devs have said that it is at that time which they will finally begin to 'market' the framework. Sure, Django doesn't have AJAX built-in, which would be nice to see, but that is something that is being looked at and it isn't hard to plug in whatever AJAX framework you want to use yourself. In that respect, you don't have scriptaculous (which has a number of criticisms of its own) shoved down your throat.

You also don't have things like PostgreSQL more or less ignored (or shall I say the database treated as retarded) because Rails is 'opinionated'. Finally, you don't have DHH. As much as I like what the guy has done for marketing Rails, he just doesn't shut up. He's constantly creating controversy and getting in a flame war with somebody, and long term that will not help Rails get adopted into serious work environments. At this point DHH is as dangerous to the future success of the Rails as he his helpful. It is a bit like a gas depot sitting right next a firing range. In the Django world, they try and keep those separate, thankfully.

These are my humble observations on using Django instead of Rails. If you are a serious developer wanting to chose the right framework for your business, or startup, you owe it to yourself to seriously consider both frameworks. And I do think that Django is a very good choice that is difficult to beat at this point in time, and will only get better as it reaches a 1.0 release, gets a finished book, and continues to attract new developers.

last updated 1 year ago #

Earlier today I submitted an article to Digg about a flaw I have found in their submission model. There is further info at the actual article itself, but essentially, a small group of Digg's top users are always among the first few people to 'digg' each others' stories. If you look at the submissions by each person in this small group of people, you will all find them voting on each other and vice versa. It essentially means that every article submitted by these guys gets an automatic 10-15 diggs. It also means that this level of control by these top users will only increase and get worse as they have undue exposure in the system that other users simply don't have. But that isn't the point of this article, that was the point of the last one, and I invite you to take a look at it yourself if interested.

The point of this article isn't a flaw in a system itself, but a flaw in how the system is run. In less than 2 hours after submitting my story to Digg, there has been great interest and over 120 diggs by interesed diggers. Most stories get on the frontpage with far less. Infact, the top 3 current stories on the frontpage have less diggs than my story. But my story isn't and won't be on the frontpage. Why is that?

It turns out that at some point after submission, the story was no longer showing up in the 'upcoming queue'. You can no longer search for it either. And if you take a look at my profile, the story actually is flagged as a 'homepage' story, meaning it should be on the frontpage. But it isn't. And unlike stories which are 'buried' by Digg users, this story doesn't have any indicator mentioning that 'diggers have found this to be innaccurate'. What this means without a doubt is that a moderator at Digg has manually removed the story from the frontpage.

What is the significance of this? It would seem that despite Digg's emphasis on being 'democratic', truly only a very small group of users get to actually have any kind of a say in the site. If you aren't a part of this group, then good luck. Even if you submit an article about a certain subject first, if one of these users in question comes along and submits it also, your story will never see the light of day. Further, it means that Digg's moderators have no problem with this abuse of the system. They seem to actually be restricting access to anything that may be critical of Digg's flaws (even if this is constructive).

It turns out for as much hyping as Digg does about how it is 'user-driven' and 'democratic', more than anything, if Digg's moderators don't want something to be seen, you won't see it. It IS controlled by editors, make no mistake. And as these editors repress Digg's flaws instead of fixing them, Digg will only continue to become a less and less valuable source of information.

We must remember that it was only a year ago that Kevin Rose and his gang were bright-eyed about how open Digg's model was and how it revolutionized the Internet. He and many users of Digg were critical of Slashdot's editorially-closed model, and of the status quo there. Why has Kevin Rose's tone all of a sudden changed? Why are certain users allowed to take advantage of the system and why do moderators have a final say on the commmunity? This is NO different than Slashdot's model, with the exception that at Slashdot, it is a rotating cycle as to what users get to decide what content is of quality, not a single set of users that always decide what gets on the frontpage. You might say that at this point in time, Digg is even worse than Slashdot when it comes to user participation. Where has Kevin Rose's bright-eyed dream gone? Well, a lot changes when your website is suddenly worth millions. Maybe you can't blame him, but you can blame Digg for continuing to harp on talking points that simply don't ring true for it.

Will Digg change? The people who regularly use Digg can only hope so. Perhaps these articles will start some kind of movement back to what Digg originally was supposed to be.

last updated 1 year ago #

Earlier this evening (Sept. 5) I decied to take a glance at Digg's frontpage. I do this fairly regularly, as do many other thousands of people.

Upon checking the frontpage, I immediately noticed a common element in several stories - the yellow nuclear symbol which served as the icon for the #2 ranked Digg user, who goes by the name digitalgopher. This user had 3 of the top 15 stories at the time. Another user had 2 frontpage stories. This isn't something I've ever really noticed before, but it did make me immediately suspicious - getting to the frontpage of Digg isn't the easiest thing in the world. So I decided to look a little closer and see who exactly were digging these frontpage stories submitted by digitalgopher. Turns out that every single one of his stories had a common group of people being the first ones to Digg his story - what made me even more suspicious was the fact that several of the users who were digging all of his stories also had stories of their own on the frontpage. This was very odd. I have heard before that a small group of users tend to get many of the top stories, but I never had considered that they may all be working together to achieve this. But before we make any accusations or come to any conclusions, let's get down to some statistics.

As I write this post, it is about 12 AM central time, and our friend digitalgopher has another 3 stories on the frontpage. Infact, digitalgopher has had 6 of the last 30 stories on the Digg frontpage (it is worth considering that most Digg users don't have a single frontpage submission). This isn't too alarming of a number, but let's 'digg' deeper, and see if there is a common theme between all of these submissions.

digitalgopher's most recent frontpage story is a happy piece about engineers breeding mice that are always cheerful. By taking a look at who else has dugg this story, we immediately can get a bigger picture. There are several names that stand out due to my research from earlier. These names are:

• hemphill81
• supernova17
• Aidenag
• johndi
• parislemon
• capn_caveman
• wayjer
• 3monkeys
• msaleem

What is significant about these names? They are all within Digg's top 30 list, that is, they are the within the 30 digg users that have the most frontpage stories. Okay, that isn't a huge deal in itself, but what makes it even more intriguing? They all, interestingly enough, have been among the first 30 users to digg digitalgopher's next most recent frontpage story. Keep in mind, these are only top 30 users I am mentioning here. There are quite a few other lower ranked users, such as laidback4evr and noname99 who similarly are among the first users to digg digitalgopher's submissions.

Infact, if you look at who exactly has dugg any of digitalgopher's links, you'll quickly see a group of regulars. See for yourself:

• http://www.digg.com/programming/Nowtheyhavetwoproblems/who
• http://www.digg.com/offbeatnews/Thehonestspeechacollegepresidentnevergottogive/who
• http://www.digg.com/technews/GoogleaddsAdWordsadstoUSmobilesearch_service/who
• http://www.digg.com/mods/WoodenTouchscreenTerminal/who
• http://www.digg.com/technews/FingerprintingWiFicouldsecureMACaddresses/who

A few regulars who stand out in these submissions are gwjc, aaaz, and tlmac59, again, all top 30 digg users (technically aaaz is 31, but close enough).

Okay, 'so what?', you say. 'Maybe he just has a lot of people who like what he submits.' And that would be a fair assessment, unless we take a look even closer. hemphill81 also has a frontpage story right below digitalgopher's. Who has dugg his story?

We again very quickly can pick out some common names:

• parislemon
• wayjer
• 3monkeys
• digitalgopher
• supernova17
• Aidenag
• laidback4evr
• tlmac59
• aaaz

That is interesting, we have several of the same people digging both digitalgopher's stories and hemphill81's stories. Also, digitalgopher and hemphill81 both regularly digg each other's stories. Let's again take a closer look at hemphill's submissions and see if he has a group of regular diggers:

• http://www.digg.com/gadgets/ResultsoftheGoogleDesktopGadgetContest/who
• http://www.digg.com/businessfinance/BillFordoutasFordCEO/who
• http://www.digg.com/offbeatnews/StudentsMomsMakeEndsMeetAs_Strippers/who

Do you see the pattern? Again here we have many of the same people digging hemphill81's stories as they were digging digitalgopher's (many of them are among the top posters in the digg community), and we have digitalgopher and hemphill81 digging each other's stories. You could literally go through every submission of their's and you'd find the same patterns. Let's check 1 more person just to make sure we aren't going crazy.

Someone who I regularly noticed digging both users (and indeed many connected users) was someone by the name of tlmac59. He too is within the top 30 digg users. Let's check out one of his stories on the frontpage and see who has dugg it. Apparently it is more of the usual:

• johndi
• digitalgopher
• supernova17
• Aidenag
• hemphill81
• laidback4evr
• 3monkeys
• wayjer

Again, let's see if there is a pattern:

• http://www.digg.com/worldnews/OKtosellanofrillsF22ARaptorto_allies/who
• http://www.digg.com/generalsciences/NanoscientistsCreateBiologicalSwitchfromSpinach_Molecule/who
• http://www.digg.com/gadgets/Top10hightechvehiclesforthefilthyrich/who

And again, we have a large selection of these exact same people having dugg all of tlmac59's stories, hemphill81's stories, and digitalgopher's stories.

Infact, if you were to look further at many of the people who are digging the stories, you'll find the same connections. But don't just take my word for it. Take a look at the history of some of these users. It is bizarre how intertwined a lot of them are (particularly the top users).

UPDATE: I wrote this entry last night but was going to delay posting it until today (Sept. 6). I checked the frontpage again, and I noticed that hemphill81, Aidenag, and supernova17 all have frontpage stories yet again (hemphill and Aidenag both with 2). Let's do a quick look at who is digging their stories.

Hemphill81's most recent frontpage story. Who has dugg it?

• 3monkeys
• wayjer
• tlmac59
• digitalgopher
• johndi
• gwjc
• Aidenag
• supernova17
• p9 (Digg's top user) - I have refrained from mentioning him as of yet, but he is found often digging these exact same stories as well
• and more of the 'lesser' regulars

This isn't odd? What about supernova17's latest frontpage story. Who has dugg it?

• johndi
• p9
• aaaz
• digitalgopher
• 3monkeys
• wayjer
• gwjc
• tlmac59
• Aidenag

Why are the same exact users among the first 30 to Digg every single story? Why are they all digging each other's stories? How exactly have all of these users ended up in the Digg top 30? It wouldn't be because of the setup we've been looking at, would it?

Let's check one more current frontpage story. This one is by 3monkeys (a user we've noticed throughout). Who has dugg her story?

Are you suprised to find the same exact group of people (you shouldn't be at this point)?

• johndi
• hemphill81
• ps9
• Aidenag
• digitalgopher
• noname99
• gwjc
• tlmac59
• supernova17

What it comes down to is there very literally is a group that controls Digg. If you are within this group and you submit a story, you are more or less guaranteed 10-15 (or more) automatic diggs from this group. What happens to the people who don't have such a luxury and only get the default single vote like everyone else? This only encourages a cycle where those who are getting votes will continue to get more and more, as they feed each off each other and pat each other on the back. These users will remain the top users, and even when people like digitalgopher find a story later than someone else, digital gopher's story is the one that will end up on the frontpage due to the influx of votes. Not only this, but every single digg from someone of this group causes the story to show up on 'DiggSpy' and cause an unproportional amount of exposure that other stories don't get.

Now something should be said here: this could all be totally innocent and there is no actual intention of gaming the Digg system. It is a bit odd that virtually all of the top users and another small group vote on each other's stories, however, that doesn't mean they are doing it wrongly. I think the real fault here is Digg's 'Friends' system. You can also check and notice that virtually every one of these users has each other added as a friend (you can look at this via their profile). What this means is when a story gets submitted, it automatically shows up in the friends' list of each of these users. They then will probably digg the story, and as they do, every friend connected to them will see the story in their friends' list as well. And thus it is a never-ending cycle. People with smaller friends' lists (or no 'friends' at all) simply can't compete.

What does this say about Digg? It means a small 'aristocracy' controls the vast majority of the content that gets on Digg, and it means that every day it gets harder and harder for new users to have any kind of an impact. I think Digg made a fatal flaw by adding their friend system because it turned Digg into more of a popularity contest than it did into who has the most quality content. But, it also shows Digg is a true democracy. And like a true democracy, the crowds making the decisions tend to not make the best choices en-masse, and it follows the saying, 'the richer get richer and the poorer get poorer'.

Happy digging.

Note: There are alternatives to Digg, notably Reddit, but it too faces many of the same problems Digg does, maybe not to the same extent because it has a smaller community. It is worth checking out, though it definitely has a different flavor than Digg does. There is also del.icio.us, but it serves a slightly different purpose.

There is no telling what it is going to take, but the way these link-share systems are currently setup isn't going to cut it as time goes on. Undoubtedly they can, and must evolve just like every other system.

last updated 1 year ago #

Background

You look around the web today and Ruby and its offspring Rails are the talk of everyone. You see former Java advocates moving over to Ruby. You see former Python developers checking it out. You see people who have never coded checking it out. On top of this, there is a massive influx of Ruby and Rails (particularly Rails books) flooding the market.

And all of this excitement and hype is for good reason. Ruby is a simple and beautiful language which is easy to learn yet hard to master. It gives power to the pros without being hard on the newbies. For this reason, it's a great language to pick up. It also has a great mix of imperative and functional qualities, and it has some very nice syntax. Ruby deserves much of the attention it is getting.

And then there is Rails. Everyone knows of Rails, if you don't, you apparently haven't been online since the end of 2004. Rails did for web applications what PHP did for web scripts 10 years ago. Rails makes developing web applications dead simple. It makes working with databases, generating dynamic and interactive views, and most important, maintaining your app, easy for anyone. Heck, to work with Rails you don't even have to know how to use Ruby. That's how I did it. Nearly a year ago I had very limited programming experience. My programming skills were so poor that I had a hard time understanding how methods worked. I didn't really understand object-oriented programming. But I saw Rails, and I was interested, so I bought the very popular Agile Web Development with Rails, and followed along. Just by slowly picking up bits and pieces, I was very quickly able to create my own little applications, and it gave me great pride to do so. This just shows you how much of a DSL Rails is on top of Ruby, in that someone doesn't even have to have a deep knowledge of Ruby to understand the techniques behind Rails. Rails changed how people looked at web apps. Rails wasn't revolutionary because it did anything new, it was revolutionary because it took many solid older concepts which had been overlooked by others and packaged them in one sleek product. And for this, Rails, too deserves much of the attention it is getting. It took a middle ground between overcomplicated systems such as Struts and very basic PHP scripting. It jump-started framework development and got people interested.

Before I continue on, let me say that I personally embraced Rails and the great language behind it. Through the help of the community on #rubyonrails, I was able to slowly understand Ruby and programming in general more and more. I invested in several good books such as Learn to Program and Ruby for Rails (I highly recommend both, though suited to different audiences). This past spring I convinced by boss to give Rails a try for the internal apps we were creating, and over the past few months got to use and see great projects coming from the Rails community such as the simple and speedy server, Mongrel. In short, I drank the Ruby-colored kool aid and the kool aid was good.

Shifting opinions

So, after all my praise to Ruby and Rails, why do I chose Python over Ruby you say? It came about by a funny set of circumstances. I had checked out Python about 6 months ago and disregarded it in favor of Ruby, mainly because I had invested my time and money in learning Ruby, and I didn't have the confidence to check out some of the Python frameworks available. A few months ago I talked with my boss and we decided to rewrite a PHP app using Rails. The app basically would take Excel files, DBASE files, CSV files, etc, do some statistical analysis on them, and convert them over to a tab-delimited format. This is where my love with Ruby started to fade. I got my CSV processing done via the Ruby library, FasterCSV, and then I headed for the next stop: processing DBASE files. So I Google around for a DBASE reader library. Nothing. A few comments here and there on mailing lists, but other than that, nothing solid. I finally find a post dated two years ago from someone who said he had created his own DBF reader, but that's all I found. I actually did email him and get the code, but I decided that when processing megabytes of data for my company, I couldn't trust something that was completely unsupported and via his own words "quite sloppy". So, where to turn to next? I could write my own DBF reader, but that was something I wanted to do as a last resort. In my previous searches, I did find that Python had a very good DBF reader (and writer). So I took the next step and wrote a simple Python script which would do all the conversion for me, and my Rails app would simply take the uploaded file, have Python convert it to a tab-delmited file, and then Ruby would do the stats conversion.

This actually worked remarkably well, except for one major problem: speed. I tweaked the stats algorithm as much as I could to make it as efficient as possible, but the PHP app that I was converting from was still killing my Rails app by half the time when it came to the stats work. This wasn't going to cut it. Furthermore, I had more complications when as part of the statistical work I wanted to display the name of the current sheet for Excel files. This meant that my Python script wouldn't just be called into, it would also be passing info back out to Rails, and it was getting more and more complicated.

And it was at this point that I pondered why I was even using Ruby and Rails. In my very little usage of Python during this time, I found that it was just as friendly a language as Ruby when it came to syntax. I picked it up very quickly and I didn't have any problems with it. On top of that, it was a good deal faster than Ruby, and had much more library support than Ruby. So why use Ruby over Python? My main incentive had been Rails, but upon doing some research, I discovered that there are multiple extremely solid web frameworks in Python. The big two everyone talks about are Django and TurboGears, but there is also Pylons, web.py, and numerous others. This was not the same situation I met 6 months earlier, when all these Python frameworks either didn't exist or were rather immature. They had grown up to be powerful, well-documented frameworks which stood their own against Rails.

Ruby's warts

So I then personally have to wonder why the hype about Ruby. Doing a quick recap:

1. It's slow. Yes, I know that you can rewrite performance-sensitive parts in C, but that should only be a very last resort. I don't want to mess around with C (its the whole point most of us are using languages like Python and Ruby anyway). With Python, I haven't had these performance issues, and if I really really need to speed them up even more, I can use tools like Pyrex and Psyco before considering breaking that final barrier and using C. This is a problem in the Ruby and Rails world. People like to make excuses for Ruby being slow but it just doesn't cut it. You can cover up for a lot of these problems using caching in Rails, but that's not a desirable nor complete way of fixing this. Thankfully Ruby will be getting its own bytecode solution, but there's no telling if that will be in 6 months or 2 years.

2. Poor library support. Yes, Ruby is an excellent language. But you can have the most elegant and wonderful language in the world, and if it doesn't have library support, it is ineffective. A lot of apps in Rails are being built without the need of solid-third party libraries, and that's great, but there are those times when you need to read a DBASE file. And there are times when you'd like another templating system than erb. In Python you've got Cheetah, Kid, Myghty, Django's templates, PSP, the list goes on and on. Likewise with ORM systems. In Ruby you essentially have ActiveRecord and Og. Now the number of people using Og probably numbers in the dozens, so you'd be hard-pressed to find adequate support when you need it. In Python you have several projects from SQLObject, Django's ORM, Zope's OODB, or the very useful SQLAlchemy.

3. Poor language support. Python has a very clear-cut process for users to improve the language. These PEPs (Python Enchancement Proposals) have seen everything from a standard Python style guide (PEP 8), to the implementation of a server interface for all Python web projects, WSGI (PEP 333). With Ruby, where do I turn? Much of the core development discussion is in Japanese, and even if it weren't, where do I formally request improvements to the language? Further on this concept of poor language support, you have Python being used at places like NASA using tools such as NumPy, and Google. Where is Ruby being used heavily internally? When there are NASA engineers using the language and making improvements to tools, I know I'm getting quality. Python is used in so many different fields for so many different uses. Ruby is used mainly for web applications. This means Python has a much wider range of influence and a much greater range of knowledge converging on the language. Its not limited to web programming. And you can see this in the tools put out by the community. Where can I find something similar (and stable) like Pscyo in Ruby? Where can I find the equivalent to NumPy? Pygame?

Ruby and Python compared

So when we get down to it, it's obvious people aren't chosing Ruby over Python for speed or library support. So why are people chosing it? The two biggest reasons I see is because Ruby is a beautiful language, and Rails.

On the topic of Ruby's subjective beauty as a language, Python and Ruby have their differences, but in many ways they are extremely similar. Hell, if Python didn't have its indentation rules, 90% of Python and Ruby code would be identical. An example:

class A:
    def test(self):
        a = "test"
        b = "test 2"
        c = [a, b]
        return c

class A
    def test
        a = "test"
        b = "test 2"
        c = [a, b]
    end
end

They each have their own quirks and there are several things that don't map so cleanly from language to language, but for the most part, these are two very similar languages. It's obvious that over the years they have each borrowed heavily from each other.

Now there are some things that Ruby does a lot more cleanly, for example, class methods, but there are some things that Python does much more effectively, such as dictionaries (known as hashes in Ruby).

A Ruby hash:

a = {"key 1" => 1, "key 2" => 2, "key 3" => 3}

A Python dictionary:

a = {"key 1":1, "key 2":2, "key 3":3}

Now this is obviously very subjective to each programmer himself, but my main point is, Ruby has better syntax for some things, while Python has better syntax for others. There isn't some massive divide between the two languages where one is a ton cleaner than the other (we aren't comparing Java and Ruby, for example). There are also language features that Ruby is better in, such as blocks (I do love and miss blocks), but there are other areas where Python is a lot better, such as its module support, and Unicode.

Two areas where the two languages do diverge quite drastically is on the topic of metaprogramming and also the idea of explicit over implicit. In Ruby, metaprogramming (programs that write programs) is embraced, while in Python it is seen as something to be avoided for the most part, in favor of code simplicity and readability. This follows the Ruby philosophy of TIMTOWTDI (there is more than one way to do it), versus Python's "there should be one obvious way to do it". Ruby's open-endedness on this is nice in some regards. I enjoyed writing little things like (this is an extremely simple example):

counter += 1 if x == 2

In Python you could also write something much like it (but keep in mind, this is considered a bad thing to do - check PEP 8, under Whitespace in Statements and Expressions):

if x == 2: counter += 1

However, when you get down to it, lots of little shorthand pieces throughout your code will force anyone reading your code to first understand all your own unique coding conventions. It can quickly make reading code more of a process of translating as opposed to just reading.

And that's something I've learned to value in Python. I always had a difficulty reading other people's Ruby code. A lot of times it could be very "hackish", much like people used to complain about Perl. But when I pick up someone else's Python code, because there is one preferred way of doing things, if I understand those community-wide conventions, then I'm going to have no problem reading their code. And this is something that is entirely too underrated.

This is why I've come to the conclusion that Python's stance on "no magic", is invaluable and ultimately is much better for a language than having a hundred different ways of doing things (do you know how many ways you can append to an array in Ruby?).

Web frameworks

And finally, let's get down to what everyone wants to talk about: Rails. As I said before, 6 months ago, Rails really didn't have a lot of competition. Django and TurboGears were gearing up (bad pun, I know), but they weren't at the level of maturity Rails was, and their documentation was rather lacking. This just isn't the case anymore, and not just for these two projects.

Python has something Ruby doesn't, and it's a little something that I mentioned earier, WSGI (Web Server Gateway Interface). WSGI is a standard interface between servers and web applications. If your web framework supports WSGI (and its extremely simple to support), it automatically can run on mod_python, FCGI, SCGI, basically any server that also supports the WSGI standard (and that list is growing). That immediately removes quite a bit of the complexity in making Python frameworks. Some frameworks, such as Pylons, are taking that a step further and integrating WSGI throughout their entire framework stack. This essentially means that any ORM, templating language, session manager, whatever, can be switched out with a few lines of code. Try using something other than ActiveRecord with Rails, and see how easy that is. This is an area where Ruby could learn quite a bit.

Python also has another great little tool built with WSGI, called Paste. Paste essentially makes creating, maintaining, and deploying your app dead-simple. As all of you Rails users know, to create a new Rails app, you simply say:

rails newapp

Using Paste, I can say:

paster newapp --template=pylons

But its longer, and what is that template suffix? With that command we have told Paste to generate a new project, using the template for the Pylons framework. I could just as easily specify the template for a project using TurboGears. I could even go so far as to decide that I love Pylons, and in each Pylons application I make, I always want a specific stylesheet and I want a company login system. No problem, I simply customize a Pylons project and create my own unique template. This is extremely cool. As I said earlier, Paste does many more things than that, but its just another example of a great Python tool that many of these Python frameworks are embracing.

Speaking of tools, I mentioned it earlier, but I must again mention SQLAlchemy. SQLAlchemy is an extremely flexible and powerful Python tool for working with databases. You ever run into legacy databases that simply won't work with ActiveRecord? SQLAlchemy has absolutely no problem with these, because you specify your table metadata using Python code, and then you create a mapper for that metadata. This allows unlimited flexibility. On top of that, if you just want something cut-and-dry simple like ActiveRecord, just use ActiveMapper, a nice little tool built on top of SQLAlchemy. You just don't have these kind of choices in Ruby, certainly not choices that are this heavily used and well-documented.

And this is the great things about all of these Python frameworks; must of them have solid documentation and most of them are being used in production already (much like Rails). Heck, Django powers parts of washingtonpost.com. And if one of these Python frameworks doesn't fit your taste, you aren't forced to accept it or leave the language behind, you have dozens of others to chose from. Want something for content-heavy sites? Use Django. Want something extremely simple? Check out web.py (it runs the very popular Reddit.com, btw). Or hey, maybe you love Rails? You are in in luck there, too - check out Pylons or TurboGears.

Breaking it down

So wrapping up this article, I will say that while Ruby and Rails are very deserving tools and are extremely useful, they are indeed overhyped. You remember that Nike commercial from a few years ago with Michael Jordan? "Everything you can do I can do better...". When it gets down to it, I feel that's the situation with Ruby and Python. Yes, Ruby is an elegant language, yes Rails is an elegant and useful framework, but Python too is an elegant language and Python has numerous elegant and useful web frameworks. On top of that, Python has a ton more library support, is a good deal faster, and the language is extremely well supported by the core developers themselves and people in different industries throughout the world.

Don't buy into the hype. Don't use a language just because its the hot thing and there are a bunch of books being released about it right now. Do yourself a favor and take a careful look at Python and the tools available to it. Enterprise Integration with Ruby, huh (no knock at the book or author, just the title)? You really want an enterprise language? One that's been used for years in enterprises such as Google and is used for more than just creating web applications? Look no further than good old Python. A good developer doesn't just use what everyone else is, he uses what is most useful to him.

Thanks for reading and God bless,

jp

Extra

Note: I am aware that a lot of people when Ruby is criticized like this like to make comments like, "this is how Java felt 10 years ago, too, look where it is now". I'm sorry, but the situation is not comparable at all for several reasons. First of all, Java was intended as a replacement for most tasks C++ was used for, tasks where C++ was just too complex. Java was an easier C++. This simply isn't the case with Ruby. There's not some wide divide between Python and Ruby in ease of use nor in power. There's not some big reason for using Ruby over Python, infact, as this article has showed, there are quite a few reasons to use Python over Ruby. Further, Java was promoted by a massive corporation. The numerous Ruby and Rails books coming out right now are nice, but Ruby still is a pretty small language and the number of people that know of it is still very limited. For these reasons I do not think you can accurately compare Ruby's situation with Java's situation 10 years ago.

Second Note: For those of you that already use Rails and love it, and are interested in a Python Rails-ish framework, I again invite you to take a look at