Twisted Isn't Specific

I was googling for some stuff related to Twisted and greenlet and I came across this post from last year by Jean-Paul Calderone: Twisted Isn't Specific. That subject is a reference to a quote in the Twisted.Quotes file:

<sayke> Acapnotic: don't make it twisted-specific
<dash> sayke: pffft
<dash> sayke: twisted isn't specific

The post is kind of relevant to some poking around I've been doing with integrating greenlet and Twisted again, but that's not really why I wanted to point it out on my blog. I just had a really good time reading it again. I think it's really well-written, and is fairly quotable. Just to warn you: this is one of those "quote posts" that contain more quotations than commentary, often perpetrated by ineffectual bloggers with nothing original to say. I'm sorry.

The email that Jean-Paul is responding to is by Andrew Dalke, and is about how he wants to magically replace the built-in Python socket module with something that switches to an event loop and allows other application code to run at the same time. This isn't terribly reliable, of course, and Jean-Paul gives a nice explanation of the dangers of implicit context switching (which applies to pre-emptive threading just as well as the controlled but still implicit switching of stackless or greenlet).

Consider this extremely trivial example:

  x = 0
  def foo(conn):
      global x
      a = x + 1
      b = ord(conn.recv(1))
      x = a + b
      return x

Clearly, foo is not threadsafe. Global mutable state is a terrible, terrible thing. The point to note is that by introducing a context switch at the conn.recv(1) call, the same effect is achieved as by any other context switch: it becomes possible for foo to return an inconsistent result or otherwise corrupt its own state if another piece of code violates its assumptions and changes x while it is waiting for the recv call to complete.

Asyncore was also brought up. It's often compared to Twisted, since in the past it was the only prevalent event system for Python. Twisted has pretty much obsoleted it by now, or so I like to think:

2) asyncore is smaller and easier to understand than Twisted, [-- Dalke]

While I hear this a lot, applications written with Twisted _are_ shorter and contain less irrelevant noise in the form of boilerplate than the equivalent asyncore programs. This may not mean that Twisted programs are easier to understand, but it is at least an objectively measurable metric.

Andrew goes on to talk about how great it would be if we could just use the existing built-in Python libraries for NNTP and POP3. That was pretty amusing, because those were some of the first protocol implementations (client and server) that Jean-Paul did for Twisted. And you know what? He had a reason for doing them with Twisted, instead of using the standard library versions:

Yet by using the Stackless socket monkeypatch, this same code works in an async framework. And the underlying libraries have a much larger developer base than Twisted. Want NNTP? "import nntplib" Want POP3? "import poplib" Plenty of documentation about them too. [-- Dalke]

This is going to come out pretty harshly, for which I can only apologize in advance, but it bears mention. The quality of protocol implementations in the standard library is bad. As in "not good". Twisted's NNTP support is better (even if I do say so myself - despite only having been working on by myself, when I knew almost nothing about Twisted, and having essentially never been touched since). Twisted's POP3 support is fantastically awesome. Next to imaplib, twisted.mail.imap4 is a sparkling diamond. And each of these implements the server end of the protocol as well: you won't find that in the standard library for almost any protocol.

He went on to compare the Twisted POP3 client documentation to the Python poplib. Neither were very good.

Jean-Paul even had a nice bit of wisdom about our culture as programmers in general:

I feel that using the phrase "just a" in the previously quoted text is an understatement. [-- Dalke]

I think you're right. We throw around "just" a lot in our line of work, don't we? :) Twisted does also account for a raft of platform-specific quirks and inconsistencies. I take this to be a good thing.

And what a classy closing.

I apologize for writing such a long message, but I didn't have time to write a shorter one.