Twisted Radix

Kingdoms of Amalur: Reckoning interactive story

2011-10-06T21:28:00.004-05:00

The community people at Big Huge Games and 38 Studios are doing a neat little project on Facebook - a kind of crowdsourced choose-your-own-adventure story.

Here's the blurb from the initial post:

We wanted to give you a quick heads-up about an initiative we're starting on our Facebook page today. It is an interactive story that you can help us write! We post a screenshot and a poll, you help us decide what happens next. Muse has posted more info on our forums, so come take a look: link

And here's the first paragraph of the first post:

You bolt awake in an unfamiliar place, sweating and out of breath from the dream. You were in the tower again. The decay. The filth. And the Tuatha…. Panic. Fire. The gnome giving his life to save yours. So many dead. As you wipe the sleep from your eyes, you swear that you can still smell the moldy decay of Allestar Tower and the smoke of the fires that destroyed it.

It reminds me of things like Action Castle, which is kind of an interactive fiction party game. Oh, here's a great video: I was actually in the audience for this!

(edit: I see that the embedded video didn't get displayed for Planet IF readers -- here's the link: The "Action Castle" Incident at PAX East 2010)

Anyway, this is like that, but with thousands of people on Facebook. Neat!

Holy sh*t! QUAKE LIVE is launched!

2010-08-08T01:27:00.001-05:00

Alright, so what's the explanation for my long absence from the blog this time? QUAKE LIVE just went out of beta yesterday. We've released subscription services which give you access to new maps, a new game type (freeze tag), and the ability to start your own matches with your friends. It's at quakelive.com.

Since I joined the project I've been working on scalability improvements, but also new features like the invitation system and subscription management. I've even dipped into our PHP code recently. It's been really fun to work on and learn about a system that needs a lot of scalability in managing a combination of extremely dynamic and static, long-term statistical data. Python and Twisted are continuing to be a great fit for our work in the infrastructure.

Having scalability problems is a great thing, of course, so go try out the game and make me work more :-)

http://www.youtube.com/watch?v=uKtrmmIUv1k

How to send good unicode email with Python

2010-07-14T16:30:00.004-05:00


# coding: utf-8

# Python's email API is simple and easy to use!!!!!!!!!!!!!!

# Requirements:
# * UTF-8 headers
# * UTF-8 body
# * prefer quoted-printable to base64 transfer-encoding.
# * Don't escape "From" at the beginning of a line in the message - it's not
#   the 1800s any more

from cStringIO import StringIO
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.header import Header
from email import Charset
from email.generator import Generator


subject = u'Hello あ'
recipient = u'Bあb '
from_address = u'Bかb '

html = u'<html><body>Hey böb!\nFrom Jack, I got enhanced pills!</body></html>'
text = u'Hey böb!\nFrom Jack, I got enhanced pills!'

# Override python's weird assumption that utf-8 text should be encoded with
# base64, and instead use quoted-printable (for both subject and body).  I
# can't figure out a way to specify QP (quoted-printable) instead of base64 in
# a way that doesn't modify global state. :-(
Charset.add_charset('utf-8', Charset.QP, Charset.QP, 'utf-8')


# This example is of an email with text and html alternatives.
multipart = MIMEMultipart('alternative')

# We need to use Header objects here instead of just assigning the strings in
# order to get our headers properly encoded (with QP).
# You may want to avoid this if your headers are already ASCII, just so people
# can read the raw message without getting a headache.
multipart['Subject'] = Header(subject.encode('utf-8'), 'UTF-8').encode()
multipart['To'] = Header(recipient.encode('utf-8'), 'UTF-8').encode()
multipart['From'] = Header(from_address.encode('utf-8'), 'UTF-8').encode()

# Attach the parts with the given encodings.
htmlpart = MIMEText(html.encode('utf-8'), 'html', 'UTF-8')
multipart.attach(htmlpart)
textpart = MIMEText(text.encode('utf-8'), 'plain', 'UTF-8')
multipart.attach(textpart)

# And here we have to instantiate a Generator object to convert the multipart
# object to a string (can't use multipart.as_string, because that escapes
# "From" lines).

io = StringIO()
g = Generator(io, False) # second argument means "should I mangle From?"
g.flatten(multipart)

# Pass the result of this to your SMTP library of choice.
print io.getvalue()

edited: The last part in a multipart message is the preferred one, so I moved the HTML part to the bottom.

edited AGAIN: I found out that in order to avoid ridiculous "From" quoting, I needed to use a Generator object instead of multipart.as_string().

I can't believe I made it

2010-01-10T11:00:00.002-05:00

Let's see: I got to my new apartment in Uptown Dallas on Saturday the 2nd, got my driving learner's permit on Monday the 4th, got my driver's license Friday the 8th (the soonest I was allowed to, a four day wait after I got my learner's permit), and got a car yesterday, Saturday the 9th. It's been a crazy week and I got everything done both as soon and as late as possible. I can actually get to my first day of work tomorrow!

Welp, things have been happening

2009-12-19T20:22:00.004-05:00

Dang, July was my last post? That was a long time ago.

Since then, I left my job at Canonical, took a nice vacation/hiatus of a few months, and now I've got a job at id Software which I'm starting on January 11th, if all goes well. I've already ~~sorted out~~ applied for an apartment, so now all I need to do is actually move down there... and get a driver's license and a car.

In three weeks. During the holidays.

Well, we'll see how it goes.

edit: during a frickin' snow storm

Launchpad is now open source

2009-07-20T23:55:00.002-05:00

I'm sure you've already heard this in a million places, but I'll add my blog to the din: Launchpad is now open source. All of it. Including the code hosting stuff. And the part that builds Ubuntu. Everything!

I'm happy.

inform7.com

2009-04-23T10:27:00.004-05:00

So, there's a new release of Inform 7, and a new web site.

The entire web site has been redone, and I was responsible for the new extensions section. I won't say it's pretty (I'm working on it), and it still doesn't have quite all the features of the old site (the RSS feed should be available by this weekend), but there is an important new feature: you can now browse both the documentation and the source for all I7 extensions on-line. Here's Intelligent Hinting by Aaron Reed, for example.

Judith

2009-04-13T13:57:00.005-05:00

Judith is Terry Cavanagh's latest game, and this time he collaborated with Stephen Lavelle.

It's another one of his experimental story-telling games, which are my favorite. I played the game about ten minutes after it was released and about an hour after that I was writing up a long analysis of it.

I was thinking of posting it here, on my blog, but instead I decided to put it on the intfiction.org Interactive Fiction forums, because I'd rather have a conversation about the game than try to assert things about it in an exposition on my blog. Still, it's a rather wankery academic post, but my intentions are good.

(By the way, intfiction.org is having some problems with spam lately; try to be careful and don't click on any dodgy-looking topics, since they may contain rather explicit porn spam).

Games with non-expository stories

2009-03-30T14:20:00.011-05:00

At the end of this blog post you'll find a list of games by three different developers. I think you should try them out.

Edit: Pathways does actually work on Wine if you set fullscreen=0 in the config.ini file.

Vignettes? Well, that's how one of these developers describes his works. Some of them definitely aren't short (Don't Look Back took me about half an hour; Eversion took a few hours spread over a few days), but the stories they present all have in common a reflective simplicity. All of these games are more about story than gameplay: some of them have gameplay elements which try to give the player some direct, deeply effective control of the fiction (Daniel Benmergui's games), and others take tried-and-true gameplay mechanics and use them as a way to communicate story elements to the player (Don't Look Back and Eversion).

The title of this post calls them "non-expository", and I'm not sure how accurate a phrase that is, but it's the first one that came to mind when trying to describe the way they present the story. Let's take a look:

 1. The act of exposing or laying open; a setting out or
displaying to public view.
[1913 Webster]

It may be overly subtle or perhaps even punny, but I think it does make sense to refer to them as non-expository. The stories they present are not clear-cut and laid out. The author did not tell us what the story is: he showed us some scenes, depicted some characters, but didn't flesh out their details. They're much more introspective than that. When you get to the end (or one of the endings), you think to yourself: "okay, maybe I know what just happened. Whatever it was, it was cool."

I can only say that I am really happy that indie game development is becoming so widespread and successful. Certainly most indie development these days is focused on improving the essential gameplay of video games, which is awesome, but I find the experiments with story much more fascinating.

There's a more concrete property that most of these games share: there's no text or dialogue. Despite this, they instill in the player a strong sense of the narrative with subtle graphical, audial, and gameplay cues. Somewhat tangentially, there is something that I've become a bit obsessed with as I think more and more about story and games -- the way that we can communicate to the player by taking something which is commonly used as a utilitarian device of user interface (a score counter or a health meter, for example) and twisting it in some way to reinforce the impact of an event in the game's story. Eversion does this well. I'd like to write about this more, once I find a better list of examples.

The one game in the list below which does have dialogue is Pathways, and its dialogue is very limited. Each character only says one or two very simple sentences. Even so, that dialogue really has an effect on me, and I wonder if that has something to do with the nostalgia I have for those old, badly-translated Japanese video games where most of the characters you run into simply repeat the same line over and over again. I wonder if someone who's not familiar with those types of games could have the same kind of emotional response to the simple but incisively crafted dialogue in Pathways.

Anyway, here's the list. Most of these are Flash games and the links will take you directly to the page where you can play them, unless otherwise noted.

Daniel Benmergui

I wish I were the Moon. Take pictures of the things you see on the screen, and then click again to move them elsewhere. See what happens.
StoryTeller is similar in that you basically have the ability to move elements of a scene around, but this time it's broken into three distinct scenes, representing three points in the lives of three characters. You can modify the elements in any of the scenes and immediately see how it affects the later ones.

Terry Cavanagh

Don't Look Back. It's a pretty simple jump-and-run game with beautiful music and a haunting turning point.
Pathways (runs well on Wine if you change config.ini to say fullscreen=0). This one is really touching. It's all about making decisions, and it reinforces what seems to be a recurring theme in his works by not allowing the player's character to turn back. He can only move forward or turn onto another path.

Guilherme S. Töws (of Zaratustra Productions)

Eversion (another Windows-only game, works well on Wine). Another jump-and-run game which looks like a typically boring and hyper-cheery Mario-type game for the first couple of levels, but becomes gradually deeper and more bizarre. This one is the most difficult of all the games I've listed to actually complete, but you can get through the first 7 levels without having to collect all the gems. Getting them all may require the use of walkthroughs, which are readily available on Youtube.

Interactive Fiction Writing Month

2009-02-11T04:19:00.003-05:00

I've just added the Interactive Fiction Writing Month blog to Planet IF. The project looks really cool. If you know anyone who's been thinking about trying to write some IF, point them in that direction.

See also the main IF Writing Month web site.

Blueful

2009-01-26T20:50:00.002-05:00

Okay, this is really cool. I think you should check it out.

http://blueful.com/

By Aaron A. Reed.

Open-Morality

2008-11-24T18:11:00.014-05:00

I am really getting sick of open-morality games¹.

These games have a problem. It's not that they lack a wide enough range of moral decisions: I really do believe that my Fallout 3 character is a shining beacon of humanity's hope in the wretched landscape of the USA's post-apocalyptic capitol. And I realize that he could just as well have been an absolute bastard, even offending the sensibilities of the other horrible people in that world. So, okay: let's say that the game industry has figured out how to give the player some choice in how their character develops. The problem is that they haven't figured out how to construct an engaging story from those decisions. Not once have I felt any twitch of meaningful emotion or insight while playing any open-world game that focuses on allowing the player to make wide-ranging moral decisions. Okay, so most games don't give me any such reactions, but I think open-morality games actually destroy tiny bits of my brain that are responsible for emotion and insight. Maybe I'm being too harsh.

I want to talk about Fable 2 today. I picked up Fable 2 a couple of weeks ago, honestly expecting to not like it very much. I'd played through most of the first Fable game, and found its story pathetic but its gameplay fairly entertaining. After playing it, I ranted to my friends that the morality of the world was shallow: The leader of the Hero's Guild, after turning me from a village boy into a full-fledged adventurer, gave a speech indicating that whether I would be good or evil was entirely my own decision. In this world, I could create suffering or ease it. Sure, that's true, it is my decision, but you know, I would expect some encouragement in the good direction at this point (later plot events notwithstanding). When major characters don't even care whether you're good or evil, it sure does lessen the importance of that decision. So, morality in Fable is just another part of the toy.

Other than that, I thought the game was somewhat fun.

I've also been antagonistic to Peter Molyneux. He has some strong opinions about how stories in games should work, and after having played Fable and seeing how weak of a story it had, I basically condemned him as an unfortunately popular eccentric.

I've changed my mind. Fable 2 is good. I like Fable 2 a lot. And maybe I don't think Peter Molyneux is quite as crazy as I did.

Don't get me wrong: I still think open-morality is a dead-end road for narrative in games. Frankly, I think that people like Mr. Elrod who are apparently deeply emotionally affected by this game need to read some books and get a more robust personal philosophy. If a player can construct a unique, unauthored narrative out of the simple mechanics and quests of this or other open-world games, then he could probably do the same with a string tied to a stick. We all have our weaknesses, Mr. Elrod. I watch sappy anime. But I don't imbue it with an imaginary quality of depth, even if it sometimes makes me cry.

So why do I like Fable 2? Let me explain. It's incomplete to call Fable 2 an open-morality game. In addition to being an open-morality game, it is a parody of open morality.

I realized this as I was having my character join the Temple of Shadows -- the evil brotherhood of monks who receive benefits from an unnamed evil force by sacrificing innocent villagers. I should have realized it earlier (arguably even back when I played the first Fable game). But this is what made it clear to me that this game was parodying moral decisions. You see, the Dark Monk (or whatever he is) who decides whether you can join the Temple of Shadows gives you a simple task: eat five baby chicks. Including the feathers, bones, and beaks. While they're alive. He mentions that back in his day, they only had to kick a blind beggar's walking stick out from under him, but nowadays they've got more strict entry requirements. As you eat the chicks, one by one, he makes comments indicating his disgust at your actions, at one point crying "You really have no scruples, do you!?"

This scene had me cracking up. It was well-written black humor², and the subject matter was so over the top that I could have no serious emotional reaction (other than amusement) to my character taking the "evil path" by joining this temple.

Another example of this kind of thing is the Assassin's Guild. The guild will offer you jobs for killing particular (seemingly random) NPCs somewhere in the world. Each contract has the reason for the hit: reasons like the character having bad breath, or the guild needing to maintain its monthly quota.

This is not a dramatic player-generated narrative, people.

And I like Fable 2 more than Oblivion. Sure, when I took that first step out of the dungeon at the beginning of Oblivion I was excited to see the wide-open world awaiting my exploration. But when I actually found things in that big open world, the badly written and unengaging stories really detracted from the experience. I still go back to the game from time to time and enjoy a bit of a romp around the world, but who was really that excited about completing the main story? And what alternatives do we have? Farmer Brown wants you to go into a cave and kill a giant crab. Epic. Fable 2 has its moments of badly written story, but they make up a way smaller portion of the game. That is, less than all of it. Fable 2 has good comedy surrounding a bad story. Oblivion just has the bad story.

I still think Peter Molyneux is probably a bit crazy, but I can respect what he's done a lot more now. I think I conflated his seriousness about story with seriousness of story. It turns out, he is just deadly bloody serious about comedy³.

Footnotes:

1: I would say 'to coin a phrase' here, but I can't be bothered to google it up to make sure nobody else has said it yet. Anyway, it's clear that we're going to need more words than just "open-world" to explain the things happening in game development these days. Open-geography, open-morality, open-plot: these are all fairly different, and there are games to which these terms apply independently. Exercise for the reader: categorize Far Cry 2, Oblivion, Fallout 3, Grand Theft Auto, Deus Ex, and any Final Fantasy game.

2: Mr. Gijsbers, I understand that interactive dark comedy is difficult for you, and so I recommend you do not play Fable 2 to avoid wildly misinterpreting it.

3: Actually, I don't want to pretend like I'm getting at the True Intended Meaning behind this game. Mr. Molyneux might actually think that his game is a touching piece dealing head-on with hard moral problems, but if he did, he has failed. Oh, look! A stick and string!

Twisted 8.2.0pre2

2008-11-16T20:57:00.004-05:00

Please try out the second pre-release of Twisted 8.2.0, the first Twisted release since April.

Downloads

Release notes

Please try it out and report any bugs.

edit: Updated to pre2.

A meme that is not a quiz, thankfully

2008-11-14T10:45:00.006-05:00

There's a meme about books that's going around:

What you do is grab the nearest book, turn to page 56, and post the fifth sentence to your blog.

First, the proof:

This picture was taken after rotating about 120 degrees in my chair. The Kindle in the picture probably is a few inches closer than the rest of the books, but first, it's unclear that it's a book, and second, its ebooks don't have page numbers.

So I chose the book on top of the stack (not the Nintendo DS game case that's at the very top: that's The World Ends With You). The line reads as follows:

Events that do not command our attention hardly exist for us, even if they influence how we perceive, feel, or react.

It's from Artful Sentences: Syntax as Style by Virginia Tufte. This book is made up mostly of quotes, so the line is actually a quote from another work, by Gerard Roth: The Quest to Find Consciousness.

(I've added the interactive-fiction tag to this post even though it has nothing to do with IF, so that my readers who only follow that tag will see it).

Planet IF: Now slightly less buggy

2008-11-09T11:34:00.003-05:00

I've updated the software behind Planet Interactive Fiction to a new version/fork called Venus. This should fix that bug which caused some blog post titles to be replaced by the username of the person who authored the post. I think this was affecting Wordpress users who used images, or something.

Other than that, the templates are exactly the same, so you shouldn't notice a difference.

Just so I don't feel like a slacker

2008-10-30T23:45:00.003-05:00

I took a week off from work this month (and I'll do this once per month until the end of the year, since my vacation time doesn't roll over) and have been trying to actually use it productively. Here's a list of some interesting things that I've done so far:

Twisted stuff: there was a developer sprint on Sunday that Itamar, Glyph, Jean-Paul and I attended, and since then I've been sporadically (but much more than usual) reviewing branches to try to get a release out. Speaking of which, Twisted contributors: please review #3487!
Did a bit of evangelism for the new GarGlk project on Google Code, organized a mailing list and rustled up some members, and started making an Ubuntu package for the new Gargoyle. You should see something by the end of the week.
Worked on my own interactive fiction game, which is progressing at just about the right pace to be ready for the GameplayComp mini-competition in September of 2009.
Posted a couple of updated extensions to the I7 Extensions page: Emily Short's Ordinary Room Description and Jesse McGrew's Dynamic Objects.
Watched Resident Evil: Extinction (the third in the series), which did not have as good an ending as either of the first two.

Of course, in addition to this I've been spending way too much time playing video games (Far Cry 2! Fallout 3! Man, there is a ridiculously awesome influx of good games lately. And I am looking forward to Left 4 Dead). Of course, I am never one to be satisfied with simple pleasures, so all this gaming has been inspiring me to brew up an essay in my head which I'm thinking of calling The Purity of Interaction, or less wankerly, Consistency in Interaction. We'll see. I do still have three free days before I have to get back to the salt mines.

Oh yeah, and if you're in the mood for something spooky for Halloween (or as I like to call it, All Hallow's Evening), check out Dave's latest blog post.

Im in ur web site, postin ur Inform 7 extensions

2008-10-19T14:41:00.003-05:00

Hi, this is a quick note that now people who want to add extensions to the Inform 7 Extensions Page should now contact me at i7extensions@wordeology.com; Emily Short has handed off the responsibility to me, and I've just updated Liquid Handling and Supplemental Actions by Al Golden, and Epistemology by Eric Eve.

Hopefully at some point the email address will be replaced by a web form that streamlines this process a bit.

Glulxe packages for Ubuntu

2008-10-02T21:45:00.006-05:00

So I got all the IFComp 2008 games, unzipped them, and tried to play them. Then I found out I didn't have any decent interpreters for the games and couldn't find Ubuntu 64-bit binaries for them on the net. So I decided to start making packages.

Glulxe (using the glktermw backend) is now packaged and available in a Launchpad Personal Package Archive that I just set up. You can get it by adding the following software source to your Ubuntu machine:

deb http://ppa.launchpad.net/ubuntu-if/ubuntu hardy main

And installing the 'glulxe-term' package.

edit: Thanks to Peer Schaefer for the original packages, on which my packages are based. Sorry I forgot to mention in the initial post!

I hope to offer 'glulxe-gtk' package soon; ideally, I'll replace both of them with a 'glulxe' package that can dynamically use any Glk backend by making use of GlkLoader, but that'll probably require some real coding.

I'm also working on packages for Gnome Inform 7, but unfortunately they contain proprietary code so I can't publish them on the PPA. I'm hoping to be able to split that package so that I can create a 'gnome-inform7' package separate from an 'inform7' package, but that's iffy because it would make installing from .debs without a repository significantly more annoying - right now you just click the .deb link on the Inform 7 web site and then click "Install package".

Anyway, look out for more Ubuntu/IF work soon.

Interactive Fiction Competition 2008

2008-10-01T14:24:00.003-05:00

There are a slew of posts about it on Planet IF, but given that my readership is largely outside of the IF community, I'm going to mention here that IFComp '08 is now under way and is accepting votes. There are 35 games which you can get in one bundle. Most of the games can be run on any computer (Linux, Mac, Windows), with a few platform-specific (Windows-only) games.

In case you don't know what interactive fiction is, you can read Emily Short's short introduction. If you're looking for the interpreters needed to run the games, you can check out the relevant page on IFWiki.

This is the fourteenth year that the competition has been running (I was eleven when it started!), and it's the first in which I've decided to vote. Please join me.

This is a Rant

2008-08-26T09:16:00.003-05:00

Edit: see end of post

Linux.com :: Protecting your MySQL database from SQL injection attacks with GreenSQL

This is an article (written by a very clueless person) about GreenSQL which is a tool (written by a very clueless person) that acts as a proxy between an application and a MySQL database which attempts to detect malicious, likely-injected SQL statements.

Do not interpolate strings into your SQL statements.

Then, there are all the hilariously dreadful comments.

"better yet, encode the bloody data before you shove it in there" --Anonymous

Do not interpolate strings into your SQL statements.

"Do you honestly think that anybody who doesn't know how to use simple, foolproof SQL-quoting functions is really going to be able to figure out how to correctly set up a package like this?" --Anonymous

Do not interpolate strings into your SQL statements.

"Why don't you try to actually learn to secure your code instead of being a lazy (or completely unskilled) administrator? Surely mysql_real_escape_string() isn't too hard to incorporate?" --Anonymous

Do not interpolate strings into your SQL statements.

Look, it's simple. Most database interfaces have a function called 'execute' or similar which takes two arguments: a string of SQL with markers like '?' in it, and then a tuple of arguments to be used as the values of those markers.

execute('SELECT * FROM users WHERE name = ? AND email = ?', ("radix", "radix@twistedmatrix.com"))

Do that. Don't do any of the following:

execute('SELECT * FROM users WHERE name = %s and email = %s' % ("radix", "radix@twistedmatrix.com"))

name = "radix";
email = "radix@twistedmatrix.com";
execute('SELECT * FROM users WHERE name = $name and email = $email');

String escaping is an absolutely retarded alternative to this. Why would you bother escaping or "encoding" your strings when you can simply use the database API as it was intended, without interpolating strings?

Edit: This concept of passing parameters has nothing at all to do with the "prepared statements" feature of popular databases. This is a much simpler feature. This is not a new feature. This feature is commonly called "bind parameters", and it has been around for decades.

Why do so few people know about this?

Gmail and HTTPS

2008-08-08T10:05:00.003-05:00

If you use gmail, all of your email is probably going unencrypted over the Internet, allowing fairly easy snooping of all your sensitive data. You may feel good that there's an "https://" in your address bar after you type "gmail.com" into it and hit enter, but notice that after you authenticate and start reading email, it's gone back to "http://".

It's been possible for a while to work around this and use https even for email content if you knew what to do in the address bar, but now gmail has made it easier. There's a new option in the main Settings screen called "Always use https". I strongly encourage all gmail users to turn this option on.

It's still really lame that they haven't made it the default mode of operation. Come on, gmail, don't you care about privacy?

Ecru

2008-08-07T18:00:00.003-05:00

Remember how I was talking about restricted execution, and mentioned that thing "Monte" that doesn't actually exist? Well actually I was trying to refer to Ecru, a new E implementation in C. After being announced to positive reactions on the E mailing list, Allen posted on his blog about Ecru and why he started the project.

I'm excited about this project. E is a language which is specifically designed to solve the types of problems that I want to solve, but Ecru has a long way to go until it gets there. I hope it gets more contributors!

As for Lua: I do think it's feasible for some serious restricted execution work (or at least I think it might be - I haven't really proven it yet), but I'd much prefer Ecru, because E is so much more sensible a language.

python-committers

2008-07-29T17:34:00.003-05:00

Quote:

Welcome to python-committers! This publicly-archived but invite-only mailing list is for people who have commit privileges for Python to discuss issues that do not concern the general public. Example of topics to go to this list are release dates, whether a specific patch is a release blocker, etc. Essentially anything where public input serves no purpose.

-- Brett Cannon

Twisted Community Code on Launchpad

2008-07-26T09:09:00.003-05:00

We already made an official announcement about it, but I wanted to post again about the Twisted Community Code project on Launchpad.

After (nearly) two months, I think so far I'd call it successful. There are currently thirty-two projects registered, with a variety of technical areas covering concurrency, deployment, testing, protocol implementations, games, and more.

I think this is turning out way better than our ProjectsUsingTwisted wiki page, not only because it's easy to add your own Launchpad project, but also because it gives users a direct and easy way to get access to the source code and bug tracker for each of these projects.

The 'tx' naming convention is also turning out to be pretty cool; I'm amazed at how many projects have adopted it in the couple of months since it was proposed.

Anyway, if you have your own Twisted-related project, I encourage you to register it in Launchpad as a member of the tx super-project. If you're just looking for libraries to help while you're writing your Twisted-based application, it's also a great place to check.

Joss Whedon Writes Stuff

2008-07-18T22:50:00.003-05:00

Check out Dr. Horrible's Sing-Along Blog. It's a video musical. It's by Joss Whedon, famous Hollywood writer/director/terrorist. It's free to watch on that web site until Sunday, July 20th, 2008. After that you'll be able to purchase it for a nominal fee.

Here are some things which he said about this project that I think are cool.

The idea was to make it on the fly, on the cheap – but to make it. To turn out a really thrilling, professionalish piece of entertainment specifically for the internet. To show how much could be done with very little. To show the world there is another way. To give the public (and in particular you guys) something for all your support and patience. And to make a lot of silly jokes. Actually, that sentence probably should have come first.

...

It’s time for the dissemination of the artistic process. Create more for less. You are the ones that can make that happen.

Inventory Meme.

2008-06-28T20:56:00.005-05:00

Glyph inventoried -- ahem -- invented a new meme. It's called the Inventory Meme. Jean-Paul responded with a rather pathetic submission.

Now it's time for the real deal, yo.

Inventory, by Christopher Armstrong

I have to get ready for a flight to London, which is taking off in, oh, two hours. Fortunately most of the packing was done last night.

You can play it online with Parchment or download the z5 file if you have an Interactive Fiction interpreter. Please type HELP when the game starts. I've also made the source code available.

This "game" was written in Inform 7. Thanks to Glyph for inspiring me to finish my very first work of Interactive (Non-)Fiction and to Graham Nelson, Emily Short, Philip Chimento, and others for Inform 7.

(Yes, I'm being ironic. I don't actually take this that seriously.)

Requirements for a restricted execution runtime

2008-05-29T17:41:00.008-05:00

I don't care what the language is, just please someone give me a runtime with these properties:

Namespace restriction. Encapsulation. Capabilities. Whatever: the code should only be able to do what I say it can do.
CPU timeslice restriction: Either I should be able to run some code for so long, pause it, and then resume it where it left off, or I should be able to run some code asynchronously, giving it 1/Nth CPU time.
Memory allocation restrictions: this user's code should only be able to allocate up to N megabytes.
Efficient at rather high scales: I should be able to run at least 500, maybe 1000 completely isolated functions concurrently. If all they're doing is sleep()ing, then this shouldn't put the host under heavy load.
Some simple way to expose new, audited functions. This is pretty easy. If it can't run in a Python process and allow Python functions to be exposed to it, then I can at least run it in a separate process with method invocations going over a trivial RPC protocol.
A decent set of secure built-in operations, but nothing really fancy. People are going to be interfacing with my API and nothing else, so they won't need a hugely rich core language API. It does need to be secure, though: large exponentiation should either be disallowed or interruptible, for example.
Secure, ok? For example, don't talk to me about Python (as implemented by CPython). I have very little trust for any restricted execution system that's tacked onto an existing complex runtime.

In case you need some context for these requirements to make sense, think LambdaMOO or Second Life. Any user can upload code to run with their rights; it can manipulate the world through an audited "trusted" API. The code isn't allowed to interfere with the host system or other users' code. Another possible application is a wiki which allows people to upload code to be executed when a page is viewed.

Here's the list of things I've considered:

PLT looks damned close with its sandbox library, but its execution limitation is in terms of wall-clock seconds, not CPU seconds, and it's not continuable.
Monte could definitely do it some day, if its development remains steady.
Lua... well, can Lua do it? I'd love it if someone would actually make a point on this.
Haskell might be able to do it. Haskell can do anything, it seems. (I think I saw some code once which somehow implemented continuations in Haskell, for Haskell. Without being a Haskell runtime or compiler. What? I don't know.) The problem is it's inscrutable to me. I'd love it if someone commented about this.
Javascript? Is there actually a usable standalone implementation of Javascript? Which of these properties does it have?
PyPy with the sandbox translation option. It's very cool, but I'm not sure it's usable at the 1000 node scale, because I can't see a way to run multiple isolated interpreters within one process. It also misses things like CPU timeslice restriction and memory allocation restriction, as far as I can tell.

I'd love to hear your comments. If you think that an existing runtime is close and know which of these properties it lacks, I want to hear about it. If you actually know of a runtime that has all of these properties, I'll buy you fifty beers.

Search History: "R"

2008-05-16T10:50:00.004-05:00

To continue my series of search history starting with "S", here is "R":

Intelligent Hinting

2008-05-11T11:13:00.004-05:00

Aaron A. Reed recently announced an open beta for his Intelligent Hinting extension for Inform 7. This is an amazing extension that intelligently figures out how to solve puzzles in Inform 7-based games with high-level puzzle annotations in your I7 project.

You have to define "puzzles" and "tasks" in your own game, at implementation-time, and the extension provides a >SUGGEST command which indicates the next action to be taken to solve the current puzzle. It's surprisingly smart: if you've defined that a cloak must be placed on a particular hook, it will automatically figure out how to move the player to find the cloak, pick it up, and move the player to the hook. Not only that, it even knows how to completely automatically find keys for locked doors that are between the player and either the cloak or the hook.

Not only is this a good feature for end-users, it also offers very important benefits to implementors of IF: It makes it trivial to automatically test if your work is winnable, and it makes it similarly trivial to generate a walkthrough to publish with your game automatically.

Inform 7 has a rich and descriptive world model, and it's great to see tools that are starting to really take advantage of it in very useful ways.

Twisted Show 3, Raffi Krikorian of Synthesis Studios

2008-04-30T10:08:00.003-05:00

By the way, if you haven't listened to it yet, I recommend checking out the third episode of the Twisted Show, which is an interview of Raffi Krikorian of Synthesis Studios. I share space with Divmod in that back room :-)

by the way, if you have trouble with noise from the audio player at archive.org, try the .mp3 or .ogg version; they sound fine.

Planet Interactive Fiction

2008-04-29T19:30:00.005-05:00

I just set up a Planet site for the community of Interactive Fiction bloggers. It's at http://planet-if.com/ and it's called Planet IF.

Don't know what Interactive Fiction is? Emily Short explains it succinctly.

I'm not really a regular in the IF community, but I am a long-time reader and wanna-be author of IF. I want to see the medium grow in popularity and so I'll do what I can to help it out. So there you have it.

Twisted 8.1.0 prerelease 2

2008-04-26T17:10:00.002-05:00

Hey all, I've put out the second prerelease of Twisted 8.1.0. This release includes a number of bug fixes, including for some unfortunate regressions in 8.0.1, as well as a few features.

Trial's performance was improved
Fixed a bug where Failures wouldn't be constructible involving Pyrex
Fixed a couple of reactor re-entrancy bugs introduced
Fixed the conch.insults bug where gnome-terminal wouldn't scroll when output reached the bottom of the screen.

And a bunch of other miscellaneous fixes. Please download this prerelease and give it a try with your applications.

Corotwine 0.2

2008-03-31T20:19:00.003-05:00

As is my tradition, I'm doing extremely rapid releases of my software.

Here's Corotwine 0.2, which contains the following changes:

Returning from a connection-handling function now closes the connection
A bug was fixed in the example Chat server

The API documentation has been updated.

Please post any feedback.

Corotwine 0.1

2008-03-30T13:44:00.003-05:00

I've just released Corotwine, a set of Coroutine APIs for Twisted. You can download Corotwine-0.1.tar.gz. There is very thorough API documentation.

Here's a complete example chat server. Start it up and telnet to port 1025 a couple of times and you'll be able to chat.


from twisted.internet import reactor
from twisted.internet.error import ConnectionClosed
from corotwine.protocol import gListenTCP, LineBuffer


class Chat(object):
 def __init__(self):
     self.clients = []

 def handleConnection(self, transport):
     transport = LineBuffer(transport)
     self.clients.append(transport)
     try:
         try:
             for line in transport:
                 for client in self.clients:
                     if client is not transport:
                         client.writeLine(line)
         finally:
             self.clients.remove(transport)
     except ConnectionClosed:
         return

from twisted.python.log import startLogging
import sys
startLogging(sys.stdout)

gListenTCP(1025, Chat().handleConnection)

reactor.run()

There are more examples in the corotwine/examples.py file.

Twisted 8.0.1 is out

2008-03-27T10:22:00.002-05:00

Hey all. The release of Twisted 8.0 is finally done, and I was so excited about it that I just had to do another one: 8.0.1. The release announcement has been posted to the TM Labs blog.

Download and enjoy.

Twisted Isn't Specific

2008-03-24T00:11:00.004-05:00

I was googling for some stuff related to Twisted and greenlet and I came across this post from last year by Jean-Paul Calderone: Twisted Isn't Specific. That subject is a reference to a quote in the Twisted.Quotes file:

<sayke> Acapnotic: don't make it twisted-specific
<dash> sayke: pffft
<dash> sayke: twisted isn't specific

The post is kind of relevant to some poking around I've been doing with integrating greenlet and Twisted again, but that's not really why I wanted to point it out on my blog. I just had a really good time reading it again. I think it's really well-written, and is fairly quotable. Just to warn you: this is one of those "quote posts" that contain more quotations than commentary, often perpetrated by ineffectual bloggers with nothing original to say. I'm sorry.

The email that Jean-Paul is responding to is by Andrew Dalke, and is about how he wants to magically replace the built-in Python socket module with something that switches to an event loop and allows other application code to run at the same time. This isn't terribly reliable, of course, and Jean-Paul gives a nice explanation of the dangers of implicit context switching (which applies to pre-emptive threading just as well as the controlled but still implicit switching of stackless or greenlet).

Consider this extremely trivial example:
  x = 0
  def foo(conn):
      global x
      a = x + 1
      b = ord(conn.recv(1))
      x = a + b
      return x
Clearly, foo is not threadsafe. Global mutable state is a terrible, terrible thing. The point to note is that by introducing a context switch at the conn.recv(1) call, the same effect is achieved as by any other context switch: it becomes possible for foo to return an inconsistent result or otherwise corrupt its own state if another piece of code violates its assumptions and changes x while it is waiting for the recv call to complete.

Asyncore was also brought up. It's often compared to Twisted, since in the past it was the only prevalent event system for Python. Twisted has pretty much obsoleted it by now, or so I like to think:

2) asyncore is smaller and easier to understand than Twisted, [-- Dalke]

While I hear this a lot, applications written with Twisted _are_ shorter and contain less irrelevant noise in the form of boilerplate than the equivalent asyncore programs. This may not mean that Twisted programs are easier to understand, but it is at least an objectively measurable metric.

Andrew goes on to talk about how great it would be if we could just use the existing built-in Python libraries for NNTP and POP3. That was pretty amusing, because those were some of the first protocol implementations (client and server) that Jean-Paul did for Twisted. And you know what? He had a reason for doing them with Twisted, instead of using the standard library versions:

Yet by using the Stackless socket monkeypatch, this same code works in an async framework. And the underlying libraries have a much larger developer base than Twisted. Want NNTP? "import nntplib" Want POP3? "import poplib" Plenty of documentation about them too. [-- Dalke]

This is going to come out pretty harshly, for which I can only apologize in advance, but it bears mention. The quality of protocol implementations in the standard library is bad. As in "not good". Twisted's NNTP support is better (even if I do say so myself - despite only having been working on by myself, when I knew almost nothing about Twisted, and having essentially never been touched since). Twisted's POP3 support is fantastically awesome. Next to imaplib, twisted.mail.imap4 is a sparkling diamond. And each of these implements the server end of the protocol as well: you won't find that in the standard library for almost any protocol.

He went on to compare the Twisted POP3 client documentation to the Python poplib. Neither were very good.

Jean-Paul even had a nice bit of wisdom about our culture as programmers in general:

I feel that using the phrase "just a" in the previously quoted text is an understatement. [-- Dalke]

I think you're right. We throw around "just" a lot in our line of work, don't we? :) Twisted does also account for a raft of platform-specific quirks and inconsistencies. I take this to be a good thing.

And what a classy closing.

I apologize for writing such a long message, but I didn't have time to write a shorter one.

pyOpenSSL 0.7a1

2008-03-23T10:53:00.002-05:00

Jean-Paul Calderone has recently taken over maintenance of the pyOpenSSL project. He's doing development with bzr on launchpad.net/pyopenssl, but still maintaining the web site and downloads at pyopenssl.sf.net.

He's already fixed a bunch of bugs, like segfaults and things. There was a cool Open Space meeting at PyCon where he asked the community what kind of stuff they'd like to see in the bindings. I think he has plans for extending them a bit to make them more useful specifically for Twisted, but he's also very receptive to the community's patches and bug reports -- I think he's closed several bugs and patches since becoming maintainer.

Anyway, please go check out the new release - it's less buggy!

Another Twisted 8 pre-release

2008-03-22T21:06:00.003-05:00

I just put out Twisted 8.0.0 pre-release 2 (that link is temporary and will definitely not last past the release of Twisted 8). The changes are in the NEWS file.

It comes with newly improved Windows IOCP support as well as support for easy_install. You can easy_install the main Twisted-8.0.0pre2.tar.bz2, but not the subprojects. When the final 8.0 release is out I'll register it on PyPI so that you can just type "easy_install Twisted" to get it.

Please check out this pre-release and report bugs.

PyCon 2k8

2008-03-21T20:47:00.002-05:00

I'm not sure what to say. I was really happy to be at PyCon. It was bigger than ever. I only attended one talk, other than a couple of keynotes. The most important Keynote, of course, was the announcement of Twisted's membership in the Software Freedom Conservancy. You can now give Twisted tax-deductible money. Please donate!

I also released Twisted 8.0 prerelease 1. Please download it and check it out. I hope to have a second prerelease soon, which will include a much improved IOCP reactor and easy_install support.

Amazon MP3 Store now supports Ubuntu

2008-03-04T13:35:00.006-05:00

So, I noticed that Nine Inch Nails released their new album, Ghosts. The official web site that sells FLACs was slashdotted at the time but I noticed that they were also selling the album on Amazon MP3. Clicking around randomly for a while I noticed that they finally released their downloader software for Linux, including packages for Ubuntu. I think previously it was impossible (or at least hard) to buy and download entire albums without this downloader, so this is basically the first time that Linux users can use Amazon's MP3 service.

I had a problem with it, though: when I downloaded one of the .amz files to run with the downloader app, the application failed to do anything but display a blank download window. Long story short: it refuses to open files that it doesn't have write permission to. I guess firefox 3 or Hardy saves downloaded files as mode 400. Amazon downloader fails to report any error, unfortunately, so the only way I figured this out was by stracing it. Just chmod u+w foo.azw and then the downloader will happily work. I've reported the bug to Amazon support, hopefully there will be a fix soon before they start losing loads of customers who waste their money on something they can't download.

Other than this technical glitch I'm very happy with Amazon's MP3 service so far. It's great that there's actually a high-profile DRM-free music service. I just wish they would do the same thing with their eBooks.

How to create an archive

2008-01-19T13:09:00.000-05:00

Why the hell do people who write ray tracers, whether in Python, Haskell, or Erlang, always put all the files into the top level of the archive instead of creating a subdirectory?

x61 kill switch

2008-01-03T16:04:00.000-05:00

Update: Oh look, there's a kill switch down there underneath the front of the computer. I must have hit in with my stomach. Why does my computer have two kill switches? Well, whatever, I'm writing this update on my X61 with working wireless.

Hi. I received my Thinkpad x61 recently, with the Intel PRO/Wireless 3945 ABG wireless chip.

Things were going (fairly) well, but today my kill switch got activated, probably when I was trying to toggle my wireless off and then on after hibernating. And now I can't turn off the kill switch (that is to say, I can't turn my wireless back on). I hit Function-F5 and nothing happens; the wireless light remains dark. Of course, the ipw3945 and iwl3945 drivers both just say "Radio Frequency Kill Switch is On" when they're installed at boot (or when I modprobe them manually). This happened in Gutsy and in Hardy, which I upgraded to to see if it would be fixed.

Any ideas? It almost looks like bug #121415, but not really: I'm even worse off than those people, since I can't enable my wireless chip in any way.

unit testing code that uses third party libraries

2007-12-24T02:17:00.001-05:00

In the year 3000, our libraries will come with verified-equivalent implementations of their APIs that users can use to unit test their code, when the normal implementation is inappropriate.

Our libraries will also have thorough documentation on how to unit test code that interacts with them.

Kindle

2007-12-05T17:40:00.000-05:00

I was kind of embarrassed, but now I'm coming out into the open. I got a Kindle. This was a hard decision to make (making the blog post, not buying the Kindle. It was an impulse purchase). Everyone was ridiculing Amazon for producing such a stupid and crippled device. After playing around with it and actually reading some stuff on it, though, I can justify it to myself. The price was pretty stupid, but the value is pretty great. Let me list the things that aren't really well advertised on the Kindle site.

Free web browsing. This is the biggest one. They mentioned some stuff about looking up articles on Wikipedia, but it sounded like that was a special-cased service and they wouldn't allow arbitrary web browsing. They do. And it's free. You can browse to sites like manybooks.net and download ebooks. Or check up on the Twisted review queue. Whatever. Apparently there's even some javascript support, but I have a hard time believing anything of interest is supported (definitely no XHR, guys). I think the reason they didn't advertise this very well is that it makes using their blog subscription service absolutely idiotic.
The format support is better than it looks. You can download your bookwarez or your Project Gutenberg texts or whatever. You can transfer .txt and .azw files directly, and it does support the MOBI format. All you have to do is rename your mobi file to .azw and the Kindle will pick it right up. It's totally idiotic that they didn't advertise the native MOBI support (they do advertise that it works with their free conversion service. That must be one of the easier converters to write).
It looks like they have a good attitude towards future development: Right in the home menu there's an "Experimental" item which leads to three cool things: Basic Web (see point 1), Ask Kindle NowNow, and Play Music. Yeah, it'll play mp3 files, either with its built-in speaker or through headphones. The Ask Kindle NowNow thing is kind of neat, but it's not that exciting given that they have a web browser that's quite capable of viewing google.

Here are the things that they do advertise, and that they were right to advertise:

The display is damned snazzy. I'm surprised every time I glance over and can actually see what's on the screen when the Kindle is lying at some odd angle to my line of sight.
Yeah, it's full of DRM, but it is really frickin' cool to be able to find, buy, and download a book in a minute or two. Now when people recommend that I buy some book or another, I'll actually be able to get it instead of (forget to) write it down on a list somewhere and (forget to) buy it at some book store the next time I'm around one.

I'll bet that within a year or so there will be working cracks to de-DRM an AZW file. Like I said above, it seems to just be a mobi file, but with some encryption added. The DRM is the thing that worries me the most, even as a consumer, since I'm wondering if I'll be able to read these books in ten years. But given the Kindle's support for other formats, you could keep yourself busy reading nothing but non-DRMed content acquired from places other than Amazon.

The stupid things follow:

I can't copy an .html file to the Kindle and view it. Even though it has a web browser, there's no way to look at local content with it, as far as I can tell. You can send them to the free conversion service and get back an .azw, though. Or put them on the web and view them with the web browser :-).
The books are kind of expensive. I'm used to paying about $7 or $8 for a paperback, and the books seem to consistently be about $9 or $10. Given the convenience, as a consumer I'm not annoyed (as a socialist, I am. Production cost for a bunch of bits is less than $10). And it's actually *way* cheaper to buy a Kindle book than a new hardback release, since paperback releases are often delayed. I bought Pratchett's latest novel, Making Money, which I haven't seen in paperback yet, for $10 instead of the at least $20 that I'd pay at a brick and mortar store.

Ok. So I got it, and I'm pretty happy.

Supporting both asynchronous and synchronous interfaces with SynchronousDeferred

2007-12-02T22:32:00.000-05:00

Today I wrote a thing called SynchronousDeferred. The target audience is made of those who are writing libraries that want to support both blocking and non-blocking interfaces. I wrote it to allow me to use python-openid in a non-blocking way. After totally refactoring it to use SynchronousDeferred, of course.

Basically the way you use it is that you write your entire library in the way you would normally write it with Twisted: use Deferreds everywhere. There are two major differences, one at the outermost parts of your library and one at the innermost, and one rule on how they interact:

In your blocking entrypoints, return the result of SynchronousDeferred.synchronize() instead of the deferred itself

Anywhere you would do some long-running task, wrap SynchronousDeferreds around the blocking implementations but make the non-blocking implementations return regular Deferreds.

When the user is using asynchronous entrypoints, try to use asynchronous low-level APIs. When the user is using synchronous entrypoints, definitely only use synchronous (SynchronousDeferred-returning) APIs.

synchronize returns or raises the current value on a SynchronousDeferred immediately.

SynchronousDeferred doesn't have callback or errback methods like normal deferreds, as you have to pass the initial value of the Deferred straight to the constructor.

Project page: http://launchpad.net/synchronous-deferred
Code browse: http://codebrowse.launchpad.net/~radix/synchronous-deferred/trunk/files

Representing symmetric relations in a relational database

2007-11-24T21:41:00.001-05:00

I haven't searched terribly hard, but I can't find very much information on the Internet about clever and efficient ways of representing and querying for symmetric relationships in relational databases. I've thought up a few options, so I'll spew them onto my blog to get feedback. The keen researcher will probably want to do more googling for representations of undirected graphs in RDBs, since undirected edges are a prime example of a symmetric relationship.

Schema 1

CREATE TABLE some_relationship (participant1 INTEGER, participant2 INTEGER)

This is the most trivial and straightforward in-database representation of a symmetric relationship between two participants. Actually, this schema could represent an asymmetric relationship as well, but we'll write our queries and application code so that (participant1=X, participant2=Y) is treated the same as (participant1=Y, participant2=X).

This schema still leaves us with some decisions. For the simplest possible query,

SELECT participant2 FROM some_relationship WHERE participant1 = X

we would need to store each relationship in the table twice for each relationship, one of them inverted.

If you don't want to duplicate each relationship in the table (I sure don't), then we have to have a more complex query (and more complex way of processing the result).

SELECT participant1, participant2 FROM some_relationship WHERE participant1 = X OR participant2 = X

Then we must go through the results of this query, and for each of them say "If p1 = x then treat p2 as the other participant; if p2 = x then treat p1 as the other participant." I'm also not sure how well this query will perform (on any database), even if both participant columns are indexed.

Both of these are annoying. The first requires explicit duplication of data which may become inconsistent, and the second is very tedious to query for.

Here's another idea:

Schema 2

CREATE TABLE some_relationship (relation_id INTEGER, participant INTEGER)

This one allows us to avoid explicit duplication in the table and also allows us to just query against one participant column. For every new relationship (X <-> Y), you'd insert two rows consisting of a unique ID for this particular relation and each participant. The two relationships (50 <-> 51) and (50 <-> 52), you'd insert four rows::

INSERT INTO some_relationship VALUES (1, 50)
INSERT INTO some_relationship VALUES (1, 51)

INSERT INTO some_relationship VALUES (2, 50)
INSERT INTO some_relationship VALUES (2, 52)

Then, to get all participants related to 50, you'd do the query::

SELECT participant FROM some_relationship 
    WHERE relation_id IN (SELECT relation_id FROM some_relationship
                          WHERE participant = 50)
    AND participant != 50

This query is pretty complex, but at least it gives us exactly the results we want (no post-processing to decide which participant is the "other" participant, like in the second usage of Schema 1 that I discussed above. And hey, we don't have any weird duplication in our data. By the way, if you were to use this schema, you'd want to create indexes on both relation_id and participant.

Extra data and Schema 3

However, Schema 2 has some problems which Schema 1 doesn't have. What if you want to add some extra metadata to the relationship? If you're representing "similarity", for example, you may want to add a "strength" column to represent the fact that two items are more similar than two other items. Encoding this data into Schema 1 is very simple: just add the other column, and grab it out in your queries, and it's in as convenient a form as it's going to get. Encoding it into Schema 2, however, is extremely awkward because the relationship is split over multiple rows. This can be solved, however, by sticking that data into another table entirely. We can even leave Schema 2 exactly how it is, except for adding a REFERENCES annotation to the relation_id if your database supports it and you're so inclined.

CREATE TABLE similarity (
    relation_id INTEGER REFERENCES similarity_data (id),
    participant INTEGER)

CREATE TABLE similarity_data (
    id INTEGER,
    strength INTEGER)

But then getting data in a useful form out of these two schemas becomes even more complex. I'll leave that as an exercise for the reader. Doing it without application-code processing and multiple queries will probably involve DISTINCT and a join.

Well what the heck

I still don't know which option I'll use for my application. Any input, bagoblog?

Don't use Postgres's currval()

2007-11-15T09:29:00.001-05:00

Using Postgresql's currval() function in a WHERE clause is a really bad idea. It will cause a full sequence scan on your table, no matter what. If you're unfamiliar with database lingo, "full sequence scan" means "This application is too slow! You're fired".

I was using it to get the identity of the row most recently inserted. This is a bad idea. Apparently "INSERT INTO foo VALUES () RETURNING id" is a much better idea, if you're using Postgres.

Update: If you're using a version of Postgres < 8.2, you can't use RETURNING. A work around is to do "INSERT INTO foo VALUES (); SELECT id FROM foo WHERE id = (SELECT currval('foo_id_seq'))". This prevents Postgres from noticing the currval is being compared by forcing the evaluation early, so no sequence scan.

Don't expect os.environ.pop to do anything meaningful

2007-10-16T13:00:00.000-05:00

It doesn't actually remove things from the environment, just from its local copy of the environment. The Python developers seem to simply have left it out of their implementation in os.py. Gustavo Niemeyer noticed this one.

>>> import os
>>> os.system("echo $ASD")
 
0
>>> os.environ["ASD"] = "asd"
>>> os.system("echo $ASD")
asd
0
>>> os.environ.pop("ASD")
'asd'
>>> os.system("echo $ASD")
asd
0

The Orange Box

2007-10-14T11:00:00.000-05:00

As I was telling Jp yesterday, Valve has become masterful at creating near-perfect interactive multimedia experiences. That is to say, Portal is frickin' awesome. As Tycho says, they have proven the first person puzzle comedy genre.

1. It's short, but includes a satisfying amount of story content (especially if you snoop around a bit. Look for a slide show).
2. It is thoroughly and consistently hilarious, while maintaining its drama.
3. It has a very cool style and atmosphere, starting out strange and eventually growing spooky.
4. The music is excellent; Valve has always been good at timing their distinct music to maximize emotional response. Plus they got Jonathan Coulton to write the ending theme song (yes, there's an ending theme song!).

They have an interactive commentary version of each level of the game that you can play as soon as you've completed the respective level. It shows how they've thought about extremely subtle things. I would really like to see how someone who's never played a first person shooter gets along with it.

Moving on to the rest of the Half Life universe (which Portal very intriguingly refers to), Half Life 2 Episode 2 offers many of the same qualities. It's not as much of a comedy, but they make up for it. I don't know what else to say about EP2, really; if you can even tolerate first person shooters but have not yet played any Half Life 2 games, then you are missing out on a very good artistic, dramatic, and cerebral experience. And it keeps getting better with every episode.

Which reminds me: I have an extra copy each of Half Life 2 and Half Life 2: Episode 1 burning holes in my pocket; Valve has set Steam up such that I can give a copy of those to someone else since I already owned them when I bought the Orange Box, which includes them again. Who wants them?

Why thank you

2007-10-10T14:57:00.000-05:00

I came across a review of Landscape written by Kristian Erik Hermansen. He also goes on about how great Twisted is, and how he apparently used it while working at Cisco. It made me happy.

Coherence

2007-10-07T20:05:00.000-05:00

It turns out Coherence is a bit better at serving music to my Xbox 360 than uShare is. Currently my setup is improved over uShare thusly:

It doesn't include non-genre/album/artist/playlist entries in those respective categories (that was a very strange thing on uShare's part; it put individual files in those categories, or at least that's what the xbox saw)
My xbox hasn't frozen up while playing media from it yet.

I was playing Wik while listening to Sunny Day Real Estate today. That was confusing.

It's also written with Twisted, which is neat. There is also theoretically support for meaningfully listing Artists and Albums, but the plugin which is responsible for that is not working for me yet.

Fortunately a developer was around to help me out with configuring it. I'll show you how I got it working, but I'll skip all the parts about installing Coherence and its dependencies (which was very easy for me). All of the dependencies are either in Ubuntu or are part of the software stack that I have development checkouts of (Twisted, Nevow, Divmod stuff).

First create a file with the following contents:

logmode = warning
interface = eth1
web-ui = yes

[subsystem_log]

[plugins]
    [[FSStore]]
        content = /home/radix/Media
        name = Coherence Test Content

Then run ./bin/coherence -c /path/to/your-coherence.conf

Coherence has a way to go, as its web site explains: "This is still young software! It won't destroy any data - at least it isn't supposed to ;-) - but it might not yet fulfill your expectations in your particular environment." Etc.

Here's a hint, guys:

1. WRITE UNIT TESTS. Preferably before you write implementation. Please. Your software is doomed to suckiness if you don't have a good suite of unit tests.

2. Don't use 0 as a default web server port. I shouldn't have to run "netstat -nlp | grep python" and guess at ports to find out where my web UI is. Either choose something like 8080 (and document it) or show an error message if there isn't one. OTOH, this isn't a big deal right now since the web UI doesn't really do anything.

3. Make your software easier to set up and use, and write better documentation. I know this goes without saying, but here's a suggestion: every contribution should be peer reviewed and ensured to have full documentation. It's really much easier to keep the quality of your code and documentation high when you have a strict set of rules in place, as we Twisted developers have learned.

However, good job on giving me a media server that sucks less! I'll be filing bugs and working with it until I can get a movie to play :-)

playing media on an Ubuntu system with my xbox 360

2007-10-06T13:53:00.000-05:00

Just to give it a bit more google juice: ushare can act as a replacement for Windows Media Center for playing media hosted on a linux box on your XBox 360. I googled for quite a while before figuring out what exactly I needed (and I learned that UPNP has media protocols. wacky.)

1. Install the Ubuntu .deb (you're using Ubuntu, right?). The direct link on their web site to the .deb is a 404 -- I browsed their pool directory to find the directory with ushare debs. Download whichever one you need, for your 32bit or 64bit OS. You can also add their deb line to your Software Sources.

2. Edit the configuration file to change which interface to listen on, because the -i (--interface) option doesn't actually work (or the configuration file takes precedence). Pretty lame.

3. Run "ushare --xbox --content ~/Media".

4. Go to the media tab of your Xbox 360 and browse movies and music. They'll be listed.

There are several problems. First, I haven't managed to play any movies — I probably just need to transcode them to whatever codec the 360 supports with mencoder. Second, all unrecognized files seem to end up under the Music section, so there's lots of cruft in there. Third, it doesn't seem to recognize Artists, Albums, Playlists or Genres separately from Songs. That is, all of my files are showing up in all of those sections through the 360 media browser. I guess that this is a ushare bug, but I don't know. The last problem is that it seems music will just stop playing in the middle of a song. I tried starting the music again but it just froze my entire xbox. It looks like ushare still needs some tweaks to be less flaky.

Oh yeah, btw. I got an Xbox 360. Sorry guys. Halo 3 is really awesome. I'm radeex in the Recreation zone. Friend me.

Oh right, that Storm talk went well

2007-09-03T14:39:00.000-05:00

Doug Napoleone was kind enough to post my slides and a recording of my Storm talk from last month. It went pretty well. I've mirrored the slides and audio locally.

That's all!

Storm talk at the Cambridge Python Meetup

2007-08-11T17:27:00.001-05:00

I'm going to be giving a talk on Storm at this month's Cambridge Python Meetup (that's the Cambridge in Massachusetts, not the one in the UK). It's in Somerville, actually, closeish to Porter Square. If you'd like to go, sign up on the meetup site. Once you sign up you'll be able to see the address, but apparently you shouldn't trust the map that's linked, because it's wrong. Instead paste the address into google maps, I guess.

Storm 0.10

2007-08-08T17:12:00.000-05:00

I've just put out Storm 0.10. It has been Improved.

Web site: http://storm.canonical.com/
Downloads: https://launchpad.net/storm/+download

This is the first release that includes outside help! Our community is already fairly active.

Edit: I forgot to mention that I spent a lot of time yesterday and today working on API docs for Storm. They're currently hosted in my own web space, but they'll probably be moved somewhere more appropriate on http://storm.canonical.com/ soonish.

IT'S PUBLIC!

2007-07-23T08:29:00.000-05:00

This is what I have been working on for the past year and months.

Canonical Launches Web-Based Systems Management Tool For Ubuntu Deployments

On Jargon

2007-07-16T19:26:00.000-05:00

Sometimes computing jargon is pretty idiotic.

Take the words "synchronous" and "asynchronous".

A synchronous API is one which blocks until a result is available.

An asynchronous API is one which returns immediately and lets you do other things until it notifies you of the result.

Now take the English definitions:


  Synchronous \Syn"chro*nous\, a. [Gr. ?; sy`n with + ? time. Cf.
     {Chronicle}.]
     Happening at the same time; simultaneous.


  Asynchronous \A*syn"chro*nous\, a. [Gr. ? not + synchronous.]
     Not simultaneous; not concurrent in time; -- opposed to
     {synchronous}.

(taken from Webster's 1913 American English dictionary)

So clearly the jargon meanings of the words are exactly the opposite of their English meanings. How did this come to be? I'm guessing it had something to do with the invention of preemptive threading. When preemptive threading came along, suddenly it was possible to make a blocking API —one that does not allow other code to execute while it is executing, because it hasn't yet returned—to allow for some concurrency, or synchrony. Preemptive threading allows a blocking API to be used synchronously. Somewhere along the lines, I assume, people forgot about the part of the sentence "preemptive threading allows ... to be used" and mangled it into "blocking = synchronous". The opposite of this, "asynchronous", soon followed, unrelated entirely to the actual semantics of APIs described as such.

You'd better enjoy your crappy jargon while it lasts, programmers. When my revolution comes, you'll be the first up against the wall.

Using a Nintendo DS Lite expansion cartridge in a classic DS

2007-07-14T15:12:00.000-05:00

Last month I ordered a copy of the Nintendo DS Browser (by Opera) from Amazon. When it arrived I was all ready to start syphoning porns from the very air we breathe, but my dreams were to be cut short, as I discovered that the Opera browser requires a Memory Expansion Pak, which, while included, was incompatible with my *original* DS. It would only actually fit in a DS Lite. Through observation I deduced that the physical interface was basically the same and that it would probably work in my original DS, if it weren't for the lip that was on the top of the cartridge preventing me from actually pushing the thing far enough into the DS for it to make electronic contact. I decided that it was time for a hardware hack.

All I needed to find was a file to rip off the lip so that it could be jammed in there. I didn't have one, of course, so I got the closest thing I could find: a giant bolt that Jp had left over from a table he built.

After filing the lip off, my room smelled like burnt plastic all night. Oh well.

Then I tested that it would fit into the DS by half-inserting it upside down to see if the lip would fit past.

I then realized that if I actually put the cartridge into the DS, I would have no mechanism with which to remove it. I tried taping a twist-tie onto the cartridge but that proved pretty weak; I then realized that putting packing tape on both sides of the cartridge would be the best.

Going in..

I then jammed it the rest of the way in with the back end of my stylus.

It works wonderfully. There may be other DS Lite-only expansion cartridges out there, and this technique will probably work on those. Tell me if you have any luck.

Storm released

2007-07-09T15:17:00.000-05:00

Canonical just released a Python-based ORM called Storm.

http://storm.canonical.com/

Mostly it was developed by Gustavo Niemeyer, but since we're in the same team at Canonical and it's developed primarily for our project, I have contributed bits and pieces.

Communal ties will of course require me to mention Storm in relation to Axiom. While I'm still a big fan of Axiom, I do think Storm is definitely the best ORM (now) available for any project which needs to talk to a database other than SQLite, especially remotely.

The design is simple and straightforward, and the code is well-tested. As a cranky, bitter young man, I must maintain my image; frankly, I'm really shocked by the low quality of the Leading ORMs available for Python or any other language. Storm, fortunately, does many of the things that are obvious to most people who are familiar with how to write software which uses an SQL database, and leaves out the useless crap. Of course, it's not perfect, but now that it's open source, we can accept your patches :-)

Figuring out the best way to use Storm from an asynchronous codebase will be a challenge, probably, but there are techniques for it that are usable right now—one of them being "Just block because your indexes are defined to be fast"—but that isn't always appropriate. There are some other simple options, and maybe some complex ones too. I'll probably post (or get someone else to post) about those in the future.

New Universal Hall Pass release

2007-06-01T13:07:00.000-05:00

It's called Subtle Things, and it has three new tracks and three remixes of tracks on Mercury. You can get it online at Fonogenic.com.

Mercury downloads

Subtle Things downloads

The official release page is on Sneaky Records.

Subtle Things is worth the six bucks.

The real time is passing, and I'm not in bed

2007-05-23T00:39:00.000-05:00

Here's a hint: Twisted (in SVN trunk) now makes it much easier to write (and test) time-using code in a way that doesn't waste time waiting for real time to pass, and remains deterministic.

Anything which uses the reactor to schedule timed calls (with reactor.callLater) should parameterize it. Probably the only method you'll be calling on it is callLater (and then perhaps calling methods on the resultant DelayedCall), but maybe you'll also use the seconds method. The point is, you can pass an instance of twisted.internet.task.Clock instead of the real reactor, and then you have a super-easy way to test your scheduling code reliably. Just call clock.advance(numberOfSeconds) in your test to pass simulated time (thus calling any scheduled calls scheduled for that time), or you can also access the .calls attribute which has a list of all pending DelayedCalls.

Clock has been around for a while and has been in use in many tests, but the biggest new benefit is that you can now use this technique to simulate the passing of time for code which uses LoopingCall; just set your LoopingCall's .clock attribute to your Clock instance, and it'll use that for all scheduling.

Guards, Guards

2007-05-12T12:04:00.000-05:00

Pandemonium Books put on a play performance of Guards, Guards, the novel by Terry Pratchett (adapted by Stephen Briggs). It was neat. While I was there I discovered that Pandemonium Books may be a cool place to geek out, so I'll probably be visiting again.

Speaking of geeking out, I hear Tenth is moving up to Boston this weekend, and I've heard that he's threatening to run a role-playing game of some sort. Bring it on, I say.

By the way, did you know PyPy can run Twisted-based Internet servers and clients now? Our conviction is like an arrow already in flight.

It's over

2007-05-05T21:45:00.000-05:00

My three-month marathon of 50% travel at Canonical sprints around the world is finally over. I spent two weeks in Dallas in February, two weeks in Montreal in March, and two weeks in London in April. Fortunately we didn't go too overboard in London, so I'm only somewhat totally exhausted. Now to *really* kick up the work.

I'll be in Cambridge for at least a short while, hopefully. I may even be able to get back to Japanese classes; I had to cancel my previous semester for all of that travel.

Oh yeah, I started that thing again. This is, like, number eight or something. Or eighteen, depending on how you count.

I finished Jonathan Strange and Mr. Norrell. It's good. I reread The Witches by Roald Dahl. That man is a genius. I also recently read a good portion of The Cyberiad. It is not science fiction, don't believe what they say. It's about two demigods who can create machines that can do anything. The subtitle on the title page is accurate — Fables for the Cybernetic Age. The style is amazing, especially for a translation. The Fifth Sally (A) or Trurl's Prescription would be brilliant as a beat poetry performance.

Braindump over, peace out.

Things I learned in Harvard Square today

2007-04-01T22:42:00.000-05:00

There's an old hobo-looking guy who gives chess lessons on the cheap.

Don't look at the top shelf in Tokyo Kid (a shop in The Garage).

Mega-chains are destroying my ability to get tasty coffee from proud baristas.

There are many pubs and restaurants I want to try out.

Risen

2007-03-13T16:40:00.000-05:00

I'm pretty amazed that Jean-Paul has never played an MMO before - and I'm glad to be the one to inspire him to play Eve, mostly by distracting him while playing on my only Windows machine, which is at the office. If anyone wants to group, find us (is grouping a thing you do in Eve? I'm not even sure yet) - I'm Radaix the Minmatar.

PyCon 2k7

2007-03-04T13:42:00.000-05:00

I just had an exhausting two weeks in Dallas at a Canonical sprint surrounding PyCon 2007. I took just the right amount of photos to fit into one month's worth of space on Flickr (100MB) during the PyCon weekend. There are a bunch more that came from the Canonical sprint, but I guess I will have to wait a month before I can upload them.

radix's PyCon photos

The Obnoxious Activity Monitor

2007-02-14T21:18:00.000-05:00

I have also been hacking in free moments on a tool for tracking what I'm spending time on while working. It is a much more obnoxious tool than gtimelog, for two reasons: first, I was forgetting to update gtimelog when I switched to working on something else, and second, when I left my computer to get lunch or a coffee gtimelog would not stop tracking my time. That last point may be desirable for some people who sometimes work away from their computer, but I never do, so it was problematic when I left without telling gtimelog I was taking a break.

The Obnoxious Activity Monitor pops up a window every five minutes asking what you're working on, with a text entry for typing it in. The window will steal your keyboard focus, but it kindly blocks all input for one second so you don't accidentally activate anything. Whatever you typed in last time will be filled in by default, so if you are still working on that, you can just hit enter. If you have started working on something else, type in the new thing (it'll overwrite what's there because the text is selected by default). When you hit enter an ~/activity.txt file will be updated with your latest entry. If you don't respond to dialog box for more than 30 seconds, O-A-M will stop tracking your time, assuming that you have gone on a coffee break or something. When you get back, latte in hand, the dialog box will still be there awaiting your next task.

You may have noticed that I was quite detailed in describing the workflow and all the minor UI behavior of interacting with the window. This is important; if the application were not ergonomic enough, it would not be usable since I have to interact with it on such a regular basis. The common case is to simply hit the enter key, and nothing else (no clicking-to-focus, no clicking a button).

The ~/activity.txt file is compatible with gtimelog, so you can use that to generate your daily or weekly reports (although you will have spurious lines in your report called 'start'; just delete them. I might actually get around to a real fix for that).

The project is hosted on Launchpad: obnoxious-activity-monitor.

bzr and commit messages and IRC bots

2007-02-14T19:50:00.000-05:00

I was playing around with the bzrlib API because I was writing a post-commit hook for sending messages to IRC, which I don't think has been done for bzr yet.

It accompanies an IRC bot which I wrote which will hopefully be the last IRC bot I ever write (holy crap, I hate writing IRC bots): the bot simply listens on a TCP port for data to send to arbitrary IRC channels. This lets me solve a lot of IRC-notification use cases without writing any IRC code. The first application of it is the bzr commit bot.

The project is hosted on Launchpad: publish-bot. If you want to use it, you first have to set up the publish-bot on a machine accessible from where you want to do the bzr-committing, and then set up the bzr commit message post-commit hook. The README file should explain this sufficiently.

Web 1.0 mashup

2007-02-14T16:31:00.000-05:00

Remember The Milk and Google Calendar

Bazaar API

2007-02-09T20:24:00.000-05:00

I've been using the bzrlib API a bit lately to mess around with my bzr branches. Here's what I've learned.

To get a BzrBranch object representing your on-disk branch:

>>> from bzrlib.branch import BzrBranch
>>> branch =  BzrBranch.open('.')
>>> branch
BzrBranch5('file:///home/radix/Projects/test-project/')

To get the revision ID:

>>> branch.last_revision()
u'radix@twistedmatrix.com-20070210010145-s6am6e28f4lhbuyf'

To get the log message for the latest revision:

>>> revision = branch.repository.get_revision(branch.last_revision())
>>> revision.message
'testing multi\nline commits\n'

Or for an arbitrary revision number

>>> branch.repository.get_revision(branch.get_rev_id(1)).message
'hi'

How about some more data from that particular revision:

>>> revision.committer
'Christopher Armstrong <radix@twistedmatrix.com>'
>>> revision.timestamp, revision.timezone
(1171069305.8210001, -18000)

This should get you started. dir() and help() are your friend.

Search History

2007-01-04T11:48:00.000-05:00

To belatedly continue the meme represented best (so far) by Mr. XKCD, attached is a sample of my search history.

How to Do Something with RPython Part 1

2006-12-11T14:32:00.000-05:00

Ok, so here's how to set up a new independent codebase which you want to write in RPython using the C backend. I use this kind of set up when hacking on Safelisp, which is a language totally unrelated to Python which I am implementing in RPython.

This will take you through writing, translating, and running a program which prints "hello world" to standard out.

I'll assume you have an SVN checkout of pypy at ~/Projects/pypy/.

$ mkdir MyProjectInRPython
$ cd MyProjectInRPython
$ echo "This is my RPython-based project." > README
$ mkdir myproj
$ touch myproj/__init__.py
$ emacs myproj/myproj_target.py

And put the following code in the file:

# __________  Entry point  __________

def entry_point(argv):
    print "hello world"
    return 0

# _____ Define and setup target ___

def target(*args):
    return entry_point, None

Now compile and run it.

$ ~/Projects/pypy/pypy/translator/goal/translate.py  --batch myproj/myproj_target.py

[Lots and lots of output]

$ ./myproj_target-c
hello world

That translate.py command is really long, so I generally put a symlink to it in my ~/bin named "pypy-translate". The --batch flag is meant to disable the interactive debugging facilities during translation -- if you leave it off, a very nice pygame viewer will pop up showing the annotated graph of your application, which is a great way to learn more. It'll also give you a Pdb debugger on the console.

I'll post further about actually writing some real RPython code. It ain't easy.

Pre-holiday braindump

2006-12-05T22:03:00.000-05:00

Ok, here's a load of totally dissociated thoughts, technical ones last:

Woo Emma!

I am going back to Pittsburgh again for Christmas. I'll be gone from the 22nd to the 2nd.

Coldness is noticeable.

I saw Borat. I think it was pretty funny, but whenever I think about it all I can remember are the offensive bits. Which probably make up the majority of the movie, actually. I'm not a square or anything; I was laughing through most of them, but it was always a guilty laugh. Holy crap, that was so bad. Fortunately Emma, who had seen it already, was there to warn me: "This next scene is going to be REALLY GROSS".

I am reading Kafka on the Shore by Murakami Haruki. It is kind of silly but also awesome. It is FULL of (fairly explicit) literary reference, which really tickles me.

Speaking of Japanese stuff, I am only two classes away from completing my Level 1 Japanese language course. I definitely plan on going on to level two. It's amazing how little I know after two months of taking classes: it's such a different language that I only have a very basic grasp of the grammar and knowledge of maybe 50 nouns and a dozen or so verbs. For the last class, I'm going to have to write a big paragraph about myself in hiragana and read it aloud.

Efforts towards Twisted 2.5 are accelerating. Also, did you see the new web site?

Holy crap, what was I thinking with that last blog post? Porting CMUCL's readtable implementation to RPython would have been insane. I ended up just using Carl Bolz' BNF parser generator in pypy.rlib.parsing, and it was hella easy. I did it mostly while I was flying and hanging out in Pittsburgh during Thanksgiving.

That's probably enough for now.

I know what will kick your knickers, small child!

readtables.

2006-11-10T13:46:00.000-05:00

Let me start at the beginning. PyPy is neat. Also I have this language implementation called Safelisp. It is meant to be a very secure interpreter of a lisp-like language, allowing totally untrusted code to be executed in a way that won't harm the machine it's running on or effect other running code in undesirable ways.

As a way to become more involved with PyPy, I have started porting Safelisp to RPython, the Python-subset that PyPy's python interpreter is implemented in. This is not easy, because RPython is very strict in ways that pretty much all of my Safelisp code does not like.

Safelisp has this thing that I called a "readtable" when I wrote it, and while it is very superficially similar to the thing that Common Lisp calls a readtable (you associate a character with a function to parse that character and perhaps things that come after it), it is not really the same. Given that I now need to mostly rewrite the Safelisp parser so that it can fit into RPython, I have also decided to clean up the readtable implementation so that it is actually sensible.

Unfortunately I haven't been able to find any general essays on the design and implementation of readtables (The CL Hyperspec only really describes the API without much reasoning about the design, or how it ought to be implemented). Given this lack, I am left only to look at existing implementations. So, I think the next time I work on this I am going to try porting CMUCL's reader.lisp to RPython as a basis for the Safelisp parser.

Wailin'

2006-11-04T01:04:00.000-05:00

A mention of singing in an email conversation got me thinking about my old high school chorus at West Allegheny. Those were some good times. I got to googling and found this post by an alumna: Weekend Assignment #59.

Stewart Morgan. The auditorium was put up several years before I entered high school, and I sang in it every year. Our instructor, Earl Wiechelt, was his student, and Mr. Wiechelt was absolutely fantastic, but Stewart Morgan was a legend. Fortunately, he was a legend that would stop by once every week or two to give us advice and critiques, because he enjoyed it. Through their combined efforts, our chorus won regional competitions and got invited to nationals every year. Chorus was one of the few things I remember fondly of high school.

Gotta figure out an excuse to sing.

More stuff about inferring the referential transparency of code

2006-10-27T13:21:00.000-05:00

Partially for my own benefit so I can remember what to read about:

The developers of "E" (as in, the language at erights.org) propose a system of "auditors" which seems like a flexible way of adding new analysis tools to a language implementation. They propose a "Functional" auditor, which tells whether a particular piece of code is functional. I don't know if they've actually figured out how to do that (it's not clear from the article), but they're pretty smart guys. Notably, Ka-Ping Yee (aka "?!") is one of the authors.

Effect Systems (mentioned on lambda-the-ultimate) are apparently a way to annotate the *non*-functional portions of a program. I haven't really read anything about them yet, but they seem very relevant to the idea.

Functional optimizations with PyPy

2006-10-25T10:03:00.000-05:00

So, here's an idea: add a functionality inferencer to PyPy that decides whether some Python code is purely functional (doesn't contain any state-modification operations, doesn't access stuff that isn't passed to it or defined statically enough) -- or if you don't want to do that, just add a notation for declaring some code as purely functional. Then, you can disable the cyclic garbage collector for that code -- you can either just use refcounting, or use something way faster (there are tons of ways to optimize purely functional code, especially in garbage collection, like pointed out in this paper by Joe Armstrong and Robert Virding about real-time garbage collection). You can do this because it's impossible for functional code to create cyclic data structures (unless they use some sort of lazy evaluation, which doesn't really affect these optimizations). So frames created for this function can tell the cyclic garbage collector to go away, and any objects that that function creates don't need to take part in the cyclic garbage collector.

If you think that's crazy, you can also take a look at Use-Once-Variables, an idea for adding "linear objects" to non-linear languages (pretty much all programming languages are non-linear). Use-once-variables refer to linear objects, which can only ever have one reference at a time. Copying and deletion both require explicit support from the linear object. It definitely changes your style of programming to use linear objects, but they can offer some serious improvements to the ability to reason about code that uses them, not to mention that you can optimize the crap out of them.

The intuition behind linear types is the conservation of physical matter--a linear object cannot be created or destroyed without special permission, although it can be moved or transferred at will. Thus, an occurrence of a linear variable--e.g., in the argument list of a function call--indicates that the function will consume the value of the variable. If the function does not later return this value, either as a returned value, or by assigning it to a more global variable, then that function is responsible for properly disposing of any of the resources controlled by the value--even in the case of exceptions. -- Henry Baker, 'Use-Once' Variables

Alright. So PyPy is clearly the platform to implement these ideas on; what parts of the system need to be affected? Is it an object space? How many fundamental modifications does it require?

A day of mourning

2006-10-21T14:32:00.000-05:00

From inthegroove.com: "News flash: Konami has acquired In The Groove"

In The Groove is the most innovative and fun dance game in the industry. The dinosaur that couldn't keep up, Konami, has harassed Roxor Games into handing it over to them. I don't know what the future of In The Groove as a game is; it is notable that the "buy the game" links on their web site have disappeared. However, it appears it is still available from the third party vendors that already have copies. For example, check out the ITG page on Play Asia.

I was just at the MC Frontalot concert here in Cambridge last night; he's one of the artists featured in the In The Groove games. It was a very good concert. Opening for him were Shael Riley and Optimus Rhyme. They were excellent.

Upgrading to Edgy

2006-10-20T11:20:00.000-05:00

Ok, I upgraded to the pre-release of Edgy yesterday. Here are the problems so far:

Python 2.4.4 broke an interaction between Zope3 and Twisted (via changes to the 'cgi' module). Not a big deal, I only had to 'svn up' Zope3 to fix the problem (because Zope3 and Twisted developers already found the problem).
The button in my toolbar to start emacs broke, because this was the command in it: sh -c '. ~/.environment; /usr/bin/emacs-snapshot'. Apparently the default 'sh' in Edgy is now dash, which doesn't accept this syntax. I just changed it to 'bash' to get it to work.
Apparently my SMP support stopped working. Now a 'while 1: pass' loop in Python shows 100% CPU usage instead of 50%. I probably just have to apt-get the correct kernel to fix this, but I don't think I needed to do that when installing Dapper (although I may be mistaken). UPDATE: Apparently I was supposed to choose the "-generic" kernel from my bootup menu instead of the item that was selected by default and at the top of the menu, the "-386" one. UPDATE TWO: the -generic kernel didn't boot! I am going through ordeals. See bug 67256 on Launchpad. UPDATE THREE: Even after getting -generic to boot by regenerating my initrd, the nvidia drivers won't load. This might be solved soon. I don't know.

I'm not sure if any of these are bugs that ought to be filed, but I will investigate.

I am a School Student :-(

2006-10-15T14:08:00.000-05:00

I'm about to start Japanese language classes at the Boston Language institute. Excitement!

Mind

2006-09-15T14:03:00.000-05:00

Pokey the Penguin, microbreweries in Massachusetts, Interactive Fiction, simulationist virtual worlds, American classic fiction, terrifying machines.

A fine old thing is described here.

2006-09-04T01:51:00.000-05:00

It is a fine old thing to eat delicious left over potato salad at 2:30 in the morning.

Attack of the Bacon Robots!

2006-08-22T21:43:00.000-05:00

I recently indulged myself by buying a copy of the hardback limited edition of Penny Arcade's Attack of the Bacon Robots. See the bottom of This pont for extremely terse details.

The book is cool. It was $30, and it has a hard cover, so I could at least use it as a plate or something if the nuclear bombs hit. The exciting thing about this story is not the content of the book, but the *particular* book. The *instance* of the book. This book, my friends, is the one thousand three hundred and thirty-seventh copy printed. Out of Fifteen Hundred. Do I need to put that into decimal digits for you?

It is copy 1337.

My immediately surge of excitement was quickly quelled by those around me. "Hah!", said an officemate. "Surely it is a joke. It is impossible that you should be so lucky as to geet the 1337 book. It must be a ruse by the Penny-Arcade authors to trick you. There must be multiple books that say they are the 1337th."

IS IT TRUE, PENNY-ARCADE? ARE YOU TRICKING ME?

Hey look it is a natural language programming environment OH GOD STOP IT

2006-08-21T20:14:00.000-05:00

This is a bad thing.

Part 4 - Making Play More Friendly

Chapter 1 - New Commands

Section 1 - Echo-location

Understand "shout [text] to/at/for [any person]" as answering it that (with nouns reversed).

This was written by the genius (that is not sarcastic) interactive fiction author Emily Short, in her short "Beauty and the Beast" spinoff, Bronze.

As far as I can tell, it means "when the player enters a command such as "shout Hello at Bob", the 'answer' action should be invoked with the first argument being Bob and the second argument being the text "Hello". Maybe.

PyPy is self-hosting

2006-08-17T14:58:00.000-05:00

Look at this thing: Armin Rigo: "Self-hosted". Pypy can translate itself with pypy-c.

I am hoping that Twisted will have a buildslave that runs the tests on Pypy by the end of the year.

I am learning about Functional Programming

2006-08-03T16:58:00.000-05:00


(define (parse initial callback)
  ;; Construct and return a purely-functional parser of length-ten
  ;; strings.

  ;; `initial' is the initial data to process.
  ;; `callback' is a function to call with length-ten strings.

  ;; The return value is a function which accepts data as an
  ;; argument. The return value of *that* is a function with an
  ;; identical signature. When you want to push more data into the
  ;; parser, always pass it to the most recently returned function.

  (lambda (data)
    (let ([buf (string-append initial data)])
      (if (>= (string-length buf) 10)
          (begin
            (callback (substring buf 0 10))
            (parse (substring buf 10) callback))
          (parse buf callback)))))

(((((parse "hello" write) " world") " radix") " is") " here!!!!!!!!!!!!!!!")

Update 2006-07-11 09:05 -0500

2006-07-11T20:08:00.000-05:00

While I haven't been completely inactive in the blogging world since my last post here, I admit it's been a bit too long (yes, I made it back from Montréal in one piece, despite the great beer and lively night scene). So, what better than a reading-list update?

This is but a fraction of my current list. But before I go into that, here's what I've read recently:

Building Harlequin's Moon - Larry Niven / Brenda Cooper. This is surprisingly good. Brenda Cooper is a boon to Niven's characters. Not only are the characters probably the best of any Niven collaboration I've read (let alone his stand-alones), the social setting is very dramatic, and that's something that Niven is pretty good with in his other books.
The Curious Incident of The Dog in The Night-Time - Mark Haddon. It's a story about and written from the perspective of an extremely intelligent English boy. The chapters are the prime numbers. There's an appendix that describes the solution to a very difficult mathematics problem (or so I am led to believe). However, it is also a very sad story about an autistic 15-year old caught in the middle of horrible familial strife, and who also wants to find out who killed his neighbor's dog with a pitch-fork. It's good.
Shadow Puppets - Orson Scott Card. The sequel to Ender's Shadow, aka "the good Ender series", is good. Bean is a great tragic character.
A Scanner Darkly - Philip K. Dick. Sure, I admit that I read this because the movie got my attention. But I had read PKD's stuff before, and this is probably my favorite so far. Shortly after, I saw the movie, which was also awesome. In both mediums, but moreso in the book, the author really captured the paranoid moon-logic of a tripped out drug-addict excellently. It's not just a silly idea here and there: it manifests as long conversations where one strange misunderstanding of the world builds on top of another one, repeatedly, until you kind of get caught up in the logic yourself. Oh, by the way, the story is very sad.
The Last Continent - Pratchett. I'm proud to say that I got probably 85% of the references in the book. However, it was little more than that. Plot was pretty boring. The gem was the way Pratchett described Australian aboriginal art as if the artist drew the simplest thing possible, but then tried to make it come to life, by drawing it inside-out.

There have been more than that, for sure, but let's cut it to a reasonable length.

Here's what's on the plate:

Gunslinger Girl. A manga based on the anime. It's reminded me how sad the story is. I'm going to be buying the remaining two volumes of the anime that I don't have.
Metamorphosis - Kafka. Sure, I haven't read it yet.
The Book of Jhereg - Steven Brust. This is a single-bound volume of the first three of Brust's books in this series. It is turning out to be fairly good. You may have seen references to this in recent publications.
Metroid Prime: Hunters. Wait, what? That's not a book. But man, that is a bad-ass Nintendo DS game.

There's more, but those are the things I find worth mentioning.

Montreal 2006-06

2006-06-04T09:11:00.000-05:00

In case anyone in Boston didn't know, I'm off to Montreal for two weeks for work. Then on to my cousin's wedding in DC. I'll be back in Boston on June 18th.

The future of mobile communication is NOW.

2006-06-01T14:44:00.000-05:00

I ordered a phone online from Verizon a week ago. It was difficult. At first they wouldn't accept my VISA debit card, as it did not have a billing address in the zip code that I was getting the phone in. After getting my bank to change it, Verizon again wouldn't accept it since I have crappy credit. So I had to go with a monthly pre-paid plan. Sure, whatever, I don't use that many minutes.

Today I got my phone. After searching around the material that came with the phone for a frustrating 10 minutes, I figured out I had to "activate" my phone, by calling a particular three-digit number. I did that, and it was completely automated (I found it amusing that the phone said "Commit Complete" after activation).

So I tried making a phone call, and got redirected to a payment menu so I could pay for my month of service. It asked me for a four-digit passcode, which I did not know. An even more frustrating 3 hours later, I still don't know it. I tried registering an account on their web site so I could pay, but that required I know my passcode ahead of time as well. Eventually I gave up and called tech support, and it was going extremely well: The lovely voice on the other end of the line asked for my new phone number and address, and said I could immediately get a new passcode. I told her some series of four numbers, and heard the keyboard on the other end of the phone rattling, followed by an uninspiring "uh". She went on to explain that the "System" was "Updating", and that I would have to call back again in an hour to get my passcode set.

So, I guess I'm waiting for their oracle database to unlock after running the weekly backup process, or something.

Twisted 2.4.0

2006-05-27T15:32:00.000-05:00

Twisted 2.4.0 is finally out. This release has come together really well: we got a ton of subprojects released followed by the main Twisted release in a week or so. Then Cory Dodt put together the windows installers the day after the release went out.

If you want to grab it, hit twistedmatrix.com.

chibi simulationism

2006-05-11T16:18:00.000-05:00

Twenty Two Illustrations For an Unreleased Book about Physically-Based Simulation [sic]

Next Imaginary ticket: Implement a low-level physics simulation engine capable of simulating all pictured situations.

(Watch out, some of the other stuff on that site is a bit, uh, adult, but the above link is clean and extremely cute.)

Multi-Metaverse with distributed revision control

2006-04-25T22:32:00.000-05:00

I'm thinking that carrying around a digital audio recorder will be necessary when I go out with Jean-Paul or Glyph. A podcast would be way easier than a blog. So, Divmod Imaginary.

This is what Imaginary will be in N years: a Guy runs an Imaginary server. He makes an agreement with another Guy running a different Imaginary server to put a portal between their worlds. They want to allow players to take their horses with them between the worlds, but they disagree about the appropriate power of weaponry in their worlds, so while they allow players to carry Swords between the realms they don't allow them to use them to their usual effect.

This should be supported by the infrastructure: code will be transferred to the host that needs it when necessary, and that code will be run with capabilities configured by the contract between the two Guys. Restricted execution is facilitated either but a custom runtime or via a restricted PyPy-based interpreter.

We'll be storing the code in the database, of course, and using bzrlib so it's not just a glob of unversioned crap. So, when Guy B modifies the Horse class so that it supports the new action "pet" and Guy A modifies the Horse class so that it supports various lengths of hair, either guy can "merge Horse from Guy B" or "merge Horse from Guy A" to get the others' changes without overwriting their own.

Is that it? Anyway, please start working with us on Imaginary.

Tree

2006-04-20T18:09:00.000-05:00

I took some pictures of The Tree recently.

It is a pretty big tree. If you click it, you will also see a picture of a dog, a bush, and a tree.

UPDATE: The tree is a Magnolia tree.

Only Yesterday, a Studio Ghibli feature film by Isao Takahata

2006-04-10T22:10:00.000-05:00

I need to get this off of my chest all at once: Only Yesterday is a beautiful movie, Ghibli is a fantastic studio, and Takahata is a genius at character development. Do you see those italics? I mean them.

Unfortunately, this movie is unavailable anywhere in the US or other English-speaking countries, as far as I can tell, so the only way to legally acquire it is by importing from Japan at extortionate rates (don't worry, the Japanese DVDs have English subtitles). Rather than doing that, I downloaded it from Boxtorrents. But now, now I want to give Studio Ghibli all of my money. -- no, not Studio Ghibli, but instead, I'll give it to Japanese Scientists who will put all of their effort into designing a device that will keep Hayao Miyazaki and Isao Takahata alive forever, as slaves of their passionate fans.

I just can't believe how excellent a studio that is. Hayao Miyazaki is clearly one of the best creators of fantasy ever, and Takahata can give you a character that you have such a passionate connection with. Every millimeter twitch of the eyebrow or lip controls me like a puppet, and every tiny ordeal that the characters go through makes me so sad, like a sad puppy.

Anyway, I think I talked about this stuff before. Basically, Studio Ghibli is the best thing on the planet. Everyone should watch all of their movies.

New tricks!

2006-04-10T18:13:00.000-05:00

Two things. First, I learned how to do this:

(click on it to see a video of it).

Then, I did this!

This was after cleaning up. There was lots of blood.

On a happier note

2006-04-05T20:08:00.000-05:00

My stick-twirling is improving. I can do fairly fast transitions between in right and out left. Left is still a bit awkward, but getting better. For some reason, spinning in with my right hand is much easier than out, which is the opposite of my expections (and what I was told). I'm now also working on reversing a spin (in the same hand, with various contortions of the body), and doing the opposite transition: between out right and in left.

"A blessing is a curse"

2006-04-05T19:43:00.000-05:00

That's poetic, right? So, there are two indisputable facts I have learned today.

The OS X bless command's only function is to make a volume *unbootable*. This is a documentation bug, I believe: the man page shouldn't say anything about specifying boot files and such like.
Either my keyboard (A 3-year-old "Compaq"-branded USB keyboard) or this Mac Mini device has what appear to be race-conditions or just plain flakiness when it comes to boot modifier keystrokes. I did manage to have "C" boot from a CD today, and "alt" once even gave me the volume-picker menu. But these were isolated cases, in dozens of attempts.

I think, while #2 is certainly very annoying, #1 is the more fundamental problem, here. I pretty much need to successfully bless something in order to install Linux on this thing, unless either the beautiful people at mactel-linux.org or my coworkers create a bootable install CD.

Apple's recent announcement of Boot Camp (and specifically the newly-released technologies which enable it) was a glimmer of hope, but not as much to me as to others, as I have already spent countless (5) nights having my hopes crushed by this tiny, tiny machine. In fact, after installing the Firmware update from Apple, my computer did do an interesting thing: Startup Disk now lets me select the stock Dapper installation CD, and I can successfully boot it! Very interesting. The video is almost complete corrupt, though, and after about 5 seconds of spewing bootup output, the screen goes black. Now, I appear to have gotten the thing into a Legacy Mode (BIOS) which it refuses to come out of, no matter what keystroke coaxing I attempt.

got OS X booting

2006-04-04T20:29:00.000-05:00

Ok, so I'm back in an original-ish state. But now, it appears I can't boot anything but my hard drive! Why does Apple hate me. They love me when I want to use their software. Why can't they love me just for using their hardware?

So, on this Intel Mac Mini, I do not clearly remember ever having used any of the boot-up keystrokes successfully. Option? What's that? I think it's Alt, on this Windows Keyboard. Holding the Alt key down while I boot gives me no boot menu. Why do you mock me, The Web? Why do you tell me that it works? (yes, I've also tried the windows key, but I'm pretty sure that's Command, not Option). Same with "c". I booted from this CD on this very mac not 60 minutes ago -- to boot OS X when it wouldn't boot automatically. Then I got OS X booting by default again by selecting the volume in the Boot Disk. Now, I cannot boot from that very CD that saved me, no matter how furiously I depress that semi-circular letter. Maybe Apple has some secret conspiracy to not honor bootup keystrokes executed on that most hated of competitor's keyboards?

Ok, so *somehow* I booted that Live CD on this machine, what seems like ages ago -- it might have been with a C. But maybe by that time I had already destroyed the alternative method of booting.

I guess my next option is to destroy the bootability of my computer again, thus forcing the beast to look for an alternative in the optical drive. But my real goal is to get it booting from a USB stick. An impossible task which is monumental, even from this far-off vantage.

Twirling a Stick

2006-04-04T15:24:00.000-05:00

I can twirl a stick forward and reverse quite smoothly and quickly (for having picked it up this week) with my right hand, and kind of choppily forward with my left hand. I can't do reverse with my left hand yet. My next milestone will probably be a smooth reverse-right to forward-left transition, which I can clumsily do now. Then, once my reverse-left is reasonable, forward-right to reverse-left.

Then: TWO STICKS AT ONCE.

I guess I will have to find another stick.

Maa maa maa! Oh toh toh toh!

2006-04-04T10:58:00.000-05:00

Note to self: when you visit Japan, make sure to learn How to eat Sushi in Japan.