Archive for 2002

Håvard Pedersen writes:

I’d like to know if you have any idea when an nntp plugin might see the light of day?

The next release of Hep will include an NNTP server.  The reason it’s taking this long to get that release out is that I’m making some pretty big changes to the code Hep uses under the hood.  Realistically, it will probably be at least another month before that code is ready for a release.  

The result is going to be a better designed, faster, more featureful Hep, which of course I think you’ll like a lot, will be worth the wait, etc.  But if you can’t wait, there’s good news: some other people have written an RSS-to-NNTP program in Java, available today.

Russ Beattie, Les Orchard, and others are discussing an idea for something called a "Universal Personal Proxy", or UPP (because every vaporware product must have a three letter acronym).   "Personal" means "you run it yourself, on your own computer".  "Proxy" means "it’s between you and something else".   But it’s the "Universal" that’s the most important – it’s meant to express that this kind of software would work with all kinds of systems, not just the web or e-mail.

Of course, this is where I must chime in to say that my own little project, Hep Message Server, is a non-vaporware UPP that’s under active development.  Les has said that he’s considering using Twisted for his app (Java being the other possibility). I sent him an e-mail yesterday saying that if he decides to go with Python I’ll have a nice chunk of working code for him to play with.  So we’ll see what happens.

Meanwhile, I’ve started working on a GUI test app using my messaging library and PyGTK.  This is mostly to give me a chance to test my library in a different kind of application than Hep.  Also it’s a refreshing change from the pure-data, no-UI work I’ve been doing lately.

Working with a GUI also means that you notice speed problems much more quickly.  I already found (and fixed) a problem with the Twisted pop3 client code that made it use lots of CPU while downloading big messages.

Screenshots

An RSS feed:

A POP3 mailbox:

The same mailbox, accessed directly through the filesystem:

I spent a few hours the other afternoon working on a mini web application that uses my new messaging library.  You can try it out here:

http://fettig.net/playground/messaging_demo/

It works, but it’s not very useful at this point – you can give it the URL of an RSS file (or, more interestingly, a POP mailbox) and it will show you the messages available there, and let you view them.  It’s javascript-driven and probably won’t work on browsers other than the ones I tested (IE5 and Mozilla 1.2).  But it should give me a place to test my library as I add new features, and some of the code might make it back into Hep eventually.

Curious people can download the code here.

I know I haven’t been posting much lately, but I don’t have much to report.  My messaging library code is coming along well, though.  Soon I may have something to show off here.

Also, I got a Casio Exilim digital camera.  In brief: I like it a lot, and it works well with Linux (shows up as a USB hard drive).

This weekend I started working on upgrading Hep’s messaging library.  My goal is to end up with a nice abstraction layer for working with messages, that will be useful for other projects as well as Hep.  As part of this I’m splitting all the file-format bits (that deal with reading and writing RSS/mailbox/text/html files) into a seperate "parsers" library that doesn’t required Twisted or dealing with events.  So hopefully that might be useful for somebody.

Once the library is upgrade is done (and it’s coming along quite well already), it should be possible to add NNTP support to Hep in just a few hours (thanks to Twisted’s excellent news module).

A couple of interesting weblogs, culled from my inflated-by-Scripting-News referrer log:

• Live From Tokyo Tech Blog covers interesting things you can do with Python, Java, and the internet.
• K is a nice looking, mostly web-related blog from Portugal.  It’s always impressive when people who speak English as a second language still write better that I do.

Recently there’s been some buzz about Genecast News Service, which provides RSS-to-NNTP conversion.  People are realizing that when a new message format or protocol comes into existance it doesn’t mean you need an entirely new application.  All you need is a translation service for your current apps.  As Russ Beattie says: "there’s this very usable application sitting your desktop right now meant to read news just waiting to be utilized."

What the Genecast people are doing is, as regular readers of this site will know, on my TODO list for Hep.  The difference is in the approach we’re taking to deliver the functionality.  Hep is a standalone application that you install and run yourself.  Genecast is a service that other people run, and you use.  Which means that Genecast should be easier to set up, but Hep will be more flexible.

Brian Elliot sent me an e-mail earlier today pointing out that it would, of course, be possible to set up a public Hep server, which people could use without having to install anything.  I probably won’t be doing this anytime soon – I don’t like the idea of having a single point of failure, and I doubt that the current Hep code is scalable enough to handle hundreds of users.  But someday it might be interesting.

At some point in this discussion Dave Winer noticed Hep, and linked to it on Scripting News.  So I’ve been getting a lot more traffic, referrers, and Hep downloads than usual.  Cool!

At least these people are.

Here’s a fairly common programming task:  Given a string, see if it begins with another string.  For example, you may have a list of words, and you want to grab only the ones that begin with "Test".  So you may do something like this (for those who just walked in, we’re talking Python here):

for word in wordList:

    if word[:4] == "Test":
	print word

…which is really not too bad, except that you’ve hard coded your test as being the first four characters, so later if you change "Test" to something else you have to change the [:4] bit to match.  So instead you might do this:

searchWord = "Testing"

for word in wordList:
    if word[:(len(searchWord)] == searchWord:
	print word

Which is a bit more flexible, but harder to read.  Still, that’s what I had been doing most of the time.  But it turns out that Python gives you a much better way, that I’d never seen before just now:

for word in wordList:

    if word.startswith("Test"):
        print word

Yes, every Python string has a startswith method, that works just the way you’d expect it to.  Thanks to Mark for indirectly pointing this out to me through his code.

Wari figures out how Hep and PyBlosxom are similar: they’re both Python apps that deal with reading, storing, and editing messages that live in files.

So they could both use a common library for handling the nitty-gritty of parsing and generating files.  Moreover this library could provide a way to get generic Message objects out of many of the common formats for storing messages, including:

• RSS (0.9.x, 1.0, 2.0)

• maildir

• mbox

• blosxom-style text files

Having such a library would make it easier to write weblog tools, RSS aggregators, conversion utilities, and PIMs in Python. 

Then I’d like to write another library sits on top of the first.  This second library would (optionally?) use Twisted to provide asynchronous network operations, and give you the ability to access:

• RSS/maildir/mbox files on FTP and web servers

• Weblogs through the Blogger and MetaWeblog APIs

• Advogato diaries

• IMAP mailboxes

With such a library, it would be really easy to work with messages in Python without worrying about the underlying protocols involved.  For example, you could copy the contents of an RSS feed to a weblog like so:

store1 = openMessageStore("http://www.mysite.com/rss.xml")

store2 = openMessageStore("blogger://username:password@plant.blogger.com/RPC2/MyBlogID")
for messageNo in range(store1.messageCount):
	message = store1.getMessage(messageNo)
	store2.appendMessage(message)

A lot of the code to build these libraries is already in Hep.  Probably the most important thing at this point is coming up with nice APIs that will work accross many different message formats, and that other programmers will find easy to use.  I started working on a set of such interfaces last night.  If I have time later today I’ll post what I’ve done so far.