fettig.net

I’ve put a new version of Hep up for testing (see the bottom of the page).  This is an alpha release, meant for developers and curious people.  It has some nice new features:

• Uses Mark Pilgrim’s rssfinder module to find the RSS feed for a web page, searching syndic8’s database if the page itself doesn’t contain a link to the feed.  This means that Hep should be able to find the RSS feed for almost any site that has one.
• Also uses Mark’s rssparser module to parse RSS feeds, which works around some buggy feeds and eliminates the problems some people were having with Python’s XML libraries.
• Lets you set up message destinations, and has an SMTP server that accepts mail sent to those destinations.   Now you can send e-mail to your weblog (through the Blogger or MetaWeblog APIs), or your advogato diary, or both at once.
• Lets you view your Inbox on a web page, so you can read and delete messages without using your mail client if you want.

A complete changelog is here.

But before you download, let me remind you that there are reasons that this is an alpha release:

• The SMTP server has NO SECURITY, and only works for the first Hep user you create.  Don’t run it on a shared or public system unless you want the whole world to be able to post to your weblog.
• The SMTP server hasn’t been thoroughly tested and might occasionally lose an outgoing mail message.  A suggestion:  put yourself on the bcc list for any important messages that you send.  That way you can verify that your message got through.
• The first time you run the new version you’ll probably get a lot of messages you’ve already seen (the way it generates message IDs is a little different, so all RSS messages look new).

These things will all be fixed soon enough, so if you’re just using Hep as an RSS aggregator and you’re happy with the way it’s working I recommend you wait to upgrade, at least until this release gets into the beta stage.

If you’re an existing user and you want to upgrade, here’s what you need to know to get started fast.   Edit the hep.ini file that comes with the distribution (don’t replace it with your old one - there’s some new stuff in there related to the SMTP server, and it won’t work without it).  Set the SMTP upstream_server=your.smtp.server.   Set your mail client to use localhost, port 5337 for outgoing mail.   Send mail to [destination_name]@hep.   And when you set up your destinations don’t put spaces or special characters in your destination name.

If you’re interested in seeing an almost complete list of things that need to be done before 0.3 is ready, check out the TODO file in the distribution.

Sitting on a bench outside the coffee shop this afternoon, I think I finally figured out how Jabber instant messaging can be connected to Hep.  I’m putting my thoughts up here for comments.   This is a first draft.  Let me know what you think.

1. You create a Jabber account.  Say joesmith@jabber.org
2. You set up some message destinations in Hep:  your weblog, your advogato diary.
3. Hep will log into Jabber using your account, once for itself, and once for each destination you’ve got set up.  For each login it uses a different Jabber resource: joesmith@jabber.org/Hep, joesmith@jabber.org/Weblog, joesmith@jabber.org/Advogato.
4. You give Hep a list of Jabber accounts it should allow posts to come from.
5. Now anyone can send messages to joesmith@jabber.org/Hep.  These messages go to your Hep Inbox.  The /Weblog and /Advogato connections only accept messages from the accounts you’ve specified.  These messages get routed to the appropriate destinations.

Here is one man’s wish list for an XML aggregator: Written in Python, can search for RSS feeds on web sites, uses syndic8, can route messages through e-mail and/or the blogger API.  Hey Rick, I’m working on a project called Hep.  It’s written in Python.  It’s close to doing the things you’re looking for.   It’s open source.  Try it out!

I’ve been playing with my new wireless card (as you may have guessed
from my last post). I may write up something about what I’ve learned -
as a wireless newbie I was disappointed by the lack of good real-world
documentation on what to expect from wireless - things like what kind of
signal strength you need to get a connection, how to tell if the network
you’re on is using encryption, etc. Maybe I just haven’t looked in the
right places, but I’ve spent the time on Google.

I’m still working on Hep 0.3, but I think I’m going to have to push the
release back to next week, and even then it will probably be an
“0.3-alpha” release. Hep 0.3 has a built in SMTP proxy server that
watches messages going to your ISP’s SMTP server. If it sees a message
addressed to <address>@hep, it delivers it to a destination you’ve
configured. I’m sending this message to weblog@hep (my weblog), and
cc’ing advogato@hep (my advogato diary). It seems to be working so far,
but the thought of people sending all their e-mail through Hep makes me
nervous: suddenly Hep will be handling information that’s actually
important. So I want to do some good testing, and let the brave and
curious use it for a while before I do a “real” release.

posting from downtown Portland, near the new hotel.

The weather is hot and I’ve been busy.

I’m working on Hep 0.3.  It should be ready this week.  I think you’ll like it.

Mark Pilgrim introduced his new "ultra-liberal RSS locator" today.  It’s similar to the code I added to Hep 0.2.3 to do RSS auto-discovery.  But it adds one brilliant feature.  If it can’t find an RSS feed by scouring the HTML source of a site, it looks the site up in syndic8’s database - a vast repository of RSS feeds, maintained by real people.  This means that Hep should be able to find the RSS feed for virtually any web site, without any work on your part.

After trying out Mark’s module I found a bug, where it would crash on sites that had mailto: links.  Also it didn’t sort the results it got from syndic8, so sometimes you’d get a list of possible RSS feeds for a site without any way to tell which one was the best.  I wrote up a little patch that fixed these things and sent it to him.  So now I’ve got a nice little RSS auto-discovery module that works just the way I want it to, plus the satisfaction of contributing to what could be a widely used bit of code.

Also, for the first time in maybe a year I’m going to spend money on gadgets.  I read the Best Buy flyer in this weekend’s paper.  They’ve got some things on sale that I’d like to own (a wireless network card, and more memory for my laptop).

Oh, and I’m writing this post in my e-mail client.  To: weblog@hep.   One of the new features in Hep 0.3 :-).

I bought VMware last year, but haven’t really used it - nothing I’ve been doing since then has required Windows, so I’ve been perfectly happy doing everything in Linux. But now I’m using Windows at work, and soon I’m going to setting up a Windows installer for Hep. Plus I’d like to play around with .NET, now that there’s a free .NET IDE. So I’m installing Windows ME in VMware (off a CD that come with my laptop, long ago). So far, so good.

(I’m posting this more as a

favor for Dave then because it’s particularly interesting).

I’ve been getting some very good suggestions and bug reports for Hep. David Dorward wrote to say that Hep didn’t work with Fetchmail, since its POP server didn’t support the TOP command. I’ve added support for TOP in Hep 0.3.4, which I’ve just uploaded. Grab it if you’re getting lots of Hep debug messages along the lines of “I don’t know what TOP means”, otherwise you can ignore this release. David also reported that Hep messages get zapped as spam due to their lack of valid To: and From: e-mail addresses. This will be fixed in Hep 0.3. Also Mathew Yeates is reporting buggy interaction between Hep and Mozilla mail, and Troy Allen is testing Hep with Outlook. Wari Wahab is bypassing Hep’s POP3 server because he’s not fond of POP, and says that it works “the way I ‘almost’ want”. Wari suggested a better way to keep track of which messages Hep has already downloaded, which I’ll be using for Hep 0.3.

Thanks for the feedback, guys - Hep is getting better thanks to you.

I just put Hep 0.2.3 up for download. I had been planning to wait until I finished all the features for 0.3 (most notably accepting mail messages through SMTP and posting to weblogs) to make a new release of Hep. But I had already completed and tested some new features and bug fixes, and in the spirit of

“release early, release often” I thought i’d make them available.

Changes from 0.2.2

• Fixed a bug where some messages would be stored in such a way that they appeared to have no message ID. The result was that these messages would never be deleted, and lots of copies would build up on your system. If you’ve been using Hep and upgrade to 0.2.3, you’ll probably see a large amount of old messages that had accumulated, including many copies of certain messages. This should only happen once, after which everything should be back to normal.

• Added an autosubscribe feature for community sites using Slash, Virgule, or PHP-Nuke, weblogs that use either a <link> tag or an xml.gif image to point to their RSS feed, and individual diaries on Virgule-powered sites. For these sites you can give Hep the URL of a page you’re interested in and it will automatically set up a message source.

Download:

zip

tar.gz

I added a new feature to Hep. Here’s how it works. You give Hep the URL of a web page you’d like to subscribe to. Hep grabs the page from the URL, and passes it (along with the original URL and HTTP headers) to each of its message source drivers. Each driver gets a chance to analyze the page information and try to get messages from that page. If it can, it returns a working configuration. If not, Hep goes on to the next driver.

Currently I’ve added support for community sites using Slash, Virgule, or PHP-Nuke, weblogs that use either a <link> tag or an xml.gif image to point to their RSS feed, and individual diaries on Virgule-powered sites. And I wrote a simplified “Subscribe” bookmarklet that just passes Hep the URL of the web page you’re currently looking at. This makes the process of adding a message source almost completely painless:

1. While browsing, you come accross a page you’d like to subscribe to.
2. You hit the Subscribe link on your browser’s Links toolbar.
3. You’re transferred to Hep, which verifies that you want to add the page as a message source.
4. You press OK, and are returned to the original page.
5. From that point forward you get a message in your Inbox every time the page is updated.

All this and more in Hep 0.2.3, coming soon!