Andy Carvin's Waste of Bandwidth

Experimenting with Email-to-RSS Syndication

A few days ago I thought it would be interesting to see if it would be possible to use RSS feeds to syndicate my WWWEDU and DIGITALDIVIDE discussion lists over the Web. In a nutshell, RSS syndication is a way of sharing dynamic online content, such as a blog or a news feed. Normally, someone's website might feature a static link to someone's blog, for example, but RSS syndication allows them to display the actual blog entries themselves -- or at least their titles and a link to the blog entry.

I've been using RSS feeds as a way of displaying the titles of the latest photos from my photo.spotlight gallery on my blog. If you go to my blog homepage and look at the left-hand column, you'll see the RSS feed for photo.spotlight. Every time I post a new photo to that site, the list of new entries gets updated on this site. That's RSS syndication.

Getting RSS syndication to work between my two websites was one thing, but then I wondered if I could also display the latest email postings from my discussion groups. As it turns out, there are several tools on the Web that "parse" an email into RSS. It doesn't matter if your email is simply a hello to a friend, or the latest message from your discussion group; these parsers will convert your emails into RSS so you can then syndicate them to your hearts content.

Two of the most interesting parsers out there are MailBucket and DodgeIt. They both work on the same principle: you create an email address at either site -- for example, andy-rss-test@mailbucket.org and andy-rss-test@dodgeit.com. Try sending a test email to either of them. Once you've done this, you can go to each website to see the RSS feed generated by the incoming emails. Follow these links to MailBucket and DodgeIt to see the RSS feeds. (From what I can tell, the DodgeIt RSS feed gets generated almost immediately, while MailBucket takes its own sweet time before appearing as RSS.)

When you look at either of these RSS feeds through a web browser, they'll probably just look like a bunch of code and text - not very readable to the average person. But if you take the URL of each of these feeds and then go to a website like BlogLines, you can set up a personalized collection of your favorite RSS feeds and view them at Bloglines whenever you want.

Most people use sites like Bloglines to follow their favorite blogs, but why not use it to follow your favorite discussion lists? This is where tools like DodgeIt and Mailbucket come in. Basically, it's simply a matter of subscribing a new email address to your discussion list of choice that would send a message back to either DodgeIt or MailBucket. Example: pretend there's a discussion group called ANDY-COMPLAINTS for people who want to complain about me. You'd simply create a subscription to this discussion either as andy-complaints@dodgeit.com or andy-complaints@mailbucket.org, then go to their respective website to get ahold of the RSS feed generated by the website. Once you've got the RSS feed's URL, you'd simply go to bloglines.com or a similar "news aggregator" service and add that URL to your collection of favorites.

In my case, I was interested in displaying RSS feeds for my discussion groups on my blog homepage, as well as encouraging friends and colleagues to do the same on their sites. My first experiment involved utitilizing the RSS feed automatically generated by Yahoogroups.com, which I use to host my WWWEDU list. In the case of WWWEDU, you can find the RSS feed simply by adding /messages?rss=1 to whatever the homepage of the yahoogroup list is. This means you don't necessarily have to use a tool like Dodgeit or mailbucket to do it for you, since yahoogroups does it automatically. As for my DIGITALDIVIDE list, since it's not on yahoogroups, the only way to take advantage of Yahoogroup's RSS feeds was to create a new yahoo group that has only one job -- to subscribe to the DIGITALDIVIDE list. So whenever someone posts a message to my list, a copy of it gets sent to the fake list a yahoogroups. Wham, bam, instant RSS feed.

The next trick is getting these RSS feeds to display on your homepage. In my case I'm experimenting with feedroll.com, which allows you to input an RSS feed to generate a short script for you to include in the HTML of your website. Let's go back to the example I gave before, in which you can send an email to andy-rss-test@mailbucket.org and andy-rss-test@dodgeit.com. Doing this allows you to create RSS feeds for emails that get sent to those addresses: see the examples I have for MailBucket and DodgeIt.

With Feedroll.com, I can take the URL for either of these two feeds and generate a little script that can be placed into the HTML of any website. This HTML then displays a list of all the emails sent to either dodgeit or mailbucket as if they were a newswire feed. Here, for example, is the DodgeIt feed, as seen by feedroll:

As you can see, it has a problem displaying the feed; it's interpreting the date of the email as the title of the email, so it doesn't look very good.

And here's the feedroll interpretation of our test of Mailbucket:

This one looks better, though it still seems to take a while for it to be up-to-date with the latest emails.

RSS feeds generated by Yahoogroups work okay through Feedroll, but not great. The problem is that the Yahoogroup RSS feed displays the latest message at the bottom, rather than the top, whereas feedroll assumes that the latest message appears at the top, as is standard on most blogs. What does this mean in practical terms? For example, here's the Feedroll feed for my WWWEDU list, based on the RSS feed generated by yahoogroups:

As you can see above, the only way to get the most recent posts from WWWEDU to be displayed this way is to show all 30 of the posts captured by Yahoogroup's RSS feed. The most recent messages are at the bottom of the list, and the oldest are at the top.

So, if you try to show only a handful of posts - let's say five - it looks like this:

This is much easier on the eyes, but it's not up to date, since it's cutting off the 25 most recent messages because it's only showing the five oldest out of 30 messages displayed by yahoogroup's RSS feed. (got it?) Compare the two feeds to see how the second feed is displaying only relatively older messages.

So what's the solution? Good question. For now, I'm having my blog display my WWWEDU and DIGITALDIVIDE messages by using yahoogroup-generated RSS feeds processed by feedroll. I have to display over 20 messages for each list so they display fairly recent messages. The more I display, the more recent they are; the less I display, the less recent. I'm hoping the holy grail in my quest for a good email-to-rss parser may be MailBucket, but so far the website is running so slow I can't tell how it's going to work. Stay tuned to this blog to find out how the quest goes.... -ac

Posted by acarvin at February 28, 2004 04:59 PM | TrackBack