There has been some confusion about the hashtags registration process. Why do we require you to follow the bot account? Why can you just parse the public feed like other sites do? I am here to answer these queries.

Why do we require you to follow the bot account?

It boils down to the fact that we want hashtags to be an opt-in service. If you don’t want the bot to look at your messages, it didn’t. We also figure this would develop a more active user base. I mean how many people are going to use hashtags unless they know about the site? Another benefit, that we didn’t realize at the time, was that this process allowed us to handle gradual growth rather than just letting open the flood gates of twitter.  So that is why we have you follow the bot.

Why can’t you just parse the public feed like other sites do?

Simple, it just doesn’t really work. Have you ever looked at the public feed page and tried refreshing? Even if you click twice in a row, the results are almost always completely different. What that means is you inevitably miss some messages. In order to do this reliably we’d have to have multiple bots hammering the public xml feed and parsing those. Parsing xml, in general, is really slow too. We made the decision to not hammer twitter’s servers and instead took the passive approach of parsing the bot’s xmpp feed.

What’s next?

Well the beta version we’re working on runs off of the public jabber feed. So we’ll be processing every single message that comes in. The only problem is that it requires a lot more computing power. We’re processing a whole lot more messages. The beta site, feature wise, is pretty complete. Right now Ben is working on optimizing our database calls and caching the hell out of everything. Our biggest problem, if you want to call it that, is that we’re getting to much client work. Since there’s no current value proposition in hashtags itself, it’s hard to dedicate too much time to it. Any suggestions or question you have post in comments and we’ll give them a look.

This will be the first in a series of posts about the pending release of our complete rewrite of Hashtags. Why did we rewrite it? Simple, it started out as an experiment and the traffic has grown. What start out as a thought experiment with two people now has a team of four behind it. Unfortunately two of the four know ruby and the person who originally wrote it developed it in python. In order for all of us to contribute we found in necessary to move it over to ruby, specifically Merb. This allows us to a better work flow to optimize the site as well as work on new features, which we have a lot planned.

So what can you expect from the new site? Well for one, it’ll be faster. We’re using memcache to cache most of the pages as fragments and also caching slow running methods, like creating the nifty graphs. So hopefully we won’t see the slow downs that have happened on the current site. What else? Well we’re now receiving the xmpp public feed, this means we’re now tracking every message. There have been some ui tweaks as well. We’ve layout out the messages on the main page a little different, we think for the better. We’ve also changed up the navigation to make it easier for us as we start adding features.

The other big news is now you can have Hashtags user accounts. Why is this good? It’ll will allow you to track certain tags and then have a custom page that will display message from those tags. We’re also looking at the ability to recommend tags or users based on which tags you are tracking, so look forward to that.

We’ll keep you updated as we get closer to releasing it, we’re shooting towards the end of may for now. Also we’re looking for some beta testers, so let us know if you’re interested.

I’ve been playing around with Fluid, the free site specific browser for Mac OS X Leopard, for a while now and recently came up with a really interesting way to browse hashtags.org. I’ll briefly cover the steps involved to browse hashtags.org using coverflow and fluid.

(more…)

Chimeric has released a really nifty script for greasemonkey (Firefox extension) that links hashtags on the twitter website to our site. Pretty plain and simple but highly useful.

We’ve been working hard on this latest roll out. We cleaned up more parts of the already spiffy clean site as well as some really really neat looking graphs. Instead of just providing a 24 hour spark line we’ve now included a 3 month history. Check out the macworld tag for instance. The spike was the day of and the long tail afterwards is the rest of the reactions over the next few days. By now it’s pretty much stopped.

We’ve launched a new portion of hashtags.org for the specific purpose of tracking the #macworld event coming up tomorrow. Any tweet with #macworld will be index and archived along with any other hashtag into that site (along with the mainsite). We’re doing this to up the signal and down the noise. Check it out: macworld.hashtags.org.

We’re looking for some community feed back for the next round of developments. We’ve got some ideas of our own, but we wanted to see what our users had in mind. Leave a comment and let us know what you want down the road with hashtags.org.

We recently redesigned the site a bit - mainly cleaning up the sidebar and standardizing the layout so everything is uniform across the site. Brian added some nifty aggregated media stuff on the right side using jQuery & Yahoo Pipes.

CES is kicking off in a few days and we’d like to suggest a few of you who will be attending to use the #ces hashtag while you’re there. If enough folks there use the #ces tag we should be able to create a nice river of real-time information on the #ces page.

I knew that microformats were easy to implement, but the tagging microformat has to be the easiest by far. Any who, all hashtags are now marked up with microformats.

<a href="/tag/hashtags/" rel="tag">

We are going to be moving hashtags.org over to EC2 Joyent for scaling to be handled when/if we blow up and start getting traffic. We’ve seen some pretty steep growth patterns and we’re afraid that if someone like Scoble picks us up we’ll be doomed. I won’t comment on the hardware we’re using right now, but let’s just say it needs to be a thing of the past.

As well, we’re going to be adding a few features here and there through the next few days, weeks, and months… so if you have any suggestions leave us some comments. We’ll do our best to translate.

(P.S. Hashtags has officially gone into debt ($0.10/hr). Aaron upped our EC2 server and the clock is ticking!)

* UPDATE*

We’ll be moving over to Joyent. After carefully looking at the offerings a bit more we found that Joyent was a better fit for this particular app.