It's been a heck of a long time since there's been much external perception of progress around here.

Today, that all changes. Well, almost.

See, when I first started Twit Cleaner, I wanted to get something useful out there as soon as possible. A Minimum Viable Product, it's called. That worked well enough, but when I started looking at moving to a one-click-one-unfollow model, I realised that the current infrastructure - the way I'd designed things - simply wasn't going to work. It was possible to do new things, but it would have been horrible, painful & slow.

I made some (with hindsight, of course) silly technical choices, & they came back to bite me on the ass.

So, for the past 7 months I've been going through redesigning the entire of Twit Cleaner, more or less from the ground up. The first of that giant chunk of work rolled out yesterday. Believe it or not, but the previous version had almost everything just shoved into a giant directory tree*. So yes, that's a folder with many, many million files in it. It worked ok for the one task it was designed for, but it seriously hampered the ease & speed with which I could develop any neat new tools. There were a bunch of other bad technical decisions, but that was the key one.

Now, everything is in a big shiny database. Which has its own issues, of course (everything does) but I'll iron those out over the next couple of days. Once that's over, & the database is fully loaded up (it's happening as we speak, and looks like it might take a few days to complete so expect the site to be a little shaky until that's finished, please be patient), it will smooth the way to quickly & easily roll out a bunch of new tools to help you manage & explore your Twitter life. Oh yes, I have many, many great ideas I've been working on.

On the outside, things may have been serene, unbroken, just meandering along like a duck floating on a pond.. but underneath I've been, just like a duck, paddling furiously seven days a week all hours of the day & night to get things working just the way they should be - and to get you guys the help you deserve.

Unfortunately, like a duck, there's not much to look at just yet. Oh, except reports will be much, much faster.

*If you're really curious, the very first version of Twit Cleaner used to run on my desktop machine at home in Melbourne, then copy things furiously back & forth to the web server, which at that time was in London. Now THAT was nutty.

Identifying people that are high volume has been something that has taken the longest time to really settle down.

I'm getting much happier with this latest (the 3rd? 4th?) incarnation.

If you remember, the previous version simply identified anyone that tweeted on average more than 50 times in a day. That was much more reliable than earlier versions, but suffered from one major limitation. There's a huge difference between someone who tweets "hello world!" in the morning, then uses the other 49 tweets to chat to their friends, vs someone that just blasts out junk 50 times a day.

The difference is - how many of those tweets are public?

Why are high volume tweeters even a problem? Well, this is something that people tend to forget once they start following more than a few thousand people. When you're following that many people, there are so many tweets flying past it's mostly a blur. So, if you tweet like crazy who cares, it gets lost in the blur, right?

What is forgotten is this: Of the active users on twitter, most people only follow 2-400 others. One high volume user can flood an entire tweetstream, making it impossible to connect with anyone else.

With that in mind, the new high volume algorithm works like this. If you tweet publicly (ie, anything other than a reply) more than 24 times a day, you're listed as high volume. This isn't completely accurate, since if you have any overlapping friends, you'll see their conversations with them too, but it's a good estimate.

Once an hour may not sound like a lot, but once you factor in work, sleep, play - oh, and the fact that this is only public tweets, it's an absolute ton. So talk, talk away! Just connect, make some friends! Don't blather on about yourself all day :)