Today, that all changes. Well, almost.

See, when I first started Twit Cleaner, I wanted to get something useful out there as soon as possible. A Minimum Viable Product, it’s called. That worked well enough, but when I started looking at moving to a one-click-one-unfollow model, I realised that the current infrastructure – the way I’d designed things – simply wasn’t going to work. It was possible to do new things, but it would have been horrible, painful & slow.

I made some (with hindsight, of course) silly technical choices, & they came back to bite me on the ass.

So, for the past 7 months I’ve been going through redesigning the entire of Twit Cleaner, more or less from the ground up. The first of that giant chunk of work rolled out yesterday. Believe it or not, but the previous version had almost everything just shoved into a giant directory tree*. So yes, that’s a folder with many, many million files in it. It worked ok for the one task it was designed for, but it seriously hampered the ease & speed with which I could develop any neat new tools. There were a bunch of other bad technical decisions, but that was the key one.

Now, everything is in a big shiny database. Which has its own issues, of course (everything does) but I’ll iron those out over the next couple of days. Once that’s over, & the database is fully loaded up (it’s happening as we speak, and looks like it might take a few days to complete so expect the site to be a little shaky until that’s finished, please be patient), it will smooth the way to quickly & easily roll out a bunch of new tools to help you manage & explore your Twitter life. Oh yes, I have many, many great ideas I’ve been working on.

On the outside, things may have been serene, unbroken, just meandering along like a duck floating on a pond.. but underneath I’ve been, just like a duck, paddling furiously seven days a week all hours of the day & night to get things working just the way they should be – and to get you guys the help you deserve.

Unfortunately, like a duck, there’s not much to look at just yet. Oh, except reports will be much, much faster.

*If you’re really curious, the very first version of Twit Cleaner used to run on my desktop machine at home in Melbourne, then copy things furiously back & forth to the web server, which at that time was in London. Now THAT was nutty.

We’re currently working on a streamlined manual version (which will allow you to unfollow a single account at a time). This is our number 1 priority. We’ll be in touch very soon.

The official cut off date is May 21st, but we will be stopping over the next couple of days (ie, well before cut off). Our intent has always been to do the right thing by Twitter & we will continue to operate within their guidelines to the extent of our ability.

We will attempt to complete all existing unfollowing obligations before that date, although those accounts wanting 50k+ unfollowed we may just have to do as many as we can. Obviously we need to stay on the right side of Twitter, or risk having our API access removed altogether.

After that, we will still offer reports, & you’ll still be able to unfollow dodgy accounts – just one at a time. We’ve had extensive discussions with Twitter & they have told us that unfollowing one at a time is ok with them.

At this point I can’t say how much of an overlap there will be between old style (mass) & new style (one at a time) reports, so we may shift into a read only period for a while (reports but no unfollowing).

The key point right now is this: If you want to get a mass clean of your account done, I suggest doing it very soon, as we will be stopping this any day now.

Specifically, they want to avoid churning – that is, following a ton of people, then unfollowing those that don’t follow back, repeat ad nauseum. I suspect part of their motivation is to limit spammy behaviour, & part is that it’s a massive drain on their servers (getting hit with thousands of API requests in a short period of time).

I realise that with The Twit Cleaner, we skirting a fine line. However, my priority is to keep your accounts safe, operate within the guidelines of Twitter, while improving the quality of experience for everyone.

In short: I want to improve the Twitter experience as much as I possibly can – but without pissing Twitter off (or causing them any hassle) in the process. I’ve been very careful to try & ensure that the service is the least possible use to those I’m trying to rid Twitter of – those engaging in churn or other spammy practices. Ie, there’s a lot of things I could have put in, but I deliberately haven’t because of any possibility of abuse.

Of course if Twitter says jump, the only appropriate response is “how high”, but I believe we’re safe because:

1. You only ever have the option to unfollow people that are bad Twitter citizens in the first place – typically a very small percentage of anyone’s account
2. We do the unfollowing very, very slowly (only one every few seconds) to limit drain on Twitter’s servers
3. We never unfollow more than a small percent of your account per day, no matter how many you request.

To this end, I have slowed the unfollow down even further than before. It will now not unfollow more than 20% 10% 5% (or 500, whichever is smaller) of your friend count per day, as well as spacing each unfollow out much, much more slowly.

Ergo, if you want to use The Twit Cleaner to empty your account, you’re better off going somewhere else (it’s not something I’d recommend anyway). If you want to use it to trim out spammers & time wasters of course, we’re the guys for you.

It’ll still happen, just very, VERY slowly. Don’t hold your breath :)

It takes a lot of work to get something that clean yet still intuitive.

I’ve always believed that the reason we have computers is to make things simpler for us. We shouldn’t be burning our own cycles if we can get the machine to do it for us.

Now, I’m the first to admit that The Twit Cleaner is nowhere near that level of elegance yet, but it’s an iterative process, & a high goal. I’ll keep pushing towards that.

In the meantime, I’ve simplified the pricing.

Now, things are like this: All reports are free, as always. If you follow fewer than 2000 people, we’ll auto-unfollow whoever you want for free. If you follow more than 2000 people, it’s five bucks.

Yep, Five US Dollars ($5) to clean your list.

It’s hard to get cleaner & simpler than that.

[Edit: Ok, that pricing has been in place for a week or so, & some interesting things have happened. For example, I had two users in the space of half an hour, each with 60k+ lists. Given that creating a report for a list of that size involves downloading many gigabytes of data & 6-12 hours of processing, I started to think "Is it really worth doing all that work, for five measly bucks?" This is even more the case when not everybody that requests a report pays, of course.

Even a list that is 20,000 people basically takes ten times as long as a 2,000 list - due to getting the lists from Twitter, downloading the data, & running the analysis. There are no real economies of scale.

So, I've adjusted the pricing (again). I still like the $5 mark, & for most people, that'll still be it. For the whales though? Above 25k users is $10, & above 50k is $20. It's still not a hell of a lot, but I feel it's a better reflection of the costs & effort involved.

We'll see how long this pricing sticks for. Everything is a work in progress, & over time I'm sure things will settle down.]

What happened
Previously, you could enter the url for anyone else’s report, & see it. These URLs weren’t public anywhere, & the backend was deliberately setup so you couldn’t just scan a directory & find them. I was aware of this, but figured the chance of randomly guessing another user’s name was low enough that it didn’t require immediate fixing. I was wrong.

What the hacker did was search for anyone who publicly mentioned The Twit Cleaner. That then gave him (because yes, I know exactly who did it) their usernames, & he could then see their report.

So, once he was viewing their report, he entered a rude message into the “Tweet my followers” box, & hit send.

That was the extent of it. Not very clever, mostly just annoying. At no point did he have access to any of our databases, your OAuth info, or any level of control over your account.

This affected around 1.6% of our customers, & the entire thing was over in 18 minutes (before I could figure out what was going on & shut down the right bits). I started by going to Twitter & killing our OAuth access, since I figured that was the most dangerous possibility – turned out it was much more trivial than that.

Yep, that’s right, put the security back in. I’d had security in there a couple of weeks earlier addressing this very issue. However, in the first 6 hours it was in place, it managed to piss off 44 people, so I ripped it out again. I figured it was more important that people be able to easily see their reports, rather than just annoy people like crazy. I needed to think of a better way to implement the security, so I put it on the back burner while I worked on other things.

Obviously I made the wrong call.

Second. We store information locally tracking that you are who you say you are. Just something to make it harder for people to try and get around the system, & no, nothing personal or incriminating, just a marker.

Third. You must be signed into the specific user of the report you want to look at.

Fourth. Without cookies & javascript you won’t be able to access the system at all.

I think I’ve achieved that. A common issue is that Twitter (or part of it) goes down, or is inaccessible. As much as possible, I’ve made it so you’ll still be able to securely see your report (or request one).

I’ve also tried to make as much of it invisible to you as possible. There’s a lot happening automatically in the background. If you clear cookies, or move to a new browser, you will need to re-authenticate with Twitter, but I’ve made this much simpler & cleaner than before. A new window pops up, you can watch it do its thing, then it goes away again. In most cases, you’ll click two buttons & be done. Very simple.

If (when?) Twitter dies, you’ll get a little message so you can just hit the button & try again.

Obviously this happening at all is. Ahh. Hmm. Significantly sub-optimal. I was hoping for a little more time before I needed to get super hardcore about security. Security is a multi-layered, complex thing. Big chunks had already been taken care of, but obviously not enough.

As I stated at the time, I’m extremely sorry to everyone that was hit by this hacker. All affected paying customers had their money refunded.

I also apologise for the time we’ve been offline – both to existing users wanting to see their reports, & new potential users.

I’ve inserted a blog section, so as things upgrade & improve I can provide a little more more information than 140 characters allows – as much as I love Twitter.

It’s also a move to allow better feedback for & integration with you guys.

The comment system is run by Disqus, who are basically the best & largest comment system out there. You sign up with them once, register a picture etc, & can then comment without needing a sign-in on thousands of blogs across the web. A bit different if you’ve never seen it before, but thoroughly awesome.

I’ve also upgraded the hell out of the communication with Twitter. Twitter can be pretty unreliable, so I’ve done all I can to mitigate that – and make it a lot more visually obvious what’s going on.

Oh, and once you’ve requested a report, you can come back here any time & it’ll be accessible from the front page. It’ll tell you who you’re signed in as (if you have multiple accounts), & there is a direct link both to your report, or if you’d like to rerun the report, one click & it’ll be done.

Anyway, have a play. I look forward to getting your feedback.