This week I worked 62 hours straight. Actually, that's a lie, I worked 65 hours straight, with 3 hours of naps.

Twitter has stopped being fun.

I started Twit Cleaner in 2009. Given that I haven't managed to develop a livable income out of it in that time (barring donations from you many wonderful, generous people), it's pretty obvious I'm not in it for the money. It's been my primary occupation over that time, and absorbed several thousand hours of coding, designing, building, testing and supporting it. Everything on Twit Cleaner (bar some low level server decisions), I've personally done.

I wanted to make Twitter a better place for everyone. I love to code. I love helping people. That's it.

Unfortunately, the new API is so crippling that Twit Cleaner is unable to continue. If I can't even maintain the base service, I certainly can't (won't) roll out any new functionality I had planned, all of which sits on top of Twitter's APIs.

So, here's some things I had in the pipeline that won't see the light of day:

Filtered follow back - like auto-follow, but filtered through the (user customisable) Twit Cleaner reports , so if they would appear on (your choice of) report sections, you won't follow them; only if they're awesome. I've had this running on my personal account for a year or so, it's just needed some UI whipped up around it. Oh, and a billing system.

DM guard - deleting the myriad of DM spam we all receive. Making Direct Messages useful again. Additionally, catching and quarantining all the phishing attacks Twitter suffers, and feeding those back en masse to Twitter, so they can break the links and quickly quell any new attacks that arise, before they spread. This is already partly running on my Twit Cleaner account.

Spam Hunter - a neat game (a core chunk of the back end code already written) to make it fun (and super efficient) to hunt spammers on Twitter. Thus, the masses pool their efforts to remove spam from Twitter (since of course, all this would be collated and automatically fed back to Twitter's spam team). I reckon, combined? We could make Twitter spam history (or so expensive as not to be worth it for the spammers). I've tracked, identified and reported hundreds of thousands of spammers off Twitter.. just while testing this code - not even using it in force.

At the risk of sounding immodest, Twit Cleaner was a stupidly robust system (optimising systems is something I'm really good at).

Most of the downtime I did experience was because I was constantly tweaking and improving things, while working at the outer limit of my skills in some pretty arcane technical areas.

I could happily leave the system running for weeks at a time while I worked on other code. I'd do tech support, but not really touch the engine. It would gracefully handle the plethora of random errors emanating from Twitter's complex environment; interpret, adjust and retry. Nothing would be lost. Not much would slow.

It was an engineering marvel built, as it were, atop a fail-whaling ship.

And yet the v1.1 Twitter API "upgrade" hit like 20 ice bergs.

Every single Twitter related part of my system started screaming. Yes, some bits I'd made bad choices or coded less-than-perfectly, but when you fix 15 major problem sites, and there's still more, many of which I genuinely have no idea how I can correct, adjust for or code around? That's dark.

And when you factor in how smoothly everything ran before this upgrade?

It's not fun.

Now, to be really clear. I don't begrudge the decisions Twitter has made with these changes. They're running a system with a couple of hundred million users. Mine has a couple of hundred thousand. At that many orders of magnitude difference, things change. A lot. With thirty years of tech experience, I marvel that Twitter works at all - let alone as quickly and (relatively) painlessly as it does.

All that said, I'd hoped that dropping the maximum user size down would help things, but quite literally, every part of my system is still complaining (despite a ton of work by me). I'm running out of API calls (how much Twitter will let me do, per 15 minute block) everywhere. Everywhere.

The Twitter API documentation says I should check that. Oh, wait, I've run out of API calls to check how many API calls I have left. WTF? Seriously?

And so on.

Multiply that by 60+ hours in 3 days and you start to see why I'm walking away from the dead horse.

I suspect I won't be alone.

Running Twit Cleaner has been one of the most amazing experiences in my life. I've met so many incredible people. I've helped, in a small-but-significant way so many to have a better experience on Twitter. I deeply regret that I can't continue to help improve your lives. It's not feeding starving children in Africa - but you know, it wasn't nothing. And wow, you people are amazing. Thank you, from the bottom of my heart.

Thank you also to both Matt and Katy in the spam team at Twitter. It was an extreme pleasure and an honour helping them kick all those bad guys in the ass.

Through all this I've learned so, so much.

Perhaps the key thing? Never play football when someone else owns the field.

So obvious in hindsight.

Perhaps if I'd known that in Oct 2009, when I started all this, I might have taken a very different path.

But I would have missed so, so much. And I'm incredibly grateful for it all.

From here? I have no idea. I'll see what the universe brings. A holiday would be nice. In the mean time, of course, I'll keep learning how to love more and keep posting weird crap on my personal Twitter account, @sidawson.

Take care out there.

Si

 

[related: Some of the feedback Twit Cleaner has received over the last four years]

In the two weeks since I announced the shutdown of Twit Cleaner, the following has happened:

I've personally responded to:

• Somewhere between 3-4000 Twitter messages (I figure, if people cared enough to comment, they deserved a response)
• 200-ish blog comments
• 50+ emails
• a stack of meetings and proposals of various seriousness/ridiculousness ("You should write Angry Birds, but on a zipline! It'd be easy! We'll make millions!")

On top of that, I had offers from two different "competitors" (of sorts), to buy, variously - the domain, the traffic, the email list (ie, all your email addresses), and the Twitter account. One offered a trivial amount, in exchange for wanting me to spam you all, three times a day for three months.

Uhh. Yeah, really.

Needless to say, I told them (politely) to bite the wax tadpole.

Possibly Replacement Services

Unfortunately, despite looking, I haven't found any other services that with good conscience I can point you to as a reasonable replacement.

There are a dozen services that do the basics - who's not following you back, who's left Twitter (although be warned here - Twitter gives false information; a lot of these services tell you WAY more people have left than Twit Cleaner would have, because I very carefully checked this data several times, to compensate for Twitter's unreliability).

There are no services that go to the depth that Twit Cleaner did.

Why not? Simply because doing that requires an absolute ton of data. Getting that data is no longer possible in a reasonable timeframe. Thus, any service that could have done this would have had to shut down, same as I did.

The Source Code

Regarding the code-base. I've had several people suggest (*cough*demand*cough*) I open-source the whole lot, "so people could run their own reports" (aka "You've spent 10,000 hours writing this, give it to me for free!").

Unfortunately, this simply isn't practical. Twit Cleaner consists of several hundred database tables (many of which require specific maintenance processes to keep them operational and performant), and a dozen different processes all carefully balanced and interacting.

Additionally, the problems I had getting data from Twitter are actually worse if you only have access to a single account (because a lot of what I did was only possibly by spreading data requests across tens if not hundreds of accounts - but shh, don't tell Twitter that. It may have been a little naughty).

In a nutshell, the core reason Twit Cleaner has closed is because data access has dropped to 111-333 times smaller.

I do plan to roll my core Twitter library changes back into python-twitter (mainly more robust error handling code), if the maintainers want them. I'll also be writing a post or two about some of the more subtle maintenance techniques I had to figure out to keep Twit Cleaner running. I'm sure they won't be a surprise to the MySQL gurus out there, but other beginners might find them helpful.

What's Left?

It's nearing the end of the month, so the server will be shut down this weekend (otherwise I have to pay server fees for another month). I'll be moving the blog over to another server (same domain, different hardware, different ip address), but all Twit Cleaner related services will cease once that happens. Which means you only have a couple of days left to do any last minute cleaning, if you still have a report sitting around.

I will keep blogging here. There are still a few things worth saying in this space. Plus, of course, I'll still be keeping you alerted to trials and tribulations in Twitter land, over at @TwitCleaner.