You can now see all about Retweets. I added this because Twitter removed it from their site (it’s still there, sort of, under “Interactions”). The retweets page allows you to see who has retweeted you, who you’ve retweeted, and also see what any other Twitter user has been retweeting. Note that this only displays New Style RTs, since that’s all the API will give us.

You can now also see a How Am I report for any Twitter user. So yes, that’s basically directly querying our database (although if the user you ask for isn’t in the database, we’ll scoot over to Twitter, get their info and analyse it for you).

Finally, I’ve also sorted the Absent users (“Other Dodgy Behaviour, Now Absent” and “No Activity In Over A Month”) on the reports by age. Those who have been gone the longest are first. That should help you prioritise a little better.

Have fun!

Obviously there are some physical limits – how fast can we get data from Twitter, how many other people want reports at the same time, how much data do we need to analyse and so on. The answer to the last one can be A LOT, by the way. For every 100k people you follow, we need to ask Twitter for roughly 3 gigs of data.

There’s been particular problems when super large users (100k+) have asked for reports. If Twitter was being particularly grumpy, it used to create enough of a backlog it could slow down everyone’s reports for an entire day. Blergh!

If you’re only following a few hundred people, why should you have to wait just because some giant user requested a report right before you?

So, over the last few months I’ve been sitting here, all day every day, watching the systems on a second by second basis. I’ve identified roughly a dozen bottlenecks, some small, some huge.

The good news is, I’ve managed to bend the underlying technology in a few ways that are theoretically impossible (according to their documentation) to get around these limitations. I’m justifiably pleased.

You will have already seen some of the benefits as I’ve been gradually adjusting things on the live system. Most of the big improvements I’ve rolled out this evening.

Oh, I’ve also taken the opportunity to roll out a few big structural changes to allow me to offer significant new functionality. What? Ahh, you’ll just have to wait and see!

Yes, I’m super pleased with progress. Yes, I’m even more excited about what’s to come!

App Spam

Accounts where more than 50% of their tweets are auto-generated by an application. Examples of this include paper.li, 4sq, blip.fm, RunKeeper, miso etc.

Uses Advertising Networks

Accounts that are being paid to put tweets into their stream. Advertising networks include organisations such as MyLikes, ad.ly, Magpie, Sponsored Tweets, etc.

Self Obsessed

Accounts where more than 50% of the tweets are about themselves. So, either starting their tweets with “I”, “Ive”, “I’m”, “I’d” or retweeting things people have said about them.

Relatively Unpopular

People who follow more than 3.3 times as many people who follow them. Or, put another way, those who have fewer than 30% people following them back. Sometimes this is just because someone likes to follow lots of people, but occasionally it can also show someone (eg spam bots) whose behaviour is offputting or boring, so very few people choose to follow them back.

The Self Obsessed and Relatively Unpopular sub-categories go into a new category, called “Not Very Interesting.” It will appear at the bottom of your reports.

As always, this is informational. It’s always up to you who you want to follow.

Personally, I follow (& will always follow) a whole bunch of people that appear on my report. Why? Because they’re entertaining, informative and I like them. Free will – it’s not an accident!

It took quite some digging to find what was happening (since everything was operating normally) but I finally discovered what the problem was.

We were only updating our records for accounts that had already been flagged to appear on a report.

In other words, we weren’t keeping up with how people’s behaviour might change over time.

So, commenting out a single bone-headed line of SQL (What was my obviously under-caffeinated brain thinking? What?!? What??!), and voila, everything is now as it should be. It means reports will be slightly slower (in this case, this is a good thing) – but only because we’ll now be pulling down much more information from Twitter, which takes time.

The upshot of this? Your reports will now have much, much more detail on them. I ran it on my personal account (@sidawson) and my report went from having 47 people on it to 190 (I’ve been deliberately leaving them there so I can test the new version)

If you’ve run a report in, ohh, the last few months, might be worth running it again now.

My apologies for not noticing this sooner. It’s all very silly.

Today, that all changes. Well, almost.

See, when I first started Twit Cleaner, I wanted to get something useful out there as soon as possible. A Minimum Viable Product, it’s called. That worked well enough, but when I started looking at moving to a one-click-one-unfollow model, I realised that the current infrastructure – the way I’d designed things – simply wasn’t going to work. It was possible to do new things, but it would have been horrible, painful & slow.

I made some (with hindsight, of course) silly technical choices, & they came back to bite me on the ass.

So, for the past 7 months I’ve been going through redesigning the entire of Twit Cleaner, more or less from the ground up. The first of that giant chunk of work rolled out yesterday. Believe it or not, but the previous version had almost everything just shoved into a giant directory tree*. So yes, that’s a folder with many, many million files in it. It worked ok for the one task it was designed for, but it seriously hampered the ease & speed with which I could develop any neat new tools. There were a bunch of other bad technical decisions, but that was the key one.

Now, everything is in a big shiny database. Which has its own issues, of course (everything does) but I’ll iron those out over the next couple of days. Once that’s over, & the database is fully loaded up (it’s happening as we speak, and looks like it might take a few days to complete so expect the site to be a little shaky until that’s finished, please be patient), it will smooth the way to quickly & easily roll out a bunch of new tools to help you manage & explore your Twitter life. Oh yes, I have many, many great ideas I’ve been working on.

On the outside, things may have been serene, unbroken, just meandering along like a duck floating on a pond.. but underneath I’ve been, just like a duck, paddling furiously seven days a week all hours of the day & night to get things working just the way they should be – and to get you guys the help you deserve.

Unfortunately, like a duck, there’s not much to look at just yet. Oh, except reports will be much, much faster.

*If you’re really curious, the very first version of Twit Cleaner used to run on my desktop machine at home in Melbourne, then copy things furiously back & forth to the web server, which at that time was in London. Now THAT was nutty.

We’re currently working on a streamlined manual version (which will allow you to unfollow a single account at a time). This is our number 1 priority. We’ll be in touch very soon.

The official cut off date is May 21st, but we will be stopping over the next couple of days (ie, well before cut off). Our intent has always been to do the right thing by Twitter & we will continue to operate within their guidelines to the extent of our ability.

We will attempt to complete all existing unfollowing obligations before that date, although those accounts wanting 50k+ unfollowed we may just have to do as many as we can. Obviously we need to stay on the right side of Twitter, or risk having our API access removed altogether.

After that, we will still offer reports, & you’ll still be able to unfollow dodgy accounts – just one at a time. We’ve had extensive discussions with Twitter & they have told us that unfollowing one at a time is ok with them.

At this point I can’t say how much of an overlap there will be between old style (mass) & new style (one at a time) reports, so we may shift into a read only period for a while (reports but no unfollowing).

The key point right now is this: If you want to get a mass clean of your account done, I suggest doing it very soon, as we will be stopping this any day now.

I’m getting much happier with this latest (the 3rd? 4th?) incarnation.

If you remember, the previous version simply identified anyone that tweeted on average more than 50 times in a day. That was much more reliable than earlier versions, but suffered from one major limitation. There’s a huge difference between someone who tweets “hello world!” in the morning, then uses the other 49 tweets to chat to their friends, vs someone that just blasts out junk 50 times a day.

The difference is – how many of those tweets are public?

Why are high volume tweeters even a problem? Well, this is something that people tend to forget once they start following more than a few thousand people. When you’re following that many people, there are so many tweets flying past it’s mostly a blur. So, if you tweet like crazy who cares, it gets lost in the blur, right?

What is forgotten is this: Of the active users on twitter, most people only follow 2-400 others. One high volume user can flood an entire tweetstream, making it impossible to connect with anyone else.

With that in mind, the new high volume algorithm works like this. If you tweet publicly (ie, anything other than a reply) more than 24 times a day, you’re listed as high volume. This isn’t completely accurate, since if you have any overlapping friends, you’ll see their conversations with them too, but it’s a good estimate.

Once an hour may not sound like a lot, but once you factor in work, sleep, play – oh, and the fact that this is only public tweets, it’s an absolute ton. So talk, talk away! Just connect, make some friends! Don’t blather on about yourself all day :)

This covers two areas:

People who retweet 70% or more of the time

Of course, some people do find the best stuff out there, but in general, if someone is only ever RTing things by other people – why not just follow the other person? This is also something that is done a lot by spam bots, to make them appear ‘more human.’

People who post quotes more than 50% of the time

Similarly to retweeting, spam bots often intersperse their crap with quotes. It’s a zero effort way for them to have ‘fresh’ content. In reality though, if they’re quoting Epicurius, this probably isn’t something you need to be getting second-by-second Twitter updates on, the guy’s been dead 2300 years!

That said, as with everything on the reports, there will always be those you choose to follow that fit into the above categories (eg, I follow a couple of accounts that post nothing but quotes). Just click their icons & they’ll be saved.

If you don’t want to unfollow any of them,  simply click the headings, & the the entire category will be saved. As easy as ever!

Of course, there is value to following news feeds (all links) or celebrities (no follow back) etc but here are some tips on how you can be a better member of the Twitter community.

Retweet

Retweeting (whether old style “RT @name message” or the new style) is a great way to do several things. First, it shares valuable material with your followers. Second, it’s a way of saying thank you, & passing kudos to the person you’re retweeting.

Obviously the more you retweet, the more people are likely to retweet you, spreading your message far beyond simply those currently following you. This, of course, is also likely to get you more followers.

Now, you don’t want to retweet too much (or only retweet), because also important is:

Personal Content

Just posting links all the time – while informative, doesn’t give people a feel for the kind of person you are. People don’t want to connect with a company (or a bot), they want to interact with other people. That’s what’s so great about Twitter, it enables human connection in a way that blogs & old media really don’t.

Now, this doesn’t mean telling everyone what you had for lunch, necessarily (yawn!), but remember – you’re a human first, and a content spreader or news provider second.

Engage! Connect! Communicate!

@reply to people who say interesting things. Reply to those that talk to you. Firstly, it’s polite. Secondly, it’s a great way to make friends, discover exciting new opportunities, and, frankly, one of the greatest things about Twitter.

Now obviously people with zillions of followers won’t necessarily be able to reply to every single comment made to them (much like restaurants, the rules for celebrities are slightly different than for us normal folk), but at least make an effort.

If you never talk to anyone, how are you any different from an RSS feed? Youtube, TV or a newspaper? You’re not. Besides, there are so many wonderful people out there – it’s fun! Start chatting!

Variety is important

Posting the same link over & over is tedious – and people switch off from that very quickly. If you’re losing followers, or all your followers are dead accounts? That’ll explain why.

People aren’t stupid. They’re not on Twitter coz they want the same thing repetitively rammed down their throats. If they wanted that, they’d be watching adverts on TV.

Same goes for posting the exact same tweet over and over. That’s just ridiculous. Now of course, if you have a product launch, or you want to attract people in different timezones, you may need to repeat yourself a bit – but mix it up. It’s not hard to rephrase things, try different wording & so on. If nothing else it keeps you more interesting – but also, it’s basic human psychology that people respond to different stimuli (verbal, aural, visual, mental imagery, etc). So, it’s actually smart to vary your message – you’re more likely to appeal to a greater audience by doing so.

Don’t go crazy

While Twitter can be a very exciting place to be, churning out thousands of updates a day is just going to flood people’s tweetstreams.

If you’re following 40k people, it can be very easy to forget, but 90% of active users follow fewer than 2000 people. If you include all the half-arsed dead accounts out there, it’s probably closer to 98%. If you are posting 50 times a day, you’re going to flood everyone else they care about off their front page.

I’ve analysed a ton of accounts, and it’s actually pretty hard to set off the alarms in this regard – & of course, if all you’re doing is chatting to people, that’s totally fine – since conversations with them only go to them (& anyone else following them). If every tweet you do is public though? Maybe tone it down a little, for the sake of everyone else out there. Quality, not quantity, remember?

Getting feedback

If you want to know how you’re doing on all these points? When you’re logged into The Twit Cleaner now, on the front page, there’s a link “How do I look to The Twit Cleaner.” That’ll tell you exactly how you’re doing. It’ll also give you tips on how to improve your game & get even better in the future.

The basics

Other than that – try to follow people back (if they appeal). Don’t @ tons of people in one tweet (except RT’s & follow friday, of course) – either talk to one person, or just DM them.

Oh, & one last point – don’t sell crap on Twitter. Seriously, if people want white teeth, they’ll google it, or, & here’s a crazy thought, talk to a dentist.