It bothers me when you have to wait to get your reports.
Obviously there are some physical limits - how fast can we get data from Twitter, how many other people want reports at the same time, how much data do we need to analyse and so on. The answer to the last one can be A LOT, by the way. For every 100k people you follow, we need to ask Twitter for roughly 3 gigs of data.
There's been particular problems when super large users (100k+) have asked for reports. If Twitter was being particularly grumpy, it used to create enough of a backlog it could slow down everyone's reports for an entire day. Blergh!
If you're only following a few hundred people, why should you have to wait just because some giant user requested a report right before you?
So, over the last few months I've been sitting here, all day every day, watching the systems on a second by second basis. I've identified roughly a dozen bottlenecks, some small, some huge.
The good news is, I've managed to bend the underlying technology in a few ways that are theoretically impossible (according to their documentation) to get around these limitations. I'm justifiably pleased.
You will have already seen some of the benefits as I've been gradually adjusting things on the live system. Most of the big improvements I've rolled out this evening.
Oh, I've also taken the opportunity to roll out a few big structural changes to allow me to offer significant new functionality. What? Ahh, you'll just have to wait and see!
Yes, I'm super pleased with progress. Yes, I'm even more excited about what's to come!
How though? Care to share?
Wanna throw me some more words? I don't know what you're asking :)
TIL: @si_dawson:disqus I'd replied to this. It didn't email me! Cheers for the words. :)
How very odd. Well, glad it's all sorted now.
Since I haven't heard back from you, I'll take a wild guess what you're referring to.
There were a bunch of tech tweaks, in varying areas, but one key one I've blogged in detail here.
Well all I can say is I was amazed at how quickly your report came through!! Have started the unfollowing process and it is really useful to see how many people dont respond at all etc - plus makes me feel Ive been a bit indiscriminate in following so many in the first place!
The How I look report is really useful too - going to RT a bit less and talk a bit more!!
Thanks Si
Hey, that's super cool. Glad I could help.. and that all those performance fixes are noticeable!
and yeah, the first time you run the report, it tends to be one hell of an eye opener, for everyone. It's amazing how many junk accounts are out there. Still, means the second time through will be much, much faster.