It bothers me when you have to wait to get your reports.
Obviously there are some physical limits - how fast can we get data from Twitter, how many other people want reports at the same time, how much data do we need to analyse and so on. The answer to the last one can be A LOT, by the way. For every 100k people you follow, we need to ask Twitter for roughly 3 gigs of data.
There's been particular problems when super large users (100k+) have asked for reports. If Twitter was being particularly grumpy, it used to create enough of a backlog it could slow down everyone's reports for an entire day. Blergh!
If you're only following a few hundred people, why should you have to wait just because some giant user requested a report right before you?
So, over the last few months I've been sitting here, all day every day, watching the systems on a second by second basis. I've identified roughly a dozen bottlenecks, some small, some huge.
The good news is, I've managed to bend the underlying technology in a few ways that are theoretically impossible (according to their documentation) to get around these limitations. I'm justifiably pleased.
You will have already seen some of the benefits as I've been gradually adjusting things on the live system. Most of the big improvements I've rolled out this evening.
Oh, I've also taken the opportunity to roll out a few big structural changes to allow me to offer significant new functionality. What? Ahh, you'll just have to wait and see!
Yes, I'm super pleased with progress. Yes, I'm even more excited about what's to come!