Today someone told me their report didn't appear to be updating.

It took quite some digging to find what was happening (since everything was operating normally) but I finally discovered what the problem was.

We were only updating our records for accounts that had already been flagged to appear on a report.

In other words, we weren't keeping up with how people's behaviour might change over time.

So, commenting out a single bone-headed line of SQL (What was my obviously under-caffeinated brain thinking? What?!? What??!), and voila, everything is now as it should be. It means reports will be slightly slower (in this case, this is a good thing) - but only because we'll now be pulling down much more information from Twitter, which takes time.

The upshot of this? Your reports will now have much, much more detail on them. I ran it on my personal account (@sidawson) and my report went from having 47 people on it to 190 (I've been deliberately leaving them there so I can test the new version)

If you've run a report in, ohh, the last few months, might be worth running it again now.

My apologies for not noticing this sooner. It's all very silly.

It's been a heck of a long time since there's been much external perception of progress around here.

Today, that all changes. Well, almost.

See, when I first started Twit Cleaner, I wanted to get something useful out there as soon as possible. A Minimum Viable Product, it's called. That worked well enough, but when I started looking at moving to a one-click-one-unfollow model, I realised that the current infrastructure - the way I'd designed things - simply wasn't going to work. It was possible to do new things, but it would have been horrible, painful & slow.

I made some (with hindsight, of course) silly technical choices, & they came back to bite me on the ass.

So, for the past 7 months I've been going through redesigning the entire of Twit Cleaner, more or less from the ground up. The first of that giant chunk of work rolled out yesterday. Believe it or not, but the previous version had almost everything just shoved into a giant directory tree*. So yes, that's a folder with many, many million files in it. It worked ok for the one task it was designed for, but it seriously hampered the ease & speed with which I could develop any neat new tools. There were a bunch of other bad technical decisions, but that was the key one.

Now, everything is in a big shiny database. Which has its own issues, of course (everything does) but I'll iron those out over the next couple of days. Once that's over, & the database is fully loaded up (it's happening as we speak, and looks like it might take a few days to complete so expect the site to be a little shaky until that's finished, please be patient), it will smooth the way to quickly & easily roll out a bunch of new tools to help you manage & explore your Twitter life. Oh yes, I have many, many great ideas I've been working on.

On the outside, things may have been serene, unbroken, just meandering along like a duck floating on a pond.. but underneath I've been, just like a duck, paddling furiously seven days a week all hours of the day & night to get things working just the way they should be - and to get you guys the help you deserve.

Unfortunately, like a duck, there's not much to look at just yet. Oh, except reports will be much, much faster.

*If you're really curious, the very first version of Twit Cleaner used to run on my desktop machine at home in Melbourne, then copy things furiously back & forth to the web server, which at that time was in London. Now THAT was nutty.