Valuing thought

I think I already learned the lesson that is the subject of this blog post before but I’m having to relearn it in my new role as a data science manager rather than a data science do-er.

When I first started to get into data science I made the mistake I think a lot of people make where they think “I’m doing data science, data science is typing code, I want to do lots of data science, let’s type a lot of code”. And you just feverishly type code all day. Once you’ve been doing that for a few months you start to realise that you’re just making a huge mess. All your code needs refactoring, your dashboard is ugly and makes no sense, you’re using a model that really just doesn’t work with the data. You make a big pile of mess at high speed and then slowly improve it at low speed.

I twigged this after a while and started to realise data science is actually thinking and planning. The typing code bit comes right at the end. It’s the last thing you do. And I would very often walk away from my keyboard and write on a piece of paper to stop me from just banging code out.

The other lesson that I had to learn soon after was to value this thinking time. Some days I would finish a day of data science with nothing much to show for it. A diagram and a data import. A GitHub repo containing a README and a licence. Some days I would finish with nothing at all. And when you’re under pressure or rushing it’s easy to fall into the trap of cutting this time down. It feels like an indulgence, something to be minimised when you’re in a rush. And now you’re back to making a big mess again but now it’s even worse because you’re kicking yourself about “wasting” the first three hours of the day thinking.

Anyway, I learned that lesson, and sometimes when I’m talking to data scientists who need to learn it themselves or just need a reminder I will bring it up and tell them to value thought and planning more than coding, not less.

Fast forward to today and I’m a manager. I don’t write any code, ever. Now it’s my job to make decisions. I do this by talking to people and sending emails. Sometimes I write four sides of A4 if I want to influence lots of people to make a bigger decision. And despite already learning this lesson before, thoroughly, I fall into the same trap. The more decisions I make, the more emails I send, the more productive I am. The consequences aren’t as bad as they are in data science doing, I don’t think. You don’t make a big mess you have to unpick, or not usually anyway. I think the consequences here are that you end up in a hyper local and hyper optimised situation that’s not improving. Each decision is just optimised against the last thing you did with no purpose or direction. And eventually you wake up in completely the wrong place and think “Oh! I wish I was over there!” when you haven’t moved in that direction once for six months.

So I’ve learned that lesson, with help from wiser folk at my workplace. And now I’ve fallen into the second trap, AGAIN, even though I already learned it before and just fell into the first trap. Now I’m once again undervaluing time thinking and planning, especially when I’m under stress. And a day with low output feels awful. I’m finding it even harder now because there is so much to ignore. In my old life there was a code editor, sitting with the cursor flashing, and I would ignore that and write in a notebook. Now there are five or more things and I have to ignore all of them in order to think and plan. And ignoring five things all day and still finishing with low output feels really terrible. But I’m learning at the moment that, just as data science isn’t typing, managing isn’t sending emails and that the best person I can be is calm and considered and has a coherent plan based on reason and evidence, and not just a hyperactive tempest, pinging around and firing off volleys of emails to keep things on track (but I am sometimes that just as I used to sometimes sit and code for 10 hours when the situation demanded it).