Monday, June 03, 2013

The Joy of Problem-Solving

I'm in one of those time periods that crop up occasionally, in which I have an unusual number of high priority things going on both at work and at home. I can absorb a crunch time in either my work life or my home life without too much angst, but when they gang up on me, I really feel it.

The saving grace this time is that most of the high priority things going on are good- including at work. For instance, I had a great day at work on Friday. I cleared the BS work early and got to spend most of the day working on one of my favorite sorts of technical problems. We're starting to build a system to capture a new type of data, and I'm designing the database to store it. This is complex data that will be accessed in multiple ways by a variety of applications, so it is worth taking the time to think about its natural structure, rather than just letting the requirements of a single application dictate the database design.

I genuinely love this part of my job. I write down the operational requirements, as gathered from the users and the programmers, and mix in my own scientific and technical knowledge, perhaps augmented by a little research. I sketch out ideas (usually on paper), jot down questions, and add more ideas, which produce more questions... until I get stuck. I hit some aspect of the data for which the optimal storage structure is not obvious to me, and I can't immediately think of the questions I need to ask to determine the best structure. I stare at my sketches and my list of questions for a few minutes, waiting to see if something comes to me. And if it doesn't, I go do something else. I work on another task, or I read something random on the internet. (Which of these I do depends on how many other deadlines I have looming and how tired my brain feels.) If it is close to lunch, I eat and then take one more look at my papers before heading out for my walk.

I describe this process as "loading a problem into my brain and then looking away." It is by far the best problem-solving process I've found. And I love it. I love the satisfying mental effort of dissecting the work at hand until I have a clear view of what it is I do not know how to do. I love the feeling of my brain stretching to wrap itself around the gap in my understanding. And I love the almost magical way my brain can fill in that gap if I stop looking at it.

I need the mental equivalent of this
Once the gap is filled, I fill in the new details on my design and I keep going, until I hit the next non-obvious thing. Then the entire process repeats.

This problem-solving process- and the happiness it brings me- is not limited to technical problems. I have similar experiences when sorting through complex project plans, which can be like giant logic problems. My recent experiments with using kanban methods for program management also came from this process. I had a management problem I couldn't see how to solve. I loaded it into my brain and looked away for a surprisingly long time. I picked around the edges of the problem, but couldn't unravel it. I read things I thought might give me ideas, and then I looked away again. Then the idea came to me. I refined it and implemented it, and so far, I'm happy with the results.

To be honest, I get a similar kick out of solving organizational and logistical problems at home, and I use a similar process for it. Basically, anytime I confront a problem I don't know how to solve, I take the same approach: break it down, go as far as I can, then load it into my brain and let my brain magically fill in the gap or suggest new things to research. It can take a long time to work, but I have not had this process fail me yet.

I think a lot of people use a similar process for solving problems, or doing anything creative (and solving a problem is creative: you are creating the solution). A lot of people don't really appreciate how important that "look away" part is, though.  My friend @smbaxtersd tweeted a link to this blog post on the creative process today, and I think it really captures the wry, self-effacing humor about the creative process that is so common. I used to feel the same way. I'd try to skip the "look away" step, forcing myself to continue to stare at the papers with the problem. It didn't work. I now consider looking away to be an essential part of the process. In fact, it is how I know I've got a good, interesting problem to solve. If I can solve it without looking away, it was too easy. And while I'm perfectly happy to go months and months at home without a hard problem to solve, if I go too long at work with out a problem I have to look away from to solve, I get restless and unhappy, the same way I get restless if I go too long without any physical exercise. So Friday was a very good day indeed.

What is your problem solving process? Do you like it?


  1. I don't think it's quite as cut and dry for me but my process is very similar. I do as much as I can, hit a block, leave to do something purely physical for a while and get the Don't Know jitters out of my system and then come back to set my brain on it again. Physically leaving when I have that option has seemed to be the best way to shake mental fog or intellectual fatigue.

  2. That is an interesting problem you described! Taking a break is the best bet when facing a wall... Of course, I repeat to myself, there is a way, I just need to find it!

    That works.

    I am battling a huge pressure to quit. Both internal and external. Cloud, your words have kept me in the workforce till now. Keep blogging!

    Taming the work week is on my to read list!

  3. Anonymous5:11 AM

    Robert Boice pushes that "letting your subconscious do the work" part. It's also common in studies of creativity (one of the literatures I recommended to LV should she decide to write a full book on productivity), and we also covered it in a college class I took on history of math-- mathematicians swear by the process. You have to put in the grunt work, and then your subconscious will take over. There's a scientific basis for it!

    I also get restless without something for my brain to be working on.

  4. Looking away has worked for me countless times. Oftentimes, I'm driving or doing something else and the solution hits me, or a good new idea. Strange how the mind works!

  5. I go for a run. It usually helps.

  6. I need to look away, too. However, it's a blurry line between looking away and procrastinating. ;-)


Sorry for the CAPTCHA, folks. The spammers were stealing too much of my time.