Misfits Matter

How to use the trial and error method to make a scientific discovery.

 

I like moving, exploring new places, and visiting friends and family (for short manageable doses).  I can put up with traveling for work.  But one thing never ceases to annoy me:  Whenever I take a shower for the first time in a new place, I can’t for the life of me get the knobs, handles, and faucets to work right the first time.  I spend at least five minutes trying to get the water to stop being boiling or freezing, or trying to get the dribble out of the shower head to be decent enough to rinse.  Maybe you can relate.

But I’ll bet it never occurred to you that how you solve this problem is really scientific discovery skills in action:  you start fiddling with all the water controls you can see.

That’s because it’s is a classic example of doing the right kind of trial and error.  So I’ll use it to outline what I think are four key dimensions that help structure trial and error for discovery:

1. Putting in the right number of trials
2. Putting in the right kinds of trials
3. Putting in the right kind of error
4. Putting in the right amount of error

The overall theme here is this — it ain’t called “trial and success” for a reason.  The errors are part of the magic…that special sauce…the je ne sais quoi…that makes the process work.

You may have seen versions of this idea in current business-speak around innovation and start-ups (the Lean build-measure-learn cycle anyone?).  But I needed to take it out of the entrepreneurial context and put it into a science one.

So let’s get down to brass tacks and talk about important aspects of trial and error.

 

4 Goals for Thoughtful “Trial and Error”

 

I’m going to keep the shower faucet analogy going because it’s straightforward to imagine hitting the goals for each dimension.  But to give this a fuller scientific discovery context I’ll add one technical example at the end of the post.

 

Dimension #1 — On putting the right number of trials into your trial and error.

 

Goal:

Keep running trials until you gain at least one valued action-outcome insight.

 

When you start out on a round of trial and error you are really aiming for complete understanding and the skill to make it happen on demand, with fine control.

In our shower analogy, that means it’s not just enough to know how to get water to come out of the spout.  You need to be able to control the water temperature, the water pressure, and make sure it comes out of the shower head and not the tub spout (if there is one).  Ideally, you’d learn enough to be able to manipulate the handles to produce a range of outcomes:  the temperature sweet spot for a summer day shower or a winter one; the right pressure for too much soap with soft water or for sore skin from the flu.

So one of the first things you have to figure out is: how do you know when to stop making trials?

This isn’t a technical post about conducting blind trials or sample surveys.  Here we’re talking about a more qualitative definition of done; the kind of thing you might try for an “exploratory study”.  Exploratory studies are the kind where you have no hypothesis going in.  Instead, you’re trying to find your way toward an unknown valued insight, not trying to prove or disprove a previous hypothetical insight.

The whole point of trial and error is to take a bunch of actions that will teach you how to create desired results by showing you what works (called “fits”), what doesn’t work (called “misfits”), and forcing you to learn why.

The “why” is the valued insight you’re after.

If you’ve run enough trials to figure out how to make something happen, that’s good, but not enough.  For scientific discovery you need to know precisely why and precisely how it works.

So keep running trials until you’ve come up with an answer to at least one why question.

 

Dimension #2 — On putting the right kinds of trials into your trial and error.

 

Goal:

Try a mixture of fits and misfits.

 

A key facet of trial and error is that by intentionally generating mistakes it will help create insight into how to generate success.

Partly, these trials are about firsthand experience.  Your job is to move from “wrong-headed” ideas to “right-tried” experiences.  To make changes to how you operate you have to clearly label and identify two things in your trial and error scenario–“actions I can take” and “results I want to control”.

Good trial and error means that you will: (1) learn the range of actions allowed; (2) try every possible major action to confirm what’s possible and what’s not; and (3) learn from experience which actions produce what outcomes.

In the last section I brought up the terms fit and misfit: in some science work, getting a match between an equation you are trying and the data is called a “fit” and getting a mismatch between the two is called a “misfit”.

So in science terms, that means you want your trials to be a mixture of things you learn will work (fits), things you learn won’t work (misfits), and, if possible, things where you have no idea what will happen (surprises).

For my shower analogy, let’s use a concrete example: the shower in my second bathroom, which both my mom and aunt have had to use (and, rightfully, complained about).

So, for “actions I can take”: rotate left handle, rotate right handle, or pull the lever on the left handle.  And for “results I want to control”: the water temperature and the amount of water coming out of the shower head.

Then, I start moving handles and levers individually.  Every time I move a handle and don’t get the outcome I want, it’s a mistake.  But I’m doing it intentionally, so that I can learn what all the levers do.

Many of these attempts will be misfits, producing no shower at all or cold water or whatever.  Some may accidentally be fits.  Hopefully, none will produce surprises (though I have had brown water and sludge come out of faucets before).

I think this visceral experience is what allows your mind to stop rationalizing why standard approaches and methods should work and get on with seriously seeking out new and novel alternatives that actually work.

And these new and novel alternatives, with their associated insights, are the soul of scientific discovery.

So you want to move into this open-minded, curious, active participant and observer state as quickly as possible and trying fits and misfits will help you do that.

 

Dimension #3 — On putting the right kind of error into your trial and error.

 

Goal:

Make both extreme and incremental mistakes.

You know the actions you can take.  But you need to figure out why certain actions lead to certain results.

One great way to do this is to try the extreme of each action.

If it’s safe (or you have a reasonable expectation of safety) then pull the lever to the max, rotate the faucet handle all the way, cut out almost everything you thought was necessary, and see what happens.

In physics, this goes by the name “easy cases”.  What we really mean is use the extreme values, zero, negative infinity, or positive infinity.  Plug them in to your model and see what happens.  Does it break things?  Does it give wonky answers?  Does it lead to a scenario where the role of one term in the equation becomes clearer?

That’s the beauty of extreme tests when you’re doing trial and error.  They let you crank up the volume on factors so that you can pinpoint what they might do, how they might operate in your context.

So what about making “incremental” mistakes?  Just nudging things a little this way and a little that way to see what happens?

These are absolutely necessary too, and tend to happen later on in your trial and error process.  They are a great way to confirm and refine your understanding.

If you want to boil it down, making mistakes at the extreme ends of the action cycle hones your “this-does-that” knowledge, while making mistakes in small incremental steps helps clarify “how” knowledge.

So, often times, it’s best to go after extreme cases in the early trials and then move toward incremental cases later on.  For example, with the shower handles, early on you’ll probably try rotating one handle all the way to the right or left to figure out which direction brings hot water.  Later on, you’ll turn the handle a little bit at a time, until you get the right temperature.

 

Dimension #4 — On putting the right amount of error into your trial and error.

 

Goal:

Make mistakes until you can link all major actions with outcomes.

 

This one is easy enough to grasp.  To put it more bluntly: how many times should you mess up on purpose?

The goal statement says it all: make enough mistakes that you can link all major actions with outcomes in your mind, and you know why they are linked the way they are.

Just imagine if you were told that every move you made to try and set a shower, where you didn’t know the knobs at all, had to only be moving toward the right outcome (no errors allowed).  How the heck would you succeed?  You would have to look up a manual, or find someone who had used the shower before.  It would probably slow the process down to a painstaking pace.  It would stress you out.  And it would need pre-existing insight into how to do it right.

But in discovery, you won’t have that kind of prior insight.  No one does.  So you have to be willing to gets things wrong in order to start to generate that insight.

So keep getting it wrong in your trials until you really get why it doesn’t work.  Don’t avoid those misfit moments.  You should be able to make a table or a mind map of links between actions and outcomes.  If you can’t, keep making errors until you can.

 

The Four Trial and Error Dimensions in a Real Physics Research Example

 

I promised I would connect the ideas I’ve talked about to a science example, so let me do that:

For my Ph.D. neutrino physics work, at one point I had to write a piece of computer code that could reproduce a final plot and numbers in an already published paper, by the MINOS neutrino oscillation experiment, to make sure our code modeled the experiment well.  First, I wrote some code (to estimate the total number of neutrino particles we predicted this experiment to see at a certain energies) based on how my research group had always done it.  Then I wrote down in my research notebook how the existing code had previously been tweaked to produce a good match.  One value had been hand-set, by trial and error, to fit.

In the newer data published at the time, we knew this tweak no longer worked.  But at first I just tried it anyway (try misfits).  Then I started changing the values in the code (make incremental changes).  And we added a few new parameters that we could adjust and I altered those values (try unknowns).  I kept detailed hand lists of the results of my changes on the final output numbers (link actions to outcomes).

Then I synthesized these behaviors into new groupings: did it make the results too big, too small, by a little, by a lot?  Did it skew all the results or just the results at certain energies?  Was it a consistent overall effect, or some weird pattern effect?

At this point I kept many code versions to be able to have a record of the progression of my trials (fancy versioning software isn’t commonly used in small physics groups).

And I did handwritten notes where I worked through why certain outcomes weren’t produced and others were (try until you get insight).

 

Then I did it again.  And again.  And we did it for 10 more experiments totaling…well, a LOT of code.

In the end we got a good match and we were able to use it to complete my Ph.D. work, which explored the impact of a mathematical symmetry on our current picture of the neutrino particle.

So, trial and error, being able to willfully make mistakes to gain insight, can be incredibly powerful and remains a uniquely human skill.

As a 2011 study from Nature suggested, non-expert video gamers (i.e., many with no education in the topic beyond high school level biology) out-predicted a world-leading machine algorithm, designed by expert academic biochemists and computer scientists, in coming up with correct 3-D protein shapes, because they made mistakes on purpose while generating intermediate trial solutions.

Algorithms, by design, are constrained to do only one thing: get a better answer than they had before.  Every step must be forward; even temporary small failures are not allowed.

But we’re messy humans.

We can take two steps back for every one step forward, or even cartwheel off to the side when the rules say only walking is allowed.  Our ability to strategically move in “the wrong direction” (briefly taking us farther away from a goal) in order to open up options that in the long-run will move us in “the right direction” (nearer the goal) is part of our human charm and innate discovery capacity.  But that requires we acknowledge up front that in pursuit of discovery many trials will be needed, and many of them will not succeed.

 

Mantra of the Week

 

Here is this week’s one-liner; what I memorize to use as a mantra when I start to get off-track during a task that’s supposed to help me innovate, invent, and discover:

Misfits matter.

Using trial and error in a conscious, structured way can move use from having thoughts on something to experiences in something.  Notice how “thoughts on” speaks to the surface, like a tiny boat on a broad ocean; while “experiences in”, speaks to the depths, like a diver in deep water. So try.  And err.  Welcome error by remembering that misfits matter and that a deep perspective is where radical insight awaits.  In taking two steps back for every one step forward, those two steps back aren’t setbacks, they’re perspective.

 

Final Thoughts

 

So let’s recap the ideas and examples I’ve talked about in this post:

• I shared the four dimensions that help define strategic trial and error: putting in the right kind and number of trials, and putting in the right kind and amount of error.
• I shared an example of how trial and error has been used in my own physics work and in biology to get useful insights.

Have your own recipe or experiences related to trial and error?  You can share your thoughts by posting a comment below.

 

Interesting Stuff Related to This Post

 

1. Web Article – “Insight”, Wikipedia entry, https://en.m.wikipedia.org/wiki/Insight.
2. Web article – Ed Yong, “Foldit – tapping the wisdom of computer gamers to solve tough scientific puzzles” Discover magazine website, Not Exactly Rocket Science Blog, August 4, 2010, http://blogs.discovermagazine.com/notrocketscience/2010/08/04/foldit-tapping-the-wisdom-of-computer-gamers-to-solve-tough-scientific-puzzles/#.XKPkLaZ7kWo.
3. Website – MINOS neutrino oscillation experiment, http://www-numi.fnal.gov/.

 

How to cite this post in a reference list:

 

Bernadette K. Cogswell, “Putting the Error in Trial and Error”, The Insightful Scientist Blog, March 22, 2019, https://insightfulscientist.com/blog/2019/misfits-matter.

 

[Page Feature Photo:  An ornate faucet at the Hotel Royal in Aarhus, Denmark. Photo by Kirsten Marie Ebbesen on Unsplash.]