How Gamers Outperformed Biochemistry’s Best Algorithm

How Gamers Outperformed Biochemistry’s Best Algorithm

THE SHORT READ:

The Question

Can non-scientists help solve complex scientific problems that are tough to answer due to computational limitations?

 

The Answer

Yes, through innovative methods like participation in a multiplayer game where gamers direct scientific algorithms and innovate novel search strategies.

 

THE LONG READ:

 

The Research Study Set Up

A set of researchers in computer science and biochemistry had a problem:

Despite having access to a powerful computational algorithm, called Rosetta, designed to predict the structure of proteins, the algorithm was only successful a fraction of the time.

The algorithm sampled the range of available ways a protein could meet certain known requirements that constrain its structure.  But the total number of ways proteins can achieve this “native conformation” is immense.  Even with Rosetta’s sampling algorithms, finding the optimal three-dimensional structures posed an intractable problem.

To get help, the study authors took a risk:

The researchers hypothesized that human decision making could improve the protein structure prediction rate by being better at strategizing how to search the vast optimal low-energy space and when to use less than ideal configurations in order to find ideal final shapes.

To test their hypothesis they took an even bigger risk:

The researchers created a multiplayer online game to get help from non-scientists.

This scientific game, called “Foldit!” (fold-it; see link below), includes both cooperative team mode, where players work together to optimize protein folds, and individual mode, where players compete against other players using leaderboard rankings.

To set up the study, the researchers launched their game as free-to-play online.  Players were given a tutorial of already solved configurations, in order to learn how the user controls work.  This tutorial taught Foldit players:

  • How to use the game’s protein folding tools to manipulate specific features.
  • How to understand the visuals, and optional visual cues to highlight protein features and proximity to the goal solution.
  • And it gave and taught some pre-built-in algorithm actions, a base set of folding strategies already known to the researchers.

Then the researchers let players play.

To gather data on the players’ strategies and decision making, the research team used a number of different measures.  They asked for player self-reports, posing both close-ended questions (such as to gather demographic data) and open-ended questions (such as an email sent to the top scoring players asking them what strategies they used).

They also tracked the moves players made in the game (e.g., “shake”, “global wiggle”, “local wiggle”, and “backbone pull”), as well as recorded the length of time it took players to complete a puzzle.

They also gathered data on other features used in the game (player chat, help tools, etc.), and took measures of players’ final solutions, such as the distance between sections of the configuration in final shapes and the overall energy of the final configuration.

By testing players’ abilities on posted prediction problems (of solutions not publicly available) and evaluating players’ scores, as well as strategies via self-report, the researchers could compare how well players did versus Rosetta, as teams and as individuals.

 

What the Research Discovered

When they analyzed all the data, the researchers found a number of interesting trends in how the players did:

  • Players did better than computational algorithms at solutions that required many moves, or cycling through many improper configurations, in order to work the protein into a proper configuration.
  • Players stuck with the high energy (undesirable) interim configurations longer than the algorithms did.
  • Players often picked out the best starting point, whereas the algorithms did not.
  • In a total of 10 blind puzzle tests:
    • in two cases the Rosetta algorithm outperformed the players,
    • in two cases the players and Rosetta performed equally well, and
    • in five cases the players outperformed Rosetta.
  • Overall the players outscored the best algorithm (Rosetta).

Why did the players do better than the Rosetta algorithm on such a complicated problem, with literally thousands of possible solutions?

Based on the data, to the researchers it seemed that the computational algorithms got tripped up by finding a nearby low energy solution.  Since the algorithms were hard-coded (by their programmers) not to move away from the current okay solution, they got stuck in this “good enough” spot and couldn’t try a few worse solutions in order to get to a better spot.

In contrast, people were able to intentionally and strategically create worse (higher energy) set ups, in order to fold proteins nearer to the overall best (lower energy) configurations later on.

In other words, the players could make strategic choices about: (1) the best starting points in a large search space, and (2) when to sacrifice gains now in order to make larger gains later.  The algorithm couldn’t always match that high quality decision making ability.

 

What the Research Means…

The findings for some puzzles suggested that there is perhaps an optimal place for both humans and computers.  Foldit players did better when given an initial starting point while the algorithm Rosetta did better when starting from a completely blank slate.

Overall, people used a more varied range of search strategies to get to the best solution, compared to computers.  So learning from human strategies may improve automated algorithms.

Also, social aspects of the game, such as competition and the ability to share solutions, influenced overall player search strategies and suggest a social component may affect possible scientific outcomes.

The authors stressed that people learned to co-adapt: as players got better they improved the range of tools and strategies available in the game.

 

…and What Comes Next

However, the present study was limited by the fact that the researchers used informal methods to study player strategies; a more comprehensive study might lead to greater insight about the differences between human and automated prediction methods.

Counter-intuitively, some of the top performing players were not directly motivated by scientific achievement or contribution, but instead stated that they were driven simply by game mechanics like leaderboard scores or learning to solve puzzles.

 Hence, the study authors recommended keeping varied motivation sets as part of the future of scientific discovery games.

PUT IT IN ACTION:

 

Three Things to Try

 

(1)  Be willing to try approaches that might temporarily lead to lesser gains in the short run, but will lead to better solutions in the long run.

(2)  Consider motivating yourself using measures that track more than just discovery, like the number of social interactions you’ve engaged in, to make progress.

(3)  Give yourself some warm-up practice on related examples with known solutions, in order to get into the groove, before tackling new complex discovery problems.

 

THE FINAL WORD:

 

Best Quote from the Study Authors

 

 

“Our results indicate that scientific  advancement is possible if even a small fraction of the energy that goes into playing computer games can be channeled into scientific discovery.”

[Cooper et al., Nature 466, p. 760]

 

FULL CITATION:

 

Cooper, Seth, Firas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popović, and Foldit Players. “Predicting Protein Structures with a Multiplayer Online Game.” Nature 466, no. 7307 (August 2010): 756–60. https://doi.org/10.1038/nature09304. (Article was open access as of December 2019.)

 

Foldit online game web link (game still active as of December 2019):

Foldit, “Solve Puzzles for Science.” https://fold.it/portal/.

 

Categories:  Scientific Discovery

Tags:  activities, insights from research