Swarm Theory

Swarm Intelligence, or Swarm Theory, is the collective behavior of decentralized, self-organizing systems: ants in a colony, movie raters at Rotten Tomatoes, participants in a market economy, etc. By observing these systems in nature, scientists have theorized that such systems harness a sort of leaderless, collective intelligence. By leveraging these kinds of consensus-based systems, groups of independently-acting agents can solve problems more efficiently than they could if they were centrally controlled.

Ants, for example, do not use any kind of centralized management in their colonies. Organization happens organically, through millions of interactions between individual ants who are following very simple behavior rules. Some are patrollers, some are foragers, some perform maintenance, some collect waste, and so on. A forager will not leave the colony to go find food until it’s bumped into at least four patrollers and the interactions are no more than ten second apart. The fact that these patrollers have returned alive from the same area are the cue that it’s safe to travel to that area to forage for food.

Bees choose their next hive location using a similar, self-organized system. Scout bees will fly out in all directions looking for new hive locations. When a scout finds an interesting piece of real estate, it flies back to the hive to let the other bees know about his find. It communicates to other scouts using physical motion both the location of the potential new home and his enthusiasm for it (the goodness of fit). Soon, scout bees start assembling at four or five potential new hive locations. Consensus is reached once fifteen bees arrive at any single location. Those fifteen bees will then fly back to the hive to signal that the new hive location has been chosen.

Scientists started looking at this kind of theory as early as the 40’s (John Van Neumann and John Conway did the first theoretical work on “self-replicating automatons”). The field exploded in the last twenty years with the rise of computer science and the Internet. Swarm Theory lends itself perfectly to Artificial Intelligence. Computer learning is based on cycles of testing, valuation, and reiteration using simple heuristics and leveraging computational brute force. This is analogous to leveraging the many thousands of simply-programmed individual agents within a swarm. Google uses a variation of Swarm Theory to discern authorities and rank pages.

Swarm Theory was popularized in 2007, in a National Geographic article: I also spoke briefly about Swarm Intelligence in a previous article called The Superstar Trap.

There are many examples of how society has leveraged this idea, even before we knew what it was. Adam Smith’s Invisible Hand works through the collective intelligence of Swarm Theory. Market valuation is a perfect problem for this particular structure, and the fall of the Soviet Union became a great validator of the collective wisdom of decentralized markets compared with command economies.

Decentralized problem solving works better on some problems than others. According to an article from SEO Theory, swarms work in situations that involve discovery, testing, and comparing results. For example, ants finding the most efficient route to food, or iPhone users who use their Yelp App to find the highest-rated restaurant close by. By leveraging the volume of agents involved, you can act upon complex and rapidly changing sets of data. This works in situations where you are looking for the most efficient method of execution, or the most optimal process. Shipping companies, for example, use Swarm Theory computer algorithms to reallocate trucking routes based on up-to-the-minute energy prices. Swarm Theory works less effectively for creative processes like innovation, except perhaps as a broad directional pointer. A Swarm cannot paint the next Mona Lisa.

Decentralized problem solving also has preconditions. Individual agents must act on a very simple heuristic, and all agents must be capable of making the same very simple judgments. Not every person who submits a review to Yelp is a gourmet food critic, but they can say if they liked or did not like the experience. It’s alright that the judgments of individual actors be subjective, because the act of aggregating these judgments washes out the bias. Also, the actors must be diverse, and not act in unison. No swarm of bees is ever going to find the optimal hive location if all the scouts look in the same direction. When they don’t, as when all major U.S. investment banks got caught betting the wrong way on mortgage debt, the group loses the connection between consensus and wisdom. Michael Maubaussin makes this point in his excellent book on counter-intuitive wisdom, Think Twice.

In decentralized problem-solving, there must be some mechanism by which the group communicates and evaluates information. Horse races are most frequently won by the horse with the best odds leading up to race time. The odds themselves are an algorithmic representation of the group’s collective wisdom displayed through wagering, and those odds are transparently communicated to all participants. Therefore agents can distill their actions based on the group’s collective findings.

Most importantly, individual agents must act responsibly and make their own decisions. According to the National Geographic article cited earlier, “A group won’t be smart if its members imitate one another, slavishly follow fads, or wait for someone else to tell them what to do.” Yelp does not work if I give a high rating to a restaurant I’ve never been to because my friends did. Nor will a group be smart if individual agents withhold information or mislead others in order to gain a private advantage. Swarm Theory depends on all agents having access to the information produced through group discovery; it depends on complete transparency.

Swarm theory has drawbacks, even when it does provide efficient consensus. First, consensus is by no means perfect. The ratings on Yelp signify the preferences of the collective palate, and may not align with the superior preferences of the discerning palate. Likewise, Wikipedia is a great primer, but is inappropriate for scholarly citations due to the higher standards of accuracy demanded by the academic community. Evolutionarily speaking, it is quite possible that humans are supposed to refine their individual sensibilities and expertise, so as to provide higher-quality intelligence to the group.

Also, there must be a motivation to contribute. As systems become more complex, certain subgroups will be more motivated than others, which will increase the weight their input and introduce bias.

Decentralized systems will, by definition, suffer from a lack of uniform design and vision. No individual actor sees the big picture. So the swarm process would never have led to the creation of the iPhone. Instead, it creates other valuable resources like Wikis and Open Source Projects such as Linux and Drupal. These tools are useful and widely adopted, but inelegant and with little emphasis on unity and form. Also, swarms are closed systems unto themselves. Drupal, the open-source web development platform, was invented by and for its web programming community. As such, it has little interest in accommodating less savvy outsiders who would use the platform if only it were more intuitive.

Finally, Swarm Theory systems tend to show a creeping overconfidence in the correctness of the group’s consensus. This assumption leads to blind-siding by Black Swan events. A horse race, for example, with not always be won by the horse with the best odds, even though it seems that way. The 2007 financial crisis could not have happened without the staggeringly overconfident consensus on the “risklessness” of CDOs. This blinded the industry to biases introduced when individual agents started act in correlation with one another.

Here are some ways in which we can leverage the power of Swarm Intelligence:

Create Web Tools That Collect, Consolidate and Communicate Information from Users

Japan Earthquake Shelter Map

In the aftermath of the Japan earthquakes last spring, on Harvard Business Review article spoke about how volunteers came together to map out the locations of makeshift shelters for stranded drivers. The community of programmers collected information submitted by people offering homes and buildings as shelters, mapped out the information, made it available to smartphone users, and advertised it on social media. And they did it within hours. It’s a striking and heartwarming example of the positive potential that comes from helping communities to self-organize.

I am not at all talking about antiquated tools like site forums or submission forms – these are rudimentary, uncreative methods that do not provide a consolidated aggregation back to the users supplying information. I’m talking about the future tools that will evolve from today’s community review systems, wiki-style projects, and the like. How can I sign on to your site and see a concise, useful representation of your community’s input? How will that representation guide me as I submit my own input?

Value the System, Not Just the Talent

In a preceding post, I spoke about how we tend to associate group success with the individual talent of superstar group members, rather than with the structure and cohesion of the group itself. While it’s certainly important to leverage the strengths of individual team members, we must remember that teams are more than the sum of their participants. The bigger the team gets, the more it will take on its own characteristics, which are not subject to control by individual members. This is why firing a baseball manager rarely has an immediate impact on the team’s performance: the manager has less control over team dynamics than people assume. The team has become an organism, behaving in ways that are not controllable through altering one member.

Separate from the talents and abilities of individual members of a system, we must ask ourselves whether the systems themselves are as effective as possible. Is information freely communicated from one end to another, in a useful, aggregated way? Can the system react well to independent input from members, rather than simple command-and-control? Is that feedback aggregated and communicated to every other member in real time? Is the system transparent?  Are there ways that individual members can game the system or withhold information for private advantage? Is there an inherent bias towards certain high-status members, or is the system focused on receiving and valuing contributions from all members? Most importantly, how much value do members place on making the system as efficient as possible?

Create Opportunities for Team Members to Work On Private Projects, and Share Them With The Team

One of the fastest spreading ideas from Daniel Pink’s book Drive is the concept of the Atlassian “FedEx Day.” Periodically, company employees are given one day to be completely autonomous and work on anything they want, as long as it does not have to do with their actual job. The employees will then show what they’ve created to their colleagues after 24 hours. This tradition takes its name because the employees are required to deliver something overnight.

This type of “non-commissioned work” – which employees eagerly do for the fun and the challenge – has resulted in numerous innovations and patent applications that have directly benefited the company. That’s before you consider the intangible benefits to morale and cohesion.

Notice, by the way, that this is not an anarchic undertaking. This is not a random, unbounded process. This is a highly organized, finite exercise in self-determination. Innovation exercises like this do not work because they pull the company in an egalitarian direction. On the contrary, they work because they provide a limited framework within which each employee can use his own individual judgment to bring his maximum contribution to the company.

By allowing an environment where employees have complete freedom to provide autonomous contributions, we leverage the idea of Swarm Theory by consolidating the efforts of agents acting independently. With only a loose directive that the projects be “skewed” in the direction of company products and operations, employees’ creative and sometimes radical ideas inject a burst of innovation into the company.

Related articles

• Learning from Insect Swarms: Smart Cancer Targeting (scientificamerican.com)
• Swarm Intelligence and Collective Memory (magnetsnmiracles.wordpress.com)
• Swarm Theory – National Geographic Magazine (ngm.nationalgeographic.com)
• From Standard to Second Order Swarm Intelligence Phase-Space Maps ” Chemoton § Vitorino Ramos’ research notebook (chemoton.wordpress.com)