Erik McClure

Sturgeon’s law states that 90% of everything is crap. Combined with Hanlon’s Razor, we arrive at the inescapable conclusion that most problems are caused by incompetence. What’s particularly interesting is that the number of incompetent people in a system tends to increase the higher up you go. Part of this is due to the Peter Principle, where organizations promote employees until they become incompetent, but this happens in the first place because it becomes harder to measure competence’ the longer it takes the effects of actions to be felt, and as a species we have no way of measuring long-term incompetence. Instead, we rely on social cues, and tend to use whatever our local culture determines is “competent”.

One way to try to address this is to teach better critical thinking, but this almost always runs into fierce objections from parents who don’t want schools to “undermine parental authority”, which is what happened with the 2012 Republican Party of Texas platform (original). This kind of thinking is actually fairly common, and it is not a fluke of human nature - it is a feature.

To understand why humans can be inquisitive and intelligent on an individual level, but follow arbitrary and sometimes counterproductive rituals on a cultural level, you must understand that our ancestors lived in epistemic hell. My favorite example is the tribe that had a very long and complicated ritual for preparing manioc, which contained dangerous amounts of cyanide:

In the Americas, where manioc was first domesticated, societies who have relied on bitter varieties for thousands of years show no evidence of chronic cyanide poisoning. In the Colombian Amazon, for example, indigenous Tukanoans use a multistep, multiday processing technique that involves scraping, grating, and finally washing the roots in order to separate the fiber, starch, and liquid. Once separated, the liquid is boiled into a beverage, but the fiber and starch must then sit for two more days, when they can then be baked and eaten.

[..] even if the processing was ineffective, such that cases of goiter (swollen necks) or neurological problems were common, it would still be hard to recognize the link between these chronic health issues and eating manioc. Low cyanogenic varieties are typically boiled, but boiling alone is insufficient to prevent the chronic conditions for bitter varieties. Boiling does, however, remove or reduce the bitter taste and prevent the acute symptoms.

So, if one did the common-sense thing and just boiled the high-cyanogenic manioc, everything would seem fine. [..] Consider what might result if a self-reliant Tukanoan mother decided to drop any seemingly unnecessary steps from the processing of her bitter manioc. She might critically examine the procedure handed down to her from earlier generations and conclude that the goal of the procedure is to remove the bitter taste. She might then experiment with alternative procedures by dropping some of the more labor-intensive or time-consuming steps. She’d find that with a shorter and much less labor-intensive process, she could remove the bitter taste. Adopting this easier protocol, she would have more time for other activities, like caring for her children. Of course, years or decades later her family would begin to develop the symptoms of chronic cyanide poisoning.

Thus, the unwillingness of this mother to take on faith the practices handed down to her from earlier generations would result in sickness and early death for members of her family. Individual learning does not pay here, and intuitions are misleading. — "The Secret Of Our Success" by Joseph Henrich

Without modern tools, there is no possible way (other than acquiring brain damage from chronic cyanide poisoning), for an ancient human to realize that every step of the ritual is actually necessary, because without extensive experimentation over many human lifetimes, it isn’t obvious what danger the ritual is guarding against, and if it’s working as intended, no one will have seen the danger or be able to know about it in the first place! It seems that evolution always kept around enough sacrificial intelligent humans to tinker with new possible rituals, but always ensured that the majority of the population would obey the safe, known ways of doing things, without questioning them, because trying to rationally evaluate an opaque ritual meant death. Not even the culture itself knew what disaster or point of failure the ritual was actually preventing, only that it kept them alive. Religion is simply a convenient way of packaging rituals; if you look in the rules set out by many ancient religions, a lot of them start looking like “how to run a functioning society” and include things like “keep your toilet clean”. They got popular because they worked, we just had no idea why and in many cases couldn’t have possibly figured out why with the technology at the time. Even worse, if you got it wrong, it could take you decades until you finally manifested an affliction that actually started causing problems.

This is the core evolutionary drive behind religion and conservative mindsets, where obeying authority is paramount to survival. In modern times, we could communicate to our children why doing a particular thing is bad, because we know the entire chain of cause and effect. Just a few hundred years ago, we couldn’t even do that! A famous example is the effort to get iodine added to salt. Doctors didn’t resist the idea of adding iodine to salt for no reason, they resisted it because at every dosage amount that seemed like it could have an effect, it made people sick! They had experiments on fish that showed that iodine seemed to make goiters go away, but giving people iodine supplements would always make them sick. At this point in time, nobody had any evidence whatsoever that micronutrients existed. Giving people just 150 micrograms of iodine a day, accomplished by evenly mixing tiny grains of potassium iodide into a kilogram of salt, seemed like homeopathic medicine. There was no known substance that had any effect at that little concentration. Only by taking a leap of faith could Otto Bayard theorize that perhaps we needed just a tiny amount of iodine, going against all known nutritional science at the time.

Humans likely evolved culture as an alternative to animal’s reliance on old pack members to know what to do in case an extremely rare but devastating event happened every hundred-ish years. Rituals could seem completely nonsensical inside a single human lifespan, because they addressed problems at a societal level that only happened every 200 years, or slow acting chronic issues. In one case, ancient elephant matriarchs were the only ones capable of remembering waterholes far enough from a drought that only happened once every 35 years. The packs that lost their matriarchs all died because they had lost this knowledge.

We evolved logic to solve problems that had clear first-order effects, but we aren’t very good at evaluating second-order effects. Long lived humans were capable of finding cause and effect links that happened over a human lifespan, but only human culture perpetuating strange and bizarre rituals created out of random experimentation could deal with problems that had very long, unknowable cause and effect chains. It is very hard to tell if the person building your house is competent if the house only collapses every 150 years when a massive earthquake hits. Various cultures have developed all sorts of indirect methods of measuring competence, but many of them emphasize students obeying their teachers, because the teachers are often perpetuating rituals that are critically important without actually understanding why the rituals are important or what they guard against. It is culture guarding against Chesterton’s fence over enormous timespans. Another good example of epistemic hell is how we cured scurvy by accident and then ruined the cure:

Originally, the Royal Navy was given lemon juice, which works well because it contains a lot of vitamin C. But at some point between 1799 and 1870, someone switched out lemons for limes, which contain a lot less vitamin C. Worse, the lime juice was pumped through copper tubing as part of its processing, which destroyed the little vitamin C that it had to begin with.

This ended up being fine, because ships were so much faster at this point that no one had time to develop scurvy. So everything was all right until 1875, when a British arctic expedition set out on an attempt to reach the North Pole. They had plenty of lime juice and thought they were prepared — but they all got scurvy. The same thing happened a few more times on other polar voyages, and this was enough to convince everyone that citrus juice doesn’t cure scurvy.

Our ancestors weren’t stupid. They were trying to find some kind of logical progression of cause-and-effect, but they lived in epistemic hell. This is why cargo-cult programming exists. This is why urban legends persist. This is why parents simply want their children to do as they say. This is why we have youtubers chastising NASA for not reading their own Apollo 11 postmortem. This is why corporate procedures emphasize checking boxes instead of critically examining the problem. When your cause-and-effect chain is a hundred steps long and caused by something 5 years ago, economic pressure incentivizes simply trying to avoid blame instead of trying to find the actual systemic problem. The farther up the chain of management a problem is, the longer it takes for the effects to be felt, and the worse we get at finding the root cause. Software engineering has the same issue, where incompetence may only cause performance issues years later, after the original coder has left, and the system has scaled up beyond a critical breaking point. This is why we still don’t know how to hire programmers.

Only in the modern era do we have the necessary technological progress and the historical records to be able to accurately evaluate the usefulness of our rituals. Only now can we watch chemical reactions happen at an atomic level. Only now can we have Just Culture and blameless post-mortems that allows identifying actual systemic failures. Only now can I watch a YouTube video explaining how to go from a quantum simulation of particle collisions to a dynamical fluid simulation. Only now can I watch a slow-motion capture at 200000 frames per second to see exactly how a tiny filament explodes into hot globules that then fly into a nest of zirconium filings and set it aflame exactly where each one lands.

The engineers who invented these flashbulbs couldn’t see any of this. They had to infer it from experimentation, whereas I can just watch it happen and immediately understand exactly what is going on. We live in a pivotal moment of human history, on the cusp of being able to truly understand the entire chain of cause-and-effect for why things happen. We have the ability to measure events with unprecedented accuracy, to tease out tiny differences that catalyze huge reactions.

Unfortunately, the ability to merely see cause-and-effect is not sufficient when large systems tend to be chaotic. We do not yet have good mathematical frameworks for predicting emergent behavior, and our ability to analyze complex chaotic systems is still in its infancy. We know that large groups of humans consistently display emergent behavior, such as crowd dynamics closely following the equations of fluid dynamics. Likewise, large human organizations are themselves largely emergent behavior, and we never really understood how they were working in the first place. Organizational competence, and coordination problems in general are our modern epistemic hell, and it means there is no easy way for us to address the failure of our institutions, because we still have no holistic way to analyze the effectiveness of a given organization.

We are tantalizingly close to grasping the true nature of reality, to having the machinations of the universe laid out before us, but we are still missing the tools to fully analyze subtle patterns, to lift a whispered signal out of the thundering noise of spacetime. There is simply no escape from emergent behaviors evolving out of chaotic systems. Until we have the means to analyze these kinds of complex systems, we will forever be at odds with our nature, still tempted to cling on to superstitions of old, because long ago, that was the only thing that kept us alive.