Rationalist hero, AI alignment pioneer, and brilliant autodidact Eliezer Yudkowsky recently published a short book called Inadequate Equilibria, Where and How Civilizations Get Stuck. Like all Yudkowsky writing, it’s densely thought provoking and intellectually playful. It’s not for everyone, but for those who enjoy thinking through the hardest and deepest philosophical problems, his writing is a can’t-miss.
The book has two central theses: 1) there are many areas where civilizations reach suboptimal equilibria because of flawed incentive structures and 2) too often smart people are too modest in challenging those inadequate equilibria.
When it comes to flawed systems, Yudkowsky believes that this is not a rare occurrence.
[M]ost of the time systems end up dumber than the people in them due to multiple layers of terrible incentives, and that this is normal and not at all a surprising state of affairs to suggest.
The first part of the book discusses which systems are more likely to have flawed structures that are exploitable and which do not. For Yudkowsky, there are three basic types of systems: 1) efficient and not easily exploitable 2) inefficient but inexploitable, and 3) inefficient and exploitable.
For an example of a system that’s efficient and not easily exploitable, Yudkowsky points to short-term markets. There are millions of smart people who watch markets and are highly incentivized to know whether the price of Facebook stock is going up or going down from one day to the next. If you know for certain whether the price of Facebook is going up or down tomorrow, you can make millions. Since other people can too, they will also want to do the same. If you have a leg up on the market, you should already be wealthy. If you are not, then perhaps you should be more most about your ability to outperform markets.
Yudkowsky also emphasizes that there are many flawed systems that many people know are flawed, but cannot be exploited because of skewed incentives. An example of this might be NCAA college sports, particularly football and basketball. For both college football and college basketball, the NFL and the NBA rely on collegiate sports to provide minor-league systems for their professional leagues. College sports are popular, but the athletes – the very people who are busy dedicating their lives to providing the entertainment, receive no compensation. This is a screwed up system that athletes have known is screwed up for decades. But there isn’t anything anyone can do about it, because of the extraordinary coordination problems inherent in the system.
Colleges certainly have no incentive to change the system; they profit from it. Individual athletes may wish to change the system, but to do so involves a huge coordination problem. Each individual athlete only gets one chance to aspire to a professional career, and the best way to do that is to excel in the conventional system. The best athletes, the ones with the most leverage, have the least incentive to rock the boat. They just have to play nice for a year or two and then profit from a professional career. Those with the greatest incentive to make a change, excellent athletes who aren’t quite good enough to become professionals, don’t have enough leverage to force a change. And the fans, the ones who watch and pay money to see college athletes, they like to pretend that the minor-league athletes who represent their alma maters are actually somehow representative of where they went to school.
So the system lives on, even though everyone knows it’s ridiculous, because the incentives of those who make decisions and benefit from the system are separated from those within the ecosystem, and the incentives of those who make decisions favor a perpetuation of the system.
The last category of systems is the inefficient and exploitable. It is here where Yudkowsky recommends that we focus our attention. Every successful startup began when visionaries saw a flawed system and then went about fixing it. Yudkowsky here is not necessarily arguing that everyone should go about starting their own business, but rather having the confidence to trust one’s own judgment. To understand that the majority of systems outside of short-term markets have deep flaws. And if the opportunity is right for exploiting those flaws, that we should do so.
This is a central disagreement I have with modest epistemology: modest people end up believing that they live in an inexploitable world because they’re trying to avoid acting like an arrogant kind of person. Under modest epistemology, you’re not supposed to adapt rapidly and without hesitation to the realities of the situation as you observe them, because that would mean trusting yourself to assess adequacy levels; but you can’t trust yourself, because Dunning-Kruger, et cetera.
The alternative to modest epistemology isn’t an immodest epistemology where you decide that you’re higher status than doctors after all and conclude that you can now invent your own de novo medical treatments as a matter of course. The alternative is deciding for yourself whether to trust yourself more than a particular facet of your civilization at this particular time and place, checking the results whenever you can, and building up skill.
We live in a world where Donald Trump was elected president. This is not a place where the most qualified always rise to the top. As such, we ought not perpetually to defer to those with higher status. We have to rely on our own judgment to decide what to do and how to react in any given environment.
Yudkowsky does not suggest that it is easy to exploit vulnerable systems. To succeed, we have to be very cautious about picking our battles. He provides the following formulation for how often we can expect to exploit such systems.
0-2 lifetime instances of answering “Yes” to “Can I substantially improve on my civilization’s current knowledge if I put years into the attempt?”
Once per year or thereabouts, an answer of “Yes” to “Can I generate a synthesis of existing correct contrarianism which will beat my current civilization’s next-best alternative, for just myself.”
Many cases of trying to pick a previously existing side in a running dispute between experts, if you think that you can follow the object-level arguments reasonably well and there are strong meta-level cues that you can identify.
Yudkowsky then makes one final, powerful argument about why we should not have too much modesty in the face of daunting systemic challenges. Simply put, modesty is a losing strategy.
I think that’s my true rejection, in the following sense: If I saw a sensible formal epistemology underlying modesty and I saw people who advocated modesty going on to outperform myself and others, accomplishing great deeds through the strength of their diffidence, then, indeed, I would start paying very serious attention to modesty
This does not mean that we should then employ an arrogant disregard for systems or people. It means we should make a lot of small bets, assess how we do in those bets, and then reassess and move forward. In Yudkowsky’s words, we should:
Run experiments; place bets; say oops. Anything less is an act of self-sabotage.