What's So Bad About Rationalism?

Reason and logic alone are not adequate for understanding reality.

Hello and Welcome to your Dark Markets weekend read. This is a rough draft of an important section of my Sam Bankman-Fried book, “The Boy Who Wasn’t There.” Keep that “rough” part in mind - this is a work in progress. That’s the main reason these posts are partially paywalled, though quite a bit of this one is available to all readers.

One quick reminder: be sure to check out my new podcast episode discussing FTX and Tether with the Crypto Critics, Cas Piancey and Bennett Tomlin.

***

Throughout this text, you will read many references to “Rationalism.” This is easily confused with the very similar word rationality - and that’s exactly how the Rationalists get you.

Eliezer Yudkowsky, the Naked King of Reason.

In very broad strokes, Rationalism is the belief that pure logic, reasoning, and mathematics can, given present knowledge and tools, produce objectively “correct” solutions to all of humanity’s problems, not just in the present, but into the future. This requires an intense and more importantly singular faith in the scientific process; in the base physical predictability of the universe; and, in the end, in the Rationalist’s own incredible gifts of logic.

Rationality, by contrast, is not an ethos or set of beliefs. Rather, it’s one tool among many for understanding the world, and it’s a permanent feature of human existence - the Enlightenment privileged deductive reasoning, but it didn’t invent it.

Rationalism, though, makes rationality the only meaningful rubric for grasping reality, dispensing with (often useful) modes of knowledge including tradition, intuition, and emotion. The irony is that science has proven that intuition and emotion in particular are actually central and vital parts of how living human beings make decisions. It is not a coincidence that Rationalism is rooted in Silicon Valley and heavily concerned with AI. It is a mathematized and instrumental view of the world-as-computer - and more importantly, of human beings as computers.

Ultimately, this dismissal of human and social complexity is why Rationalism fails, and why it has produced such damaging, stupid, and embarrassing results for its many proponents.

It is also why ideas rooted in Rationalism so often tend towards fascism, or the denial of difference. As an ideology, orientation, and living community, Rationalism is the underpinning, both socially and in the substance of its ideas, to other extremely reductive, naive, and covertly violent belief systems, starting with Effective Altruism, but also including Longtermism, belief in the AI “singularity,” and ultimately, race science and eugenics.

Longtermists and EAs will, with nearly pathological regularity, remind anyone who asks that sometimes “sacrifices must be made” in the present, in order to improve the future. This process of excusing present harms in service to a glorious future is how violence sneaks into such a superficially benign idea as the assumption that the world is predictable and “rational.”

Because when a eugenicist or a longtermist asks for extreme sacrifices or specific harms to human beings in the present, they must argue that they can confidently predict the long-term benefits of that sacrifice or harm. This is where Rationalism, with its total belief in a predictable and reducible universe, feeds into the dangerous edge of the Silicon Valley ideology.

This is also, to make some second-order connections, why those who believe artificial intelligence is a threat to human life (“AI Doomers”) are able to argue that funding “AI Alignment” is more important than feeding the hungry or preventing disease in the present. It supports the instrumentalization of other humans that allowed Sam Bankman-Fried to effectively steal $8 billion dollars and leverage it in an attempt to singlehandedly subvert Democracy.

What Else than Reason?

Many, many people are likely to consider these arguments and immediately dismiss them, because they mistake rationalism for rationality. That same mistake, in fact, is a major part of what makes rationalism and the Rationalists so seductive and dangerous, particularly for a certain set of Silicon Valley wunderkinder.

What, you are likely to ask, is the alternative to “rationalism?” Has science not saved millions, maybe billions of lives? Is not the rational process of science the backbone of the wondrous increase in the quality and length of human life over the centuries since the Enlightenment began to push away the shadows of myth and cant?

But rejecting rationalism and the Rationalists does not in any way involve a rejection of rationality, as the considered process of weighing evidence and drawing conclusions from it.

Instead, it is the Rationalists who are rejecting any alternative ways of knowing or being that make up what it means to be human. They are rejecting anything but a very narrow set of tools, while explicitly pushing away tools including tradition, intuition, and emotion. For all of rationality’s blessings, these have also been pretty useful throughout human history.

I want to make clear that I’m not about to start doing land acknowledgments - the argument here isn’t that “The West has forgotten traditional ways of knowing and should return to it tribal whatever.” That argument is stupid in its own way.

Rather, my point is that all human societies ever have operated on some mix of all of these ways of knowing. There was plenty of intuition and emotion involved in the project of the founding of the United States, the emblematic polity of Enlightenment. Though the rationalists wouldn’t claim it, the French Revolution was a rather passionate bid for a rational society - the kind of contradiction that the Rationalists would be more likely to dismiss as “suboptimal” than to actually reckon with as a manifestation of something deeply true about the human species-being.

One illustration of this is the fact that sufficiently nuanced expressions of the human experience must ultimately find shape in art or music, or even in simply pure being, none of which even exist as analytical categories within a purely rationalist conception of time and the universe.

Furthermore, this is a problem that compounds on itself, because a rationality that excludes the irrational in human nature cannot even rationally solve humanity’s problems, because it prevents thinkers from understanding themselves, including their limitations.

Rationalism, ultimately, is about rejecting epistemic humility. It assumes we know everything, and should act on that total knowledge with total confidence - including when it comes to predicting events decades, or even centuries, into the future.

The Delusion of Total Knowledge

The most immediately flawed premise of the Rationalist platform is the implicit assumption that human scientific and social knowledge is already complete, and (perhaps more to the point) that the people wielding Rationality possess all of the correct beliefs within this complete body of knowledge, including holding the right positions on matters of ongoing controversy among professional specialists.

There is a more nuanced but perhaps even more devastating argument that, per Gödel, committing to a ‘rational’ approach for all human decision-making is a formula for disaster, on strictly mathematical grounds. It seems to assume the existence of a total system of knowledge, on which rational operations can be performed to reach accurate conclusions about the current or future ‘nature’ of the world, and of right and wrong.

But this “prior” has been disproven as a viable approach to reality by their very own precious rationality, in its purest and most trustworthy form - by Gödel’s mathematical proof, the Incompleteness Theorem.

Eliezer Yudkowsky: From Zero to One (with help from some guy’s money)

One way to draw the distinction between “Rationalism” as a mask for authoritarian impulses, and “rationality” as an often-useful set of heuristics derived from the Enlightenment, is to point out that the Rationalists do not functionally believe in markets. In particular, they do not believe in the marketplace of ideas, nor in the marketplace as ideas.

The perfect pair of matched examples are provided by my twin betes noir in this book: Eliezer Yudkowsky and Sam Bankman-Fried.

Yudkwosky is a representative of the rejection of the marketplace of ideas, insofar as his industry-billionaire funded and completely disreputable “research institute” is part of a larger agenda to discredit and defund the traditional academic world, and replace it with people who, like Yudkowsky, act unapologetically as extensions of their financial backers.

This might be worth some explication. While it has far more than its share of problems, the world of academic research is still a world in which people have these things called “careers.” Over the course of these “careers,” scholars work will be again and again judged by their peers, evaluated by many, many other experts in their field. In a steady though often unfair process, high achievers over a long enough period of time should rise to positions of prominence, having earned the substantial (but rarely unanimous) respect not just of the public or of a few allies, but of their professional peers as a whole.

This is a competitive process of exactly the sort that Peter Thiel is vaguely associated with in the public mind, insofar as he’s supposed to be some sort of free market libertarian. That’s not true at all, but he has cultivated the impression, so it’s worth highlighting this contradiction.

What Thiel did with his initial investment in Yudkowsky’s MIRI - something that he also aimed to do on a smaller scale with his Thiel Fellows program - was to remove Yudkowsky from academic competition. It is the intellectual equivalent, roughly, of his avowed favorite business strategy, spelled out in a famous opinion piece and later book.

“Competition is for losers,” Thiel famously wrote, arguing that in a developed economy, all competition is zero-sum, and the best strategy is to go, in essence, outside the rules of the market to make sure that you don’t actually have to compete with anyone.

It might seem astonishing that such a fundamentally anti-capitalist and anti-American worldview has gone so seemingly unscorned by Thiel’s billionaire peers, but that day may be coming. What Thiel is arguing goes against not just the democratic idea of fairness enshrined in law, but also against the supposedly “natural” laws of the market, according to which competition creates abundance by forcing competitors to become more efficient, or to meet the needs of more “customers.”

This vision of fruitful competition is on some level neoliberal horseshit, but there is at least general evidence for it if you focus on the thin definition of economic development as human “thriving.” Thiel, like so many right-wingers whose authoritarian daydreams are given free rein thanks to the very open democracy they seek to tear down, envisions instead a return to the middle ages, to citadels and dependencies, closed and bricked.

Sam Bankman-Fried: The One Who Knew Best

And how does Sam Bankman-Fried fit in to the premise that “Rationalists” are antidemocratic, and specifically anti-Market?

Here is where we must fill in the blanks of motivation left by Bankman-Fried’s compulsive, delusional, likely sociopathic denial of his role in the collapse of FTX. What materially happened, and what was implicitly the goal of a series of large and small decisions ultimately made by SBF, was the use of funds that were the legal property of customers to invest in assets and expenses other than what they had intended.

The implicit belief driving these actions was that even though FTX’s entire business was helping people choose what they invested their money in, Sam believed that he knew, better than each and every one of his hundreds of thousands of customers, what projects, companies, technologies, and networks would grow. He took their money, it was always implicit, because they did not know the right thing to do with it, whereas he did.

This is in contradiction with a fundamental principle of markets - that they unleash the “wisdom of the crowd.” In fact, this is one of the favorite tenets of ubercapitalists like Friedrich Hayek, in their arguments against state industrial policy.

This point is driven home by the fact that so many of the charitable funds that famously flowed out of FTX were funneled to nonprofits not just associated with effective altruism and rationalism, but explicitly devoted to developing EA and Rationalist ideologies and methods. This included, for instance, $5 million recently alleged to have flowed through an FTX charity to an entity called Lightcone Infrastructure [https://davidzmorris.substack.com/p/stolen-ftx-money-supported-richard] that now administers Yudkowsky’s Lesswrong.com. Large amounts also went to effective altruism nonprofits, so much so that SBF was essentially neck-and-neck with Facebook cofounder Dustin Moskovitz, who remains a major EA funder.

The implication of these donations is again compounding, or dual. On the one hand, the rationalist worldview implies that studying rationalist methods now is so important that it qualifies as worthy of “charity” from a man who had said his goal was to make the world a better place. But at the same time, and in a frankly more meaningful way, the donations implied that this specific group of people had a kind of copyright on rationalist methods.

The money didn’t go, for instance, to the philosophy department of large public universities. Those departments, with their open, transparent, and fair admissions processes, their rigorous and collaborative methods, and their huge institutional knowledge, are the real “marketplaces” in which rationality in the broader sense could be honed.

Or at least, that would be the case, if those in the existing Rationalist movement were self-aware enough to see the stark contradiction between their professed ideals, and the cult-like, gnostic discourse which they are actually living out.