The closing day of the 14th Bangalore Literature Festival felt less like the end of a nerd’s weekend retreat and more like the next chapter of an R.F. Kuang story. Best-selling authors—both the fresh faces and the well-known figures from the day before—now stuffed the roster like a PCB board. A key segment of it was Karen Hao’s Empire of AI, which, for many devotees, took the spot of the Sunday morning show over breakfast.
After an exhaustive discussion with fellow panelist Anil Ananthaswamy and moderator Indulekha Aravind, which nearly went into overtime, it was time for Karen to show her journalism chops and showcase the hunk of a book about OpenAI, its culture of secrecy, and the promise of artificial general intelligence.
Much like her panel talk on Day 1, listeners turned up in massive numbers in the watchtower arena. The interviewer across the table is Samanth Subramanian, another journalist with more than a few bestsellers under his belt. After a brief intro about the book and the state of AI, he asked Karen to do a time jump and tell the audience what it was like to encounter these people for the first time at OpenAI.
“It’s really hard to explain to people now how much OpenAI was not really considered a serious company at the time,” answered Karen as she recalled the time when she ended up becoming the first reporter to profile the company in 2019. “Within the AI world, it (OpenAI) was seen as this strange stepchild because they had, just straight out of the gate, said, ‘We are going to pursue artificial general intelligence, AGI,’ which, as a concept, was largely unpopular within the scientific community because it was seen as almost a pseudoscientific idea. Anyone who was talking about AGI was not a real researcher, not actually part of the field. And in fact, many of the people who worked within OpenAI at that time were not part of the fields. They were tech entrepreneurs, like Greg Brockman, who was the chief technology officer, had never done AI research in his life, and had never even graduated college. Sam Altman, himself, you know, also a Stanford dropout,” she began to continue as Subramanian chipped in to add the fact that Altman was also a failed writer. Karen affirmed that as well.
She mentioned the cultural rhythm surrounding OpenAI in the early days, which portrayed the company as an underdog, making efforts to take on Google, the pioneer of AI research. But unlike Google, OpenAI was campaigning for making all their research public and transparent, to share the knowledge with the world.
“But what I quickly realized when I was interviewing the executives is that while they position themselves as completely different from Silicon Valley, they were actually just a product of it,” highlighted Karen as she revealed her findings on OpenAI before it was the juggernaut that it is today. “When I was in that very first meeting with Greg Brockman, Chief Technology Officer, and Ilya Sutskever, Chief Scientist, and just asked them simple questions, like, ‘Why AGI? Why spend billions of dollars building AGI rather than, you know, distributing that money to help alleviate poverty, or trying to tackle already known technologies that we can use to mitigate climate change?’ They could not articulate an answer to that question. And when I asked, ‘Okay, well, part of the founding mythology of OpenAI is that you need to build it—the good AGI—first before the evil people build a bad AGI. And I said, ‘So, what’s the worst that could happen if the bad AGI comes?’ And they couldn’t articulate that either.”
This preliminary impression of OpenAI served as an eye-opener for Karen. According to her, they had not even thought about the extremely basic questions about what they were doing. And this was like most Silicon Valley startups, where they were given billions of dollars solely because of a compelling idea, and they could sell it to the right people in power and hold their purse strings. She became suspicious from the start because OpenAI didn’t care to consider the impact they were promising to make on the world, let alone make it positive.
On that note, Subramanian pushed further to ask whether OpenAI anticipated the pace of technological change that was yet to come, in the form of ChatGPT-3 in just four years.
“They did, and they didn’t,” stated Karen as she began to open an explanation. “In the sense that, in the very early years of OpenAI, they were already far more bullish than anyone else in AI research about the idea that there could be a very transformative AI system coming in just a few years. And they were projecting, internally, what they thought might happen with this particular approach of AI development that they were pursuing, which is taking existing techniques that had already been developed, specifically a technique from Google called the Transformer Neural Network, and then scaling it with ever more training data, with the largest supercomputers that had ever been built in human history. And they were projecting these out. They call them scaling laws, like mapping out how they thought the capabilities would continue if they dramatically jumped the scale of the data and the compute. And so, they already anticipated that they would be requiring very rapidly far more than just billions of dollars, but actually hundreds of billions of dollars. And I think based on that, they had a sense that if this worked, if their scaling approach worked, they would be talking about a very dramatic change in AI capabilities within only a few years.”
Subramanian then jumped to a part in the Empire of AI that he found quite strange. There was a set of private correspondences in the book in the form of emails and exchanges between people that include Elon Musk, Sam Altman, and Dario Amadei. In that part, Karen sensed the fear of the worst outcomes of AGI. The question he asked was, “At what point did your concerns start to become secondary or tertiary to these people?”
The answer received a warm round of applause and laughter, but didn’t surprise the audience. Karen said, “If I were to summarize, like, a large umbrella to encompass all of these different people’s perspectives, all of them generally think it’ll go well as long as I do it, and I need to do it first.”
“We run an entire country based on that principle,” remarked Subramanian promptly. Karen mentioned that most big names associated had gone on to create their own company and eventually become billionaires in the process. She outlined that a lot of these people genuinely were fearful of AGI, but the same might not apply to Altman. “People are never sure (about Sam Altman). Like, does he really believe that this is a catastrophic future that might happen? Or is it just a mythology that has worked really, really well and is helping him amass a lot of resources and a lot of power for OpenAI?” she added before mentioning an expert from Empire of AI on a particular observation Altman made in 2013: “The most successful founders don’t set out to create companies, they set out to create religions, and it turns out that a company is the easiest way to do so.”
The title of Karen’s book prompted the next question, which asked her to draw parallels between an Empire of AI and orthodox imperialism.
Karen compared OpenAI to a modern version of the East India Company and then provided four stark parallels that she observed.
- They lay claim to resources that are not their own. Training their models on the intellectual property of our artists and traders without their credit or compensation, training on public data available online, and saying that it’s just free, even though, you know, people are posting their children’s photos online, not thinking that they’re getting ensnared in their web of GPT models.
- They exploit an extraordinary amount of labor—tens of thousands of workers that they contract, primarily in the global majority, including in India, where people are asked to do really horrific tasks in order to content moderate the models that OpenAI builds, in order to even teach the models how to respond.
The fact that ChatGPT can chat is because there are tens of thousands of people around the world who are teaching it. This is what a conversation looks like: Person A speaks—person B responds. Person A asks a question—person B answers. And they’re paid a pittance, and the working conditions are awful.
- The empire always monopolizes knowledge production. That’s part of the way that they perpetuate themselves, by only allowing certain information that enhances their imperial agenda to exist. And anything that would undermine their expansion, they censor where they quash.
OpenAI in the AI industry does the same thing. They have become so resource-rich in the last 10 years that most of the AI researchers now are bankrolled by these companies. Either they are academic labs that are funded by them, or they are employees of these organizations. So, there are both soft and hard ways that these companies distort what kind of research they’re even allowed to produce. There is a case where one researcher, the ethical AI code lead of Google, was just fired straight up because they tried to write a paper about the harms of large language models.
- All empires engage in this moralizing narrative that they are the good empire, on a civilizing mission to bring progress and modernity to all of humanity. And they are competing with the evil empire. Silicon Valley often nods to China as the evil empire. If the evil empire gets there first, then humanity’s going to end up in some kind of hell, and that’s why the good empire needs unfettered access to resources, to labor, to all of these things, because that’s the only way that humanity will finally attain heaven.
After this extensive answer, Subramanian posed a general question along the same lines about the segregation of AI companies in the US as compared to the rest of the world.
“You have to understand that when these executives are talking, they’re talking to a specific audience, which is the U.S. government,” answered Karen as she began to explain how internal decisions are carried out in typical AI companies. “Meta, for example, has hired an enormous number of Chinese and Chinese-American researchers, to the point that sometimes Meta is now called the People’s Republic of Meta. And yet, Mark Zuckerberg is the number one executive who, when he speaks to the U.S. government, is like, ‘We need to beat China; we’re in a race against China.’ But behind the scenes, he’s like hiring up all of these Chinese researchers because they’re really good.”
OpenAI was not much off the mark here as well, she suggested. “In the beginning, a huge share of OpenAI researchers were not American. They were from all over the world. But when they had to figure out how to position themselves with the government, and I cite a lot of internal documents where OpenAI is, like, talking about the strategy for how to make sure they essentially become backed by the U.S. government, essentially, and get the allegiance of governments around the world. They talk about emphasizing these narratives, like, ‘We are in this race with evil companies like Google and with evil countries like China.’ And so it is all like a political game that they’re playing, a cute political game.”
Taking the lead from this response, Subramanian then asked about concordance between American politics and the politics of the AI industry.
Karen explained how the new Trump administration is trying to build a relationship with Silicon Valley companies to make sure America maintains an upper hand in global politics as an economic power. On the surface, this relationship seems aligned, but there are also tensions, where Silicon Valley is trying to build out its own empire, irrespective of American interests. With the unstated intention of surpassing the US government, the clustering ideology that democracy is no longer a means of organizing a society is spreading among these corporations. In the meantime, the U.S. government is attempting to control Silicon Valley as an American asset by implementing various global initiatives in the hopes of advancing the American imperial agenda. This allows companies like OpenAI to ask for backing for all the debt they piled up to build a computational infrastructure, essentially a government bailout and nationalization of OpenAI.
The conversation turned toward China when Subramanian mentioned the argument he read in a book by Dan Wang: “The US has become a country of lawyers, and China is still a country of engineers, and everything that is everything about their financial industry and about the way their society is structured, about their politics even, comes down fundamentally to these two differences.” This paved the way to a question of DeepSeek and the AI culture in China.
Karen mentioned the infamous chip ban the US enforced on China and how it compelled researchers to one-up OpenAI, a company founded by fundraisers rather than engineers. She called the development of AIs like DeepSeek a minor alternative path that branched off the main road of still producing LLMs. It used a fraction of computational resources compared to OpenAI to build GPT-equivalent models, while using already existing techniques in the AI literature.
Karen also stated that most of the problems of the world are not solved by LLMs. “Sometimes I like to use the analogy that AI is like the word ‘transportation,’ where the word ‘transportation’ can refer to anything from a bicycle to a rocket. They’re all different modes of getting from point A to point B, but clearly, they have different cost-benefit trade-offs, and they’re designed to target very specific things. Like, if I were using a rocket to travel from Bombay to here, that would make literally no sense,” she remarked. “And in the same way, I think most of the heavy-duty work that AI can do for us as a society will come from specialized modes of AI, and we’ve had that long before ChatGPT came out.” Karen brought to attention the absurd extent of capital misallocation that resulted in environmental degradation. These funds could have been used to improve supply chains, optimize a building’s or even a city’s energy consumption, perform more accurate weather forecasts, integrate more renewable energy into the grid, etc. And all of that could be done on a handful of laptops.
In the final minutes of the session, Subramanian revealed to the audience that he had asked Karen backstage if she used AI at all, to which she replied that she didn’t. Karen later added that she doesn’t use generative AI, but she uses specialised AI models. She recalled an instance of using Google’s reverse image search to discern the spike in prices of some of the furniture that OpenAI upgraded, when they flipped from being a non-profit to the Microsoft-docked juggernaut.
Karen maintained her stand against AGI till the end. “I don’t believe that anything is ever inevitable, but I do think that this pursuit of AGI is completely unwarranted. Not only because most scientists right now don’t even think we’re on our way to AGI, so, like, all of these resources that companies are justifying by saying it’s worth it because we’re getting to AGI,” she added, and bookended with a statement that reverberated with the audience in the watchtower arena. “Why are we building a system that is replicating human intelligence when we have human intelligence? Why are we using all of the world’s resources to ultimately copy us? That’s not the purpose of technology at all.”
Maynk
I also write here.