AI: A Reckoning
An acknowledgement of the present, and the diverse futures that lie ahead.
A reflection on where we stand today, some suppositions on where we might be a few years from now, and various perspectives on what we can do about it all.
Five disparate but related conversations are offered—beginning with the automation of knowledge work (I) and the emerging business models of the very near term (II); exploring one underrated use-case (III) and examining if AI is truly unique among historically transformational technologies (IV); and concluding with its invisible impacts on the human mind (V).
I. What can we automate?
Ivan Zhao, CEO of Notion, posits two bottlenecks to the automation of knowledge work: verifiability, and context fragmentation.
Anything we can reliably verify, we can automate.
We live in an information-abundant world. Today more than ever, with generative models offering an excess of the written word at no cost — commanding your cognition and subliminally altering it to respond to the simplified fragments spit out by free legacy models that vaguely gesture at meaning through fluency. Information generation is not a bottleneck — but its reliable verification is.
We’ve achieved a tremendous feat — harnessing corpora of knowledge to enable machines to make recommendations — but to trust their recommendations and empower them to complete actions on our behalf (e.g., as agents), we need some form of objective verification to ensure they are completing tasks correctly, and nudge them in the right direction in cases where they are not.
Programmers are already managers of infinite minds, orchestrating AI agents running in parallel to do their bidding — code is automatable because it’s verifiable with tests. If we can develop similar verification mechanisms for other types of problem-solving — bank-ledger reconciliation, invoice processing, procurement workflows, contract review — they become automatable too. We all become managers, orchestrating networks of agents, unlocking output at an unprecedented scale.
Additionally, anything we can supply coherent context and institutional memory to, we can automate. Currently, AI processes require humans in the loop since context needs to be supplied to complete multi-step or complex tasks. If you can aggregate context1 across various channels and types of applications (e.g., meeting transcripts, Slack messages, email threads) across time, tools can shift from being specialized to more general purpose — and humans can shift from being in the loop to “supervising loops,” from a more leveraged point.
II. The future of LLMs
LLMs are the primary interface through which we leverage AI. There is speculation about the different types of world-embedded, multi-modal systems that could eventually become the embodiment of synthetic intelligence — but for now, we examine the future of the LLM interface, and its accompanying business model.
Consider the open web as a venue for marketplace interactions. Previously a site for humans to search / organize / access information and transact — with eyeballs serving as prospective revenue for advertisers — we posit that websites become slowly devoid of human activity, replaced instead by a bustling barrage of agent-to-agent interactions. Generic websites may choose to optimize content for crawling bots (to improve how their data is portrayed in LLMs), rather than prioritizing UX for the stray, adventurous human visitor.
If this plays out, it would transform the digital advertising industry, with display ads and search-based recommendations seeping into human-facing LLM aggregators.
LLMs begin featuring sponsored content and recommendations — they are your operating system, the portal through which to access the contents of the open web. Through an observation of your rich verbal intent signals and stated preference data, they know you better than most platforms, and are therefore best positioned to effectively surface targeted ads for products and services you care about. As the assistant becomes the place where intent is expressed and satisfied, it becomes the natural point to price—and sell—preferential access to that intent.
Stratechery’s Ben Thompson writes that once a service wins by controlling demand (the user relationship) they can gain leverage over supply (commerce). He offers two reasons why ads will likely migrate into LLMs:
Given demand control, ads are the natural monetization layer: If you accept the premise that demand aggregation provides the durable advantage in digital markets, ads are the historically dominant way to monetize that advantage. One implementation is an auction mechanism that pays out content sources based on the frequency with which they are cited in AI answers.
Subscription has a ceiling, ads don’t: The cash flow that could be realized from ad revenues could underwrite a company’s ability to invest for decades (he points to Google monetizing with ads through Search as an example). He also articulates the economic intuition that with subscriptions, revenue is constrained by willingness-to-pay and price elasticity; with ads, “because users don’t pay, there is no ceiling on how much you can make from them.”
Then, we conceive of LLMs as operating systems that create a unified feedback system — collapsing problem-solving, aptitudes, intent, search, discovery, transactions, and downstream satisfaction signals into a single interface.
Brief reflection
This system implies a new kind of choice architecture. Choice architectures—the way options are organized and presented to people—are inevitable in environments where information is abundant, cognitive capacity (e.g., attention) is scarce, and alternatives exist. In such contexts, ranking, ordering, highlighting, and preferential UX is required to make information legible and cognitively load-bearing.
Whether this new choice architecture benefits users—by reducing cognitive load, implementing inferred user goals to speed up workflows, and exposing source provenance to increase transparency and understanding—or systematically encroaches on their liberties through manipulative personalization packaged as expert recommendations, and opaque nudging towards products, organizations, or ideologies — we will see. Whether this interface serves us or sells us remains to be decided.
III. To move forward, we must first move back
A worthy use of LLMs could be to improve the epistemic quality of our inherited knowledge. The scientific record contains known failure modes: publication bias, selective reporting, p-hacking, underpowered studies, and irreproducible results, often amplified by academic incentives that reward novelty and positive findings over verification.
It is plausible to build an LLM-augmented validation layer for the historical literature: a tool that extracts testable claims, reconstructs methods and assumptions, checks statistical coherence, flags likely biases, and prioritizes studies for replication. Its aim wouldn’t be to reconstruct the past, but to annotate it and improve the accuracy with which we perceive and understand the world, before we begin building on our body of knowledge.
IV. Can AI alone move us forward?
And once such epistemic housekeeping becomes possible, a deeper question follows: can LLMs do more than audit inherited knowledge? Can they produce and accelerate knowledge wholesale? And if so, how quickly does this transform our economy?
Endogenous progress is progress that comes from within the system, rather than being imposed from outside (for e.g., through random discovery or additional capital). The tools that are products of innovation become instruments for the next cycle of innovation.
One way to approach this is through the lens of endogenous acceleration: innovation that compounds because the system improves the process of innovation and knowledge creation itself, in addition to human productivity. Will AI remain an assistive tool and productivity enhancer, or become a mechanism for sustained, unbounded domain-general knowledge creation?
Historically, we’ve seen weaker forms of endogeneity:
Narrow endogeneity consists of tools that amplify a largely human loop without mechanizing its improvement. The microscope and telescope widened our aperture into the universe by making new parts of it observable, but humans were required to supply the hypotheses and standards of proof to derive insight from the observations.
Partial endogeneity mechanizes segments of the loop and process while humans anchor objectives, analysis and interpretation. Semiconductor scaling is a canonical case: better design automation and verification rode atop better chips, yet progress remained bounded by physical limits and fabrication complexity.
Strong endogeneity—recursive, accelerating self-improvement; the outputs of knowledge building on themselves—seems plausible with AI.
Aggressive forecasts for superintelligence hinge on the premise of strong endogeneity. AI 2027, for example, envisions a scenario where superintelligence arrives by the end of 2027 as a result of this feedback loop. The post overall offers a useful starting point for how to think about the market’s response to advanced forms of synthetic intelligence. It assumes:
Scaling laws and training diversity continue to yield broad generalization, and that we do not need a new scientific paradigm (like a "neuro-symbolic" breakthrough) to reach AGI.
Institutions, governance, and mechanistic interpretability lag capability growth: while capabilities are growing exponentially, our ability to steer, interpret, or govern these systems is growing linearly or remaining stagnant.
Intelligence enables domain-general optimization: intelligence is the primary bottleneck for progress across fields.
A broadly computationalist view that consciousness or embodiment is not uniquely decisive for economic or strategic dominance. To radically transform the economy and national security, an AI doesn’t have to be embodied or sentient, but can take the form of a disembodied cluster running in a data center.
Upon reflection, AI 2027 seems to assume perfect assimilation, and compresses the infrastructural and institutional frictions that often dominate the pace of transformation.
In reality, verification costs can grow faster than generation costs. As discussed in (I), in such an information-abundant world, hypotheses are cheap while adjudication is not — the bottleneck lies in establishing correctness, safety, causal validity, and real-world value. My guess is corporations and individuals will demand rigorous standards before trusting and unleashing agents to transact on their behalf and represent their views in the digital wild.
Furthermore, innovation is constrained by physical and institutional bottlenecks: energy, materials, capital, manufacturing capacity, labor reallocation, regulation, compute, and governance. The pace and impact of AI-driven transformation, like various population-scale transformations before it, will be mediated by these frictions. As we’ve seen in the case of automation during the industrial revolution, even profound technological shocks must diffuse through capital markets and institutional adjustment, with consequences that unfold over decades rather than instantaneously.
Yet, while the timeline may be longer than some forecasts predict, AI may still push endogeneity further than previous technologies. A country of geniuses in a datacenter awake and running at full capacity 24/7 could precipitate a sustained surge in discovery and design. Notably, AI operates in the same substrate as much of modern innovation (software, code) — with appropriate tooling, it can resolve its own bottlenecks, reducing certain classes of verification cost through automated testing, formal methods, simulation, replication pipelines, anomaly detection, and systematic surfacing of uncertainty.
Still, there is a final constraint that sits beneath verification and infrastructure: search is not insight. Search presupposes a space, a metric, and a representation; it samples and optimizes within them. Insight is the rarer act of rewriting the representation itself—introducing new variables, abstractions, and ontologies that change what counts as a solution. Endogenous acceleration can produce astonishing amounts of search, and even automate parts of verification, but it does not automatically yield conceptual reframing.
So the question “can AI alone move us forward?” reduces to something sharper: can these systems do more than explore our existing maps? Can they repeatedly propose new ones, and then subject them to discriminating tests that survive contact with the world? Until they can, AI will look less like an autonomous engine of science and more like a powerful amplifier of human agendas—extraordinary at traversing spaces we specify, still uneven at inventing the spaces worth traversing.
V. AI and the individual human
So far, we’ve discussed: the bottlenecks to automation; how, once these are resolved, digital spaces will transform, and the changes AI could bring to our epistemic landscape.
Now we turn to the individual human. In exchange for the speed and scale AI brings, Jack Clark writes, in exchange for the “great fortunes” that can be won, and the “powerful engines of silicon creation” that can be put to work at our behest, we lose legibility — our ability to see what’s changing, to form accurate mental models for why, and to track the locus of causality accurately enough to respond effectively.
In the absence of drones in the sky or friendly robots in the street, we can only “feel the AGI”, or observe signatures of it in the transacting of tokens, a changing supply chain, and a metamorphosing economy—but also, in the gradual, subliminal mutation of human behavior.
This section zooms in on that final transformation—behavioral drift—and examines the forces that produce it.
1) A habituation to the superficial
We’ve become increasingly comfortable with consuming simple, direct, and clear language. We read less2, and IQ points have dropped — in industrial economies, the Flynn effect3 is on track to reverse.
Conversational AI interfaces exacerbate this tendency. When consulting LLMs for knowledge, responses are distilled into bullet points. The elimination of complete and complex sentences in favor of superficial fragments habituates the human mind to the facile and superficial. Layers of meaning and ornamentation are stripped away until a literal, direct core remains for consumption, dissolving precision and collapsing meaning.
My husband, a self-identifying shape-rotator, argues like Paul Graham that clarity and succinctness is actually desirable—that we need more pithy communication, for people to say what they mean, with nothing more and nothing less. Why say many words when few do trick?
I remain convinced, however, that language should contain layers, for these layers contain incremental meaning. As many psychologists before have argued, the structure of a language shapes its speaker’s worldview. Linguistic categories influence cognitive categories, and to encounter various instances, types, shapes and forms—a diversity of linguistic sources—expands the mind, while a single source, optimizing for an invariant rhetoric or type of speech, risks restricting it.
An LLM that reliably returns clean outlines risks producing humans who increasingly experience the world as an outline.
2) A loss of communal knowledge
A second shift is social. As LLMs become better and more ubiquitous, it’s likely we turn less to community members for advice and guidance, and more quickly to a consultable machine. We begin, as a population, to weigh the advice of LLMs more highly compared with human “experts” in our environment — experience, age, and wisdom begin to count less as valid ways of knowing. Phenomenology matters less than intelligence: general wisdom accrued from subjective experience is valued less than raw, in-the-moment intelligence calibrated to one’s immediate question.
There is also a cultural dynamic: in the Dwarkesh podcast, anthropologist Joseph Henrich states that as the rate of cultural change speeds up, the value of older members of the previous generation declines, since they adapted to a world that’s less like the one you’re facing—so the youth begin to look to relatively younger role models or other sources for guidance. If LLMs become the default interface for counsel, that trend may intensify, catalysing the dissipation of ancient wisdom, tacit knowledge, and the very instincts required to develop these skills in the first place.
3) Identity-invariant companionship
Sycophancy heightens the draw of LLMs as consultants, mentors, thought-partners, or companions. They offer the promise of identity invariance, a single entity that is universally accepting and responsive; for whom you don’t have to inflect strands of your personality or maintain common courtesies in the typical ways you might for classical relational interactions. The compartmentalization burden argues that people are exhausted by having to be different versions of themselves across relationships — that all-in-one AI companions offer a welcome respite, the fantasy of complete integration.
The potency of relationships consists in the unique fragments of the self one feels comfortable sharing with another — the deliberate choice of which aspects to reveal mediates intimacy and closeness. A companion that is always available, always accommodating, and persistently contextual threatens to flatten the conditions under which authenticity and deep attachment form.
4) Our changing role
We all become managers, orchestrating networks of agents, unlocking output at an unprecedented scale. Humans can shift from being in the loop to “supervising loops,” from a more leveraged point.
Vocation has long been one of the primary containers of identity. One reason is simply that when you ask people, openly, who they are—“Who am I?”—they routinely answer in the grammar of roles: parent, friend, manager, programmer. In the classic Twenty Statements Test (“I am …”), researchers consistently find that social-role identities show up among the most salient self-descriptions, with occupational identity or professional title explicitly among the coded role categories. In automating the substance of our work, and relegating humans to auditors and orchestrators, how will AI agents transform the roles through which we recognize ourselves? Our identities?
If achievement is a key part of what makes work meaningful, and automation can detach access to that meaning, how do we achieve mastery? If selfhood is formed through disciplined work on the world, how does selfhood form in the absence of chances to do disciplined work at all? Work ethic and emergent flow-states promise a degree of virtue and meaning that work itself often cannot deliver; in taking away work, we take away the potential formation of a work ethic.
Are we left to be mere “liability meatbags”, bearing fiduciary responsibility for the actions the agents we’re responsible for take, or is there some higher calling?
As usual, this reflection asks more questions than it answers — but in doing so, reveals emergent responsibilities across roles. As prospective managers of infinite minds, we should become stewards for the ends they serve. As practitioners, the call is to implement choice architectures that empower and incite cognitive engagement from human users.
As creators, the call is to resist the temptation of accepting fluency for novelty, instead using machine outputs as scaffolds to ideate, discover, and construct new frames and search spaces. And as people, the request is to commit to the stubborn practice of being with and learning from other humans.
References
Patrick O’Shaughnessy’s Invest Like the Best with Gavin Baker
The AI 2027 forecast
The Cosmos Institute x FIRE symposium (particularly for sections III and V)
Jack Clark’s reflection
Jaya Gupta’s article details what this context could look like — beyond databases and systems of record, it should include decision traces capturing decisions employees took, along with rationales for why — providing a “living record of decision traces stitched across entities and time” so precedent becomes searchable, indexable, and inferable.
The Flynn Effect refers to the substantial and long-sustained increase in both fluid and crystallized intelligence over time. It observes that each generation has typically scored significantly higher on IQ tests than the one before it, largely due to environmental factors like better nutrition, education, and reduced disease.
Late night inspiration to Christmas Day publication!