Gell-Mann Amnesia and AI – Econlib

Tyler Cowen recently challenged us to try to stump an AI named “o1”:

1. o1 is a very good economist.  Try to stump it if you can.

I could not find o1, so I used ChatGPT 4o mini.  I hope someone will try the following question with an o1 and report the results in the comment section:

The thing I keep noticing with AIs is that they are like a super smart student that is capable of memorizing vast quantities of information, but lack an ability to actually see the essence of a problem.  In the example above I gave one hint after another, practically begging the AI to avoid reasoning from a price change.  It had so little awareness of what it was doing that it ended up telling me that Qd fell while Qs rose and yet the market remained at equilibrium. 

Perhaps I used the wrong AI, or perhaps this question is unusually tricky.  (To a college student, “tricky” means a question where you actually have to understand something, and cannot just regurgitate what you’ve read in your notes.) 

So let’s take a question that Tyler asked the o1 economist:

Please write an essay on how current macroeconomists find inflation dynamics so very difficult to predict, and why that has made them reject various forms of monetarism, even as approximations of what is going on behind price level behavior.

The answer is very long, so I’ll just focus on a few key portions of the response:

Several factors have contributed to the increasing complexity and unpredictability of inflation. First, the financial innovations and regulatory changes of the late twentieth and early twenty-first centuries dramatically altered the relationship between money and economic activity. Monetary aggregates—like M1 or M2—that once served as dependable indicators of policy stance and future inflation now behave erratically due to shifts in the velocity of money, the proliferation of shadow banking, and the globalization of financial flows. Simply put, where money resides and how quickly it moves through the economy has become too fluid and too complex for older monetarist simple rules to capture.

Second, the nature of central banking and fiscal policymaking has changed. Central banks now intervene in a host of unconventional ways, from massive purchases of financial assets to the forward guidance of policy expectations. These tools are not well-explained by the classic monetarist perspective, which centered on controlling a particular measure of the money supply. The recent experience following the Global Financial Crisis vividly illustrates this: The Federal Reserve and other central banks undertook unprecedented quantitative easing programs, dramatically expanding their balance sheets. According to traditional monetarist logic, this rapid increase in the monetary base should have led to substantial inflation. Yet inflation remained persistently below target levels in many advanced economies for years, confounding those who relied on old monetary aggregates as a guide.

Third, the determination of prices today involves a bewildering interplay of global supply chains, technological advances, labor market transformations, and shifts in consumer behavior. Globalization means that prices for goods and services are influenced not just by domestic monetary conditions, but also by distant supply shocks, currency fluctuations, and international competition. Technological change increases productivity and can compress prices in certain sectors, while leaving other parts of the economy less affected. Labor markets have also evolved, with changes in union power, demographic shifts, and altered labor-force participation patterns influencing wage formation and cost pressures. These micro-level frictions and structural changes make the older macro-level equations linking money supply growth to inflation too coarse and imprecise.

Sorry, but this is mostly a mix of empty clichés and irrelevant comments.  Again, it’s the sort of thing that I’d expect a student to put down if they didn’t know the answer.  I’ll explain why in a moment, but first let me clarify an important point.  I am pretty sure that o1 is more intelligent than I am, at least in the way that most people define “intelligence”.  In other words, o1 knows way more than I do.  It is far more intelligent than I am in roughly the sense that the New York Times is far more intelligent than I am.  And that’s really, really impressive.

But o1 is not a good economist.  Sorry, it just isn’t.  

Let’s first dispose of the final paragraph, which is utter nonsense.  Technology has been advancing for more than 100 years, and the pace of productivity growth is not accelerating.  So that’s not a factor making inflation harder to forecast.  Union strength goes up and down, and that fact does not make inflation harder to forecast.  Shifts in consumer behavior?  Now the student is really getting desperate.  It’s an “I need to fill up the page with something” sort of comment. Global supply chains?  Again, it doesn’t make inflation harder to predict.  Not at all.  Global commodity prices had a bigger impact on US inflation 100 years ago than today. None of those four factors have made it any more difficult to predict inflation.  Indeed I’m not even sure it is becoming more difficult to predict inflation.  It seems to me that inflation was far harder to predict when I was young than it is today. 

The first sentence in the middle paragraph is a bit less bad.  Central banking really has changed in one respect.  And the relationship between the base and the aggregates really has gotten looser.  Unfortunately, the AI seems to have no idea why the relationship has gotten looser.  It mentions a bunch of irrelevant stuff like QE, and misses the key point that the payment of interest on reserves and the zero lower bound problem have made the money multiplier far more unstable.  The AI also says that fiscal policymaking has changed.  That was probably just a wild guess.  I cannot think of any changes in fiscal policy that make inflation harder to predict.  Given that the AI is silent on the issue, I suspect that it also has no idea what has changed about fiscal policy, or why those changes would make inflation harder to predict. 

In the first paragraph the AI says:

Monetary aggregates—like M1 or M2—that once served as dependable indicators of policy stance and future inflation now behave erratically due to shifts in the velocity of money, the proliferation of shadow banking, and the globalization of financial flows.

False.  The aggregates were never dependable indicators of inflation.  Money velocity has always been unstable.  But don’t the textbooks say that money velocity used to be stable?  Yes, many do.  That’s probably where the AI got its misinformation.  It might be more unstable now, but it’s never been stable enough for the aggregates to be good predictors.

Also notice the odd list, which includes velocity, shadow banking and financial flows.  That’s kind of silly.  An AI that understood what it was talking about would have put a period after velocity, and then had a separate sentence explaining some reasons why velocity changed.  Right now, it’s a confusing jumble of concepts.

If you are a math-oriented person who is confused by economics, this equation might help you see my point:

V = PY/M

The AI is basically saying that the PY/M ratio is changing due to things like changing V, shadow banking and financial flows.  The ratio is V!!  The other variables help explain why V might change.

Even though the NYT is much smarter than I am, when I happen upon a NYT article in an area where I have some expertise, the paper suddenly seems much less smart.  I won’t say “dumb”, because even at its worst the NYT employs talented reporters.  And I cannot expect them to have devoted their entire life to studying monetary economics.

In my view, the o1 response provided by Tyler is modestly below the level of the NYT.  At a minimum, the Times would have mentioned interest on bank reserves or the zero lower bound.  Nonetheless, AIs are truly impressive when it comes to the quantity of information they have memorized.  Even within monetary economics, an AI will often have more information at its fingertips than I do.  An AI could write a better essay on all the new Fed policy tools than I could write.  Where it falls down is in a certain type of understanding.  I’m not sure I can even put this concept into words.  Maybe understanding doesn’t really exist, and we just fool ourselves into thinking we have a deep understanding of something.  Perhaps a super-intelligence would mock my writing in the same way that I mock the AIs that engage in reasoning from a price change. 

All I know is that AIs have a long way to go before they impress me in the domain of understanding monetary economics.  But outside my field, I’m already highly impressed.

Here’s how AI Overview defines Gell-Mann Amnesia:

Gell-Mann Amnesia (GMA) is a defense mechanism that occurs when someone reads a news article about a subject they know well and finds it to be full of errors, but then proceeds to read the rest of the newspaper as if it is more accurate. The term was coined by Michael Crichton after discussing it with Murray Gell-Mann