When you ask people to trade an asset with an unknown terminal payout in a lab experiment, it’s really common to observe a boom in the asset’s price followed by a sudden crash right before the trading session ends. In other words, it’s very common to observe price paths that resemble a bubble. Bubbles are a regular occurrence in the lab even when participants are given no additional info about the assets they are trading and are not allowed to talk with one another.

By contrast, the recent literature on real-world asset bubbles tends to emphasize the role played by specific asset characteristics and social interactions. For example, Greenwood, Shleifer, and You (2019) document that price booms in industries with lots of young firms are more likely to followed by a crash. In my recent paper, The Ex Ante Likelihood of Bubbles, I show that speculative bubbles are more likely in assets where small price increases make excited speculators much more persuasive to their friends.

What are we to make of these two contradictory sets of results? I have a hunch that the boom/bust price paths observed in lab experiments are not the same phenomenon as the asset bubbles observed out in the wild. But this is just a hunch. And, as highlighted above, I’m definitely not an impartial judge on this topic.

So how might you check if the trading phenomenon that you’re studying in the lab is the same as the one you observe in real-world financial markets? It’s common to hear asset-pricing researchers say something like: “There’s *ALWAYS* a chance that a lab experiment has missed some important real-world consideration.” It’s certainly true that researchers should worry about omitted variables. But is it really the case that this problem is totally unsolvable? The *ALWAYS* part of the claim strikes me as too strong.

In this post, I describe an analogous problem from the physical sciences and show how you might solve it using dimensional analysis. My point is not that asset-pricing researchers should use dimensional analysis (but that ain’t a terrible idea). My point is that you can theoretically analyze the relationship between the lab and real-world asset markets. Asset-pricing researchers should be doing more of that. Is trading in the lab governed by the same economic forces as trading in real life? This need not be a philosophical question.

An analogous problem

Imagine you’re an engineer who works at a company that makes personal submarines, and your team’s most recent prototype turned out to be much slower in field tests than expected. Something about the craft must be generating excess drag, and you want to figure out what.

You have the data on the prototype submarine’s performance during the recent field tests done out in the ocean. But this craft is 18ft long and weighs several tons. It’s entirely impractical to tinker with. So you create a 1ft mock-up that you can quickly and cheaply experiment on in a wave pool.

But how can you be sure that these results will apply to the full-sized submarine? How can you be sure that your lab experiment isn’t omitting an important real-world consideration? And what is the correct way to scale up the results in the wave pool to results in the ocean?

This an analogous problem to the one outlined above in the introduction. Instead of worrying about the drag force experienced by a submarine, asset-pricing researchers are interested in some property of speculative bubbles (i.e., timing/severity of the crash, likelihood of future occurrence, etc). And, instead of using a 1ft model in a wave pool, they run market simulations in university labs. But the resulting question are identical: are the findings in the lab informative about the real world?

Here are the relevant variables for determining the submarine’s drag, : the length of the ship, , the speed at which the ship is traveling, , the density of the liquid it is traveling through, , and the viscosity of this liquid, . The functional form of the relationship between these variables

(1)  

will be determined by the nitty-gritty details of the submarine design. You don’t know this function. But, for your wave-pool experiments to be informative about the drag experienced by a full-sized submarine, this function needs to be the same for both. Any differences between the drag experienced by an 18ft submarine in the ocean and the drag experienced by a 1ft in your lab should be determined by the inputs , , , and .

Consider the alternative. Suppose there are other important factors in determining drag in the open ocean besides , , , and . Or suppose your wave pool is introducing a new variable that is irrelevant to drag force in the open ocean. In either case, your analysis of a toy 1ft submarine will not generalize to the real-world 18ft object. If it turns out to be the case that

(2)  

then your lab experiments will not be informative.

Dimensional analysis

You want to know when/how/whether there is any way to address this concern. Dimensional analysis offers one way to do this. “We are allowed to add, or subtract, various quantities together only if they are expressed in the same units. Thus, the left hand and right hand sides of an equation… must have the same physical dimensions.” And it turns out that it’s possible to leverage this seemingly banal observation to test whether . Here’s how.

I will use to denote the dimension of a variable. The length of the ship, , has dimension . It doesn’t matter whether this length has units of feet, meters, or parsecs. The speed at which the submarine is traveling, , has dimensions of length per unit of time, . Drag, , is a force, and Newton taught us that force equals mass times acceleration:

(3)  

Density, , measures how much stuff there is per unit of volume, . Finally, viscosity, , measures how much force is needed to deform a given area per unit of time, .

Suppose it’s possible to input , , , and . When you do this, the output is . If the functional form in Equation (1) holds across a wide-range of values, then it should also hold in the special case where , , and . The original values of , , , and were just arbitrary choices from the domain of .

But notice the problem that this creates. By setting and , we’ve just stripped the dimension of from the inputs to : , , and . So there is no way for us to balance the dimensions on the right-hand and left-hand sides of Equation (1) without making an additional change.

The simplest possible change would be to multiply by . If we do that, we get:

(4)  

That way, we would have times . This yields a combined dimension of for the right-hand side, which is exactly what we’re after.

Of course, there’s nothing special about the density input to . We could do a similar trick for length, , and speed, . If we set , , and , then we would need to multiply the right-hand side by an additional factor of to preserve dimensional consistency:

(5)  

Again, and are chosen to balance the dimensions on either side of the equal sign in Equation (1).

What’s more, there was nothing special about our original choice of inputs: , , , and . These could have been anything in domain so long as we used the same values throughout. So, if we drop the ones as inputs to and rearrange things a bit, we get:

(6)  

The left-hand side is dimensionless, , and commonly called the “drag coefficient”. The input to is called the Reynolds number, , and is also dimensionless, .

Why this helps

We didn’t know anything about the functional form of when we started our analysis. And we still don’t know anything about the functional form of . We have to empirically estimate both function based on observed data. So it might not be immediately obvious how dimensional analysis has improved things.

What’s more, every textbook I’ve read has talked about the benefits of dimensional analysis in terms of variable elimination. was a function of four variables. is a function of a single variable. These textbooks argue that it’s easier to estimate than because , , , and have been combined into a single variable, . This is true. But it’s not why dimensional analysis is helpful here.

Dimensional analysis is helpful for assessing the external validity of your wave-pool experiments because you know the way that the single variable, , combines the four other inputs , , , and . As a result, you can compare predictions made using Reynolds numbers that were arrived at in very different empirical settings.

For example, suppose that you observe a particular drag coefficient in the data from the recent open-ocean field tests involving the full-sized 18ft submarine:

(7)  

If you fill your wave pool with a liquid that is -times as viscous as sea water, like honey , do you observe a similar drag coefficient on your 1ft model submarine?

(8)  

The first calculation comes from real-world data. The second comes from your lab.

Do they match? If yes: great!! Your lab experiment captures the main forces at work in your open-ocean field test, . If no: bummer. Either your lab experiment is introducing unwanted variables or it is missing an important consideration. Either way, you should not assume your lab results can be applied to the next full-sized prototype submarine your company turns out.

The original problem

Asset-pricing researchers regularly see prices that boom and bust in trading experiments. These episodes look a whole helluva lot like speculative bubbles. But are they? Are the forces that explain these episodes the same ones that produced the Dotcom bubble? Or are these two different phenomenon? I really want to know the answer to this question.

The point of this post is not to argue that dimensional analysis is the obvious way forward. I wrote about an “analogous problem” rather than the problem I’m actually interested in because it’s not immediately obvious how to apply dimensional analysis to study speculative bubbles. It might be possible. But I haven’t managed to select the right variables. If I knew how to do this, this would be a paper and not a blog post.

Instead, my goal is to argue that it’s possible to assess whether the phenomenon observed in the lab is the same as the one observed out in the real world. This is true when the lab experiment involves a 1ft submarine and a wave pool. It’s also true when the lab experiment involves undergrads buying and selling a fictitious stock. Lab experiments are a major source of knowledge in the social sciences. Rather than complaining that a given experiment might be missing something important about the real world, asset-pricing researchers should work towards developing methods for verifying external validity. The application of dimensional analysis to the submarine example above is a proof of concept. It can be done. It is done in other fields.

Imagine you’re an asset-pricing researcher. You’ve just thought up a new variable, , that might predict the cross-section of returns. And you’ve regressed returns on in a market environment of your choosing (i.e., using data on some specific time period, country, asset class, set of test assets, etc):

(1)  

If differences in predict differences in returns in your chosen market environment , the estimated slope coefficient will large, . It would’ve been profitable to trade on the predictor in sample.

Suppose you find . Assets with higher values today tend to have higher returns tomorrow. You now face a choice about whether to publish this finding. If you do, then other researchers will read your paper and try to replicate it in other market environments you haven’t yet looked at, . Let denote the collection of all out-of-sample market environments that your colleagues might examine.

Obviously, you shouldn’t publish if isn’t a good cross-sectional predictor in most of these out-of-sample environments—i.e., you shouldn’t publish if . But, even if is a good predictor on average, you still worry about worst-case scenarios. If there’s one market environment where , then one of your colleagues will surely discover it and you’ll look utterly foolish when he tells the world. You only want to publish if robustly predicts returns out-of-sample:

(2)  

captures the relative importance of these two considerations to your publication decision. The larger the , the more you care about saving face by not publishing any really bad predictions.

Importantly, let’s assume that all you care about when doing research is solving this robust out-of-sample prediction problem. You don’t care at all about whether investors actually price assets based on . All that matters is whether reliably predicts returns out-of-sample. You’re completely drunk on Friedman’s “as if” Kool-Aid. Before deciding whether to publish, you have a choice as to which market environment to examine. What sort of environment should you choose? What should your empirical strategy be?

The key insight in this post is that, even if all you care about is robust out-of-sample performance, causal inference still turns out to be a useful tool for achieving this goal. If investors always use the same model to price assets, then understanding this model will allow you to always make good predictions. Your empirical strategy should be to choose an empirical environment that identifies the causal effect of on returns.

Investors’ model

I begin by defining investors’ model. Suppose that, in every market environment, investors price each asset so that its returns are governed by the following linear structural model:

(3)  

Moreover, assume that the parameters, , are the same in every market environment. is the cross-sectional predictor that you’re working on, and is an omitted variable. This is a variable that investors might be using to price assets but researchers have yet to discover. If it’s 1981 and is firm size, then might be liquidity. is a noise term and captures its affect on returns.

Crucially, either affects returns, , or affects returns, , but not both in investors’ model. If , then reliably predicts the cross-section of returns since is the same in every environment—i.e., in every time period, country, etc. If , then any predictability associated with is spurious. Let

(4)  

denote the entire range of possible values that and might take on.

To keep things simple, suppose that the realized values of , , and for each asset in a given market environment are drawn IID normal:

(5)  

, , and all have mean zero and unit variance. The noise term is uncorrelated with both and in every market environment, . However, and may be correlated across stocks, . Moreover, this correlation can differ across market environments. In other words, and may be highly correlated in one time period/country/asset class/etc but not in another.

Note that Equations (3) and (5) imply asset returns are zero on average, , in every market environment. I’m making this assumption to keep the math simple. If it really bothers you, just think about as an asset’s residual return that unexplained by other trading signals. Let’s also assume that in every market environment for the same reasons. This assumption implies that .

Two explanations

When you regressed the cross-section of returns on in your chosen market environment , you found that . Given the structure of investors’ model, we know that either predicts the cross-section of returns or predicts the cross-section of returns but not both:

(6)  

It might be that you estimated in market environment because in every environment. Or it might be that you estimated in market environment because happened to be correlated with an omitted variable in that environment, . These are the two possible explanations.

Since you’re focused on robust out-of-sample performance in other market environments , the reason why in-sample is very important. If merely because , then will only be a good cross-sectional predictor in other market environments where and are similarly correlated, . Under this explanation, it’s possible to imagine market environments where is an abysmal predictor. Just look for environments where .

Causal inference

What needs to be true about market environment if you want to be able to distinguish between these two explanations? The answer boils down to an identifying assumption about the range of values that might take on:

(7)  

A market environment, , consists of a set of structural parameters, ; a range of possible values for the correlation between and , ; and, a particular choice for this value, .

If market environments and have the same structural parameters, , then the cross-sectional slope coefficient will be the same in both environments, . Yet, you will interpret the slope coefficient differently in each environment if . By analogy, medical researchers will draw different conclusions about a drug’s efficacy from an RCT than from an observational study even if the joint distribution of patient outcomes and observable characteristics is the same in both datasets. If , then identifies as the correct explanation. There’s no way to have in such an environment. By contrast, if , then could be explained either by or by .

Note that it isn’t possible to choose a market environment where consists of an arbitrarily small neighborhood around zero. The omitted variable can explain no more than of the variation in returns across assets. That would occur if since we are assuming . Hence, if and due to a spurious correlation, then this correlation must be bounded away from zero:

(8)  

This digital zero/non-zero distinction is why it’s possible to map out causal effects using path diagrams. A path between two variables must be contemplated whenever they could have a non-zero correlation.

Out-of-sample environments

When you regressed the cross-section of returns on in market environment , you found . We can now give a precise definition for the set of all out-of-sample market environments that other researchers might try to replicate this finding in. Let

(9)  

denote the range of possible values for and that are consistent with your initial estimate given . If is guaranteed to be uncorrelated with the omitted variable, , then and we say that market environment identifies changes in as having a causal affect on the cross-section of returns.

Given the set of all and values that are consistent with your initial result , the range of potential out-of-sample market environments is defined as follows:

(10)  

This collection of market environments consists of any environment which could be generated by some and that’s consistent with your initial result combined with any possible value of .

Research strategy

If you chose a market environment that identified the causal effect of on the cross-section of returns for your initial test, then your estimate of would imply that in every out-of-sample market environment. The left-hand side of Equation (2) would reduce to:

(11)  

The finding would be robust out-of-sample, and you should publish it.

By contrast, if you chose a market environment that did not identify the causal effect of on the cross-section of returns, then your estimate of would be harder to interpret. It could be that or it could be that . If the latter is true, then we would say that reflects a spurious correlation. And to make this spurious correlation look as bad as possible out-of-sample, other researchers should look for a market environment where :

(12)  

Absent identification, you have to entertain this possibility. So you may refuse to publish strong results with good average-case out-of-sample performance for fear of being embarrassed by worst-case predictions.

Thus, as outlined at the beginning, even if all you care about as a researcher is publishing results that have robust out-of-sample performance, causal inference still turns out to be relevant. It’s a very a useful tool for achieving this goal. If investors are always using the same model to price assets, then understanding this model will allow you to always make good predictions. So you should consider adopting a research strategy whereby you insist on testing each new predictor in an identified market environment .

No free lunch

I recognize that identifying causal effects is hard. Running RCTs is hard. Finding valid instrumental variables is hard. It’s hard to find a market environment that identifies the causal effect of a change in on returns—i.e., to find a market environment where it’s reasonable to assume that .

So you might be thinking: “Can’t I just get around the problem by checking lots of different market environments before publishing? If in lots of different market environments , then shouldn’t I be more confident in ‘s out-of-sample performance? After all, in real life, no researcher would (or could!) publish a result about cross-sectional predictability based on one regression.”

It’s absolutely true that you do learn something about ‘s out-of-sample performance when you verify that in many different market environments . Unfortunately, the something that you learn only applies to ‘s average-case performance, . For example, if in more than half of all possible out-of-sample environments, then there’s no way for since we know that in every remaining market environment as we saw in Equation (12).

Yet, until you check every imaginable out-of-sample environment, you can say nothing new about the worst-case outcome. No matter how many environments you check in , you can never be certain that in one of the remaining environments in that you haven’t checked. Thus, if you care about never publishing a result that makes an embarrassingly bad out-of-sample prediction in some situation, , then simply doing lots of in-sample checks isn’t a viable research strategy on its own. It certainly doesn’t hurt. But it DOES NOT tell you anything about out-of-sample robustness in the setup thusfar.

The thing that makes causal inference difficult is that it requires making a strong assumption about the joint distribution of and an unobserved/unknown/omitted variable in a particular market environment . You have to assume that . Such identifying assumptions can be hard to stomach. However, the assumption that you would need to make for in-sample robustness to guarantee out-of-sample robustness is even less palatable. TANSTAAFL. Instead of requiring that in some specific market environment, you would need to assume that in all out-of-sample environments . Such an assumption is tantamount to simply assuming the result you’re after—namely, out-of-sample robustness.

An asset-pricing model is a claim about which optimization problem people are solving when they choose their investment portfolios. One way to make such a claim testable is to derive a condition that should hold if people were actually solving this optimization problem. And the standard approach to testing whether an asset-pricing model is correct involves using market data to estimate the key parameters in this condition, plugging in the results, and checking if it does hold.

For example, the consumption capital asset-pricing model (CCAPM) argues that people view the stock market as a way to insure consumption shocks. If this is correct, then the expected return on the market should be equal to the riskfree rate plus a multiple of the market’s consumption beta, :

(1)  

The standard test of the CCAPM involves computing average stock returns and the market’s consumption beta using historical data, plugging in these values to Equation (1), and checking whether it holds.

Of course, this isn’t the only way to test an asset-pricing model. Researchers can use surveys to learn about which optimization problem people are solving (see here, here, and here). They can also run lab experiments where participants trade with one another in a simulated market environment (see here, here, and here). These two alternative approaches have gained popularity in recent years (here, here, and here).

Unfortunately, many researchers view investor surveys and lab experiments as just another way of generating the data needed to test the moment conditions implied by an asset-pricing model. The prevailing view is that the data produced by these two empirical approaches is just a poor substitute for market data. Not so! These empirical approaches are not substitutes. They are each good for doing different things. Investor surveys and lab experiments can answer questions that market data can’t address.

What sorts of empirical questions are best suited to being answered via sample statistics computed using market data? Where do survey responses have a comparative advantage? When can we learn the most from a well-designed lab experiment? This post outlines how I think about the answers to these questions.

Canonical asset-pricing model

To make things concrete, I’m going to talk about testing a particular asset-pricing model—namely, the CCAPM. This model says that a representative investor solves the following optimization problem:

(2)  

Here’s a line-by-line breakdown of what all this means. (Line 1; Objective) The investor tries to maximize the present discounted value of his future utility from consumption given time preference parameter . (Line 2; Preferences) The investor has power utility with risk aversion parameter, . (Line 3; Budget Constraint) At time , the investor must have enough wealth to cover his consumption AND to pay for the shares of the market that he buys at a price of dollars per share AND to pay for the dollars he invests in riskless bonds. (Line 4; Wealth Evolution) The investor’s will receive a dollar dividend for each share of the market he bought at time AND he will be able to sell the share for dollars AND he will earn the riskfree rate of on each dollar he invested in bonds at time . (Line 5; Dividend Process) The total market dividend grows at an average rate of per period, and it has a per period volatility of .

The solution to this model reveals that:

(3)  

The price an investor is willing to pay at time for a share of the stock market (think: level of the S&P 500) is equal to his expectation of the total future payout to owning this share at time (total payout = the dividend plus the sale price ) discounted back at a rate that accounts for both his preference for getting paid earlier rather than later, , and his aversion to fluctuations in consumption, .

According to the CCAPM, the investor should be willing to pay more for the aggregate stock market if it provides insurance against drops in consumption, . And, if we define the return to holding a share of the market portfolio as , then standard manipulations of Equation (3) result in the moment condition in Equation (1). Thus, if investors are solving the optimization problem in Equation (2), then the expected return on the market, , should be equal to the riskfree rate plus a term proportional to the market’s consumption beta, .

Why we care if investors are using this model

Physicists use the principle of least action to model the path that an object will take through a field. Under this approach, it is as if the object chooses the route that minimizes the total difference between its kinetic and potential energies through time. What will the arc of a baseball look like as it travels from the center fielder’s hand to the catcher’s mitt? We know that a baseball cannot actually CHOOSE anything. Yet the predictions made by acting as if it can describe the ball’s trajectory so well that physicists find it productive to pretend otherwise.

You can imagine a world where the same reasoning applies to the CCAPM. It could be that the CCAPM makes predictions that fit the data so well that it’s useful to pretend that investors solve the optimization problem in Equation (2) regardless of whether or not they actually do. We do not live in such a world. The central empirical prediction of the CCAPM (Equation 1) is not true on average. And, even if it were, most of the variation in expected returns over time and across assets cannot be explained by differences in consumption betas. Consumption growth is nowhere near volatile enough.

In general, even when an asset-pricing model’s predictions are correct on average, the fit is poor. Financial markets are complex. Asset-pricing models are simple caricatures of the forces at work. The goal is to write down a model that captures some small kernel of truth about these forces. An asset-pricing model is only useful to the extent that we can count on its noisy predictions holding up in novel as-yet-unseen time periods, countries, asset classes, etc. What would asset prices look like if changed? Researchers write down asset-pricing models so that we can analyze such counterfactual settings. This is the point of a model.

Researchers care about whether investors are actually solving the optimization problem associated with a given asset-pricing model because this is what gives us confidence that we can count on the model’s noisy predictions in those settings. If investors are usually solving the optimization problem in Equation (2), then we can trust the model’s predictions to be roughly correct in a new market setting where the input parameters had changed. Why? Investors are usually solving that problem. A researcher who knows which optimization problem investors are solving can make robust out-of-sample predictions.

With this background in place, I can now describe the sort of question that’s best suited to be answered by each kind of empirical approach: market data, investor surveys, and lab experiments.

Are the model’s empirical predictions correct?

An asset-pricing model is a claim about which optimization problem investors are solving. The resulting first-order conditions imply that certain moment conditions should hold in the data. If your goal is to test whether these predictions are true in a given market setting, then start by looking at market data. For instance, if you want to test whether the expected return on the market is equal to the riskfree rate plus a multiple of the market’s consumption beta as predicted by the CCAPM in Equation (1), then it makes sense to estimate and using historical data on market returns and aggregate consumption.

Of course, the in Equation (1) represents INVESTORS’ beliefs about what future returns will look like on average. Likewise, the represents INVESTORS’ beliefs about what the covariance between market returns and consumption growth will be. So you should also be able to ask investors questions about these two statistical objects. And the results should also satisfy Equation (1).

This is the usual way that researchers use investor surveys to test asset-pricing models (see here, here, and here). They ask investors about the key parameters in some moment condition. Then, they plug the results into this condition and check whether it holds. This is a perfectly fine thing to do. But it’s not an application where surveys have a comparative advantage over other empirical approaches.

Are they correct for the right reasons?

An asset-pricing model’s predictions can be correct on average for reasons that have nothing to do with the ones in the model. We know that differences in consumption betas do not explain differences in expected returns as predicted by the CCAPM. But, even if they did, it would not imply investors were actually solving the optimization problem in Equation (2) when choosing investment portfolios. Any observed correlation between expected returns and consumption betas could be a spurious correlation.

As I point out in my recent JF paper with Sam Hartzmark and Abby Sussman, investor surveys give researchers a way to investigate why a model’s predictions do or don’t hold in the data. Surveys give investors an opportunity to show their work—to describe how they arrived at the outcomes we observe in the data. This is something that you can’t examine using market data alone. This is where investor surveys have a comparative advantage over other empirical approaches (see this post for more details).

At this point, you might be thinking: “But prices can move for reasons that no individual investor can understand.” I totally agree. But this is beside the point when testing an asset-pricing model. Asset prices can move for lots of reasons. An asset-pricing model makes a claim about what one of those reasons is. The fact that there are also other reasons is neither here nor there.

We expect the set of moment conditions implied by an asset-pricing model to hold in the data because investors are maximizing the objective function in the model. So there should be some evidence that investors are actually trying to do this. And surveys are the best way to gather such evidence. Every investor doesn’t need to be thinking exactly like the agents in a model. But it’s problem if no investor thinks that way.

For what parameters does the model apply?

This brings us to the last empirical approach: lab experiments. The main concern with this sort of approach is that the experimental setup in a lab might be missing something important about how real-world markets operate, and it’s hard to know what that missing something is. The only evidence might be that your results from the lab don’t line up with what you observe in the real world.

This is a feature not a bug.

Suppose you think that some optimization problem like the one in Equation (2) captures the essence of some important market phenomenon. The resulting moment conditions are satisfied when you plug in parameter estimates using market data. And, when you ask traders about how they are trading, you get responses that are consistent with the logic of your model. But, when you set up a trading game where your model should govern participant behavior, you find no evidence of the phenomenon you’re interested in. You’ve just learned that you’re missing something. At the very least, you need tweak the input parameters.

When you run a trading game, do you get results that are consistent with the model for sensible input parameters? Do you tend to get similar results when you run lab experiments with the same input parameters? When you adjust these input parameters, at what point do the predictions of the asset-pricing model start to diverge from the results of the experiments? These are the right sorts of questions to be asking via lab experiments. A lab experiment SHOULD NOT be the first test of an asset-pricing model. In epidemiology, “Check whether a healthy person gets sick if you inject him with the cultured microorganism.” is the third of Koch’s postulates not the first.

If you show that some outcome can be generated by a trading game, you have no idea whether the conditions responsible for the outcome occur in the wild. Lab experiments are useful when working in the other direction. Suppose you think you fully understand the conditions that lead to a certain outcome in real-world financial markets. Ok, then you should be able to reproduce this outcome by simulating those conditions in the lab. Once you do that, you can also use the experiment to see how far can you adjust the input parameters and still get valid predictions. In addition to verifying your understanding, lab experiments also offer a way to assess the effective range of an asset-pricing model.

Asset-pricing research consistently produces many different explanations for the same empirical facts. As a rule of thumb, you should expect asset-pricing researchers to wildly overachieve. Behavioral researchers can typically point to several psychological biases which might explain the same anomaly. e.g., it is possible to argue that the excess trading puzzle is due to a preference for gambling, the disposition effect, and social interactions among other things. In cross-sectional asset-pricing, there is an entire zoo of different explanations for the size and value effects, most of which “have little in common economically with each other.” In a 2017 review article, John Cochrane lists ten different explanations for the equity premium puzzle.

Unfortunately, the existence of so many different explanations for the same few facts is a real problem for the field. With so many to choose from, which explanation should a researcher use when evaluating counterfactuals or doing policy analysis? While each explanation is consistent with observed market data, different explanations will generally make different predictions out-of-sample in novel market environments. e.g., even if firm size and liquidity were perfectly correlated in all past data, researchers would still care which was the correct explanation for the size effect because they’d want to know what to expect if they ever encountered a bunch of large illiquid stocks.

So what is it about the asset-pricing research process that consistently produces multiple explanations for the same set of facts? That’s the topic of this post.

The Research Process

To answer this question, I need a working model of what it means to do asset-pricing research. What does the process entail? What counts as a new empirical fact? When asset-pricing researchers encounter one, how do they go about constructing a model to explain it? What primitives do they start with? What’s the general form of the model they eventually arrive at? How do different models differ from one another?

Asset-pricing models study the behavior of investors who can take actions today and want things tomorrow. I will use to denote an action that investors can take today. e.g., think about allocating a portfolio, deciding how much to consume, or making a capital improvement. I will use to denote a future outcome that investors want. e.g., think about consumption, wealth, or eternal happiness in the afterlife.

An asset-pricing model is a constrained optimization problem of the form

(1)  

The operator captures the idea that investors strategically choose which actions to take today, . They realize that their choice of actions today, , affects how much of the thing they want they get to enjoy tomorrow, . The function denotes the utility that investors get from having a given amount of tomorrow. The expectation operator represents their conditional beliefs about the likelihood of tomorrow given their choice of actions today with denoting their subjective probability distribution function. This is the distribution that investors have in their heads, and because it is a subjective distribution, it might not be objectively correct. Investors also realize that they face various constraints, which I write as the requirement . Not all actions are possible today.

All asset-pricing models agree on this basic setup. Different asset-pricing models just make different claims about the functional forms of investor preferences, ; beliefs, ; and, constraints, . Models based on habit formation, recursive utility, hyperbolic discounting, and loss aversion all monkey around with investor preferences. When a model says that investors have rational expectations, that they extrapolate, or that they seek sparsity, it is making a choice about the functional form of investors’ beliefs. The limits-to-arbitrage literature gives a taxonomy of investor constraints (short sale, margin, etc).

Researchers test whether an asset-pricing model’s specific choices for , , and fit the data by examining the model’s first-order conditions (FOCs). e.g., the stochastic discount factor (SDF) implied by an asset-pricing model comes from the FOC with respect to consumption. See Chapter 1 verse 1 of the Book of John. The function is an upside down parabola, which has a maximum value of . You could double check that maximizes by verifying that . Likewise, you can double check whether investors’ actions are optimal by verifying that

(2)  

If an asset-pricing model’s FOCs aren’t satisfied in the data, then the model’s choice of functional forms must be missing something about investor behavior. The new parameter in the equation above is a Lagrange multiplier, which captures how much a particular constraint distorts investors’ optimal choice of actions.

Here’s the asset-pricing research process in action. At time , the literature contains a bunch of models and data on various empirical settings (different countries, time periods, asset classes, etc). Each asset-pricing model makes different assumptions about , , and . But they all take the general form outlined in Equation (1). Asset-pricing researchers check whether the FOCs implied by each existing model hold in the various empirical settings observed at time . If no model adequately explains market data in some important empirical setting, then we call it a new empirical fact. Researchers then work backwards from the non-zero values of the FOCs in Equation (2) to try and guess the correct functional forms for , , and . If successful, we add a new asset-pricing model to the literature and the process repeats at time , perhaps with data on a few more empirical settings.

Ill-Posed Inverse Problem

The whole goal of the asset-pricing research process is to guess the correct functional forms for , , and to plug into Equation (1). Researchers do this by working backwards from violations of known models’ FOCs in Equation (2). If researchers guess correctly, they will find that the resulting model’s FOCs fit the observed market data in all empirical settings. But this excellent empirical fit is just a signature verifying that the model is correct. The point of guessing the true asset-pricing model is not to maximize empirical fit. With enough free parameters, any model can fit the data arbitrarily well.

Researchers care about guessing the true model (i.e., the optimization problem that investors are actually trying to solve) because the true model will allow them to predict how investors will behave in novel market environments that they haven’t encountered before. Put differently, knowing the correct functional forms for , , and allows researchers to evaluate counterfactual scenarios and do policy analysis. Knowing the correct asset-pricing model allows you to answer questions like, ‘What should the expected return of a large illiquid stock be?’, even if you have never encountered such a stock in the past.

This is an inverse problem. Asset-pricing research doesn’t involve finding the maximum of some known widely-agreed-upon optimization program. It involves staring at a bunch of data that is assumed to maximize some optimization problem and trying to figure out the details of that unknown problem. Put another way, asset-pricing researchers aren’t in the business of finding the maximum of a known function, . They observe and assume this data is the maximum of some unknown function, . Then they try to guess the details of this unknown function.

The study of inverse optimization problems represents an entire mathematical field of inquiry. But there’s one detail in particular that’s relevant here: inverse problems are typically ill-posed. Put simply, many different functions can share the same derivative values. Functions with quite different global behavior can have the same slope locally. e.g., , , and all achieve a maximum value of at . These curves are all roughly the same when . Yet, they have very different behavior globally: , , and .

Asset-pricing researchers are solving an inverse problem. They are trying to reverse engineer an entire optimization problem by studying finite data about its first-order conditions. And inverse problems are typically ill-posed. This is why they consistently produce multiple explanations for the same facts. We should expect many different optimization problems to produce equivalent FOCs in the observed market data. Moreover, we should expect this to be true even though the optimization problems, which look equivalent locally, will generally display quite different global behavior in novel as-yet-unseen market environments. And these global differences are what matter when evaluating counterfactuals and doing policy analysis.

A Potential Solution

If asset-pricing researchers generate multiple explanations for the same fact because they’re solving an ill-posed inverse problem, then how might we make this problem well-posed? The mathematical literature on inverse optimization problems makes one suggestion: severely limit the class of functions you are willing to consider. If you see derivative data in the neighborhood of that was generated by either , , or , you must find some grounds for ruling out two of the three options. e.g., if you were only willing to consider polynomial solutions, then and would be off limits. Your only remaining choice would be , making the inverse problem well-posed.

Unfortunately, this isn’t a particularly promising route for asset-pricing researchers. There aren’t good economic grounds for ruling out functional forms for , , and that yield similar FOCs in the observed data. In fact, some of the most cited papers in asset pricing involve dreaming new and exotic functional forms for these objects. e.g., think about models involving recursive preferences, which argue that investors’ utility takes the functional form:

(3)  

There’s simply no way this heifer would have been accepted into the asset-pricing cannon if researchers had decided to severely restrict the kinds of utility functions they were willing to consider sometime back in the early 1980s. If you think the discovery of recursive utility was progress, then that’s a problem.

Luckily, this isn’t the only way forward. Asset-pricing models do more than just make predictions about which FOCs should hold in the observed market data. They also say why these FOCs should be satisfied: because investors are optimizing with a specific tradeoff in mind. There is economic content in the operator as well as in the FOCs this operator produces. An asset-pricing model doesn’t say that a set of FOCs should just happen to hold in the data. It says that these FOCs should hold because investors are optimizing the tradeoff embodied by the model’s choice of , , and . So in most cases it should be possible to ask investors whether they are thinking about this tradeoff.

This represents a practical step we could take towards converting the asset-pricing research process into a well-posed inverse problem. It is a straightforward way of ruling out lots of spurious solutions. e.g., Cross-sectional differences in expected returns can’t be explained by investors demanding a risk premium for holding assets with exposure to X if real-world investors show no desire to hedge their exposure to X when given the chance. Likewise, if the excess trading puzzle is due to a widespread preference for gambling among retail investors, then it should be easy to find retail investors who express a preference for gambling. Yes, it is possible for markets to move in ways that no individual investor understands. But asset-pricing models don’t explain those kinds of market fluctuations. Asset-pricing models make predictions about how investors strategically respond to market fluctuations they do understand, so it should be possible to ask them about these fluctuations and the logic behind their responses.

Milton Friedman laid out his methodological approach to doing economics in his 1953 essay, The Methodology of Positive Economics. This essay gives his answer to the question: What constitutes a good economic model? Or, put differently, how would you recognize a good economic model if you saw one?

According to Friedman, “the only relevant test of the validity of a hypothesis is the comparison of its predictions with experience. The hypothesis is rejected if its predictions are contradicted; it is accepted if its predictions are not contradicted.” All that matters is whether or not a model fits the data. Assumptions? Priors? Intuition? All that stuff is just moonshine. Empirical fit reigns supreme. This is an extreme view!

For example, in Friedman’s eyes, a good model of how leaves are distributed about the canopy of an oak tree is a model in which each leaf optimally chooses its position and orientation relative to its neighbors. Yes, we know that leaves don’t have brains. They can’t actually make decisions like this. But it is ‘as if’ they could. So a model in which each leaf strategically chooses where to grow is a good model of leaf placement.

A good model of how an expert billiards player makes difficult shots would be a model in which “he knew the complicated mathematical formulas that would give the optimum directions of travel, could estimate accurately by eye the angles, could make lightning calculations from the formulas, and could then make the balls travel in the direction indicated by the formulas.” So what if the player can’t do these things? We know he regularly makes difficult shots, so it’s ‘as if’ he can. Friedman tells us to just model him like that anyway.

In Friedman’s view, “a theory cannot be tested by comparing its ‘assumptions’ directly with ‘reality.’ Indeed, there is no meaningful way in which this can be done.” In fact, Friedman argues that insisting on reasonable assumptions can be misleading. “The more significant the theory, the more unrealistic the assumptions.”

Every economist knows about Friedman’s ‘as if’ approach to model evaluation. If asked, most economists will say that Friedman’s methodological approach is, if not correct, then at least reasonable. They will argue that it’s at least important to consider ‘as if’ justifications when evaluating a model.

But here’s the thing: no working economist actually evaluates models this way! Aside from one glaring exception, no economist actually thinks ‘as if’ models are helpful. Ask yourself: Is the factor zoo a problem for asset pricing? Yes. But what is a spurious factor? It’s a factor that fits the data for wrong reasons. It is ‘as if’ investors were using it to price assets even though they aren’t. And that’s precisely the problem!

The idea that we can’t test (or shouldn’t even bother testing) the assumptions behind our economic models is simply preposterous. It’s a claim that Steven Pinker would call a “conventional absurdity: a statement that goes against all common sense but that everyone believes because they dimly recall having heard it somewhere and because it is so pregnant with implications.” No economist does research this way!

Why not replace all economic models with uninterpretable machine-learning (ML) algorithms? ML algorithms can fit the data well precisely because they contain no economic assumptions. But TANSTAAFL! It is precisely the economic assumptions about what agents are trying to do that give us confidence a model’s predictions will hold up when conditions change. In other words, these assumptions are what allow economists to use the model for counterfactual analysis—i.e., to make predictions in new and as-yet-unseen environments. The right assumptions embedded in a good economic model are responsible for its robust predictions. If you’re going to ignore all such economic restrictions, then there’s no point in writing down an economic model in the first place. There are better ways to do pure prediction.

I’m by no means the first person to highlight these issues. They long predate the factor zoo and the popularity of ML algorithms. If I had to pick one person to judge the quality of an economic model, that person would be Paul Samuelson. And Samuelson strongly disagreed with Friedman’s ‘as if’ approach. Samuelson clearly recognized the importance of evaluating your assumptions, disparagingly referring to Friedman’s ‘as if’ methodology as the “F-Twist” in a 1963 discussion paper.

Moreover, in almost every context, economists approach research in a manner more consist with Samuelson than with Friedman. They firmly believe it’s important to verify one’s assumptions. This is why we see papers with titles like Do Measures of Financial Constraints Measure Financial Constraints? getting hundreds of cites a year. This influential paper is entirely concerned with testing our working assumptions.

As far as I can tell, there is only one context in which economists actually use ‘as if’ reasoning to constrain the research process—namely, when interpreting survey data. Standard asset-pricing models assume investors are solving an optimization problem that looks something like

(1)  

Economists regularly test assumptions about investor preferences , the law of motion for wealth, , and various other kinds of economic constraints, . However, for some reason, it’s entirely taboo to ask investors whether they are actually trying to this problem in the first place.

Friedman directly calls out survey data in his 1953 essay, writing that “questionnaire studies of businessmen’s or others’ motives or beliefs about the forces affecting their behavior… seem to me almost entirely useless as a means of testing the validity of economic hypotheses.” However, he offers no concrete reasons why economists should think about the “maximize” part of investors’ optimization problem any differently than the “subject to” part. Both are assumptions. In Friedman’s eyes, both are untestable.

Yes, survey data can be misleading. Above I describe a situation where surveying economists about their views on ‘as if’ reasoning would yield specious evidence. But all data can be misleading. It’s not like NOT using survey data has resolved the factor zoo. Sometimes investors give uninformative answers which might lead researchers down the wrong path. But this doesn’t mean that we can’t learn anything concrete about how investors price assets from a well-constructed survey. Not every regression result is informative. Some regression estimates can even be misleading. None of this implies that regression analysis is worthless.

Friedman’s 1953 essay outlines a bad approach to model evaluation. There’s more to a good model than . Paul Samuelson knew this to be true. And, except when they’re looking at survey data, every other economist knows it to be true as well. There’s no reason for us to continue applying ‘as if’ reasoning only in this particular context. It’s just not a valid argument for dismissing survey evidence about a model.