Motivation

Trading has gotten a lot faster over the last two decades. The term “short-run trader” used to refer to people who traded multiple times a day. Now, it refers to algorithms that trade multiple times a second.

Some people are worried about this new breed of short-run trader making stock prices less informative about firm fundamentals by trading so often on information that’s unrelated to companies’ long-term prospects. But, this is a red herring. By the same logic, market-neutral strategies should be making market indexes like the Russell 3000 less informative about macro fundamentals. And, no one believes this.

Short-run traders aren’t ignoring fundamentals; they’re learning about fundamentals before everyone else by studying order flow. And, this post shows why they also make trading volume look asymmetric and lumpy as a result. The logic is simple. If short-run traders get additional information from order flow, then they’ll use this information to cluster their trading at times when everyone else is moving in the opposite direction.

Benchmark Model

Consider a market with a single company that’s going to pay a dividend, , in future periods And, suppose that there’s a unit mass of small agents, , who have noisy priors about these dividends,

   

which are correct on average, . This assumption means that, by aggregating agents’ demand, equilibrium prices can contain information about dividends that isn’t known by any individual agent.

This unit mass of agents is split into two different groups: long-term investors and short-run traders. At time , each group of agents trades shares in its own separate fund, , that offers frequency-specific exposure to the company’s dividends at times . The long-term investors, , trade the low-frequency fund which has a payout:

   

And, the short-run traders, , trade the high-frequency fund which has a payout:

   

At time , each agent observes the equilibrium price of a frequency-specific fund, , and then chooses the number of shares to buy, , in order to maximize his expected utility at the end of time :

(1)  

Above, denotes agent ‘s conditional expectation, and denotes his risk-aversion parameter.

Let denote the number of shares of the company’s stock that are available for purchase at time . We say that “markets clear” at time if the dividend payout from each available share at times has been unambiguously assigned to exactly one trader via their fund holdings:

(2)  

Let and denote the number of available shares at each frequency.

An equilibrium then consists of a demand rule, , and a price function, , such that 1) demand maximizes the expected utility of each agent given the price and 2) markets clear.

Because the equilibrium price of each fund only depends on its promised payout and the number of available shares, if agents knew the number of available shares, then they could reverse engineer a fund’s future payout at times by studying its equilibrium price at time . And, an equilibrium in such a market wouldn’t be well-defined. So, to make sure that equilibrium prices at time aren’t fully revealing, let’s assume that the number of available shares in each period is a random variable:

   

This means thinking about “available” shares as shares that haven’t already been purchased by noise traders.

The key fact about this benchmark model is that agents don’t use order-flow information at time to update their time beliefs. As a result, the model really isn’t about short-run traders in spite of how the variables are named. Crouzet/Dew-Becker/Nathanson shows that with some clever relabeling you could just as easily think about the low- and high-frequency funds as index and market-neutral funds, respectively.

Trading Volume

Although agents are only active at time in the benchmark model, each fund has to trade at times in order to deliver frequency-specific payouts. Let’s use and to denote aggregate demand:

   

To deliver to every one of its shareholders, the low-frequency fund has to buy shares of the company’s stock between times and and then liquidate this position between and . And, to deliver to every one of its shareholders, the high-frequency fund has to buy shares between times and , sell shares between and , and then buy back shares between times and .

So, trading volume in the benchmark model is:

   

The key thing to notice is that trading volume is symmetric, , because short-run traders don’t get any new information after time .

Model Solution

So, how many shares of each frequency-specific fund are agents going to demand in the benchmark model? To solve the model and answer this question, let’s first guess that the price function is linear:

   

This guess introduces a new parameter, , which represents the signal-to-noise ratio of fund prices at time . If this parameter is large, then the time prices of the low- and high-frequency funds will reveal a lot of the information about the company’s time dividend payouts.

Here’s the upshot of this guess. It implies that each fund’s price is a normally-distributed signal about its future payout, . And, with normally-distributed signals, we know how to compute agents’ posterior beliefs about after seeing :

   

Above, denotes agent ‘s priors about a particular fund, either or . And, we can then use these posterior beliefs to compute agents’ equilibrium demand rule by solving the first-order condition of Equation (1) with respect to :

(3)  

Finally, to verify that our original guess about a linear price function was correct, we can plug this equilibrium demand rule into the market-clearing conditions in Equation (2) and solve for :

   

The resulting price function is indeed linear, so our solution is internally consistent (…though not unique). And, by matching coefficients, we can solve for the equilibrium signal-to-noise ratio, . In other words, fund prices at time reveal more information about a company’s dividend at times when agents are less risk averse ( small) or when they have more precise priors ( small).

Order-Flow Info

How would this solution have to change if short-run traders could learn from time order flow?

For markets to clear at time , the aggregate demand for the low-frequency fund plus the aggregate demand for the high-frequency fund has to equal the total number of available shares, . And, from Equation (3), we know that the aggregate demand for the low-frequency fund is related to the company’s total dividend payout:

   

So, by looking at the time order flow, short-run traders can get a signal about the company’s time dividend since :

   

And, this information about the time dividend is helpful since .

With this additional signal, the short-run traders who previously invested in the high-frequency fund would now rather trade the company’s stock directly at times . Let denote their demand at time . When they observe a high price for the low-frequency asset at time , , and a large dividend payout at time , , they know that is likely small. And, as a result, they’ll short more shares at time , . By contrast, when they observe a high price for the low-frequency asset at time , , and a small dividend payout at time , , they know that is likely large. So, they’ll short fewer shares at time , .

Either way, trading volume is going to look asymmetric and lumpy as a result, with relatively more of the trading volume clustered at one of the end points. If and , then relatively more of the trading volume will occur at between time and because is unchanged and:

   

Whereas, if and , then relatively more of the trading volume will occur between time and because is unchanged and now .

What’s more, to long-term investors who can’t see short-run order flow, short-run traders are going to add execution risk. The price at which the low-frequency fund executes its time orders will now vary. And, this variation will be related to the magnitude (but not the sign) of their time demand.

Finally, note that this analysis shows why it’s easier to model indexers and stock pickers than long-term investors and short-run traders. An equilibrium in either model has to contain a demand rule and a price function (e.g., see the setup in the benchmark model). But, an equilibrium in a model with multi-frequency trade also has to contain a rule for how long-term investors think short-run traders will affect their order execution. And, this rule is the crux of any model with long-term investors and short-run traders.

1. Motivation

Imagine we’re traders in a market where the cross-section of returns is related to variables:

   

In the equation above, denotes the mean return, and each captures the relationship between returns and the th right-hand-side variable. Some notation: I’m going to be using a “star” to denote true parameter values and a “hat” to denote in-sample estimates of these values. e.g., denotes an in-sample estimate of . To make things simple, let’s assume that , , , and .

Notice that learning about the most likely parameter values,

   

is really easy in this setting because . These maximum-likelihood estimates are just the coefficients from a cross-sectional regression,

   

So, finding is homework question from Econometrics 101. Nothing could be simpler.

But, what if we’re interested in predicting returns,

   

rather than learning about the most likely parameter values? It might seem like this is the same exact problem. And, if we’re only considering right-hand-side variable, then it is the same exact problem. When the best predictions come from using . But, it turns out that when there’s or more right-hand-side variables (and an unknown mean), this is no longer true. When we can make better predictions with less likely parameters. When there’s a tension between the learning and predicting.

Why? That’s the topic of today’s post.

2. Maximum Likelihood

Finding the most likely (ML) parameter values is equivalent to minimizing the negative log likelihood. So, because we’re assume that , this is just

   

where the “” at the end denotes a bunch of terms that don’t include any of the parameters that we’re optimization over. Optimizing each parameter value then gives the first-order conditions below:

   

And, solving this system of equations and unknowns gives the most likely parameter values:

   

Clearly, the most likely parameter values are just the coefficients from a cross-sectional regression.

Now, for the sake of argument, let’s imagine there’s an oracle who knows the true parameter values, . With access to this oracle, we can compute our mean squared error when using the maximum-likelihood estimates to predict returns given any choice of true parameter values:

   

The first term, , captures the unavoidable error. i.e., even if we knew the true parameter values, we still wouldn’t be able to predict . And, the second and third terms, and , capture the error that comes from using the most likely parameter values rather than the true parameter values.

3. A Better Predictor

With this benchmark in mind, let’s take a look at a variant of the James-Stein (JS) estimator:

   

In the definition above, denotes a bias factor that shrinks the maximum-likelihood estimates towards zero whenever . So, with access to an oracle, we can compute our mean squared error when using the James-Stein estimates to predict returns just like before:

   

Now the third term is more complicated. If we use a more biased estimator, and , then using the most likely parameter values rather than the true parameter values to predict returns is going to cause less damage. But, bias is going to generate really bad predictions whenever the true parameter value is large . The and terms capture these two opposing forces.

Comparing the maximum-likelihood and James-Stein prediction errors reveals that we should prefer the James-Stein estimator to the maximum-likelihood estimator if there’s a such that:

   

But, here’s the thing: if we have access to an oracle, then there’s always going to be some that satisfies this inequality. This is easier to see if we rearrange things a bit:

   

So, there’s always a sufficiently small such that this inequality holds. Thus, for any , there’s some such that the James-Stein estimates gives better predictions than the most likely parameter values.

4. Abandoning the Oracle

Perhaps this isn’t a useful comparison? In the real world, we can’t see the true parameter values when deciding which estimator to use. So, we can’t know ahead of time whether or not we’ve picked a small enough . It turns out that having to estimate changes things, but only when there’s just right-hand-side variable. When there are variables, James-Stein with an estimated still gives better predictions.

To see where this distinction comes from, let’s first solve for the optimal choice of when we still have access to the oracle. This will tell us what we have to estimate when we abandon the oracle. The optimal choice of solves:

   

So, if we differentiate, then we can solve the first-order condition for :

   

If the maximum-likelihood estimates are really noisy relative to the size of the true parameter values (i.e., is close to ), then using the most likely parameter values rather than the true parameter values is going to increase our prediction error a lot. So, we should use more biased parameter estimates.

But, notice what this formula is telling us. To estimate the right amount of bias, all we have to do is estimate the variance of the true parameter values. We don’t have to estimate every single one. So, we can estimate the variance of the true parameter values as follows,

   

where the factor of on the left-hand side is a degrees-of-freedom correction. To see why we need this correction, recall that

   

so not all maximum-likelihood estimates can move independently. There is an adding-up constraint.

Thus, if we have to estimate the right amount of bias to use, then we should choose:

   

Notice that, when we have to estimate the right amount of bias, only if . If , then the denominator is infinite and . After all, if there’s only right-hand-side variable, then the equation to estimate the variance of the true parameter values has the same first-order condition as the equation to estimate . With this estimated amount of bias, our prediction error becomes

   

which is always less than the maximum-likelihood prediction error whenever .

5. What This Means

My last post looked at one reason why it’s harder to predict the cross-section of returns when : Bayesian variable selection doesn’t scale. If we’re not sure which subset of variables actually predict returns, then finding the subset of variables that’s the most likely to predict returns means solving a non-convex optimization problem. It turns out that solving this optimization problem means doing an exhaustive search over the powerset containing all possible subsets of variables. And, this just isn’t feasible when .

But, this scaling problem isn’t the only reason why it’s harder to predict the cross-section of returns when . And, this post points out another one of these reasons: even if you could solve this non-convex optimization problem and find the most likely parameter values, these parameter values wouldn’t give the best predictions. When , there’s a fundamental tension between making good predictions and learning about the most likely parameter values in the data-generating process for returns. So, when traders are going to solve the prediction problem and live with the resulting biased beliefs about the underlying parameter values. What’s more, the with the best out-of-sample fit in the data is going to quantify how much the desire to make good predictions distorts traders’ beliefs.

1. Motivation

Traders are constantly looking for variables that predict returns. If is the only candidate variable traders are considering, then it’s easy to use the Bayesian information criterion to check whether predicts returns. Previously, I showed that using the univariate version of the Bayesian information criterion means solving

(●)  

after standardizing things so that and . If the solution is some , then predicts returns. Notation: Parameters with hats are in-sample estimates. e.g., if , then .

But, what if there’s more than variable? There’s an obvious multivariate extension of (●):

(⣿)  

So, you might guess that it’d be easy to check which subset of variables predicts returns by evaluating (⣿). But, it’s not. To evaluate the multivariate version of the Bayesian information criterion, traders would have to check different parameter values. That’s a combinatorial nightmare when . Thus, traders can’t take a strictly Bayesian approach to variable selection when there are lots of variables to choose from.

Why is evaluating (⣿) so hard? That’s the topic of today’s post.

2. Non-Convex Problem

Let’s start by looking at what makes (●) so easy. The key insight is that you face the same-sized penalty no matter what you choose when solving the univariate version of the Bayesian information criterion:

   

So, if you’re going set , then you might as well choose the value that minimizes your prediction error:

   

Thus, to evaluate (●), all you have to do is check parameter values, and , and see which one gives a smaller result. Practically speaking, this means running an OLS regression, , and checking whether or not the penalized residual variance, , is smaller than the raw return variance, .

Most explanations for why (⣿) is hard to evaluate focus on the fact that (⣿) is a non-convex optimization problem (e.g., see here and here). But, the univariate version of the Bayesian information criterion is also a non-convex optimization problem. Just look at the region around in the figure to the left, which shows the objective function from (●). So, non-convexity can only part of the explanation for why (⣿) is hard to evaluate. Increasing the number of variables must be add a missing ingredient.

3. The Missing Ingredient

DATA + CODE

If there are many variables to consider, then these variables can be correlated. Correlation. This is the missing ingredient that makes evaluating (⣿) hard. Let’s look at a numerical example to see why.

Suppose there are only stocks and variables. The diagram above summarizes the correlation structure between these variables and returns. The red bar is the total variation in returns. The blue bars represent the portion of this variation that’s related to each of the variables. If you can draw a vertical line through a pair of bars (i.e., the bars overlap), then the associated variables are correlated. So, because the first blue bars don’t overlap, and are perfectly uncorrelated in-sample:

   

Whereas, because the first blue bars both overlap the third, and are both correlated with :

   

Finally, longer overlaps denote larger correlations. So, is the single best predictor of returns since the third blue bar has the longest overlap with the top red bar:

   

And, this creates a problem. If you had to pick only variable to predict returns, then you’d pick :

   

But, is actually the subset of variables that minimizes (⣿) in this example:

   

In other words, the variable that best predicts returns on its own isn’t part of the subset of variables that collectively best predict returns. In other examples it might be, but there’s no quick way to figure out which kind of example we’re in because (⣿) is a non-convex optimization problem. Until you actually plug into (⣿), there’s absolutely no reason to suspect that either variable belongs to subset that minimizes (⣿). Think about it. and are both worse choices than on their own. And, if you start with and add either variable, things get even worse:

   

With many correlated variables, you can never tell how close you are to the subset of variables that best predicts returns. To evaluate (⣿), you’ve got to check all possible combinations. There are no shortcuts.

4. Birthday-Paradox Math

If correlation makes it hard to evaluate (⣿), then shouldn’t we be able to fix the problem by only considering independent variables? Yes… but only in a fairytale world where there are an infinite number of stocks, . The problem is unavoidable in the real world where there are almost as many candidate variables as there are stocks because independent variables are still going to be correlated in finite samples.

Suppose there are independent variables that might predict returns. Although these variables are independent, they won’t be perfectly uncorrelated in finite samples. So, let’s characterize the maximum in-sample correlation between any pair of variables. After standardizing each variable so that and , the in-sample correlation between and when is roughly:

   

Although , our estimates won’t be exactly zero in finite samples.

CODE

Since the normal distribution is symmetric, the probability that and have an in-sample correlation more extreme than is:

   

So, since there are pairs of variables, we know that the probability that no pair has a correlation more extreme than is:

   

Here’s the punchline. Because shows up as an exponent in the equation above, the probability that all pairwise in-sample correlations happen to be really small is going to shrink exponentially fast as traders consider more and more variables. This means that finite-sample effects are always going to make evaluating (⣿) computationally intractable in real-world settings with many variables, even when the variables are truly uncorrelated as . e.g., the in-sample correlation of from the previous example might seem like an unreasonably high number for independent random variables, something that only happens when . But, the figure above shows that even when there are stocks, there’s still a chance of observing an in-sample correlation of at least when considering candidate variables.

5. What This Means

Market efficiency has been an “organizing principle for 30 years of empirical work” in academic finance. The principle is based on on a negative feedback loop: predictable returns suggest profitable trading strategies, but implementing these strategies eliminates the initial predictability. So, if there are enough really smart traders, then they’re going to find the subset of variables that predicts returns and eliminate this predictability. That’s market efficiency.

Even for really smart traders, finding the subset of variables that predicts returns is hard. And, this problem gets harder when there are more candidate variables to choose from. But, while researchers have thought about this problem in the past, they’ve primarily focused on the dangers of p-hacking (e.g., see here and here). If traders regress returns on different variables, then they should expect that of these regressions is going to produce a statistically significant coefficient with a even if none of the variables actually predicts returns. So, researchers have focused on correcting p-values.

But, this misses the point. Searching for the subset of variables that predicts returns isn’t hard because you have to adjust your p-values. It’s hard because it requires a brute-force search through the powerset of all possible subsets of predictors. It’s hard because any optimization problem with a hard-thresholding rule like (⣿) can be re-written as an integer programming problem,

   

which means that it’s NP-hard (e.g., see here, here, or here). It’s hard because, in the diagram above finding the subset of variables that predicts returns is equivalent to finding the cheapest way to cover the red bar with blue bars at a cost of per blue bar, which means solving the knapsack problem.

<

p style=”text-indent: 15px;”>So, the fact that Bayesian variable selection doesn’t scale is a really central problem. It means that, even if there are lots and lots of really smart traders, they may not be able to find the best subset of variables for predicting returns. You’re probably on a secure wifi network right now. This network is considered “secure” because cracking its -bit passcode would involve a brute-force search over parameter values, which would take 1 billion billion years. So, if there are over V = 313 predictors documented in top academic journals, why shouldn’t we consider the subset of variables that actually predicts returns “secure”, too? After all, finding it would involve a brute-force search over parameter values. We might be able to approximate it. But, the exact collection of variables may as well be in a vault somewhere.

1. Motivation

Imagine that we’re trying to predict the cross-section of expected returns, and we’ve got a sneaking suspicion that might be a good predictor. So, we regress today’s returns on to see if our hunch is right,

   

The logic is straightforward. If explains enough of the variation in today’s returns, then must be a good predictor and we should include it in our model of tomorrow’s returns, .

But, how much variation is “enough variation”? After all, even if doesn’t actually predict tomorrow’s returns, we’re still going to fit today’s returns better if we use an additional right-hand-side variable,

   

The effect is mechanical. If we want to explain all of the variation in today’s returns, then all we have to do is include right-hand-side variables in our OLS regression. With linearly independent right-hand-side variables we can always perfectly predict stock returns, no matter what variables we choose.

The Bayesian information criterion (BIC) tells us that we should include as a right-hand-side variable if it explains at least of the residual variation,

   

But, where does this penalty come from? And, why is following this rule the Bayesian thing to do? Bayesian updating involves learning about a parameter value by combining prior beliefs with evidence from realized data. So, what parameter are we learning about when using the Bayesian information criterion? And, what are our priors beliefs? These questions are the topic of today’s post.

2. Estimation

Instead of diving directly into our predictor-selection problem (should we include in our model?), let’s pause for a second and solve our parameter-estimation problem (how should we estimate the coefficient on ?). Suppose the data-generating process for returns is

   

where , , and is normalized so that . For simplicity, let’s also assume that in the analysis below.

If we see returns from this data-generating process, , then we can estimate by choosing the parameter value that would maximize the posterior probability of realizing these returns:

   

This is known as maximum a posteriori (MAP) estimation, and the second equality in the expression above points out how we can either maximize the posterior probability or minimize of this function,

   

We can think about as the average improbability of the realized returns given .

So, what is this answer? Because and , we know that

   

where the first line is and the second line is . What’s more, because we’re specifically choosing to minimize , we also know that

   

And, solving this first-order condition for tells us exactly how to estimate :

   

3. Selection

Now that we’ve seen the solution to our parameter-estimation problem, let’s get back to solving our predictor-selection problem. Should we include in our predictive model of tomorrow’s returns? It turns out that answering this question means learning about the prior variance of . Is is equally likely to take on any value, ? Or, should we assume that regardless of the evidence, ?

To see where the first choice comes from, let’s think about the priors we’re implicitly adopting when we include in our predictive model. Since , this means looking for a such that . Inspecting the solution to our parameter-estimation problem reveals that

   

So, by including , we’re adopting an agnostic prior that is equally likely to be any value under the sun.

To see where the second choice comes from, let’s think about the priors we’re implicitly adopting when we exclude from our predictive model. This means looking for a such that regardless of the realized data, . Again, inspecting the formula for reveals that

   

So, by excluding , we’re adopting a religious prior that regardless of any new evidence.

Thus, when we decide whether to include in our predictive model, what we’re really doing is learning about our priors. So, after seeing returns, , we can decide whether to include in our predictive model by choosing the prior variance, , that maximizes the posterior probability of realizing these returns,

   

where the second equality in the expression above points out how we can either maximize the posterior probability or minimize of this function—i.e., its average improbability. Either way, if we estimate , then we should include ; whereas, if we estimate , then we shouldn’t.

4. Why log(N)/N?

The posterior probability of the realized returns given our choice of priors is given by

   

In this section, we’re going to see how to evaluate this integral. And, in the process, we’re going to see precisely where that penalty term in the Bayesian information criterion comes from.

Here’s the key insight in plain English. The realized returns are affected by noise shocks. By definition, excluding from our predictive model means that we aren’t learning about from the realized returns, so there’s no way for these noise shocks to affect either our estimate of or our posterior-probability calculations. By contrast, if we include in our predictive model, then we are learning about from the realized returns, so these noise shocks will distort both our estimate of and our posterior-probability calculations. The distortion caused by these noise shocks are going to be the source of the penalty term in the Bayesian information criterion.

Now, here’s the same insight in Mathese. Take a look at the Taylor expansion of around ,

   

There’s no first-order term because is chosen to minimize , and there are no higher-order terms because both and are normally distributed. From the formula for we can calculate that

   

Recall that measures the average improbability of realizing given that . So, if for a given choice of priors, then having any is infinitely improbable under those priors. And, this is exactly what we find when we exclude from our predictive model, . By contrast, if we include in our predictive model, then , meaning that we are willing to entertain the idea that due to distortions caused by the noise shocks.

To see why these distortions warrant a penalty, all we have to do then is evaluate the integral. First, let’s think about the case where we exclude from our predictive model. We just saw that, if , then we are unwilling to consider any parameter values besides . So, the integral equation for our posteriors given that simplifies to

   

This means that the average improbability of realizing given the priors is given by

   

To calculate our posterior beliefs when we include , let’s use this Taylor expansion around again,

   

The first term is the probability of observing the realized returns assuming that . The second term is a penalty that accounts for the fact that might be different from the estimated in finite samples. Due to the central-limit theorem, this difference between and is going to shrink at a rate of :

   

So, the average improbability of realizing given the priors is given by

   

where is big-“O” notation denoting terms that shrink faster than as .

5. Formatting

Bringing everything together, hopefully it’s now clear why we can decide whether to include in our predictive model by checking whether

   

The penalty term accounts for the fact that you’re going to be overfitting the data in sample when you include more right-hand-side variables. This criterion was first proposed in Schwarz (1978), who showed that the criterion becomes exact as . The Bayesian information criterion is often written as an optimization problem as well:

   

Both ways of writing down the criterion are the same. They just look different due to formatting. There is one interesting idea that pops out of writing down the Bayesian information criterion as a optimization problem. Solving for the suggests that you should completely ignore any predictors with sufficiently small OLS coefficients:

   

1. Motivating Examples

Trading strategies can interact with one another to amplify small initial shocks to fundamentals:

• Quant Crisis, Aug 2007: “During the week of August 6, 2007, a number of [quantitative hedge funds] experienced unprecedented losses… Initial losses [were] due to the forced liquidation of one or more large equity market-neutral portfolios… and the subsequent price impact… caused other similarly constructed portfolios to experience losses. These losses, in turn, caused other funds to deleverage their portfolios [and] led to further losses [and] more deleveraging and so on.”
• Flash Crash, May 2010: “The Dow Jones industrial average plunged more than 600 points in a matter of minutes that day and then recovered in a blink… [it] began with the sale by Waddell & Reed of E-Mini S&P 500 futures contracts… late in the trading day… [with] many of the contracts bought by… computerized traders who… [then] traded contracts back and forth [like a] ‘hot potato’.”
• Drop in Oil Prices, May 2011: “Never before had crude oil plummeted so deeply during the course of a day… prices were off by nearly a barrel… [and] market players were unable to identify any single bank or fund orchestrating a massive sale to liquidate positions… [rather] computerized trading just kicked in when key price levels were reached.”
• End-of-Day Volume, Oct 2011: “In the last minutes of trading, the S&P 500-stock index jumped more than points with no news to account for the rally. If you were left scratching your head, you were not alone… [and, the] culprit behind the late-day market swings: exchange-traded funds or ETFs.”
• Sterling after Brexit, Oct 2016: “If a country’s exchange rate represents international investors’ confidence in its government’s policies, the markets have given Britain the thumbs-down… The most likely explanation for the plunge lies in the action of algorithmic trades… These sales can be contagious, with one program’s trades setting off the sell signals of other algorithms.”

I refer to these sorts of events as rebalancing cascades. Stock ‘s fundamentals change, so a trading strategy sells stock and replaces it with stock . This purchase of stock forces another trading strategy to buy stock and sell stock . And, this sale forces…

The examples above seem to suggest that you don’t need a very big initial change in stock ‘s fundamentals to trigger a cascade. For instance, when oil prices suddenly dropped in May 2011, traders were “unable to identify any single bank or fund orchestrating a massive sale”. They were left “scratching [their] heads”, to use the language of the next example. So, in this post, I write down a random-networks model à la Watts (2002) to understand when we should expect small changes in fundamentals to trigger these sorts of long rebalancing cascades.

2. Market Structure

Consider a market with stocks, , where is a really big number. If a change in the fundamentals of stock will force some trading strategy to rebalance and buy stock instead, then let’s say that stock and stock are neighbors. Suppose that two randomly selected stocks are neighbors with probability . This uniform-random-matching assumption means that the number of neighbors that each stock has, , is Poisson distributed with mean ,

(1)  

The fact that each stock is only neighbors with a fraction of the market captures the idea that different trading strategies rebalance in different ways. For technical reasons, let’s assume that . The figure below gives examples of this sort of random network when .

Now, suppose that is an indicator variable for whether or not stock ‘s fundamentals have changed. If a bunch of strategies start trading stock because of changes in its neighboring stocks’ fundamentals, then this additional trading activity can affect stock ‘s fundamentals. For example, if lots of funds buy a stock and push it into the S&P 500, then the stock will have a higher market beta. Let’s model neighboring stocks’ effect on stock ‘s fundamentals as follows,

(2)  

where captures the vulnerability of a stock’s fundamentals to rebalancing. If there are lots of different strategies trading stock in lots of different ways, , then no single rebalancing decision will be important enough to change stock ‘s fundamentals. But, if stock has at most neighbors, then a change in the fundamentals of a single neighbor will generate enough rebalancing to cause stock ‘s fundamentals to change too. Let’s say that stock has such vulnerable neighbors.

Here’s the exercise I have in mind. Imagine that we select a stock at random, , and exogenously change its fundamentals, . If happens to have a vulnerable neighbor, , then the rebalancing caused by our initial shock will change the fundamentals of a second stock, . And, if happens to have an additional vulnerable neighbor of its own, , then the second wave of rebalancing caused by our initial shock to stock will change the fundamentals of a third stock as well, . If stock doesn’t have any additional vulnerable neighbors, then we will have triggered a rebalancing cascade of length with our initial shock to a single stock’s fundamentals. I want to characterize the distribution of cascade lengths for a randomly selected initial stock ,

(3)  

as a function of the market’s average connectivity, , and vulnerability threshold, .

3. Generating Functions

Generating functions make it possible to compute the distribution of cascade lengths. Here’s the basic idea. Take a look at Graham, Knuth, and Patashnik (1994, Ch 7) for more details. Suppose we’re flipping coins and counting the number of heads. The distribution of the number of heads, , after one flip is given by:

(4)  

If , then the coin is fair. The generating function for this same distribution is:

(5)  

Each term in the series is associated with one possible outcome for the total number of heads. is the probability of realizing heads. is the probability of realizing heads. We say that generates the distribution because we can compute all its moments by evaluating the derivatives of the generating function at . For example, the th-order derivative, , says that the coin never lands on its edge or winks out of existence. And, if we want to compute the expected number of heads, then we can use the st-order derivative:

(6)  

The fact that derivatives of the generating function give the moments of the associated distribution will be useful below. Let’s call this Property 1: derivatives are moments.

Here are two additional properties to keep in mind. Property 2: multiple samples. If we raise the generating function for the number of heads in one flip to the th power, then we get the generating function for the total number of heads in flips. To illustrate, look at what happens if we square the generating function for the number of heads in one flip:

(7)  

The result is just the generating function for the number of heads in two flips, .

Property 3: partial information. If we multiply the generating function for the total number of heads in flips by , then we have the generating function for the total number of heads in all flips conditional on having already seen heads on the first flip. To illustrate, notice what happens when we multiply the generating function for the number of heads in one flip by :

(8)  

If we’ve already seen heads on the first flip, then there’s no way to realize heads. The lowest we can go is heads now. So, the first term is now . And, this outcome occurs if we see tails on the second flip, which happens with probability .

4. Cascade Lengths

Now, let’s return to our original problem. Let be the generating function distribution of cascade lengths that we would start with an exogenous shock to stock ‘s fundamentals:

(9)  

The coefficient gives the probability that a shock to stock ‘s fundamentals would set off a cascade of length . If stock doesn’t have any vulnerable neighbors, then a shock to stock ‘s fundamentals can only affect stock , . Whereas, if a shock to stock ‘s fundamentals would set off a cascade affecting every other stock in the market, then . Next, let be the generating function for the number of vulnerable neighbors that stock has:

(10)  

So, the coefficient is the probability that stock has vulnerable neighbors.

Notice how these two generating functions are linked. If stock has vulnerable neighbor, , then an initial shock to stock ‘s fundamentals will set off a cascade of length if a shock to its one vulnerable neighbor will set off a cascade of length excluding stock . If stock has vulnerable neighbors, and , then an initial shock to stock will set off a cascade of length if shocks to its two vulnerable neighbors will set off cascades of combined length excluding stock . And, if stock has vulnerable neighbors, then an initial shock to stock will set off a cascade of length if shocks to its three vulnerable neighbors will set off cascades of combined length .

We know from the previous section (property 2: multiple samples) that is the generating function for the probability that different shocks set of cascades of combined length . And, we also know from the previous section (property 3: partial information) that we have to multiply through by if we want the generating function for the probability that different shocks set of cascades of combined length . So, the generating function for the distribution of cascade lengths has to satisfy the following internal-consistency condition as the number of stocks gets large, :

(11)  

The outer function, , gives the probability that the initial stock has vulnerable neighbors. The inner function, , gives the probability that shocks to these vulnerable neighbors would set of cascades of combined length . And, the multiplication by accounts for the fact that we want to compute the probability that shocks to these vulnerable neighbors would set of cascades of combined length not .

With this equation in hand, we can now compute the expected length of the rebalancing cascade that would follow from an initial shock to randomly selected stock :

(12)  

Rearranging yields an expression for the expected cascade length:

(13)  

And, in the exact same way that (property 1: derivatives are moments), the expected number of vulnerable neighbors that each stock has is given by . When stocks typically have less than vulnerable neighbor, , we have an expression for the average rebalancing-cascade length as a function of the market’s connectivity, , and vulnerability threshold, .

The figure above plots the average length of the rebalancing-cascade that would emerge if we selected an initial stock at random and shocked its fundamentals. It’s got a really interesting shape. A little math shows exactly why. We should expect short rebalancing cascades whenever stocks don’t have that many vulnerable neighbors. Here’s the expression for the average number of vulnerable neighbors that each stock has:

(14)  

Notice that stocks can have less than vulnerable neighbor on average for either of two reasons. First, they could have very few neighbors—that is, could be less than . Think about this as a fragmented market where very few people trade. This is the region on the bottom of the figure. Second, even if there are lots of people trading, stocks could have fundamentals that aren’t very vulnerable to the effects of rebalancing—that is, is large. This is the region in the upper right of the figure. But, if the market isn’t too fragmented and stocks’ fundamentals are a little vulnerable to the effects of rebalancing, then long rebalancing cascades can emerge. In fact, they can be infinitely long…

5. Infinite Cascades

…but what does that even mean? It’s actually much more reasonable than it first sounds. In a large market, , an infinitely long rebalancing cascade is just a cascade that affects a non-infinitesimal fraction of all stocks. If we now specify that is the generating function for the distribution of finite-length rebalancing cascades, then we can define as the fraction of all stocks affected by an infinitely long rebalancing cascade,

(15)  

Think back to the coin-flipping example where we said that because the coin never landed on its edge or magically winked out of existence. If the coin didn’t obey the laws of physics and disappeared of the time, then we would have said that . So, if is the generating function for the distribution of finite-length rebalancing cascades, then realizing an infinitely long cascade is like realizing a magical event that’s not characterized by . And, this way of framing the problem, , gives us a way to solve for the fraction of the market that’s typically affected by an infinitely long rebalancing cascade, .

Finally, notice how sharp the phase transition is. Tiny changes in the market’s connectivity, , and vulnerability, , can make all the difference between expecting infinitely long rebalancing cascades and expecting -stock long rebalancing cascades. Take a look at the figure above. Each panel has stocks (the dots) and represents a single realization of trading-strategy rebalancing rules (the lines) in markets where each stock has neighbors (left) and neighbors (right) respectively. Both these markets are observationally equivalent. But, as shown in the figure below, when an initial shock to stock ‘s fundamentals will yield a huge rebalancing cascade when (left) but not when (right). Small changes that push a market over the transition point where can have huge effects on the cascade-length distribution. What’s more, right at this transition point where , the sizes of the rebalancing cascades follow a power-law distribution,

(16)  

as shown in Newman et al. (2002). Slight differences in how the market happens to be wired up today can affect whether or not a stock on the other side of the market will be affected by an initial shock to stock ‘s fundamentals.