Market impact of orders, and models that predict it

Introduction

At Adaptive, we have the technical expertise to design, build and operate bespoke trading venues implementing different types of market models: central limit order books (CLOB), request for quotation (RFQ), indication of interest (IOI) and variations of these. Clients who want to move from voice or manual trading to a digital solution often ask for a CLOB venue, which is a great fit for mature, liquid markets with many participants. However, a newly electronified marketplace is unlikely to have the liquidity needed to support a CLOB. In these cases, we would recommend a marketplace built on a market-model that supports the discovery of liquidity in less-liquid circumstances.

Regardless of the market model, however, the trading venues we deliver allow our clients to retain ownership of the data it generates. We can help them understand the behaviour of the venue and its market participants through the analysis of this data, as well as design and train machine learning predictive models leveraging this data. This article talks about different models that predict the impact of orders on market price, and how recent techniques of AI and ML could be applied to improve these predictions.

Order books and market impact

In a previous post about machine learning in finance[1], we reviewed published deep learning models and found that ~79% of them address tasks related to trading, mostly price prediction. But the return of a trade not only depends on the price, but also the costs of executing an investment strategy have to be taken into account[2] as they could heavily decrease the returns of the investment. These costs include commissions, bid/ask spreads, opportunity costs of waiting and market impact on price. But before describing machine learning models applied to predicting market impact, let’s first introduce the concepts of order books, types of orders and market impact.

Order books

In the past 50 years, all major stock exchanges have implemented electronic communication networks (ECN)[3]. ECNs are used to place bid/buy and ask/sell orders, publish the best bid/ask prices, and match bid/ask orders automatically without a middleman. They rely on order books, electronic lists of orders for a specific financial instrument grouped in bid/ask orders and organized by price level. In an order book, each price level has a total quantity (or “depth”) which is the sum of the quantities for all orders at that price level.

Types of orders

There are different types of orders that can be placed in an order book depending on the price, the duration, the undisclosed quantity, etc. Market orders do not specify a price and are executed immediately at the best market price. Limit orders specify a limit price and are placed in the order book in a first-in-first-out queue for the given limit price. Limit orders are executed only when the specified limit price or better is met (Fig. 1). Market orders are usually executed fast, but there is no guarantee on the price at which will be traded, whereas limit orders restrict the price at which the order will be filled, but it is not guaranteed that the market will reach the limit price, or when. There are also market-to-limit orders, which are market orders that are converted to limit orders if they are not fully filled[4].

In the example order book in Fig. 1, there is a single bid order for a quantity of 10 at a price of 1.28, therefore if an ask market order were placed it would first have a quantity of 10 filled at a price of 1.28, then the following 15 would be filled at a price of 1.27 and so on until the market order is fully filled. In the same order book, if an ask market order for a quantity of 8 were placed, the limit bid order at a price of 1.28 would have a quantity of 8 partially filled whereas the remaining 2 would be left unfilled until the market can fill it at the requested limit price or is cancelled.

According to their duration, orders can be classified in day orders which are closed at the end of the trading day, “good-til-cancel” (GTC) orders, which last until explicitly cancelled, “immediate-or-cancel” (IOC) orders, that must be partially or fully filled immediately or else be cancelled and “fill-or-kill” (FOK) orders, which are similar to IOC but have to be fully filled immediately or cancelled[5].

Figure 1. Order book. A new ask limit order for a quantity of 30 is added to the “first-in-first-out” stack for its limit price 1.31 (arrow in the Ask section). The spread (double-headed arrow) shows the price difference between the best ask and bid.

Finally, orders can be classified according to the undisclosed amount as regular orders, where the full quantity of the order appears in the order book and hidden orders where only a part of their total quantity is disclosed in the order book. Hidden orders are used to buy or sell large quantities without revealing the intention because the full undisclosed amount is split in orders with a smaller, disclosed amount. When the full undisclosed amount of a hidden order is filled, a new one is placed in the queue for the same price, and so on until the full amount is filled. Hidden orders make it difficult to estimate the depth of each price level, adding uncertainty on the final price at which orders will be executed.

Market impact

ECNs share different levels of information regarding the order book: level I access includes the highest bid and lowest ask price values (for instance, in Fig. 1 it would be bid: 1.28, ask: 1.31), level II includes also the quantity (depth) at each price whereas level III not only show information from previous levels but also is used by registered brokers and financial institutions to enter or change quotes, execute orders and send confirmation of trades. A mid price can be calculated from level I information taking the average between best bid and ask price, or a quantity-averaged price if level II information is available. In the order book of Figure 1, the mid price would be 1.295 and the quantity-averaged price would be 1.2796, showing a slight imbalance towards the bid side of the book (there is more demand than supply).

Best bid and ask prices of an order book change with time as its orders are filled, new orders are placed or existing limit orders are cancelled. In the order book in Fig. 1, there is a single bid order for a quantity of 10 at a price of 1.28, if an ask market order for a quantity of 10 were placed it would be fully filled at a price of 1.28, and the best bid price would be 1.27 after this trade. Similarly, if all ask limit orders at a price of 1.31 were cancelled, the new best ask price would be 1.32. Placing a new limit order in an order book always shifts the quantity-averaged price towards the limit price of the order, and if the limit price value lies within the bid and ask values (the spread) it also moves the best bid or ask price. In the order book of Figure 1, placing a new limit bid (ask) order at a price of 1.29 within the spread would move the best bid (ask) price to 1.29. Also, traders might change their investment strategy, placing new orders or canceling existing ones, according to the trades and orders they observe. The change in prices observed when orders are placed and executed is named market impact. Transaction cost is one of the important factors that affect the investment performance and is usually classified into two major categories: explicit costs and implicit costs. Explicit costs, also called direct costs, are transaction costs that can be explicitly stated and measured. These costs include commissions, transaction fees, and taxes. Implicit costs, or indirect costs, are costs that cannot be measured directly but can be improvable by an appropriate trading strategy. They include bid-ask spreads, time risk costs, and market impact costs.

In the next section, we give an historical tour on market impact models and how they have been modified to provide predictions that better fit the observed market impact, and hint at some machine learning algorithms which could further improve the quality of these predictions.

Market impact models

An early model of market impact[6] published in 1985, proposed a permanent impact, increasing linearly with the total size “v” of the order (Table 1). This model has several shortcomings, for instance it assumes that the market impact is proportional to the size of the order, whereas instead it has a concave shape, where small and large orders have less impact than medium sized ones. Also, the model does not reflect the observed decay on market impact over time.

An improved model was published in 2005 that addresses the problems described above[7]. This improved model has a permanent and a temporary market impact functions added to give the realized market impact (Table 1). In this model, both the permanent and temporary market impacts are assumed to be power-law functions, which have a concave market impact shape with regard to the total size “v” of the order. In this model, the permanent impact exponent was set to 1 (linear permanent impact) because it is the only exponent value for permanent impact that prevents arbitrage (ie. lock in profits by buying an asset in a market and selling it in another market for a higher price) and also gives an impact which is independent of the trading time[6]. The temporary impact in this model has an exponent greater than zero but lower than 1 (between 0.4 and 0.7, but typically 0.5, ie. a square root), giving the concave shape empirically observed.

The square root power-law model of market impact has been successfully fitted to market impact independently of the traded asset (equities, FX, futures), epoch (from mid nineties to present day’s electronic markets) or stock capitalization[8, 9]. Nevertheless, research published in 2015 over a very large dataset of 7M trades (~7 times larger than the largest dataset used in previous studies) showed that the square-root power law model does not fit well the market impact observed for the smaller or largest trades[8], and instead a logarithmic model seems to better fit both the mid-sized trades as well as both extremes (Fig. 2)

Figure 2 Market impact of different metaorders spanning a wide range of sizes 𝛟 (normalized by daily volume). Red dashed line corresponds to the best fit of the power-law model, whereas the blue line corresponds to the logarithmic model best fit. Taken from [8]

Based on this approach, increasingly more complex models have been used to predict market impact, like the I-STAR model[9] (Table 1). This model factors in not only the size of the order, but also the imbalance between the buy and sell orders as well as the volatility of returns in its market impact predictions. The rationale is that, given two orders of the same size, a larger market impact is expected when the order book has a large order flow imbalance, or a large volatility of returns, or both.

In essence, the models described are all based on choosing a function and finding the parameters of the function that best fit the observed market impact (Table 1). In this approach to model market impact, choosing a function up-front imposes a certain relationship between the features that contribute to market impact, and facilitates understanding how each feature contributes to the market impact. On the other hand, choosing a function that does not have undesirable side effects is not easy, and also fixes the set of input features that this model will consider as relevant for market impact: the function cannot be fitted if we remove one feature, and will ignore any new feature we might want to add.

Table 1. Market impact models relating the size of the order v to the market impact. In these models, v is the rate of the order size to the average daily traded volume.

A different, data-driven approach to obtain a model of market impact would not suffer from this limitation because it would not assume a particular function. Machine learning (ML) algorithms (which we described previously[1]) fall within this data-driven approach and can be fed many features including ones that might not even have an effect on market impact. These algorithms only use the relevant features to build some function to predict the market impact. Therefore, large amounts of data are required to “train” them in order to obtain their best performance, whereas fitting the functions of the models described above typically does not require as much data to obtain the best fit. This essential difference difficulties a fair comparison between both approaches, leaving only the option of training ML algorithms using the same limited set of input features used to fit the functions of Table 1, which is unlikely to provide the best quality in ML models.

In this research article [10], different neural networks and a support vector regression model were compared to the I-STAR model of market impact cost. Each ML model was trained using only the same 3 input variables used to fit the I-STAR model (OFI, VOL and POV, see Table 1). These three input variables were extracted from ~18M transactions from the Bloomberg Terminal for the period from 2014/06/02 to 2014/06/26 for equity of 51 different firms, grouped into 17 large, 17 mid and another 17 small capitalization firms. Each model predicts market impact cost, in basis points units of the mid-price before the first trade. In order to assess the quality of the predictions, the market impact prediction for each model was compared to the real market impact using a dataset not used to train or fit the model, and its error was assessed in different ways, which ultimately measure how far the predicted value is from the observed one (Figure 3). Even with this limited dataset, which only considers transactions within a single month, most of the ML models outperformed the I-STAR model in any of the three market cap groups, which can be seen by the smaller error values compared to the error of the I-STAR model. For instance, for the large cap firms, the I-STAR model root mean squared (RMS) error of their market impact predictions was 0.2823 basis points of the mid-price before starting the trade, whereas for the predictions of the Bayesian neural network model was 0.1712, which is 40% less error. In the medium and small cap groups, the best ML algorithm still outperformed I-STAR predictions, but to a minor extent, with 16% and 7% lower RMS error values.

Figure 3. Errors of predicted market impact, in basis points from mid price when first order was submitted. Mean absolute error (MAE), relative MAE (RMAE), root mean squared error (RMS), and relative RMS (RRMS) of neural network (NN), Bayesian neural network (BNN), Gaussian process (GP), support vector regression (SVR) and I-star models of market impact. Bold font highlights the lowest error model for each error measure[10].

Concluding remarks

An accurate prediction of market impact cost, particularly of large orders, helps fund managers, investment banks and traders choose a trading strategy without unaccounted and unexpected market impact costs. This prediction could be improved by using ML algorithms trained on larger datasets including transactions for a long period of time, more than a few input variables, level-II order book data if available (instead of just the mid-price). It also could be improved by using neural networks designed for sequential data like financial time series (long short term memory, recurrent and some types of convolutional neural networks) to include previous prices in the input to the algorithm. Reach out to us if you have data analysis needs or would like to leverage trading data to train predictive models.