Overfitting is the curse of anyone who tries to use the past to predict the future. Forecasting is a delicate balance between two extremes: at one end we risk creating a model which is too simplistic to be of practical use, whilst at the other the model is too complex and closely fitted to the past. The desire to create more complex models is usually the more tempting. A successful attempt to predict the past produces something immediately tangible which can be sold to prospective clients or academic journals. That seems preferable to the more ephemeral promise of jam tomorrow in the form of better future performance.
This problem of overfitting has been endemic to both the academic and industrial wings of finance for many decades; and with each advance in computing power it has become possible to overfit in more esoteric and interesting ways. The latest fad in the discipline of quantitative finance is Machine Learning. This is a rather vague and ill defined term which covers some fairly ancient methods like the half century old Support Vector Machines, but also newer techniques drawn from Artificial Intelligence that have only become practical recently as they require considerable processing power.
Irrespective of definition there is no doubt that machine learning has been badly used by many practitioners who have mostly repeated the overfitting mistakes of classical statistical learning. The more ‘black box’ nature of Machine Learning also means that the sins of overfitting may be easily hidden from view.
Dr Lopez’s book is an attempt to bring rigour and discipline to users of Machine Learning in finance, and the fight against overfitting is the common thread that runs throughout the book. Optimising the profitability of a trading strategy involves a process called backtesting. In the author’s opinion, which I share, backtesting is a dangerous but necessary activity whose use should be severely limited; and it should only be done in a carefully controlled fashion.
Consider for example the initial stage of data analysis. At this stage raw data is converted into features which seem to have some predictive power. This initial step disintermediates the final trading strategy design from the underlying data and, if done properly, should reduce the likelihood of overfitting.
Thus we can delay the inevitable, but at some point we will have to backtest. Various well known techniques are discussed to ameliorate the problem of overfitting through backtesting; including bagging, cross validation, and the use of statistics which account for short data history and the sort of unusual statistical distributions which plague financial data.
There are more novel ideas in chapter 13 which discusses backtesting on synthetic data – one of my own favourite techniques. We begin by assuming there is a particular characteristic of a market, such as strong trends or mean reversion, and then construct artificial price series that contain that characteristic. Backtesting will then produce strategies which will be profitable in scenarios which have the relevant characteristics. Left unsaid is the assumption that such scenarios existed in past data, and will continue to exist in the future.
Another less well-known method which I am inordinately fond of is Hierarchical Risk Parity (HRP). This is a specific technique to deal with the problem of finding the optimal portfolio allocations to a group of assets. As with other forms of backtesting the standard techniques are prone to producing portfolios which do very well in historic tests by dint of extreme allocations to just one or two of the assets that are available. Such portfolios are extremely fragile and heavily exposed to idiosyncratic risk. Many methods have since been developed which deal with this problem, but at the cost of creating highly unintuitive black boxes which humans do not easily trust.
HRP does a good job of creating portfolios that look like those an experienced human asset allocator would create, but in a scalable, objective and reproducible fashion. Personally I have found that portfolios which could have been created by humans are more likely to be accepted by humans. Sticking firmly to outcomes produced by the system is one of the most difficult and important tasks in systematic trading and investing. Using a technique like HRP makes it much easier for everyone to understand and be comfortable with asset allocations: not just the quantitative portfolio managers themselves, but everyone with a stake in the result including clients and regulators.
This seems to be Artificial Intelligence at its best: making decisions in a transparent and fair way.
A prospective financial wizard who is hoping that Machine Learning is an easy path to untold riches will find themselves heavily disheartened after reading this book. This is no bad thing – the odds are heavily stacked against naive and inexperienced aspiring traders, and the cost of this book will probably save them many thousands in trading losses.
In contrast Dr Lopez believes that working in this space requires teams of highly trained and experienced specialists. Such teams will find his book an invaluable reference guide.
Robert Carver is a former quantitative hedge fund portfolio manager and author. His homepage is www.systematicmoney.org