The devil in the detail

For many decades artifical intelligence (AI) was the buzzword that never delivered anything of practical use, except perhaps for incremental improvements in chess playing software. More recently however the availability of significant cloud computing power has enabled everyone to access the power of AI. Every time you speak to a connected device, ask for a foreign language to be translated, or avail yourself of autocorrect or autocomplete; you are leveraging the power of AI.

But is this truly democratic? Although accessible the relevant AI code is actually buried inside the private data centres of giants like Alphabet Inc, and to use it requires giving up any notion of privacy. A local grocery shop that wants to solve a bespoke problem such as predicting foot traffic so it can optimise its shelf displays or staffing levels cannot just ask Siri or Alexa. For a small business or consumer to develop their own AI forecasting tools would require a substantial investment in software and expertise. This would be overkill on a grand scale for decisions where the marginal improvement that could be expected might only be a few hundred dollars, or even less.

In his new book, Microprediction, Peter Cotton offers an alternative: an open AI network that anyone can tap into or contribute to. Within such a network the corner shop can publish its raw data which will then be picked up by a network of automated forecasters, all competing to do the best possible job of working out whether tomorrow will be especially busy.

How can such a network make accurate predictions? Judging by the use of jargon this book is aimed at experts already working in the area of AI and data science, and thus they will be familiar with genetic algorithims (GA); a well-known methodology in the field of AI. To make accurate predictions in GA we set up competing randomly created algorithms to see which does the best job of prediction. Those which perform well according to some value function are rewarded with survival and the chance to breed with other successful algorithms; poor predictors die out.

In contrast a microprediction network takes the algorithms out of the controlled ecosystem of a GA simulator, and puts them into the real world. Rather than being randomly spawned, these prediction engines would be carefully created by a wide network of freelance or moonlighting data scientists, or firms specialising in the business of prediction. Instead of survival, good algos are rewarded with hard cash – or rather their creators are – whilst bad ones are quietly ignored and eventually wither away. Importantly, as the marginal cost of making a single prediction is likely to be extremely low, it’s possible for even the smallest grocery store to access a distributed network of expert forecasters that would be the envy of any tech company behemoth.

The book covers a great deal of ground in considering many different aspects of how a microprediction network would work, such as the likely denziens of such a network, including ‘Oracles’ (which actually do the prediction) and ‘Micromanagers’ (which sit between consumers and suppliers of prediction services). It also cycles through a number of particular examples covering everything from financial markets (the authors original area of expertise, and my own) to predicting the survival time of a Schroedinger’s cat.

This is a visionary book, but in setting out its vision it skates over much of the practical details of implementation. The author addresses some of these practical difficulties, for example in the chapter on privacy, but too much of the book assumes the existence of network ‘plumbing’, thus allowing a focus on more esoteric topics. For the vision to be realised will require not only the existence of motivated potential creators of micropredictors, but also multiple customers for prediction services able to publish the required data, plus services and protocols to link these two groups together.

The author has set up a modest Python language-based ecosystem for microprediction which is sufficient for an experienced data scientist to begin work on creating predictors for the network. But this is far from a ‘plug and play’ system and requires a significant amount of expertise for potential customers of prediction services. It is unlikely that the local grocery store owner will get very far with this book, or with a set of instructions that begins with the requirement to install a specific Python library within a bash shell.

In reality the closest we have to a microprediction network are the data science contests run by the likes of Kaggle (a subsidiary of Google) where it is relatively straightforward to set up a contest to solve a specific problem. However their ease of use comes at the cost of a degree of inflexibility, they aren’t suitable for the type of continuous prediction envisaged in the book, and Kaggle is a walled garden rather than an open network.

It might be that a worldwide microprediction network will emerge spontaneously, but this seems unlikely without the buy in of technology gatekeepers whose interests are unlikely to be aligned with the idea of open access to AI. But there may be another way. The original internet was incubated by the US government, with standards such as TCP/IP agreed amongst a small group of experts. Creating a “prediction internet” may require a similar initiative.

Robert Carver is an independent trader and visiting lecturer at Queen Mary, University of London. He is also a former investment bank options market-making trader, and former hedge fund portfolio manager. Robert is the author of three books: “Leveraged Trading”, “Systematic Trading” and “Smart Portfolios”. His website is systematicmoney.org