21st October 2010
The collaboration between the information scientists at University of Manchester in the UK and Indiana University (IU) in the US has allowed them to predict the rise and fall of the Dow Jones up to a week in advance with an accuracy approaching 90%.
Led by Johan Bollen, associate professor at IU's school of informatics and computing, the research found the extraordinary correlation between the value of the Dow and public sentiment after analysis of more than 9.8 million tweets from 2.7 million users during 10 months in 2008. Their full research paper can be viewed here (opens in PDF format)
The researchers used two mood-tracking tools to analyse the text content of the large-scale collection of Twitter feeds, allowing the researchers to measure variations in public mood and then compare them to closing stock market values.
One tool, OpinionFinder, analysed the tweets to provide a positive or negative daily time series of public mood. The second tool, Google-Profile of Mood States (GPOMS), measured the mood of tweets in six dimensions: calm, alert, sure, vital, kind, and happy.
Together, the two tools provided the researchers with seven public mood time series that could then be set against a similar daily time series of Dow Jones closing values.
In an interview with Mindful lMoney, research collaborator Xiao-Jun Zeng of the University of Manchester's school of computer science, says the team had been investigating novel methods for predicting market movements and commodities like oil through analysis of financial news and market data.
"What we quickly realised was that something was missing from the parameters, namely moods, psychology. While news, data mining is quite easy, capturing mood information is more difficult. That is why we began to focus on Twitter."
The researchers then correlated the two sets of values captured – the Dow Jones and public mood – and used a self-organising network model to test their hypothesis that predicting stock market closing values could be improved by including public mood measurements.
What they uncovered was an accuracy of 87.6% in predicting the daily up and down changes in the closing values of the Dow.
The researchers made use of neural networks, which aim to mimic the working of human brain. The type of neural network deployed by the researchers is already used successfully in the forecasting of electrical load needs and has the ability to self-organise its own ‘neurons' during a learning process that included information of past Dow Jones and public mood time series values.
Of the two mood-tracking tools used, the researchers found that the Calm and the Calm-Happy combination of the GPOMS had the highest prediction accuracy. Bollen at Indiana University says: "In fact, the calmness index appears to be a good predictor of whether the Dow Jones Industrial Average goes up or down between two and six days later."
An Bloomberg interview with Johan Bollen can be viewed here.
The researchers say the odds of the prediction accuracy rate of 87.6% being sheer chance were then calculated for a random period of 20 days and determined to be just 3.4%.
Manchester's Zeng stresses however that the research is at "a very, very early stage… it is really more in a testing phase".
While they are considering other indices, including the FTSE 100, for future work, Zeng says the next big step for the team will be to develop "adaptive" forecasting models.
He explains: "At the moment the model provides ‘snapshots' of all the data and news we mine and their integration with mood information. But mood in particular can change very quickly and so we would like to develop real time solutions to help with forecasting of indices, individual shares and so on."
But while accurate prediction of market movement and stock prices sounds like the holy grail for investors, some experts warn it could prove very dangerous.
Stephen Fitzpatrick, an expert on social networks based at Social Business Group, publisher of Mindful Money says of the research: "What we currently have in the online financial space is a nervous social system and there is a danger that twitter can increase the extent to which markets are correlated which is precisely what happened during the crash."
Ftizpatrick adds: "Greater connectivity is not always a good thing unless it is accompanied by mindfulness and without that twitter alone will not help us to develop what we really need, which is a social nervous system."
The kind of techniques being used by the Manchester and Indiana researchers have been investigated for application in other areas for a few years now, including in fields such health and disease.
For instance Google Flu Trends, uses a cloud of keywords to determine how sick a population is. At the more trivial end there are applications like this map of a United Kingdom snowstorm based on Tweets about snow – have also had some success tracking the real world.
This article on Wired Science considers the various applications of web analysis and social networks and asks specifically whether web crawling methodologies might help to monitor the health of the ecosystem.