Minimum Spanning Trees in Python¶
In this notebook, we'll explore some of the graphing and visualization tools within SliceMatrix-IO, including the popular Minimum Spanning Tree, a graphing algorithm that is useful for estimating and visualizng the correlation structure of the market and revealing the hidden herding behavior of investors
To do this, we begin by importing the SliceMatrix-IO Python client.
If you haven't installed the client yet, the easiest way is with pip:
pip install slicematrixIO
First lets import slicematrixIO and create our client which will do the heavy lifting. Make sure to replace the api key below with your own key.
Don't have a key yet? Get your api key here
from slicematrixIO import SliceMatrix
api_key = "insert your key here"
sm = SliceMatrix(api_key)
Minimum Spanning Trees provide a compact representation of the correlation structure of a dataset in one graph. Because they are derived from the correlation matrix of the input dataset, MST’s quickly reveal the underlying statistical structure of the data.
Points which are connected to one another share a high degree of similarity.
This is especially useful in time series analysis, where points that are connected to one another move as a flock or herd together over time. MST’s selectively filter the connections within the full correlation matrix using Kruskal’s algorithm.
MST’s were initially used to solve problems such as:
- how to link up a telecommunications network using the shortest path
- image registration and segmentation (think: feature detection and clustering)
- industrial process control
Like many mathematical concepts with practical value, it took a few decades before the MST eventually percolated into the financial markets and academia. Financial network theory developed, showing the world how to create topological road maps of the stock market. This reduced the complexity of visualizing large groups of assets, opening the door to new ways of perceiving the financial markets. In the cod that follows, we'll use MST's to visualize the correlation structure of the Dow 30.
Let's move forward by importing some useful libraries:
%matplotlib inline
import pandas as pd
from pandas_datareader import data as web
import datetime as dt
import numpy as np
import matplotlib.pyplot as plt
Then we can read in a csv file with the trading symbols in our universe
data = pd.read_csv("https://s3.amazonaws.com/static.quandl.com/tickers/dowjonesA.csv")
data.head(10)
Now we can grab trading price data from Yahoo for our list of stocks using Pandas' Data-Reader
The following code will print out the current symbol so we can keep track of progress
start = dt.datetime(2016, 3, 7)
end = dt.datetime(2017, 3, 7)
volume = []
closes = []
good_tickers = []
for ticker in data['ticker'].values.tolist():
print ticker,
try:
vdata = web.DataReader(ticker, 'yahoo', start, end)
cdata = vdata[['Close']]
closes.append(cdata)
vdata = vdata[['Volume']]
volume.append(vdata)
good_tickers.append(ticker)
except:
print "x",
As is standard in dealing with financial time series, we are going to take the log differences of the input series before feeding it into our MST algo:
closes = pd.concat(closes, axis = 1)
closes.columns = good_tickers
diffs = np.log(closes).diff().dropna(axis = 0, how = "all").dropna(axis = 1, how = "any")
diffs.head()
Now let's create and visualize the MST
mst = sm.MinimumSpanningTree(dataset = diffs.T)
from slicematrixIO.notebook import GraphEngine
viz = GraphEngine(sm)
viz.init_style()
viz.init_data()
viz.drawNetworkGraph(mst, min_node_size = 15, charge = -350, graph_style = "dark", label_color = "rgba(255,255,255,0.9)")
As you can see, not all stocks are connected equally to the rest of the index. Some like United Technology (UTX) and Coca-Cola (KO) are connected to multiple other companies via share price correlation. Other stocks live at the edge of the graph in their own world, only connected to one other node. This has the effect of creating tradeable clusters in the graph. We could use this graph, for example, to then create dynamic hedge ratios for each pair we wish to trade in the graph.
For another example of using MST's and other correlation base network graph algo's, check out Bloomberg and SliceMatrix
Note: to run these examples you'll need an api key. Don't have a SliceMatrix-IO api key yet? Get your api key here