AI for Trading Series №3: Regression

Photo by Markus Winkler on Unsplash

This series focuses on two main points:

  • Checking and transforming data.
  • Regression.

There are some visual ways to check if a distribution is normally distributed or not:

  • Box-whisker plots helps us visually check if a distribution is symmetric or skewed.
  • A histogram lets us check if a distribution is symmetric/skewed, and if it has fat tails.
  • QQ plots help us compare any two distributions, so they can be used to compare distributions other than the normal distribution.

There are three hypothesis tests that can be used to decide if a data distribution is normal:

  • Shapiro-Wilk test
  • D’Agostino-Pearson test
  • Kolmogorov-Smirnov test.

Each of these produce p-value, and if the p-value is small enough, say 0.05 or less, we can say with a 95% confidence that the data is not normally distributed.

Shapiro-Wilk tends to perform better in a broader set of cases compared to the D’Agostino-Pearson test. In part, this is because the D’Agostino-Pearson test is used to look for skewness and kurtosis that do not match a normal distribution, so there are some odd non-normal distributions for which it doesn’t detect non-normality, where the Sharpiro-Wilk would give the correct answer.

The Kolmogorov Smirnov test can be used to compare distributions other than the normal distribution, so it’s similar to the QQ plot in its generality. To do a normality test, we would first rescale the data distribution (subtract the mean and divide by its standard deviation), then compare the rescaled data distribution with the standard normal distribution (which has a mean of zero and standard deviation of 1). In general, the Shapiro-WIlk test tends to be a better test than the Kolmogorov Smirnov test, but not in all cases.

So in summary, if you want to be thorough, you can use all three tests. If you only want to use one test, use the Shapiro-Wilk test. For a sanity check, visualize your data distribution with a histogram, box-whisker plot, and/or a QQ plot.

Corresponding python implementation of Normality can be found here

Heteroskedasticity

One of the assumptions of linear regression is that its input data are homoscedastic. A visual way to check if the our data is homoscedastic is a scatter plot.

Homoscedasticity and Heteroskedasticity (Source: AI for Trading nano degree course on Udacity)

If our data is heteroscedastic, a linear regression estimate of the coefficients may be less accurate (further from the actual value), and we may get a smaller p-value than should be expected, which means we may assume (incorrectly) that we have an accurate estimate of the regression coefficient, and assume that it’s statistically significant when it’s not.

Breusch-Pagan Test

The Breusch-Pagan (BP) test is one of the most common tests for heteroskedasticity. It begins by allowing the heteroskedasticity process to be a function of one or more of your independent variables, and it’s usually applied by assuming that heteroskedasticity may be a linear function of all the independent variables in the model.

Transforming Data

In order to transform our data into a normal data, we can feed our data into log function.

Normal Data (Source: AI for Trading nano degree course on Udacity)

In order to make data homoscedastic, we usually take the rate of return from one day to the next and then pass that data through log function.

Box-cox Transformation

Box-cox transformation is used to make our data normal + homoscedastic. A monotonic transformation changes the values of dataset but preserves their relative order. Box-cox dataset takes in dataset and outputs a more normally distributed dataset.

Box-Cox Transformation (Source: AI for Trading nano degree course on Udacity)

Linear Regression

Taking an example of purchasing the house. Here, we can say that the price of the house depends upon area. For this, we will find area covered by each house as well as their price. We find a coefficient to multiply by the area and add constant term (intercept). This is equation of a straight line. Upon plotting this straight line we want it to pass through almost all the data points.

We can measure how well the line fits the data by measuring the vertical difference between the points and the line. An optimal line is the one that reduces these differences. The process of finding the optimal regression line is called ordinary least squares. After plotting the optimal line, there might be some outliers. The difference between best fit line at each point is called Residual (error terms).

Evaluating Model

We can evaluate our model to check if its the best fit by the following methods

  1. R-squared
  2. Adjusted R-squared
  3. F-test

Multi-variate Regression

If we chose more than one depended variable that we wish to predict. For example, we try to predict not just housing price but also home’s electricity consumption and gas consumption. This is known as multi-variate regression. When we are using more than one in-depended variable to predict more than one depended variable, it is called multi-variate multiple regression.

The corresponding python implementation using sklearn’s LinearRegression can be found here

--

--

--

Hi! I am a tech enthusiast currently working on leveraging language technologies to solve financial use-cases! View my work here: https://purvasingh96.github.io

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Love Island, an algorithm explored (part 1)

Linear Regression connection with Interpolation, parametric model and important Topics for…

Data Has a Better Idea™: Learn why and how machine learning can discover new ideas with your data

“Data has a better idea” is a registered trademark of HIVERY (Red Analytics Pty Ltd)

Strategy Analytics in Python | #1 Churn Analysis

“How can a beginner data scientist like me gain experience?”

person reading a book whilst sitting in front of a computer screen

Arvato Financial Customer Segmentation

The Pandemic’s Future: A Book Review on ‘From Killer to Common Cold’

When should we use the log-linear model?

logarithm properties

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Purva Singh

Purva Singh

Hi! I am a tech enthusiast currently working on leveraging language technologies to solve financial use-cases! View my work here: https://purvasingh96.github.io

More from Medium

10 Things You must Know about NLP

Simple Competitive Learning with Python

Calculating kmeans on a GPU

Good Resources — Data Science/ML/AI/Mathematics, Applications in Portfolio Construction/Finance.