Filtering or Detrending using Regression

We will be discussing filtering in-depth in subsequent sections, but one filtering or detrending technique is to use linear regression.

Consider the decomposition of a variable $y(t)$ into two parts: the fraction that is linearly related to $x(t)$ and the fraction that is uncorrelated with $x(t)$:

$$y(t)=y(t{)}_{fitted}+y(t{)}_{residual}$$

where $y(t{)}_{fitted}$ is the best-fit linear relationship between $y(t)$ and $x(t)$.

Thus,

• the fraction of variance of $y(t)$ explained by $x(t)$ is $r^2$: the ratio of variance of $y(t{)}_{fitted}$ to $y(t)$ and,

• the fraction of variance of $y(t)$ that is not explained by $x(t)$ is 1 - $r^2$: the ratio of variance of $y(t{)}_{residual}$ to $y(t)$.

Detrending

The above decomposition allows us to isolate $y(t{)}_{residual}$ which we sometimes want to do. If $y(t{)}_{fitted}$ represents a linear trend, $y(t{)}_{residual}$ is then the detrended component of $y(t)$.

Filtering

Let’s take a look at a filtering example using our ENSO and California precipitation data.

# load packages
import numpy as np
import pickle as pickle
import scipy.stats as st
import matplotlib.pyplot as plt
import matplotlib as mpl
mpl.rc('font',size=16,weight='bold') #set default font size and weight for plots

# ENSO Data:
# unpickle NINO34 (units of degC)
with open('NINO34_NDJF_2021.pickle','rb') as fp: #.pickle files are a python file type
    nino34 = pickle.load(fp,encoding='latin1')

#  Precipitation Data:
#  unpickle CA_PRECIP_DJFM (units of mm/day)
with open('CA_PRECIP_DJFM.pickle','rb') as fp: #.pickle files are a python file type
    precip_djfm = pickle.load(fp,encoding='latin1')

The $y(t{)}_{fitted}$ component is simply the best-fit line between $y(t)$ and $x(t)$ - in this case, CA precipitation and ENSO.

# calculate best-fit line

# np.polyfit(predictor, predictand, degree)
a = np.polyfit(nino34,precip_djfm,1) #polynomial fit, degree = 1 means a linear fit

# np.polyval(coeffs,predictor)
y_hat = np.polyval(a,nino34)

Now, we can filter out the influence of ENSO on California precipitation. Why would one want to do this? We might want to do this if we are looking for a relationship between California precipitation and something other than ENSO, but the ENSO influence is too strong to allow us to identify this secondary relationship.

This filtered out part is $y(t{)}_{residual}$.

# Filter out the ENSO signal from CA precip
precip_djfm_noENSO = precip_djfm - y_hat

Let’s see how the original, the fitted and filtered precipitation time series compare.

# plot the precip and filterd precip time series
plt.figure(figsize=(20,6))
plt.plot(precip_djfm,label="original")
plt.plot(y_hat,label="best-fit")
plt.plot(precip_djfm_noENSO,label="residual")
plt.title("CA DJFM Precipitation")
plt.ylabel("mm day$^{-1}$")
plt.xlabel("time")
plt.legend(loc="upper right")
plt.ylim(-3,8)

(-3.0, 8.0)