I am dealing with a pandas dataframe where the index is a DateTime object and the columns represent minute-by-minute returns on several stocks from the SP500 index, together with a column of returns from the index. It’s fairly long (100 stocks, 1510 trading days, minute-by-minute data each day) and looks like this (only three stocks for the sake of example):

DateTime         SPY     AAPL    AMZN     T 2014-01-02 9:30  0.032  -0.01   0.164   0.007 2014-01-02 9:31  -0.012  0.02   0.001   -0.004 2014-01-02 9:32  -0.015  0.031  0.004   -0.001 

I am trying to compute the betas of each stock for each different day and for each 30-minute window. The beta of a stock in this case is defined as the covariance between its returns and the SPY returns divided by the variance of SPY in the same period. My desired output is a 3-dimensional numpy array beta_HF where beta_HF[s, i, j], for instance, means the beta of stock s at day i at window j. At this moment, I am computing the betas in the following way (let returns be full dataframe):

trading_days = pd.unique(returns.index.date) window = "30min" moments = pd.date_range(start = "9:30", end = "16:00", freq = window).time def dispersion(trading_days, moments, df, verbose = True):     index = 'SPY'     beta_HF = np.zeros((df.shape[1] - 1, len(trading_days), len(moments) - 1))      for i, day in enumerate(trading_days):         daily_data = df[df.index.date == day]         start_time = dt.time(9,30)              for j, end_time in enumerate(moments[1:]):             moment_data = daily_data.between_time(start_time, end_time)             covariances = np.array([moment_data[index].cov(moment_data[symbol]) for symbol in df])             beta_HF[:, i,j] = covariances[1:]/covariances[0]              if verbose == True:             if np.remainder(i, 100) == 0:                 print("Current Trading Day: {}".format(day))               return(beta_HF) 

The dispersion() function generates the correct output. However, I understand that I am looping over long iterables and this is not very efficient. I seek a more efficient way to "slice" the dataframe at each 30-minute window for each day in the sample and compute the covariances. Effectively, for each slice, I need to compute 101 numbers (100 covariances + 1 variance). On my local machine (a 2013 Retina i5 Macbook Pro) it’s taking around 8 minutes to compute everything. I tested it on a research server of my university and the computing time was basically the same, which probably implies that computing power is not the bottleneck but my code has low quality in this part. I would appreciate any ideas on how to make this faster.

One might point out that parallelization is the way to go here since the elements in beta_HF never interact with each other. So this seems to be easy to parallelize. However, I have never implemented anything with parallelization so I am very new to these concepts. Any ideas on how to make the code run faster? Thanks a lot!