How to efficiently create a dataframe summarising key statistics found in 2 pandas dataframes with identical dimensions
I currently have 2 pandas dataframes consisting of 500 000 simulation paths (1 to 500 000) each with 30 time steps (0 to 29). Now at each time step I have a price in the first dataframe, and a delta value on the other dataframe. Therefore I have both the price and the delta value for each 30 time steps for 500 000 simulations. Is there a computationally efficient way to create a data frame that has 30 * 500 000 observations noting the price, time step and delta value for each observation (Here an observation means each individual point on each simulation path). I know that there must be a more computationally efficient method to achieve this, but I am not very versed in python. The following is the code I’ve tried to implement so far:
priceDeltaDataFrame = pd.DataFrame([], columns = ['price', 'time', 'delta']) for i in range(1, 500000 + 1,1): for j in range(0,30): temp = pd.DataFrame([[futuresPriceDataFrame.iloc[j][i], j, deltasDataFrame.iloc[j][i]]], columns = ['price', 'time', 'delta']) priceDeltaDataFrame.append(temp) if (i % 100 == 0): print(i) But I know from experience that nested loops are inefficient and that there must be a better way to proceed. Any help is greatly appreciated.
