Disclaimer: Before making any investment decisions based on this (or any financial content on the internet) analysis, consult with a financial professional (https://www.youtube.com/watch?v=ILsXSJeF9Xc). This blog post is for informational and educational purposes only.

In the world of investing, choosing the right index fund can be a daunting task. Two popular options are the Dow Jones Industrial Average (DJIA or sometimes abbreviated here as DOW) and the S&P 500. This post explores their historical performance to see if one might be a better choice.   If you are interested in learning how to perform analyses like this, consider one of my data science courses: https://wp.me/P5xMk4-5p

Methods

We utilized different methods to compare and contrast the indexes.  Some of these were traditional, such as computing helpful summary statistics of the returns (such as the average return).  We also calculated different moving averages as a useful benchmark to obtain medium to long term performance.  

We also performed a more sophisticated approach to understand the returns of both indexes called the bootstrap.  To understand the bootstrap, imagine you have the historical return data for both the Dow Jones and the S&P 500. The bootstrap algorithm is a fancy way to create many “fake histories” of returns, helping us understand how reliable the statistics we calculate from the real data might be.

Here’s how it works:

1. Resampling with Replacement: Think of randomly grabbing returns from your data/historical returns/history, but with a twist: you can put them back in each time! This allows you to create a new “fake history” with potentially duplicate years. 
2. Creating Many Fake Histories: We repeat this grabbing and replacing process hundreds of thousands of times, creating a whole collection of these “fake histories” for both DJIA and S&P 500. Each fake history has the same number of years (like 30) for each run, but the order and sometimes even the specific years might be shuffled around.
3. Analyzing Each Fake History: Now, for each fake history, we calculate statistics like average return, just like we did with the real data. This gives us a sense of how much these statistics would vary if things like random chance influenced which years we picked.  From these statistics, we can even describe the distributions of these statistics.

Why it’s Useful for Comparing Distributions:

By repeating this resampling process a large number of times, the bootstrap generates a distribution of summary statistics (mean, median, standard deviation etc.) for both DJIA and S&P 500 returns. This allows you to:

• Compare Variability: Analyze how much the summary statistics (like mean return) vary between the two indexes. A wider spread in the bootstrap distribution suggests more variability in the statistic.
• Distribution Shape: Visualize the distribution of these statistics using techniques like histograms or density plots. This can reveal if one index has a more skewed distribution of returns compared to the other.

By comparing the bootstrap distributions of the DJIA and S&P 500, we’ll gain a deeper understanding of how consistent their returns are, how much they might fluctuate, and any potential differences in their return distributions.  If you are interested in learning how to perform the bootstrap in situations like this, consider my data science bootcamp: https://wp.me/P5xMk4-5p

Initial Analysis

We compared the annualized historical returns of the DJIA and the S&P 500 over the same time period (1928 to 2023). While initial plots suggested the Dow Jones might even outperform the S&P 500, further analysis revealed a different story. The code and output from this analysis is provided at my GitHub link: https://github.com/billyl320/sp500_dow_compare 

The above plot represents the returns as “proportion change”.  Proportion change is defined to be the annualized return change.  For instance, a return of 40% would equate to a proportion change of 1+0.40 = 1.40.  A return of -40% would equate to a proportion change of 1-0.40 = 0.60.  We can see that the histograms of the proportion change is fairly similar.  There is even an (albite very very unlikely) chance that the DJIA may have years of the greatest return due to have the maximum value across both indexes.  

Average Returns Are Similar

Financial experts often cite similar average returns for both indexes across different time periods [1, 2, 3]. Our analysis confirms this. While both indexes have experienced periods of strong growth and decline, their long-term average returns tend to be close.  Below are the 10 year and 30 year moving averages of both indexes.  (This is especially true in more recent years.)  The x axis is time where the larger values indicate more recent years.  

Median Returns Tell a Different Story

However, looking deeper, we found a larger difference in the medians. Recall that the median represents the “middle” value in a dataset, where half is less than the median and the other half is greater.  The average or mean does not guarantee to have half of the data on either side of it. 

The above histogram compares the distributions of the means of the DJIA and S&P 500 (in blue and red, respectively).  The distributions are very similar and would suggest that either index would give similar rates of performance.  

The above histogram compares the distributions of the medians of the DJIA and S&P 500 (in blue and red, respectively).  The distributions are very different and would suggest that either index behaves differently.  The DJIA has two peaks, one closer to a 5% annualized return and one closer to 15%.  The S&P 500 has one primary peak around a 15% annualized return.  This suggests that it is more likely to have higher returns with the S&P 500 than the DJIA, despite having a similar range for the median.  

This got me thinking, “Is there a good way to represent how different these distributions are?”.  I created a plot comparing the proportion of medians less than a given value for both of these (this is very similar to an estimated CDF plot).  I first looked at this plot for the means, and then the median.  

The plot for the means are very similar with no noticeable differences in the plot.  Generally speaking, we would expect a better performing index to have a line closer to the x axis for a longer stretch of proportion change.   

However, when we look at the median counterpoint, we can see some deviations between the x axis values of about 1.06 to 1.14. The next plot looks as a zoomed in section of this part of the plot.  

This plot shows that there is a large discrepancy between the two indexes.  For example, the distribution of the S&P 500 median has less than 20% of its distribution less than 8%.  Conversely, the distribution of the DJIA median has over 40% of its distribution less than 8%.  The distributions begin to converge after 1.14.

So, Which Index is Better?

Based purely on historical data and risk tolerance, the S&P 500 appears to be the better choice. Here’s why:

• More Consistent Growth: The distribution of the S&P 500’s median return suggests steadier growth potential since it only has one major peak.
• Greater Diversification: The S&P 500 tracks 500 companies, offering broader diversification compared to the Dow Jones’s 30 companies.

Are There Reasons to Choose the Dow Jones?

While the S&P 500 might be the analytical favorite, there are potential reasons to consider the Dow Jones:

• Social Investing: An investor might have issues investing in many of the companies in the S&P 500.  Investing in the DJIA might provide a list of less problematic companies for the investor.  
• Easier for Self Management: It also might be easier to create a self-driven portfolio without using investment products like ETFs using the DJIA than the S&P 500.  
• Potential for Higher Returns (with Higher Risk): The Dow Jones has experienced some periods of higher returns than the S&P 500 (as due to the one year with outsized returns for the DJIA in the historical data). However, this comes with the risk of larger potential losses.

The Final Word

The “better” index depends on your investment goals. If you prioritize consistent growth and diversification, the S&P 500 might be ideal. However, if you’re comfortable with potentially higher risk for potentially higher rewards, the Dow Jones could be an option.

Is there anything that I didn’t consider that you would have? Anything that you might have done differently? Let me know and perhaps I can do a follow up to this!

Remember: Regardless of which index you choose, consulting with a financial professional is crucial before making any investment decisions. They can help you create a personalized investment plan that aligns with your financial goals and risk tolerance.