Switch now and save!

It seems that every time I turn on the TV there is some insurance company that is saying something to the effect of “People who switch from insurance from Company A to Company B save on average $500 a year for the same coverage.” but then the next ad is “People who switch from insurance Company B to Company A save on average $500 a year for the same coverage.”  How is this possible?  Does this mean that if everyone switched insurance companies that everyone's rates would go down?  The truth is sadly no (if it were true I would just switch companies enough times that I would get free insurance), but interestingly the ads are correct in their statements.  So what is going on?

First let's take a hypothetical sample:

Each shaded square is the quote from the insurance company that each person is currently using, while the unshaded squares are the quoted prices for the same coverage with the other company.  So Person 1 is paying Company A $1,320 for insurance while he could get the same policy from Company 2 for $845.

The first thing to note is that the rates for each company are the same on average ($1455.88 for Company A and B).  So the insurance companies offer the same coverage for the same price overall, and yet they both make the correct claim of “people who switch from them to us save $500”.

Next let's just see what happens if everyone switches policies:

Here are the savings if everyone switched insurance policies.  Now we see something strange; on average if everyone were to switch polices then each person would expect to save -$500 on their insurance, or everyone would pay $500 more overall.  This is quite the opposite of what the ads seem to claim.  So were are the savings?

  

Well we need to pay close attention to the wording in the insurance companies statement “Those who switch save on average $500”.  Now who would switch?  It will only be the people that have quotes that are lower (and arguably significantly lower) than their current rate.  So obviously the average rate for any switch will save money.  

So if we go back to our data now and have the people switch that received lower quotes we see:

Where the light green cells are customers who switched and the dark green are the customers who stayed with their old policy.  Now when we calculate the money saved from those who switched we see an average savings of $500 for each customer overall, no matter which policy they had and switched to.  Case closed.

This insurance example highlights a common problem that we have interpreting numerical results.  Statements are crafted in ways which are true but could be misleading (even if they are not meant to be).  It is the job of the person who reports the results to be as clear as possible, but it is also the responsibility of the people who rely on results to make sure they fully understand the statements.  Comments such as “the samples were randomly selected”, “outliers were thrown out”, or “out of our initial testing cohort these three results were found to be highly significant (p-value<0.01)”, seem to be run of the mill, but without fully understanding what actually happened they may cause the analysis to be of little practical use.  When something doesn't make sense, or is not spelled out clearly, we need to be sure to ask questions so that we can keep ourselves pointed toward the truth.