Variance

Theme from  Batman TV Show

Batman, Batman, Batman
Da da da da da da da da da da da da da da da da da
Batman!

No, Not Batman! Flatman!

In 2006, my firm, Cambridge Systematics, was undertaking a study for the Massachusetts Governor’s Highway Safety Board, GHSB.  The GHSB was submitting  that study to the National Transportation Safety Board.  It required that the result be confirmed  by a certified statistician.  At the time, my former colleague, Kevin Tierney, and I were “chosen” as the potential certified statisticians.  Since it required attesting that the sampling plan for the survey in the memo was correct and since Kevin Tierney was a national expert ( although not certified) on survey design, the required attestation fell to him.  I joked with Kevin at the time that it was a good thing that  it did not require my signature because I was only certifiable, while he could claim to be certified. I am still certifiable (as insane) , but  I am not certified as a statistician, if there is such a thing. But I would like to think that I know something about statistics.

In Flatland, which was written in 1884, an analogy was proposed that if you limit your perception to a Flat, two-dimensional, plane then you will have a tough time explaining things such as spheres, etc.  In other words, perception matters.  Einstein’s Theory of General Relativity, in 1916,  explained the same concept in mathematical terms.  There can be an absolute, (i.e. the speed of light) and our perception of such things as length, weight, time etc., depend on the relationship to that absolute. Despite NOT being a statistician, I would like to use statistics, in the same manner as Flatland did, to make an analogy

If a group thinks that there is an absolute, i.e. the mean, and that group accepts no deviations, i.e. variance, from that absolute truth, then the group has a mean that is greater than 0, but variance of the group is zero. ( This probably also makes all other statistical moments of the distribution, such as the skew or the kurtosis, also zero).  What this defines is a single point. If there are any observations away from this point, then those observations can NOT be members of the group whose variance is zero.  In addition to many other things, distributions with zero variance are inherently unstable, that is they are NOT resilient.  If the truth, e.g. mean, is contradicted, for example, if the group absolutely believes that the World would end on December 21, 2012 when the Mayan Calendar Cycle ended, then when the world did not end, it does not require that the group be dissolved.  In a normal distribution the mean, median, and mode are equal.  If the variance is 0, then 100% of the group, every member, has the mean as their value.  In a uniform normal distribution, the variance is 1.0  The mean, median and mode are still equal, but only 40% of  members of that group are at that mean.  It is thus possible to be a member of the group and NOT have a value that is equal to the mean.  If the mean changes, is found not to be TRUE,  then the group does not have to dissolve.

A problem with a normal distribution is that it allows any observation to be less than zero.  In observations of real data, for example income, what is often required is a distribution that allows only non-zero values.  The exponential distribution is such a distribution, but it has the problem that the closer you get to zero, the larger the number of observations are expected at zero.  If the number of observations at zero should also be zero, but the distribution is still expected to be normal,  then an exponentially modified Gaussian ( normal) distribution is often used.  This distribution can be defined by three attributes: the mean, µ; the variance, σ; and the exponential parameter, λ.  Unlike a pure normal distribution, this distribution can be skewed.  The closer the distribution is to normal, the closer the skew is to zero.  But if the distribution tends towards the exponential, the maximum value of the skew is 0.31. 

So by analogy, a group should not have a variance of zero, belief in an absolute, and accept no deviation.  A normal group can be skewed, but that skew should not exceed 0.31.  If the skew is greater than zero, then there is no way that the variance can also be zero. If Flatland used mathematics to show that not everything could be explained by a flat plane, then statistics  says you can not accept a skew and expect a variance of zero.  Now you know, and knowing is half the battle! Go Flatman!