Regression towards (to) the mean

I was reading a book last night, one that I bought a few months ago but never got round to finishing it. Anyway, I finished a chapter that talked about Regression to the Mean.  What does “mean” mean? I looked up Wikipedia and they began with this definition,

In mathematics, mean has several different definitions depending on the context.

In probability and statistics, mean and expected value are used synonymously to refer to one measure of the central tendency either of a probability distribution or of the random variable characterized by that distribution.^[1] In the case of a discrete probability distribution of a random variable X, the mean is equal to the sum over every possible value weighted by the probability of that value; that is, it is computed by taking the product of each possible value x of X and its probability P(x), and then adding all these products together, giving. An analogous formula applies to the case of a continuous probability distribution. Not every probability distribution has a defined mean; see the Cauchy distribution for an example. Moreover, for some distributions the mean is infinite: for example, when the probability of the value  is  for n = 1, 2, 3, ....

For a data set, the terms arithmetic mean, mathematical expectation, and sometimes average are used synonymously to refer to a central value of a discrete set of numbers: specifically, the sum of the values divided by the number of values. The arithmetic mean of a set of numbers x₁, x₂, ..., x_n  is typically denoted by , pronounced "x bar". If the data set were based on a series of observations obtained by sampling from a statistical population, the arithmetic mean is termed the sample mean (denoted) to distinguish it from the population mean(denoted  or ).^[3]

For a finite population, the population mean of a property is equal to the arithmetic mean of the given property while considering every member of the population. For example, the population mean height is equal to the sum of the heights of every individual divided by the total number of individuals. The sample mean may differ from the population mean, especially for small samples. The law of large numbers dictates that the larger the size of the sample, the more likely it is that the sample mean will be close to the population mean.^[4]

Outside of probability and statistics, a wide range of other notions of "mean" are often used in geometry and analysis; examples are given below.

Okay, so that is a headache in the making. I prefer the term we used when we studied statistics long long ago in high school, we called it, “the middle number”. Its not the average but the number that come in the middle between the highest and the lowest. My definition may be in accurate but it works. So bear with me for a while.

The Regression to the mean is the notion that we all perform at close to the mean.  Sir Francis Walton who supposedly realised this issue noticed it when he studied genetics. By the way, I think he was Darwin’s cousin, which may partially explain his findings.

The concept of regression comes from genetics and was popularized by Sir Francis Galton during the late 19th century with the publication of Regression towards mediocrity in hereditary stature.^[6] Galton observed that extreme characteristics (e.g., height) in parents are not passed on completely to their offspring. Rather, the characteristics in the offspring regress towards a mediocre point (a point which has since been identified as the mean).

Applied to our work and everyday life, particularly in education, the notion works like this: we reward or punish incidents that deviate from the mean and then we lament the return to the mean.  One student who usually gets 70% in exams suddenly gets 95% while another who usually gets 70% suddenly gets 50% both are sanctioned. The former gets a pat on the back and congratulated while the latter gets a talking to about working harder.  The next semester comes along and both regress towards the mean.  We now say that the former has lost his steam and say that the latter has been working harder.  Statistically, however, if we look at the statistics of all of the past performances of both of these students, we may find that 70% is their normal performance level.  What we should do then is work on the average score not on the abnormal instances – the deviations from the mean.

We subscribe meaning to the deviations from the mean but the meaning of the stories or justifications for the deviation is often false because they are not the norm: they are deviations from the mean and they are followed by regression to the mean.

In an extended thought, the same applies to our understanding of history and expectations of the future: we see the deviations (e.g. the rebellions) but we do not see the norm (the working in the system e.g. of the colonials). We see the successes but we don’t see the daily grind.

When we look at history we observe the peaks and valleys of human endeavour and achievement but we overlook / glaze over / ignore / obfuscate / brush under the carper the daily occurrences at the historical points that we are observing.  Worse still, we seek to ascribe meaning and value to the things that we observe, usually, based on our values without acknowledging that our own values are shaped by our times and our context.  We are often arrogantly believe that our judgments and values are absolute and unquestionable.   So, the meaning and explanations of these abnormal occurrences may be there to help up replicate its occurrences often they do not.  A motivational theory will tell you of their 200 successes but they glaze over the 10,000 attendees whose lives remain unchanged: the occurrences when those who had their nosed to the daily grind stone remain bowed with their nosed to that very same stone after the heat and passion of the motivator’s presentation has dissipated.

Walton was right but Walton was also pessimistic.  He seems to have forgotten that the mean is not a stable number. Statisticians take snapshots of the system at a particular time and then make generalizations on the basis of the snapshot: they make the system static so that they can measure it but systems are dynamic. Every new entry alters the system. The new entry which deviates from the norm, deviates from the norm at the last batch of entries.  When the new entry appears, the mean has also changed. We can show this in our computer analysed statistics but not in paper based calculations. So the mean by which we judge the deviating entry is a false comparison: it is deviating from a mean that it has altered by its very existence.  This is how societies progress of degress: by shifting the mean little by little: it is evolution.  People keep thinking of change being revolutions that demand major upheaval of the system but nature does not work like that, neither does society in general. Both society and nature favour evolution, not revolution.  Major changes (deviations from the mean) happens occasionally and of they are highly notable: 9-11, US invasion of Iraq, Alawite massacre of Sunni Muslims in Syria, Israeli’s gross violations of human rights and their inhuman treatment of the Palestinians, the genocide by the Buddhists in Myanmar and the list extends.  Some of these we observe and react to, some we ignore.  The same will happen in the future when our deeds become history: some of our deeds are ignored especially those atrocities perpetuated by the powerful; some of our deeds get celebrated, like the minor deeds of those in / with power.

Just watch how a small thing can bring about huge changes but when these small things were introduced, they were abhorred by many. Later they become the norm. take for example, the handphone, the computer, democracy, etc

A subaltern is a primarily British military term for a junior officer.^[1] Literally meaning "subordinate," subaltern is used to describe commissioned officers below the rank of captain and generally comprises the various grades of lieutenant.^[2]

Ensign stands for standard or standard-bearer and was, therefore, the rank given to the junior officer who carried, or was responsible for, the flag in battle. This rank has generally been replaced in Army ranks by Second lieutenant.^[3] Ensigns were generally the lowest ranking commissioned officer, except where the rank of subaltern itself existed.^[4]

In postcolonial studies however, they use the term to mean something completely (almost) different.

In critical theory and post-colonialism, subaltern is the social group who is socially, politically, and geographically outside of the hegemonic power structure of the colony and of the colonial homeland. In describing “history told from below”, the term subaltern derived from the cultural hegemony work of Antonio Gramsci, which identified the social groups who are excluded from a society’s established structures for political representation, the means by which people have a voice in their society.

The terms subaltern and subaltern studies entered the field of post-colonial studies through the works of the Subaltern Studies Group, a collection of South Asian historians who explored the political-actor role of the men and women who are the mass population — rather than the political roles of the social and economic élites — in the history of South Asia. In the 1970s, the application of subaltern began to denote the colonized peoples of the South Asian Subcontinent, and described a new perspective of the history of an imperial colony, told from the point of view of the colonized man and woman, rather than from the points of view of the colonizers; in which respect, Marxist historians already had been investigating colonial history told from the perspective of the proletariat. In the 1980s, the scope of enquiry of Subaltern Studies was applied as an “intervention in South Asian historiography”.

As a method of intellectual discourse, the concept of the subaltern is problematic because it remained a Eurocentric method of historical enquiry when studying the non–Western people of Africa, Asia, and the Middle East. From having originated as an historical-research model for studying the colonial experience of South Asian peoples, the applicability of the techniques of subaltern studies transformed a model of intellectual discourse into a method of “vigorous post-colonial critique”. The term “subaltern” is used in the fields of history, anthropology, sociology, human geography, and literary criticism.^[1]

To my understanding, the easiest way to understand this term is to think of them as the people doing the daily grind.  The people who are working within the system.  They are the ones who are trying to make a living and trying hard to fulfill whatever aspirations, no matter how mediocre or how grand, by working with the options that they have.  The masses, if you will.  The people who forms the bulk of the community: the people whose votes you crave.

What does this have to do with regression towards the mean?

The thought came to me because after reading the book, I switched to Facebook and I saw a few entries about that Dyana Sofya woman.  What exactly is this young woman? One thing certain is that she is a deviation from the norm which makes her notable but does she herald the beginning of a new age for Malaysia politics: an age beyond politics driven by prejudices of race, ethnic sensibilities, religious zealots and celebration of the weird? Could she be another gate opener? Cordoba fell because a Shiite opened their gates to the crusaders, Melaka fell partly because there were Malays who worked for the Portuguese invaders, some Malayan states fell because there were Malay leaders who colluded with the English.  Either way time will tell, history will show what she is: the flag bearer of a new Malaysian age of egalitarianism or the turncoat who led the next enslavement of the Malays in their own land.

Focusing on her however, is forgetting that it all regresses to the mean. We actually need to pay attention to the peripherals of our political / societal vision where the subalterns trudge on with their daily lives. They are the ones who will usher in that new age.  

Quotations from Wikipedia.