B-School Admission Criteria: A Critique, Part Two

Continuing from where I left off yesterday.

THE PROFILE

Now, the profile weightage, which is the winner of the most flawed of all criteria. This includes weightage for the following parameters:
(a) 10^th standard marks
(b) 12^th standard marks
(c) Graduation marks
(d) Post-graduation marks, if applicable
(e) Work experience
(f) Extra-curricular activities
(g) Achievements

What is the rationale behind this?
To quote IIMB, “IIMB has found over the years that students who perform well in the academic program are typically those who have a consistently good academic record during their school, high school and graduation level, besides exhibiting sufficiently high aptitude as measured by the CAT. Therefore IIMB uses multiple parameters, namely academic performance in school, high school and graduation programs as well as candidates’ scores in CAT to judge the suitability of candidates for the PGP program.”

And, “Evaluation by multiple criteria is also consistent with empirical research on recruitment and selection that shows greater efficacy of recruitment processes that use multiple criteria. Multiple criteria are used to arrive at a composite score for every candidate, which is used to select candidates for the subsequent stage.”

That’s the rationale in a nutshell. There are quite a few holes in it. I’ll start with the basics.

1. What do marks have to do with an MBA program?
An MBA program is not one in which you write exams, get marks and graduate. It is supposed to prepare you for the business world. Businesses don’t use marks as a test; they judge performance. They don’t want you to write solutions to hypothetical problems with inadequate data for making the right decisions; they expect you to actually solve real ones.

Marks are awarded by an examiner based on his judgment of your answer (i.e. subjective evaluation). In business, the mark equivalents, bonuses and incentives, are awarded based on the tangible value added to the company through your work (i.e. objective evaluation). I add that these are general statements. There are exceptions, but they don’t significantly affect the purpose of my observations.

In sum, if we accept that an MBA program is supposed to prepare a student for a corporate career (an implicit assumption) then marks have no relevance whatsoever.

2. The problem with empirical research
Empirical research is research based on observations or experiments. The problem with empirical research is the problem of induction. I’ll introduce it in an easy to understand manner by using The Black Swan Theory.

The Black Swan Theory
Let’s say you have observed 4000 swans and found all of them to be white in color. You publish the result of your empirical study as: All swans are white.

All it takes to refute your conclusion is for me to show you a black coloured swan. If I can successfully do that, then I can confidently say “Not all swans are white.” This finding of mine would totally invalidate the conclusions you formed from your empirical research. Incidentally, black swans do exist.

Basing anything on empirical research alone without logic is like building a house without a foundation. It looks fine, till it collapses.

If the empirical research on which IIMB bases its selection criteria is someday invalidated by a new study, they would have been ‘unfair’ in their selection process. But that’s not the worst of it. The worst problem is that they would have created a self-fulfilling prophecy, which we shall see in a while.

Also, even if using multiple criteria makes the process more efficacious, the question of what criteria to use still remains. And here, we get into the criteria of marks, and why it is flawed selection criteria.

3. What do marks project?
I studied in a really good CBSE school. My school marks have always been average among my school peers. I never did learn by rote (save for Hindi in 10^th standard). I used to understand and analyze concepts and their implications. I have sometimes even gone beyond what is ‘required’ in order to get marks out of my curiosity. I used to love studying some subjects, though I didn’t score greatly in them.

Do my 10^th and 12^th standard scores reflect my academic development? Not performance in useless tests, but real intellectual development. The answer is an emphatic NO. What do they actually reflect? The mood of the examiner, the presence of certain ‘keywords’ in each answer, my level of boredom at that time and in a small way, my knowledge about the questions asked. As a judge of my caliber, they are merely some meaningless numbers.

College was even worse. I could graduate with a B. Tech. in Biotechnology without knowing anything relevant whatsoever about biotechnology. How? Because I just had to write some answers to some questions on an exam paper and clear the cut-off, and I am now an engineer with a specialization in biotechnology.

I don’t have a ‘consistently good academic record’, so IIMB concludes that I won’t perform well in their academic program. I may have, I may not have. They’ll never know. But I can say this with certainty: if I did do well, it would not be ‘in spite of’ my poor academic record. And if I didn’t do well, it would not be ‘because of’ my poor academic record.

Doesn’t past performance matter?

Monkeys on a typewriter
If you have a billion monkeys typing away on a billion typewriters for a period of a billion years, by sheer chance, one among them may produce the complete works of Shakespeare. Now, let us take this impressively performing monkey and tell it to produce the complete works of Homer next. What are the odds you would give it?

Clearly, past performance does not matter when things happen by sheer chance.

But IIMB claims to have found a correlation between their selection criteria and student performance at IIMB. To again quote their selection document, “IIMB has found over the years that students who perform well in the academic program are typically those who have a consistently good academic record during their school, high school and graduation level, besides exhibiting sufficiently high aptitude as measured by the CAT. Therefore IIMB uses multiple parameters, namely academic performance in school, high school and graduation programs as well as candidates’ scores in CAT to judge the suitability of candidates for the PGP program.” So why not use those criteria for selection?

From these statements, we can see that IIMB has made a basic logic error which any scientist, statistician, academic researcher, or real economist knows he should avoid: correlation does not imply causation.

First, the correlation
If, when A happens, B happens, can we say B and A are correlated? For example: It rains on election days. Are elections and rains correlated? Should rains be regarded more highly when they occur on election days?

Now let us look at this statement:
“IIMB has found over the years that students who perform well in the academic program are typically those who have a consistently good academic record.”

Does this imply that good academic performance at IIMB and past academic record are correlated? Should past academic performance be regarded highly? (There is actually a correlation, but not for the reasons you may think. I’ll address it in a while).

Second, implying causation
A happens. Then, B happens. A caused B. This is called post hoc ergo propter hoc logic. B happened after A, therefore B happened because of A. Example: The performance of bank stocks is correlated with interest rates. RBI hiked repo rates. The BSE Bankex (Banking index) fell 1%. Therefore, the banking index fell 1% because RBI hiked repo rates. If you think that statement looks correct, try the reverse: The BSE Bankex (Banking index) fell 1%. RBI hiked repo rates. Therefore, RBI hiked repo rates because the banking index fell 1%. Now you’ll see the absurdity of this logic much more clearly.

As we see from their selection document:
“IIMB has found over the years that students who perform well in the academic program are typically those who have a consistently good academic record. Therefore IIMB uses multiple parameters, namely academic performance in school, high school and graduation programs as well as candidates’ scores in CAT to judge the suitability of candidates for the PGP program.”

First, they assume they have found a correlation between past academic performance and IIMB performance. Second, they determine there is a causal link between the two i.e. good track records in school and graduation implies good performance at IIMB (if they didn’t think so, they wouldn’t try to get more students with a good track record). And finally, they reward consistently good academic records, thus ensuring they get more students who have it. They have successfully created a self-fulfilling prophecy.

Self-Fulfilling Prophecy
Let’s understand this concept with an example. I use one loosely based on one from the book Outliers by Malcolm Gladwell. An anthropologist found that most Canadian hockey players were born during the first three months of the calendar year. Upon researching, he found that the selection criteria employed an age cut-off date of Jan 1.

What this means is that if you are born on January 2^nd 2005, you are eligible for the hockey tryouts in the Under-6 category in 2011. You’ll be the oldest in the group, and much more physically mature than someone born on December 31^st 2005 (who is over 5 but under 6 on that arbitrary cut-off date and so competes in the same category).

The extra physical development the January born player has passes off as talent. At that age, size really matters. So even if the January and December born are equally talented, or the December born is slightly more talented, the January born gets selected.

Selected players are given rigorous training, due to which they actually become better. If the coach then measures that January born player against that December born player, he will think his selection is validated because that January born player is much better than that December born. Thus, the slight initial advantage of that January born player due to his being born just after that random cut-off date, turns into an actual advantage because of the training he gets.

The coach then observes over the years that people who perform well in the rink are typically those who were born in the month of January, besides exhibiting sufficiently high aptitude for hockey as measured by the selection tryouts. Therefore, he decides to give people born in January an additional profile weightage in the selection tryouts, which boosts the initial advantage they got by being born in January, after that random cut-off date of Jan 1^st. This profile weightage will make even the less talented January born players to appear more talented than they actually are.

With the odds so markedly stacked against the February to December born players, they eventually stop trying, and even when they do try, they lose out unless they have a really massive skill advantage and they get the opportunity to exhibit it during the tryouts.

While the coach thinks that he is picking the best among the players, he is actually only picking the best among the January born players! But, all his observations still validate his selection criteria! Such is the power of the self-fulfilling prophecy.

Won’t some really talented February or March born players surmount their handicap and make it through the tryouts? Yes, that’s a possibility. We’ll see how the coach handles that.

If typically only the January borns play good hockey, but I show the coach a February born player on his team who excels, won’t that make the coach change his mind about his selection criteria? No, it wouldn’t. He would say that player is statistically insignificant. If he employs smoothing techniques, that player would be the noise he would eliminate from his data set.

The mind only sees that which it wishes to see. If you think only January borns make good hockey players, you’ll dismiss all data to the contrary as ‘noise’, ‘outliers’ or ‘statistically insignificant’. This is called the confirmation bias: seeing only the information which supports your preconceived notions, regardless of whether that information is true or not.

I’ll now proceed to relating this analogy to IIMB profile criteria.

Having compared the arbitrary January 1 cut-off date to ‘consistently good academic record in school, high school and graduation’, I have to show why this criteria is also arbitrary i.e. has no basis in logic.

My personal example won’t suffice to prove this. In fact, attempting to use it as a ‘proof’ would be a fallacy. Why? Because I’ll then be guilty of using a local example to dispute a global criteria.

For example, if Chennai has a spell of cold weather for two years (it actually being a hot city) I cannot cite that as a ‘proof’ that global warming is a myth. Nor can I cite it as a proof that there is now global cooling. It is obvious how ridiculous such a statement would be.

(It is ironic that we accept the reverse to be a proof: If Chennai has an unusually warm climate for two years, it is ‘because of’ global warming, and if Chennai has a cold spell, then Oh! It is because global warming caused local cooling due to certain natural phenomena. Warming or cooling, global warming can never be proved false if we accept such proofs. Alas, the mind sees only what it wants to see.

Here, it should be noted that some global warming zealots try to use logic to prove their beliefs. They say: global events have local consequences but local events do not have global consequences so the second example above is true while the first one isn’t. My answer is: you see it that way because you have already accepted the ‘global event’ i.e. global warming and you now see Chennai becoming warmer (or even colder) as the ‘local consequence’. You assume A precedes B, and so when B happens you say it happened because of A, applying the fallacious post hoc ergo propter hoc type of reasoning. Another example of the mind seeing only what it wants to see!)

I had to digress to address the critiques of that example, since failing to do so would have given an entry point for the global warming cult to attack my logic. They never rest in their bid to unleash some eco-fascism couched in the name of science and utilitarianism.

I won’t use my personal example to show why ‘consistently good academic record’ is not an objective criterion. Rather, I’m going to introduce another concept here to create doubt on the efficacy of the criterion.

The Survivorship Bias
I’ll explain the concept with an example related to the criteria we’re analyzing: marks.

First, I’ll assume that people who do well in school (as measured by marks) do so because:
1. Their parents force them to (most likely)
2. They love to score high marks since it gives them an identity or sense of worth or respect (somewhat likely)
3. They love their studies, and also do well in tests (least likely)

All these categories of students move on to college and do well there in terms of marks. After this stage, most of category 1 students move on in life. Some are forced into preparing for CAT. Some among category 2 see it as the ultimate exam, and revel in the challenge so they too prepare for it. Some from category 3 love the learning opportunity and they too prepare for CAT.

In the end, 375 of them make it to IIMB, helped along by lots of luck as we saw earlier. By analyzing these candidates’ profiles, what can we conclude?

IIMB likes to conclude that their consistently good academic performance is linked to their performance at IIMB. But what is IIMB actually doing? It is fooled by the survivorship bias.

It sees only the students who make it in, not the entire sample of students from which these students emerged.

If they cared to see the entire sample, they’d find students with the same ‘consistently good academic performance’ but yet not in IIMB. Since they only look at the qualities of the survivors (i.e. the students actually in IIMB), they miss the point that there are candidates with the same qualities as the survivors outside of IIMB, and that it is not these qualities of the survivors which matter for their academic performance at IIMB.

If the people both inside and outside IIMB have the same qualities, then how do these qualities predict someone’s performance there? It cannot.

I’ll make it simpler. All students have a pen. IIMB selects some students, who then write great essays. Can IIMB then say that it is because their students have a pen that they write great essays? Could it not be the case that they write great essays because IIMB trains them to write great essays? As we saw with the coach example, the initial advantage becomes a real advantage due to training. Looking for explanations elsewhere misses the whole point. When you’re fooled by the survivorship bias, you tend to attribute success to certain qualities of the survivors, missing the point that it is not these qualities which mattered for their success but something else altogether. If only these qualities mattered, then everyone, not just these survivors, would be as successful.

It’s a little tricky to understand, but well worth the effort. When you get stuck in understanding certain logic, draw a parallel. For instance, this statement of mine: If the people both inside and outside IIMB have the same qualities, then how do these qualities predict someone’s performance there? A parallel would be: If everyone in the world had a great pair of lungs, then how does this predict someone’s sprinting ability? Yes, it is a necessary, but not a sufficient condition, and definitely not the predictor of a sprinter’s ability. (I have addressed why consistently good academic performance is not a necessary condition earlier).

This is the survivorship bias. You tend to notice the qualities of the achievers and erroneously conclude that it is these factors which made the achievers what they are. If I find the same qualities among non-achievers, your conclusion goes out the window. This is why statistics without logic is dangerous. You tend to get fooled by what you see. Retro-fitting logic to statistics doesn’t work so well either, but it can be helpful at times.

I have also shown how school and graduation marks have no relevance to an MBA program. Even if marks objectively measured your subject knowledge, my knowledge of physics and engineering has no relevance to my management potential. A good doctor need not be a good actuary. The exception is math, but since CAT tests your current level of math, why go back to scores from your history?

Thus, we have seen that the rationale behind using marks as a criterion is sufficiently flawed.

4. The final nail in the coffin
Even if marks mattered, how will they be judged? Not all universities are the same. Not all boards are the same. Marks over years vary due to the difficulty level of the paper, the type of correction, the test-taking ability of the students (on aggregate) that year etc. Apart from that, a student who did very well in an easy paper may do equally well in a tough paper. We cannot judge his ability based on the difficulty level of the paper.

The solution? More statistics using student data accumulated over the years! Let’s commit more of the same mistakes!

Logic without statistics is fine, but statistics without logic is dangerous. There is no logic behind academic profile weightage. However good the statistical tools employed, the final results are bound to be flawed.

THE WORK EX MYTH
MBA education at top B-schools abroad is prohibitively expensive. Only working executives can afford it. Not wanting dissatisfied freshers (students with no work ex) with huge loans on their backs bringing down the B-school’s image, they started to enroll only working people who could afford the fee or the loan burden.

Indian B-schools only saw that foreign B-schools demanded work ex. Without thinking through the logic behind it, they simply aped the west, and concocted their own logic: that work experience matters for getting the most of an MBA program. Some went a step further and admitted only those with work ex (and ironically, also charged prohibitively high fees), completely fooled by the trend abroad.

And so the myth of work ex was born. Just like the myth that engineering plus MBA is a great combination (which sprouted up because engineers who hated engineering jumped into the non-technical field of management and did well there because a failure to do so would have meant a career as an engineer; and then people applied induction to deduce that engineers make good managers), a new myth was created: work experience is essential for an MBA.

Aside from the myth’s impact on a fresher’s chances of admission to an MBA program, most of the impact is social.

A 22-year old female engineer works for 2 years (2 years is a ‘magic number’ for work ex) in a software company, does her MBA by the age of 26 and spends 1.5 grueling years as a management trainee, only after which she gets married and contemplates starting a family. The full-time residential requirement at MBA programs affects personal life. Late marriage becomes the norm. The same is true for men, but the social impact is most distinctly felt for women.

The other problems with the work ex criterion

1. Work ex is measured in units of time
A year of data entry work is apparently ‘more valuable’ in terms of MBA admission than 10 months of BPO tech support work, which in turn is ‘more valuable’ than 8 months of Project Engineer work at an IT company, which in turn is ‘more valuable’ than 6 months of managerial work ex at a tech startup, and so on. You get the point. It is not the quality of the work ex which gets measured, but the quantity.

IIMC had a unique criterion this year: rewarding work ex in IT and Telecom sector more. Is it because mainly IT and Telecom sector employees populate B-schools? Seems like they want to firmly plonk their self-fulfilling prophecy in place: more IT and Telecom employees do MBA, they do well because they come with IT and Telecom background, so let’s reward IT and Telecom work ex more. This may not be the actual reason (for instance, recruiters may actually prefer candidates with work ex in these sectors) but this whole thing sounds fishy to me.

2. Informal sector work ex is not counted
Let’s say my friend makes around Rs. 1 lakh/- a year doing Internet Marketing part-time, while still pursuing his under-graduation. He finishes his studies and gets into Internet Marketing full-time, increasing his earnings as well as his learning in this new field. He is an entrepreneur, with no company to his name. He has no joining letter, no salary sheet, no visiting card, nothing. All he can show is a screenshot with his earnings and a bank balance. According to the proof of work ex criteria, he is unemployed.

If he does this business for 2-3 years, he can’t show it in his B-school application form and he will get summarily rejected by all B-schools simply because a person who doesn’t seek employment in the formal sector for such a long period of time is not ‘serious enough about his life’ to be considered for an MBA program. The same fate awaits successful day traders, vendors and all the self-employed who don’t have a ‘proof’ of their work ex.

I say the proof of the pudding is in the eating. Joining letters, pay slips etc. are mere bureaucratic red tape. (On that issue, one of the most annoying parts of the admission process is asking for attestation for mark sheets, admit card, score card etc.) Judge the self-employed on their terms.

3. Quality of work ex is not measurable
This is why B-schools measure quantity. No one can objectively measure the quality of an intangible attribute. However, ignoring things just because they are not measurable is a fatal flaw, since these things do have a measurable impact. (Incidentally, this is a flaw of neo-classical economics). The best way to measure quality of work ex, if it must be measured, is during the interview. Subjective evaluations have their place there, since the whole interview is in itself a subjective evaluation of a candidate’s entire life in 5-30 minutes.

4. Incentives to cheat
Now let me make this clear before I start: I am not criticizing the B-schools for this. Incentives to take the easy way out always exist. This criterion of work ex just adds to the already existing incentives. How?

I took a break from formal academics after graduating in May 2010. I didn’t seek employment in the formal sector. This practice is generally frowned upon, since only ‘not-so-serious’ people ‘waste their time’ away from college and workplace. I like the implicit assumption here: that college and workplace don’t ‘waste your time’ but doing something on your own does.

The ultimate purpose of formal education is to make money. If you don’t accept that, I challenge you to opt out of placements in your engineering/MBA (and to prove me wrong, all students have to opt out of placements). We make money so that we can live a life of our choosing. And we may also make money by living the life of our choosing. The point is that a job or education is not an end in itself to be looked upon so highly: the lifestyle you choose is. If I say no to the means (work) because I get the ends directly (leisure or my choice of lifestyle), can I be faulted for it?

But B-schools apparently value conformity (much like socialist bureaucrats) more than individual choice: You study in school and college, you go to work and you come here to do your MBA. We don’t want people who do things differently; because we are so trained in conforming that we cannot look beyond it. It is the same with the classical music snobs. All those who prefer rock and roll are tasteless, because they don’t conform to these snobs’ tastes. Just to be clear, I have nothing against either type of music, just people’s attitudes.

Some of those who took a break from the formal sector take the easy way out: fake some work ex. This is only because there is an incentive to cheat.

But all said and done, this enforcement of conformity, though subtle, plays havoc with the students’ minds: chase your dreams or conform. Most choose the latter.

What’s the link between extra-curricular activities and MBA? Nothing. Some say it is just a way to start the interview, by making the candidates comfortable by asking questions in their areas of interest. I’m all for making candidates feel comfortable. The problem is that it actually doesn’t. People who have no extra-curricular activities feel threatened. People who do have them feel like they have to know all about it since they have mentioned that they like it. It’s akin to the principal of a school saying attendance at his Saturday afternoon speech is optional. He may really mean it, but that’s not the way the teachers and students see it. Can we blame the B-schools for that? I don’t know.

And seriously, what do your achievements in sports or music have to do with your admission into an MBA program? How does an ‘achievement’ give the impression that you’re talented, and that you’ll apply this talent in a different field? It really doesn’t. As I mentioned earlier, a good surgeon need not be a good actuary.

With so many flaws in profile based selection, why do B-schools even choose to adopt it? I can think of two reasons: they’re ignorant and just want to continue business as usual, or they want students who conform to their tastes. Both aren’t objective selection measures.

The entrance test and the profile weightage are the supposedly objective selection criteria, which I have addressed so far. In my next article, I’ll examine the subjective criteria: the interview, group task, group discussion, essay writing, etc. I’m taking it up separately because it is my subjective evaluation of these subjective criteria.

B-School Admission Criteria: A Critique, Part One

Disclaimer: I have spent well over two years trying to gain admission into a top-notch B-school in India, because I think an MBA education is necessary for my career progression.

A small note
Until a month or so ago, I never paid much attention to the logic behind the admission criteria of top B-schools. All I knew was that such criteria existed, and in order to gain an admission, I needed to satisfy them.

While going through the rigmarole of admission processes in 4 Symbiosis institutes in Pune, I happened to buy a book called Fooled by Randomness by Nassim Nicholas Taleb at a roadside bookstall. Reading that book changed my thinking. It made me realize the uselessness of B-school admission criteria.

That knowledge can now be yours, for free, if you take the time out to read this somewhat lengthy article.

Even non-MBA aspirants can gain from reading this. How? I am outlining a way of thinking here, which can be universally applied, i.e. in practical situations faced in daily life.

I hope you gain as much pleasure and understanding reading this as I did from writing it.

THE PROCESS
B-schools give weightage to a lot of different criteria. Such division of weightage among ‘relevant’ parameters is considered to be a positive.

While the relative weightage of the criteria differ, all B-schools have the following criteria:
1. Written Assessment Test (WAT) or Computer Based Test (CBT) with Multiple Choice Questions (MCQs)
2. Profile of the candidate (Academics, achievements, work experience etc.)
3. Interview (Group/ Personal)

Other commonly used criteria:
1. Essay writing/ Précis writing
2. Group discussion
3. Case study analysis
4. Extempore
5. Group task/ group exercise

The final selection is based on an overall composite score, with minimum qualifying marks in the interview.

So, what’s wrong with this process?
At first glance, everything seems fine. All the above criteria appear to be objective, and will ensure that the B-school gets a great candidate overall, with competence in a lot of relevant aspects.

Delving into them a bit though, we start treading murky water. All the above criteria fail as a test for measuring a candidate’s potential in handling an MBA program and a career beyond. The rest of this article will be devoted to explaining why.

First, the assumptions
The obvious assumption I have made is that the B-schools employ these criteria in order to get the ‘right’ talent for their institute. They may have other motives, but if so, it remains a secret to the aspirants.

As an MBA aspirant myself, and speaking on behalf of all aspirants, I can safely say that we assume that the selection criteria is intended to get the ‘best’ students.

Debunking the Process
First, the WAT (or CBT) with MCQs in the areas of Language Comprehension, Quantitative Ability, Reasoning and Data Analysis. The mark allotment and weightage given to each area differs, but all entrance exams have these. Some also include a General Awareness (GA) section.

To explain the logic behind these parameters, I have to digress a little. The primary aim of schooling was to educate children in math, logic and language. These 3 are regarded as the fundamental skills every man needs (that school has an etymology that translates into luxury is a different issue).

An MBA program is most probably the last formal education a student will receive in his life. To weed out those who have had at least 17 years (2+10+2+3) of formal education and still have not acquired these skills, B-schools employ such objective tests of math, logic and language (English). I’ll come to the GA part later.

I am in favour of testing for these skills, but not on the method employed in using these test scores.

Note: The relative weightage to be given to each section is subjective, and cannot be argued against provided there is not too huge a bias in favour of or against a section.

Let’s look at two types of entrance tests: a paper pencil (PP) based one (say SNAP, with +1 mark for a right answer and -0.25 for an incorrect answer) and a computer based one with ‘normalized’ scoring (say CAT).
In the case of SNAP, everyone gets the same paper, which translates into an equal opportunity for all to prove their mettle vis-à-vis his competitors in that paper. The only unknowns at play are luck (which cannot be controlled) and preparedness of the candidate. It is a good way of testing.

Note: No man is perfect. No test designed by man can be perfect. A good test is therefore one which is the least imperfect.

The problem with SNAP (leaving GA apart for now) lies not in the test itself, but in the interpretation of the test score. Say SIBM-P has a cut-off score of 118, and calls candidates at and over that threshold score for due process. They think candidates who meet the cut-off are worthy of further consideration.

But how is a candidate with a score of 118 so much better than one with 117.75 that one gets an interview call and the other does not. Clearly, he isn’t.

I don’t deny that for the sake of convenience, the admissions committee has to pick some random cut-off number, which will invariably be a round number. I only say it is not an objective measure of a candidate’s ability.

Yes, the candidate with a 117.75 is out of luck. It can’t be helped. Or can it? I say yes. Let us now look at how the marking system in SNAP is skewed in favour of risk takers.

The problem with negative marking
Why negative marking? Obvious answer: to minimize guessing, so that only real ability is rewarded. By penalizing a candidate for incorrect answers, they hope to ensure that people who make it to the cut-off do so due to their competence, without the help of lucky guesses.

But negative marking actually has the opposite effect: it rewards the guessers and penalizes the talented.

How?

Let us say you’re a salaried employee working as a coder in an IT company. You work hard, spend frugally and save your money in a bank. Would you gamble with it, knowing that you may lose all the fruits of your labour? I assume not.

But say your job is to gamble with other people’s money. You then do it probabilistically, with expected value calculations, hedges and stop losses in place. This is called investing.

Getting back to the SNAP scenario, consider two students A and B, both at a score of 100 due to sheer effort.

A is prudent, B has the investor mindset. A does not guess the answers he doesn’t know. His score stops at 100.

B is a risk-taker. He decides to randomly guess 10 answers. He is risking a maximum loss of 2.5 marks for a maximum gain of 10 marks. And his expected value is positive, at 0.625 marks.
[Expected value: (0.25x1-0.25x0.75)10= 0.625]
A net positive, just from guessing!

But wait. If guessing can be so rewarding, why doesn’t B just guess all the questions he doesn’t know? If he is intelligent, he won’t. An intelligent investor always has a stop loss in place.

Understanding stop loss
Let us say you own a share worth Rs. 100/- at present market conditions. You expect it to rise in value, to a maximum of Rs. 110/-, at which point you will surely sell it, which, incidentally, is called profit booking (realizing the gain). In case the market moves against you, you don’t want to take too much loss on your investment. So, you place a stop loss order at Rs. 97.5/-.

The moment the share price touches Rs. 97.5/- (called stop price), your broker sells your share for you and gets you out with a loss of Rs. 2.5/-. (This is a simplified version; sometimes your broker may be unable to sell at the stop price and you may make a greater loss). Your net risk is 2.5% of your investment. Your net reward can be up to 10% of your investment. Does the risk seem to be worth it? It depends on how much risk you can stomach.

Consider a few more scenarios with the same analogy:
Current Stock Price: Rs. 100/-
Stop Loss: Rs. 95/-
Expected Profit: Up to Rs. 20/-
Expected Value: Rs. 1.25/-
Risking up to a 5% downside for up to a 20% upside.

Current Stock Price: Rs. 100/-
Stop Loss: Rs. 92.5/-
Expected Profit: Up to Rs. 30/-
Expected Value: Rs. 1.875/-
Risking up to a 7.5% downside for up to a 30% upside.

Current Stock Price: Rs. 100/-
Stop Loss: Rs. 90/-
Expected Profit: Up to Rs. 40/-
Expected Value: Rs. 2.5/-
Risking up to a 10% downside for up to a 40% upside.

Current Stock Price: Rs. 100/-
Stop Loss: Rs. 87.5/-
Expected Profit: Up to Rs. 50/-
Expected Value: Rs. 3.125/-
Risking up to a 12.5% downside for up to a 50% upside.

These scenarios I have outlined are basically the same as guessing in SNAP. I stopped at a gain of 50 since SNAP has only 150 questions and B has already answered 100 through his talent.

B has decided to place a stop loss at 97.5 because of his subjective assessment. A has placed his stop loss at 100 by not taking any risk.

If B thinks the cut-off is surely more than 100, he is definitely motivated to guess some answers. He either makes it, luck being in his favour, or does not make it, luck being against him. If the cut-off is above 100, A has no chance of making it, while B does. The risk-taker has been successfully rewarded by SNAP.

More worryingly, if B has access to past data regarding cut-offs in SNAP for various institutes, he can do a time-series analysis and estimate the current cut-off (if he knows what he is doing, he can do it with remarkable accuracy). Armed with this estimate, and depending on his score due to talent, he can determine the optimal level of risk he needs to take. And if he can eliminate one or two choices per question he guesses, with 100% certainty, his expected value will shoot up!

Thus, we see how the intelligent risk-takers are rewarded and the prudent non-guessers are punished.

Lesson: Negative marking makes luck play a more important role and worse still, luck helps only a few (the risk-takers), than all. A better way to eliminate the role of luck would be to give everyone an equal opportunity to use it: stop penalizing incorrect answers.

The problem in no negative marking
If guesses are not penalized, then everybody is incentivized to guess. No candidate’s score will therefore be an accurate measure of his talent! Is not having negative marking worse than the problem it is trying to solve?

No.

Consider the following table:

Type of candidate	Scores	Reflect
	Negative marking	No negative marking
A (No Risk)	Talent	Talent/More than talent
B (Risk-taker)	More than /Less than/ Equal to talent	Talent/More than talent

When there is no negative marking, both A and B have an equal opportunity to use luck. And their scores will definitely be at least equal to their level of talent. When there is negative marking, A’s score reflects his real talent while B’s score may not. Also, under negative marking, if you don’t know who A is and who B is, how will you judge who is merely talented, and who is talented plus a risk-taker? You can’t. Whereas, under no negative marking, you can more accurately discount the influence of luck of each candidate’s score. (If you do that in a negative marking scenario, you penalize A, and make his score go lower than that which he got by talent).

Sure, discounting the influence of luck (under no negative marking) may still harm those who were favored less by luck than the discounted value and benefit those who were overly favored by luck. But it is less imperfect than the current negative marking system.

(I have to make a confession here. There is a statistical concept called “Regression to the Mean” which can be used to take my explanation of luck further, but I haven’t understood it sufficiently well to attempt to invoke the concept here).

And so I say, since luck influences your chances of an interview call anyway, everyone should be given an equal opportunity to use it, and not just the risk-takers. There may be outliers who make it on sheer luck, but that is true in both negative marking and no negative marking scenarios.

Consider this example: while it is entirely possible that among a large group of music illiterates hitting 60 keys on a piano randomly, one may produce the starting notes of a symphony, the chances of him performing at Carnegie Hall are very, very slim. His lucky performance does not invalidate the ability of the talented performers. That is, a test should not be judged based on outliers. (There is a concept in statistics called smoothing, which tries to capture patterns while eliminating the effects of the outliers. While this has some defects, it holds good for analyzing data sets where the effects of noise (here, luck) on the actual data need to be minimized).

Also, it is more ‘fair’ to those MBA aspirants who just need to be good at their 3 fundamental skills in order to do an MBA, such as those who do not want to specialize in finance. They don’t need to cultivate the investor mindset for their profession; they shouldn’t be forced to do so just to take an entrance exam.

Moving on to CAT
If even a PP based test like SNAP suffers from a lack of objectivity, what of CBTs with normalized (scaled) scores like CAT and NMAT? NMAT has rightly decided to stop penalizing incorrect answers (and has allowed multiple attempts), so I stick to CAT.

Statistics is reliable only at the macro level. It doesn’t truly represent any individual sample. Let’s say you flip a coin twice in succession. According to statistics, you should get 1 Tail (T) and 1 Head (H) in any order (A more technical statement would be that the results converge to that). The possible outcomes are {HH, HT, TH, and TT}. Of these, {HT, TH} individually satisfy the condition. But {HH, TT} do not, although when taken together, they do. After 4 experiments, if all the 4 results (collectively called as the sample space) materialize, we will have 4[1H and 1T] as predicted by statistics. When this experiment is repeated a number of times, the ratio of H to T converge to 1:1, though you will find numerous strings of {HH} and {TT}.

What we understand from this is that while statistics ‘work’, they can still misrepresent an individual sample. A statistical scoring technique which is 99.95% reliable will tend to mess up the scores of 5 out of every 10,000 candidates. At over 2 lakh test takers, around 100 may get short shrift. And you won’t know if you’re one among them.

The more complicated something is, the greater the chances of failure. Do you trust an opaque algorithm to evaluate you accurately?

The problem is exacerbated with the presence of negative marking.

For CAT, where a 0.01 percentile could be the difference between an IIM-A call and no IIM-A call, how many deserving candidates are excluded because they ran out of luck? (I would not say the converse though, that many undeserving candidates got an IIM-A call because Lady Fortuna smiled on them. You need both talent and luck to get an IIM-A call, and frankly, you can’t differentiate between the two. And beyond a threshold level of talent, talent doesn’t really matter for your performance).

A better selection system would be a lottery for all those who passed a threshold percentile. Skip the illusion of objective selection, acknowledge the role of chance. It’s much more ‘fair’ to the candidates than the current system.

How does CAT compare students across time slots?
They use ‘similar’ questions to judge the caliber of the test takers in each slot. Based on this, all scores are brought to the same scale. I have my reservations on the use of these ‘similar’ questions, and how they are scored. I’m not critiquing it; since they keep their methods a secret it is not possible to critique it.

Consider two similar questions to understand my point.

Q1. Suppose I have a pack of cards, each of which has a letter written on one side and a number written on the other side. Suppose, in addition, I claim that the following rule is true: If a card has a vowel on one side, then it has an even number on the other side.

Imagine that I now show you 4 cards from the pack: E 6 K 9
Which card(s) should you turn over in order to decide whether the rule is true or false?

Q2. You are a bartender in a town where the legal age for drinking is 21 and you feel responsible for the violations of the rules. You are confronted with the following situations and would have to ask the patron to show you either his age or what he is drinking. Which of the 4 patrons would you have to question?

1. Drinking beer
2. 2. Over 21
3. Drinking soft drinks
4. Under 21

While both these problems are ‘similar’, the second question is much easier to answer. (The answers are to check 1 and 4 in both cases). Why? Because it is easier for us to associate with people than cards, though the logic is the same. My question now is: how will these types of ‘similar’ questions be evaluated?

How well do entrance tests test your 3 fundamental skills?
It can be argued that any question in math, reasoning and data analysis helps test for your math and logic skills, but what about language? Does the answer to this question really test anything useful?

A baby deer is called a ____.
(a) Foal
(b) Fawn
(c) Calf
(d) Joe

That apart, in many questions, the answers are subjective. The ‘right’ answer is anybody’s guess. This is also true of the decision-making caselets in XAT.

Imagine you are driving a motorbike, wearing a helmet, and at a red signal, you are asked by a traffic cop to produce your license. You don’t have it with you at that moment. Do you…

(a) Plead with the cop saying you have it at home and you would produce it if only he lets you go get it? (And actually do so if he lets you go).
(b) Stall and wait for the signal to turn green and just zoom away on your motorbike.
(c) Pay him a bribe which is less than the fine.
(d) Pay the fine.

Which is the ‘right’ decision to take? Clearly, it depends. If I am sure I could zoom away, I would. If I’m uncertain about that, I would either plead or pay a bribe (depending on whether my time or my money is more important at that moment). I would never pay the fine, if I could avoid it. Now, which choice should I tick? (The worst choice of all, pay the full fine, would probably be the ‘best’ one from an admission point of view though. Does ticking that tell the examiner anything about my actual decision-making ability? What are the odds that a person does in real life what he says he will do in an exam paper?)

A test of decision-making is probably the worst contrivable, save for the test of general awareness.

What’s wrong with testing a candidate’s general awareness? Surely a manager should know what’s going on around him. In fact, everyone should know what’s going on around them. Why not test for it?

I counter-question: How will you test ‘general’ awareness with ‘specific’ questions? Can specific questions test for general answers? A specific question can only test for a specific answer. Only open-ended questions can test your general awareness. Let me explain the difference with an example.

Specific question: What was India’s real GDP growth in the last fiscal year?

Open-ended question: What do you think of India’s real GDP growth in the last fiscal year?

The first type of question tests your memory; the second type tests your awareness as well as gives an insight into your understanding of the data. A rote learner can answer the first question, but not the second. A ‘generally aware’ person can most definitely answer the second question, and he may be able to answer the first one with approximate statistics. Whom does the GA section of the entrance exam benefit? The rote learner! The one who mugs up a CSR yearbook excels, a real thinker who doesn’t care for the exact numbers (which, in any case, are made up and manipulated so much that they may as well as be fictitious) is penalized. The insidious practice of rewarding rote learning over thinking continues well beyond school and under-graduation!

An even more saddening criterion
There are institutes whose placement eligibility criteria include test scores on ‘business awareness.’ That’s taking stupidity to new highs. I wouldn’t be surprised if they churned out rote learners by the dozen. I pity the thinkers who aspire to be real managers but make the mistake of pursuing their MBA in such institutes. I’m glad I’m not among them. (Full disclosure: I got rejected by one such institute).

All in all though, there isn’t much to criticize of the entrance test. It is the most objective among all the selection criteria (though still flawed).

To be continued....