GMAT Experimental Questions

January 10, 2007

It's an odd thing to quibble over, but there's never been agreement among GMAT professionals on one major part of the test. Depending on who you ask, between 10% and 25% of multiple-choice questions (both Quantitative and Verbal) are experimental. In other words: they don't count. You won't know which ones they are, and several questions on the test won't affect your score at all.

Understand that mine is just one opinion: I don't have any insider knowledge of the construction of the test. But, based on my experiences taking the GMAT, I think that the number of experimental questions is on the high side of the range I mentioned above. I wouldn't be surprised if at least 7 or 8 questions per section (about 20%) were experimental.

Why Does It Matter?

Does it make much of a difference whether four or ten questions per section are experimental? If you're a median-level GMAT scorer, it doesn't. However, if you're consistently testing in the 80th percentile or above, it does. To explain why, we need to take a detour to understand why the exam contains experimental questions at all.

As I explained in my introduction to the GMAT computer-adaptive test, each question in the database correlates to a certain ability level. We can refer to particularly tough questions as "700-level" questions, because test-takers who can get those consistently right will score at the 700 level or higher. The GMAT database consists of far more questions than you'll ever see on one test (or even five of them), to ensure that it feeds you questions designed to determine your specific ability level.

The difficulty level isn't ultimately determined by some psychometrics expert sitting in an office–it's decided by you. A 700-level question is one that about 8% of test-takers get right, and 92% get wrong. The whole system is doubtless more complicated, but that's all you need to know. It doesn't matter why 92% get it wrong and 8% get it right, only that those percentages are established.

How the Experimental Process Works

When a question is placed in the experimental pool, it isn't assigned a difficulty level. It pops up at random on GMATs, whether the test-taker is getting a 350 or a 650. Eventually, a large population will have seen and attempted that question, and the testmaker will have some idea of its difficulty. But that isn't all they're trying to do: they also want to determine whether the question is testing the same things that the rest of the GMAT is testing.

Here's what I mean. Let's say a problem appears as an experimental question on 1,000 administrations of the GMAT. It's answered correctly 200 times, incorrectly 800 times. In other words, it's about a 640-level question. Ideally, the people getting that question right are scoring between 640 and 800, while the people getting it wrong are scoring between 200 and 640. It's never going to work that way in practice, but the goal is to get close.

However, they don't always get close. A student of mine several months ago reported that he saw a question on his test that referred to a diagram–but didn't have a diagram! Obviously, it was a mistake. He took it in stride, realized it was an experimental question, and made his best guess. Now, some test-takers are going to get that question right; if everyone guesses randomly, 20% will answer correctly. Does that mean it's a 640-level question? No. Since the test-takers who got it right will run the gamut from the worst to the best among them, that's a question that the testmaker will throw out or revise.

Back to Strategy

That's a lot of explanation for a little nugget of relevance, but I'm fascinated by the process, and you ought to be too. The point to take away from this is that, on average, experimental questions are of average difficulty. If you score a 700 on the GMAT, most of the questions you see will be difficult. In fact, most will be among the hardest 10% of questions in the database. However, the experimental questions are fed to you at random, so they will not be as difficult.

Without experimental questions, here's what a sequence of ten questions in the middle of a test might look like, where each number stands for the difficulty level of the question:

• 650-660-650-640-650-660-660-650-650-660

It's pretty straightforward: this person has established a skill level and is sticking to it. By contrast, here's what it might look like if 20% of the test is experimental:

• 650-660-420-640-650-660-660-590-650-660

In terms of the experience of the test-taker, that's a huge difference. Naturally, someone who is scoring at the 650 level will find those 420 and 590 questions relatively easy.

Taking Full Advantage

Getting the experimental questions right doesn't make that person score any higher, but they do have benefits:

• They give you a break from high-stress questions that are at the edge of your ability level.
• Because an easier question, on average, will not take as long as a more difficult one, they allow you to save crucial time.

You'll only have access to those benefits, though, if you take the test with the right mindset:

• You know several questions are experimental.
• You're aware that you won't know which ones those are.
• You realize that they may be extremely easy in comparison to other questions.
• When you see an easy question, you recognize that it maybe experimental.

In short, you don't take your knowledge of the computer-adaptive test and allow yourself to self-destruct. Too many people see an experimental question and think, "This is so easy! I must have just missed a bunch of questions!" Even worse, they let their mind run with the thought, and continue: "I have no chance to score well on the test; I might as well give up now."

Don't fall into that trap. Having read this article and better understood how the test is constructed, you're much less likely to do so.

About the author: Jeff Sackmann is a GMAT tutor based in New York City. He has created many resources for GMAT preparation, including the popular Total GMAT Math and Total GMAT Verbal, as well as 1,800 practice GMAT math questions.

• Older: How To Use GMAT HACKS
• Newer: Your Guide to The Official Guide
• CAT Strategy: more articles
• All articles at GMAT HACKS