The AI That 'Passed' the Bar Exam: What the Headline Didn't Tell You | inicIA

Direct answer

In March 2023, OpenAI announced that GPT-4 had scored "around the top 10% of test takers" on a simulated US bar examination (Uniform Bar Examination), with 297 points out of 400 —enough to pass in every jurisdiction—. The headline traveled the world. A year later, MIT researcher Eric Martínez re-evaluated it in the journal Artificial Intelligence and Law: the comparison group was heavily skewed toward test-takers repeating the exam after failing it (a much lower-scoring group). Recalculated against all test-takers, GPT-4 landed in the 69th percentile; against first-time takers, the 48th; and on the essay section, the 15th percentile against those who passed on the first try. The lesson for your business: vendor benchmarks are marketing. The only test that matters is your real task.

The numbers, honestly

The verifiable figures in the case:

Martínez further warns that the bar exam is a poor proxy for a real lawyer's competence: the content is general, reflects no jurisdiction's doctrine, and multiple-choice questions do not resemble a lawyer's daily work.

Why this matters when you buy AI

Every vendor publishes benchmarks where its model "beats" the rest. The bar-exam case teaches three things:

  1. The comparison group is everything. A percentile without context about the compared group says nothing. Always ask: against whom, on what task, by what criterion?
  2. Passing an exam is not doing the job. A model can answer exam questions and fail in your business's real flow: messy data, exceptions, difficult customers, consequential decisions.
  3. Benchmarks are chosen to sell. Nobody publishes the benchmark where they lose. Your evaluation must be independent of vendor marketing.

How to really evaluate AI for your business (4 steps)

  1. Define your task in one sentence. Not "customer support": "answer questions about our plans and book the call with the right salesperson."
  2. Build 50 real cases. Pull 50 typical situations from your operation, including the hard ones: the angry customer, the edge case, the trick question.
  3. Test blind. Run the same 50 cases on two or three tools without telling your team which is which. Score with a written rubric.
  4. Measure in production for 30 days. The pilot with your real data and real customers is the only benchmark that counts. If it does not move your metric (sales, time, cost), the vendor's percentile is irrelevant.

Related articles

Frequently asked questions

So is GPT-4 bad?

No: passing the US bar exam with 297/400 is a real technical achievement, far above its predecessor. The problem was the headline's interpretation, not the model's capability.

Are benchmarks useless then?

They work as an initial filter: if a model fails basic benchmarks, discard it. But never as the final purchase decision. The decision is made with your cases.

What is the Uniform Bar Examination?

The standardized exam to practice law in most US states: multiple-choice questions, essays, and performance tests. Maximum score 400.

Does this apply to other industries?

Fully: medical exams, technical certifications, coding tests. Every vendor benchmark deserves the same healthy suspicion and the same test with your cases.

Take the next step

If you want to evaluate AI for your business with your real cases —not the vendor's benchmarks— talk to inicIA on WhatsApp: https://wa.me/17866120930

By inicIA