The AI That 'Passed' the Bar Exam: What the Headline Didn't Tell You | inicIA
Direct answer
In March 2023, OpenAI announced that GPT-4 had scored "around the top 10% of test takers" on a simulated US bar examination (Uniform Bar Examination), with 297 points out of 400 —enough to pass in every jurisdiction—. The headline traveled the world. A year later, MIT researcher Eric Martínez re-evaluated it in the journal Artificial Intelligence and Law: the comparison group was heavily skewed toward test-takers repeating the exam after failing it (a much lower-scoring group). Recalculated against all test-takers, GPT-4 landed in the 69th percentile; against first-time takers, the 48th; and on the essay section, the 15th percentile against those who passed on the first try. The lesson for your business: vendor benchmarks are marketing. The only test that matters is your real task.
The numbers, honestly
The verifiable figures in the case:
- 297/400: GPT-4's score on the Uniform Bar Examination (Daniel Martin Katz study, Chicago-Kent). It passes in every jurisdiction (minimums of 260 to 272).
- "Top 10%": OpenAI's claim, based on comparing against February Illinois test-takers, a group skewed toward repeaters.
- 69th percentile: the real position against all test-takers, per Martínez's 2024 re-evaluation.
- 48th percentile: against first-time test-takers.
- Bottom 10%: where GPT-3.5, the previous model, had landed. The leap was real; the headline's interpretation was not.
Martínez further warns that the bar exam is a poor proxy for a real lawyer's competence: the content is general, reflects no jurisdiction's doctrine, and multiple-choice questions do not resemble a lawyer's daily work.
Why this matters when you buy AI
Every vendor publishes benchmarks where its model "beats" the rest. The bar-exam case teaches three things:
- The comparison group is everything. A percentile without context about the compared group says nothing. Always ask: against whom, on what task, by what criterion?
- Passing an exam is not doing the job. A model can answer exam questions and fail in your business's real flow: messy data, exceptions, difficult customers, consequential decisions.
- Benchmarks are chosen to sell. Nobody publishes the benchmark where they lose. Your evaluation must be independent of vendor marketing.
How to really evaluate AI for your business (4 steps)
- Define your task in one sentence. Not "customer support": "answer questions about our plans and book the call with the right salesperson."
- Build 50 real cases. Pull 50 typical situations from your operation, including the hard ones: the angry customer, the edge case, the trick question.
- Test blind. Run the same 50 cases on two or three tools without telling your team which is which. Score with a written rubric.
- Measure in production for 30 days. The pilot with your real data and real customers is the only benchmark that counts. If it does not move your metric (sales, time, cost), the vendor's percentile is irrelevant.
Related articles
- How to know if your AI is working: the 5 KPIs that matter
- What to do when your AI agent makes a mistake
- How to choose an AI vendor: 7 questions before signing
Frequently asked questions
So is GPT-4 bad?
No: passing the US bar exam with 297/400 is a real technical achievement, far above its predecessor. The problem was the headline's interpretation, not the model's capability.
Are benchmarks useless then?
They work as an initial filter: if a model fails basic benchmarks, discard it. But never as the final purchase decision. The decision is made with your cases.
What is the Uniform Bar Examination?
The standardized exam to practice law in most US states: multiple-choice questions, essays, and performance tests. Maximum score 400.
Does this apply to other industries?
Fully: medical exams, technical certifications, coding tests. Every vendor benchmark deserves the same healthy suspicion and the same test with your cases.
Take the next step
If you want to evaluate AI for your business with your real cases —not the vendor's benchmarks— talk to inicIA on WhatsApp: https://wa.me/17866120930
By inicIA
