Chat GTP-4 Could Pass the Bar Exam
The U.S. requires all lawyers to take what is known as the bar exam to become licensed to the bar of a U.S. state. Researchers recently put Chat GPT 3.5 through this rigorous test to see how well it performed.
Human applicants need to complete at least seven years of post-secondary education and three years at an accredited law school before taking the bar exam. It takes weeks and even months for an applicant to prepare for this test, and one in five tend to fail on the first try.
Chat GPT and the Bar Exam
Researchers at Chicago Kent College of Law, Bucerius Law School Hamburg, and the Stanford Center for Legal Informatics (CodeX) examined the Open AI GPT 3.5 model to see if it was capable of passing this difficult test. The researchers used the multistate multiple choice section of the Bar Exam.
GPT 3.5 ended up trailing humans by about 17%. In the Criminal law category, it was up to 36%. In the Torts and Evidence categories, however, Chat GPT 3.5 got a passing grade. It exceeded the baseline chance of 50% with random guessing. Without fine-tuning, according to research, it passed two bar categories. It managed to reach parity with humans on one. It exceeded the researcher's expectations.
Researchers now indicate the possibility that Chat GPT-4, which is said to come out sometime in 2023, will be able to pass the bar exam.
It's clear that this technology is readily evolving if it can get questions right on the bar exam, a comprehensive test for lawyers. We must remember that this technology is in its infancy, and there's no telling what is possible in the near future. This test is an indication of what is possible.
Full test results are indicated in the link below: