Using language models in legal higher education

I asked for ChatGPT, not CatGPT.

It is trite to point out that the launch of ChatGPT and the subsequent deployment of other language models such as Bard have created quite a stir in different fields. The capability of asking questions and writing prompts, and receiving a large chunk of text, has proven to be useful for many people, evidenced by the meteoric rise in adoption. One of the industries that has been impacted by large language models (LLMs) has been the education sector, particularly higher education.

Anecdotally, I can report that there is some level of anxiety and apprehension in academia, particularly with regards to assessment. I will not be dealing with that touchy subject in this blog post, as how to respond will be handled at an institutional level with the deployment of specific policies. Moreover, this is a complex issue that is way above my pay grade, I have opinions, but I will not cover those yet. I do want to share a few experiences that I have had during last term, as I tried to use language models in some of my teaching. I’m not expecting other colleagues to follow my experiments, in many ways my own practice has benefited from the fact that a lot of my teaching involves AI as a legal and regulatory subject, so it is easier to incorporate LLMs into my own teaching. Your mileage may vary.

When ChatGPT was released in November, it became clear immediately to me that it was going to be used by students, but I also started seeing potential benefits in teaching. I signed up and started experimenting myself, and I was impressed with the tool, but also started seeing the many shortcomings that it had. By asking questions on topics that I knew about, I became familiar with hallucinations, and outright fabrication. Asking for references to my own work, ChatGPT made up entire articles. So this was a limited language tool, not an omniscient mechanical brain.

So when classes resumed at the end of January, it was clear that academia was going to have to respond to this new challenge. The new term started with a class on AI, so I took it as an opportunity to start having serious conversations about LLMs.  When I first brought up the topic of ChatGPT,  it was clear that many people in the room knew about it, and some of those who did seemed taken aback by its mention. The initial reaction was filled with a combination of panic and feigned ignorance. “ChatGPT? What is that?” As I clarified that using AI tools wasn’t strictly forbidden (yet), the atmosphere changed, the initial shock wore off and more honesty surfaced. Students began to engage more openly, asking questions and sharing their thoughts and ideas about the possible applications of AI in their studies. Some suggested that they were using it to check their spelling and grammar, some were using it to summarise large and complex chunks of text, some were looking at definitions, some to prepare notes for seminars, and some to get ideas for their assessments.

One of the most important things that I felt I had to do was indeed to dispel the air of infallibility that language models were acquiring. I recall specifically one class in which a student was very vocal that ChatGPT was never wrong. So I made it a point to showcase just how wrong it can be. Luckily, I found a perfect example by chance. I have seminar notes for each class, sometimes we go through hundreds of cases, and I like to keep a Word document with the relevant facts, citations, and quotes from some of the most used decisions. I was going through my notes before one class, and I realised that I was missing a description for the important copyright case of Hadley v Kemp, I know the facts by heart, but I wanted to have notes written down for completeness. I could have Googled it, but I tried ChatGPT. Initially it didn’t know the case, so I gave the full case citation: Hadley v Kemp [1999] EMLR 589. Here is the response:

“I apologize for the confusion in my previous response. I found the case you mentioned, Hadley v Kemp [1999] EMLR 589. It is indeed an English copyright case that was heard in the High Court of Justice, Chancery Division.

In this case, the claimant, Mr. Hadley, was a photographer who took a photograph of the famous actor Peter Ustinov. Mr. Kemp, the defendant, used the photograph without permission in a promotional brochure for his theatrical production called “An Audience with Peter Ustinov.” The brochure featured the photograph along with other images and text.

The court ruled in favor of Mr. Hadley, stating that Mr. Kemp had infringed on the claimant’s copyright by reproducing and publishing the photograph without obtaining proper authorization. The court held that the use of the photograph in the promotional brochure was not covered by the defense of fair dealing and that it constituted copyright infringement.

The case of Hadley v Kemp [1999] EMLR 589 serves as an example of the importance of obtaining proper permissions and licenses when using copyrighted material to avoid potential copyright infringement claims.”

Bullseye! Anyone familiar with the case will know that this is a complete fabrication, the real Hadley v Kemp is a case involving the 80s band Spandau Ballet, and it’s not about infringement, it’s an important authority on joint authorship. I have found several other cases that contain errors, both big and small. In the case of Designer’s Guild v Russell Williams, ChatGPT continues to insist that it never went to the House of Lords, and was decided by the Court of Appeal. I could list many other mistakes such as these, and they provide excellent didactic value about the limitations of language models (and also help with the assessment question as students realise that these tools can be unreliable, but I digress).

There were other useful teaching uses of LLMs. Sometimes I would use ChatGPT and Bard to get instant definitions. Whether we’re dealing with jargon from various industries or complex technical terms, having a quick definition was useful, even if the definitions could be wrong, which can be in itself a good didactic tool. I also used it as a form of “vox populi”, if you need to know what the mainstream understanding of a subject is, LLMs can help to give that level of knowledge. Sometimes I used it to draft seminar questions precisely on the topic of AI, letting the students know that the question had been drafted by a language model. Sometimes I would compare how two models would respond to a specific question. Sometimes I would use as an ice-breaker if the class was quiet.

But personally, the most useful use of language models has not been in class, but during preparation. If you treat them as what they are, namely language tools, they are great at summarising. It is easy to forget that sometimes some complex pieces of legislation or case law can be dense to navigate, and having a tool that can explain things can be quite useful. In a busy term it has helped me to get on top of class preparation by helping me in drafting my class notes. If I’m struggling to break down a large document into chunks that I can use in class, language tools are great. There are also tools such as AskYourPDF where you can upload a document and ask questions of it. I’ve found tools such as these invaluable.

Concluding

There has been quite a lot of talk recently about a case in the US where one of the parties used ChatGPT to draft a brief, which proceeded to make-up a bunch of cases. Hilarity ensued. Language models are not factual, they hallucinate and make up stuff. But that doesn’t mean that they’re not useful, particularly used as intended, as language models. My own practice with them in class has been rewarding precisely because I have tried to utilise them as tools, not as sources of knowledge. I am satisfied with the results, maybe the novelty will dissipate, and students will roll their eyes just as when I make yet another Dad joke. I think that I will for sure continue to incorporate them in my own personal workflow, particularly in class preparation.

While I mentioned that I would not discuss assessment, I have found one benefit of using LLMs in class, and it is that my students learned early on just how inaccurate they can be. One thing that I have noticed while marking is precisely the absence of hallucinated cases and literature (with a couple of exceptions). If at least I have taught that one thing to this cohort, my work here is done.

Now, let’s ask ChatGPT what it knows about Donoghue v Stevenson…