Generative AI is now capable of grading law school exams; what’s next?

Episode Notes

Transcript

Announcer:

Welcome to the ABA Journal Legal Rebels podcast where we talk to men and women who are remaking the legal profession, changing the way the law is practiced, and setting standards that will guide us into the future.

Victor Li:

Let’s talk about every lawyer’s favorite subject exams. It seems like every day there’s another threshold that generative AI crosses. First, it was able to take a bar exam and do reasonably well, then it was able to ace it. Same with law school exams. Right now, AI could probably graduate at the top of their class, edit law review and land a six figure associate’s job with an AM law 50 firm. And now comes another milestone. Generative AI can apparently grade law school exams with close to the same accuracy as a human. So what does that mean for law school professors? Should they be worried about AI taking their jobs, assuming they don’t already have tenure? What does this mean for law students? Where is this technology heading? My name is Victor Lee and I’m assistant managing editor of the A B ABA Journal. On today’s episode of the A b ABA Journal Legal Rebels podcast, we’re joined by Professor Daniel Schwarcz. Dan is a professor at the University of Minnesota Law School and he teaches insurance law and writes extensively about AI in the law. And he’s here today to talk about some of these issues and more. Welcome to the show, Dan.

Professor Daniel Schwarcz:

Thanks so much for having me.

Victor Li:

So before we start, I have to disclose this. Dan and I went to college together, Amherst College class 2000. So if we get tired of talking about ai, we can always talk about our memories of tap, Valentine’s dinner and nineties pop culture. Does that sound good?

Professor Daniel Schwarcz:

Well, yeah, though there are some memories from college that perhaps shouldn’t surface in podcast,

Victor Li:

So that’s probably fair. So anyway, I don’t think I’ve seen it since graduation. So how been?

Professor Daniel Schwarcz:

I’ve been doing great. I went straight from college to law school and after that I sort of was the pretty quickly into the legal academy. Only four years between when I graduated and when I started at University of Minnesota Law School. And so I’ve really enjoyed my career. I’m really happy with it and feel very blessed to be doing what I do.

Victor Li:

Yeah. So there is one thing about you that I remember from college. I remember I got a class with you, I think it was intro to econ, and I remember you would often know more than the professor or teach or bring up topics that even the professor would be like, how’d about this kind of stuff? And so I guess I wasn’t surprised to see you excelling in academia and doing so well. But I mean, even with everything that you’ve gone through, do you ever kind of stop and think about what things were back for you back then and how things have evolved for you since you’ve been at school?

Professor Daniel Schwarcz:

Of course. I mean, I thought a lot about actually going into graduate school and economics, and I really loved studying economics in college. And ultimately I had sort of heard all these horror stories about economics. Graduate school and law school really appealed to me because I figured that at the end of the day, law and economics obviously interlinked so much. I had been a debater both in high school and college, and so everyone I knew was going to law school. And then the sort of dirty secret that I only sometimes disclose is, interestingly enough, my father had been a partner at a major New York City law firm when I was growing up. And the same year I went off to college sort of very fortuitously, he went and became a law professor and he became this much happier person, much less tightly wound. And so I sort of saw that and I thought, well, maybe I can get the best of both worlds, go to law school, but be a law professor, which seems like a pretty good gig given the trajectory my father’s been on. And so I certainly think how could things have worked out differently? But at the end of the day, and there’s been a lot of fortuity in my career that I think luck is a huge part of how people end up where they are. But I’m very grateful.

Victor Li:

So lemme get this straight law school. Was the lesser of two evils for you as far as what you were considering?

Professor Daniel Schwarcz:

Oh, absolutely. And I have no doubt I actually loved law school. I have no doubt that I enjoyed law school way more than I would’ve enjoyed an economics PhD program.

Victor Li:

Gotcha.

Professor Daniel Schwarcz:

I think I would’ve seen an economics PhD program purely as a means to an end in a very economic, I guess that’s probably very economic. And then I really enjoyed economic thinking, but my math skills were good. They were not the top top that were out there. And I had that perception that I think probably true that a lot of times success in economics is correlated very strongly with math skills. And I saw my limit there, whereas in law it’s harder to sort of know where you stand. And so I figured, well, who knows? Maybe I’ll be able to excel beyond my limitations that I perceived at the time.

Victor Li:

Well, I looked at your background a little bit. So you teach insurance law, well that was your specialty when you first got the law school. So how did you get from insurance law to, so first of all, how did you get into that in the first place? Why did you choose that as opposed to some other topics?

Professor Daniel Schwarcz:

Yeah. Well, again, there was a lot of fortuity there. I mean, obviously I was interested in economics, and so there’s a lot of overlap between insurance concepts and economic concepts. But then really, I mean, I wrote my law school note about criminal law and I knew I was interested in the legal academy, but I had no idea what the field would be at clerked. And again, I got exposed to a lot of different areas that were interesting to me. I was thinking about writing about Fed Courts, but then when I went right after my clerkship to start at Ropes and Gray as it happened, I got placed in insurance litigation and my main client was Lloyds of London that I was working for. And I just got exposed to some really interesting issues from the sort of 9 11, 1 or two occurrence litigation that every law firm was involved in at the time, at one layer or another, or issues involving Elliot Spitzer’s investigation of Marcia McClellan.

And they had been rigging bids, which sort of was very shocking at the time. And so I got really interested in it because I thought the legal issues were fascinating. I thought it sort of played to some of my economic strengths. And this was probably the biggest thing I realized. I realized how massively important insurance was in practice, and I had never encountered it in law school. I didn’t take an insurance law class. And when I started my mento fellowship, which my fellowship where I was trying to become a law professor, I was strategic and I thought, well, look, here’s an area of law that is really important that very few people are writing on. And that seemed like a great sort of something I was interested in. It was something I had some minimal experience in, but better than none. And it was an area that I thought was incredibly important and consequential, not withstanding the fact that there was really very little attention being paid to it in the legal academy. So that was sort of how I ended up on that initial course.

Victor Li:

Gotcha. And then how do you go from that to AI then?

Professor Daniel Schwarcz:

Yeah, well that really dates back to 20 11, 20 12. And at the end of the day, a huge part of the insurance business is predicting, predicting losses, predicting fraud, predicting what’s going to be litigated. And so insurers were always very much at the forefront of considering using AI tools, machine learning tools to make those predictions. One of the obvious contexts, so much early machine learning tools really excelled, especially compared to sort of old insurance technologies that were based on human intuitions and limited data sources. At the same time, I had been writing and interested pretty early on issues surrounding insurance discrimination. I guess it’s a really interesting area because discrimination is actually how the insurance industry works in a lot of domains. I mean, insurers have to discriminate in order to price accurately, but at the same time, obviously discrimination of some types is normatively troubling. And so I had written and started writing about these types of dilemmas pretty early on in my career, like 20 10, 20 11, 20 12, just looking at what are the rules surrounding when insurers can discriminate when they can’t, how do those rules fall short or not and how can they be improved?

And so I started thinking early on about issues surrounding at first sort of more straightforward algorithms. So the big one there would be credit score insurers relied a lot on, not credit score per se, but a lot of the same elements within credit score to come up with an insurance score. And that’s been a controversial topic. And so I became interested in that. I started writing in that space, and then from there I started thinking, okay, well what about ai? And so that was sort of my earliest foray into thinking about AI and thinking about what types of regulatory challenges does it create? And I thought a lot about how can you implement the, let’s say, relatively rudimentary approach to in insurance in a context where insurers are using these tools that really depart from the historical approaches they’ve used and in which the regulatory approaches are based. So that’s how I got interested in AI initially, it was really from my interest in insurance law and from the fact that AI posed these really interesting challenges in that insurance space.

Victor Li:

And so when you first saw whatever iteration of chat to EPT was when it first came out, or maybe even got to see it before it was commercially released, what was your first thought about it? Was it like, oh my god, this is going to change everything, or we just kind of like it’s a fad, it’ll pass, or

Professor Daniel Schwarcz:

I was blown away. I was blown away. I did not have pre-release access. I was interested in ai, but I was really interested in AI in so far as it impacted insurance. And so it was just sort of one of several topics I had been writing on. I had an interest in, I had sort of rudimentary understanding of how it worked, but when I first saw it, I was totally blown away and just really interested in it as so many were. And then I just got, again, totally lucky. It was just a colleague of mine, John Choi, who’s fantastic, who I owe a great debt to, who got me interested in studying it empirically. He was the one who initially sort of it’s like, Hey, let’s see how this performs on a law school exam. And he was just looking for co-authors and given my interest in AI and how I was like, yes, this sounds like a great idea, let’s do it.

And so from that time till the present, I would say that AI takes up the bulk of my research agenda just because I keep encountering interesting issues and wanting to study them. I guess another thing to add there is I also was always very interested in just legal writing. And part of that stems from my fellowship where I taught legal writing when I was Alamento fellow from oh five to oh seven. And I’d also empirically studied how do we teach legal writing? How can we teach students to write well, what is the approach we should use in training our students? I’d always been interested in those issues. I’d written about them some, and actually again, fortuitously my wife has long worked in the educational space and higher education just in biology. So we had just talked for some time about that. So I was also just immediately interested in, okay, what does this mean for law students? What does this mean for legal pedagogy? What does this mean for how we think about training lawyers? And so those issues also, very quickly, once I saw the capabilities of the first public version of chat, GPT started coming to me and saying, I want to understand this better.

Victor Li:

Alright. Alright. So let’s take a quick break for a word from response there and we’ll get back to our talk. And we’re back. Before we talk about your research or the generative AI research, I did want to ask you about the intersection between AI and insurance. Obviously with lawyers who need all kinds of malpractice insurance and things like that. What are ways that they can navigate this sort of uncertain terrain? Because there isn’t a technically, I mean I know there are some AI insurance policies out there, but they might differ coverage or they might even differ in what the definition of it is. So how should lawyers protect themselves if they want to use this technology but they don’t want to get in trouble if something goes wrong?

Professor Daniel Schwarcz:

Well, I think the key thing is to look at the terms of your malpractice insurance policy. And I would say that right now the insurance industry is trying to figure out how to deal with AI risk. And there are two different approaches. The first approach really is not to create new coverage, but instead to clarify the coverage associated with AI risks within existing policies. And I would say that that is almost certainly my prediction about what will happen and is happening in the legal malpractice insurance setting. And I think that’s true. I think at the end of the day, most of the risks associated with AI that can result in malpractice are not different in kind than historic concerns with malpractice. So they may impact premiums, there may be some specific terms, but it would shock me, and I’m not aware of any malpractice insurance policies to say, well, we’re not going to cover review if you get sued or in trouble for connection with ai, whether it’s confidentiality problems or hallucinated citations.

I suppose the hallucinated citations thing is a little bit different because some of the concern really is not actually tort liability. The concern is reputational concern, reputational risk, maybe even court fines. And those very well may not be covered by a liability insurance policy. So there are some risks that are not risks that would be covered by a traditional liability policy, but to the extent you’re dealing with say, malpractice lawsuits, what I would assume though I would ask people to confirm is that their policies don’t contain AI exclusions. And that’s pretty common in a number of domains. So a good example would be cyber insurance. Most cyber insurers are not putting in AI specific exclusions in their policies, in part because the risk is not fundamentally different and in purpose, it’s really hard to identify what role AI played in, say, cyber incursion. You’ve been hacked, you don’t necessarily know how much your hacker relied on chat GPT to generate the code.

And I think the same is probably true for many legal malpractice claims. There may be some element of my lawyer relied on ai, but there may be some element of, my lawyer was incompetent in all sorts of other ways. And again, to me, a lot of those allegations are probably not different in kind right. Lawyers have always sometimes been lazy about thinking through cases and checking their source material and thinking about who they release information to. So at least with respect to lawsuits, tort lawsuits, I think that the risk is probably one folks are covered for and things aren’t going to change terribly much, even if premiums increase a little bit. I do think the risks are different in kind when you’re dealing with reputational concerns, when you’re dealing with court sanctions. And that’s an interesting question, whether an insurance market would arise in that setting, I don’t think it could for reputational concerns, and I don’t even necessarily think it could or would in the context of court fines.

So I think that is an important risk that folks have to manage. And I think that the key thing to do is actually not terribly difficult to check the veracity of your sources and to have pretty reliable mechanisms that are built into your systems whenever you’re using AI to ensure that you don’t have hallucinated citations. And again, to me it’s almost an overblown risk because I think it’s very easy to prevent. I think that it gets a lot of attention, so weird that you would just completely cite a case that doesn’t exist. But I think that the actual risk to most practicing lawyers is pretty negligible.

Victor Li:

I mean, we’ve covered this topic a lot in this show, and I think the consensus is sort of like, yeah, if you’re already a bad lawyer, AI is not going to make you a good lawyer, but if you’re already a good lawyer, it could make you a more efficient one and give you time and whatnot. And it kind of begs the question, if you’re saving so much time with your practice by using these tools, then why aren’t you checking the citations? You have plenty of time, right?

Professor Daniel Schwarcz:

Yeah, no, absolutely. Absolutely. So I think it gets a lot of headlines because obviously it is sort of weird that people would completely fabricate sources. That’s not something we’ve seen. But in terms of analytically what’s going on, I don’t think it’s that different than lawyers have always mischaracterized cases. Lawyers have always cited cases and said they stand for one thing and actually they stand for another. And so to me it’s just sort of an example of that, but even sort of more obvious to spot as opposed to different in terms of what the underlying mechanisms are.

Victor Li:

Gotcha. So let’s talk about AI’s capabilities right now. So how would you describe them? How would you describe AI’s capabilities? What is it good at, what is it not good at, and where is it heading?

Professor Daniel Schwarcz:

Yeah, so I’ve approached this in my research by doing randomized control trials. And in my mind, randomized control trials are really the best way to understand how AI is going to affect the legal profession. A lot of folks out there have been using benchmarks which make a lot of sense. You can sort of test how the models are improving over time and they’re a bit easier to administer. But the problem with the benchmark, which is okay, you give the AI essentially an exam, see how it scores is, it’s not clear to me what to extrapolate from that. I mean, the fact that AI passes theBar exam is great, but that doesn’t necessarily mean it’s going to be helpful for me. It may be that it takes me more time to check the AI’s work than to do it myself. It may be that it’s not good at collaborating with me or I’m not good at collaborating with it.

And so I’ve always been somewhat skeptical of the benchmarks which suggest these tools are very capable, as you mentioned in the introduction, and much more interested in what happens when a human lawyer is using AI compared to when a human lawyer is not using ai. So the research I’ve done in that space suggests pretty amazing effects on average effects, increases in productivity in the order of 30 to 60%. And that is both not only from increases in speed, but for many folks, increases in the quality of the work that they’re doing is judged by a blind grader. It’s not even right, it’s not even across people, it’s not even across tasks. And I do think it’s true that one, obviously as people gain more fluency with these tools, they get better at using them. Two, it’s probably the case that some people just sort of their workflows and work processes are more natural fits for using ai.

So I really like to use AI to clean up my writing because I generally think in a sort very step-by-step way, and I have a sense of what my argument is and what I want to say before I start saying it. And so in that setting, I find AI very useful because I don’t have to worry about fine tuning every sentence and making sure every sentence is perfectly grammatically correct. I sort of have a dialogue with it and I find that very useful. But then there are other tasks and other people where it’s less useful. And I think we’re still learning that. In one of my experiments that we did, we saw less of a gain from using AI when it came to revising a non-disclosure agreement. And we had various hypotheses for why that might’ve been. Part of it might’ve been that our subjects were law students since they didn’t really know how to use an NDA in the first place or draft an NDA in the first place.

But those aren’t really skills one learns in law school. Part of it may have been that we actually gave them a model form to use, and so they didn’t need an AI to actually use that form. Well, relative to an AI and the folks who just took the model form and put in the relevant information and adapted it to the situation actually worked out fine. So we’re still learning, and I don’t think that they’re going to be simple answers about, okay, this type of person, this type of task, AI is optimized versus not. It also just depends, I guess on other variables what type of AI you’re using. So there’s just so many variables that it’s hard to give definitive conclusions. But I think if I’m going to say what overarching, what my research suggests, it’s that across a wide spectrum of tasks, across a wide spectrum of lawyers using AI significantly increases both the speed at which you can do work and the quality of that work in many domains.

Victor Li:

Gotcha. So let’s talk about the current paper that you wrote, the one about the grading. So you talk a little bit about that. How did it come about? And also you had mentioned before a professor, it might even hamper them a little bit. They have to go back and check what the AI did. How can professors use AI then to help them grade if that’s what is heading and that’s what they want to do?

Professor Daniel Schwarcz:

Yeah, absolutely. So this is another one of those fortuity where I had been interested in this topic for a very long time because back in, I guess it was 2017, and I think I alluded to this earlier, I had written a paper looking at the effect of providing feedback to law students and showing that when professors provided individualized feedback to law students, not only did they do better in that class, they actually did better in their other classes as well. And I sort of had an intuition that that might be true because what you’re doing when you’re providing students feedback is you’re not just teaching them about tort law or contract law, which are both classes I’ve taught, but you’re actually teaching them about how to write a law school exam and more generally how to engage in legal reasoning. And those are things first semester law students struggle with, they struggle with what does it mean to identify an issue they struggle with, how do you sort of distill the relevant rule they struggle with?

How do you explain the rule based on the facts of a fact pattern and analogize and distinguish all the things that we want ’em to do? So I have for many, many years always been providing individualized feedback to my first semester law students, which is of course just a difficult and thankless task, but I felt was worth it. And when chat GPT came out, I’ve just been continuously trying to see if better and better versions of the AI can help with this task. And for a while I was not getting very good results. Then I was starting to get okay results. And then it was really pretty recently that I was like, wow, actually I think we’ve crossed a threshold where this can provide really good results. So I was going along that track and I’ve been sort of talking with a variety of people about writing that paper.

And then meanwhile, I sort of encountered some of my co-authors in this project who were looking at this independently. So I think it was first Scott Hurst at BU who I sort of encountered through one group where we were chatting about, and he was saying he was working on this issue and he was working on it with Kevin Cope and Y Franken Re and Dane Thorley. And so we had sort of just been having conversations working on separate tracks. They had made some progress on it, frankly, I was sort of thinking about it and working on it on an independent track. And then we sort of decided to join efforts. We got Eric Posner to come in as well. He had been thinking about it and trying a lot of this on his end. And so that’s how that collaboration was born. And again, a lot of it, frankly, I owe a lot of credit to my co-authors on this one because as we discussed, I did not get that PhD in economics. And at the end of the day, while I think I have okay instincts about pretty good instincts, I guess about empirical strategies at a high level and types of questions that are amenable to empirical testing, to the extent I ever had those sort of more detail oriented quantitative skills, they have eroded. And so I’ve relied on my co-authors to help me with that. And really it’s been a great process and I’m hope and expect that we’re going to continue working on it.

Victor Li:

So obviously, look, it’s not going to take professor’s jobs, it’s not going to replace you guys, but how can a professor use a tool, if not chat GBT or something similar, a generative AI tool to help them be more efficient as far as either what you said, giving feedback or grading exams or things like that. How can they use that best to help with them to be more efficient the way lawyers might be more efficient by using these tools?

Professor Daniel Schwarcz:

Yeah, absolutely. So to highlight the bottom line results that we had, we tried to test a few simple approaches. Our goal wasn’t to optimize the grading. We wanted to try simple approaches that would be available to law professors. And what we found is that if you use an AI system where you’re providing it with the exam question itself, the student answer and a grading rubric, and you essentially prompt the AI to use these materials to fill in the grading rubric, you can get correlations between 80 and 90. I think our top was 93% of what the ai, how the AI scores the exam relative to how the human scored the exam. Now, what can you do with that? I think you’re right. I at least am not on board with just saying, oh, I’m not going to grade anymore. I’m just going to run your exams through an AI system.

And I don’t think my students would be on board for that either. But going back to what I was mentioning earlier about the importance of providing feedback to law students and frankly the importance of providing feedback to lawyers. I mean, I remember when I was practicing, I got very little feedback because the lawyers are charging some outrageous amount. They want to mostly, they don’t have the time say, well, here’s how you could have improved things. Here’s what you did. Well, so the main sort of application I see and I’m excited about is the idea of now being able to tell all students, look, you should take lots of practice exams, you should try lots of things, and I can give you a very good automatic sense of how good it’s, and that’s incredibly valuable. Now, what I want to take it and where I think the next sort of stage of research is going to be is, okay, not only can we use AI not only to give very accurate top level assessments of this is an A level exam, this is a B level exam, this is a C level exam, which is still very useful for students to just know that and then be able to sort of think through why.

But can we use the AI to provide more tailored feedback? And there are two different ways that might happen. So one that I’m very optimistic about, and I think we’re going to be able to test very easily, and I hope and expect I need to talk to my co-authors, but I’ve been pushing them a lot to start thinking about this as a next paper or next step is to what extent is actually, are we getting correlations not just in the top line grade and score, which is what we focused on, but to what extent are we getting really good scoring on the individual rubric level? Because that’s actually a very good way to provide feedback. If you have a very detailed good rating rubric, which is not one, not hard to create, and one, once you create it, you can just keep using it for practice.

So you can sort of over time just generate a huge library of these which I now have, then students can actually get a very good sense. Okay, here’s the issue I missed. Oh, here, I messed up this rule statement here. I didn’t actually link the rule to the facts well enough. I missed that this fact is really important in this way. I missed that this case that we studied is right on point. I missed this really important counter argument. And so you can get more tailored sort of feedback from the AI and exactly which components of the exam that you did well on versus not. And if you have a very good grading rubric, that can be immensely valuable and allowing students not just to see, okay, here’s how I performed to see here’s what I need to improve on the next level. And this is something else I’ve been exploring and thinking about with multiple co-authors or potential.

And again, I’m hoping that my current group of co-authors on this project will eventually want to do this. It’s harder though, is can we use AI to not only fill out the grading rubric quantitatively, but to provide really good qualitative feedback to law students? And again, that’s what I do when I have my students and they take their midterm, I give them the grade, I fill the grade number rubric, and then I provide several paragraphs of written feedback. Here’s what you did well, here’s what you can improve on. Let me explain it. See this thing over here that now I anecdotally think AI is already capable of doing this, especially if you’re using rubrics. But the difficulty is empirically testing it is much harder. And part of why it’s hard is that there are many, many different ways of providing excellent qualitative feedback. And so even if the qualitative feedback that the AI provides is very good, it may not at all match the qualitative feedback that I provide.

And so you can’t really use the matching technique of saying, oh, well here’s the qualitative feedback I provided to student X on this. Let’s get an AI to generate that, and then let’s compare them and sort of see how similar they are because they may not be similar, but the AI actually might be quite good, might even be better. And so that’s sort of the next frontier I’m really interested in looking at. But to answer the initial question, it’s very easy to use AI in this way. All you do is you upload your exam, you upload a grading rubric, and then you upload a student answer anonymized to preserve, and you should probably get consent from the student and you ask it to fill in the rubric. Now, if you’re doing it at scale, you need to use the API, and it’s not quite as easy, but it’s pretty easy. And actually I think even we’re debating or whether or not to even include sort of a step-by-step instruction on how to use the API to do this, but it’s not challenging. And if you don’t know how to do it, you can say as chat, GPT had to do it. I’ll explain it to you.

Victor Li:

Yeah, I think as a law student, I would’ve loved to have had that level of feedback, but it’s just like, especially back then, professors just didn’t have the bandwidth or they didn’t have the time, or some of them just didn’t want to do it, frankly.

Professor Daniel Schwarcz:

Well, that’s what I’ve encountered the last seven or eight years. I mean, I’ve been on the bandwagon of trying to get professors to provide individualized feedback for a long time,

And it’s very hard because this is not a fun thing to do when you have many students. I mean, frankly, still every year I regret the fact that I’ve sort of locked myself into having to do it. And I think, oh gosh, why have I championed this issue now? I really can’t back out of it. So I’m very sympathetic to all the professors out there that don’t want to do it. At the end of the day, I sort of think about it as, well, look, this basically takes me a week of time and I think one of the most valuable things I do for my students. So I think, okay, rip it off, rip off the bandaid, get it done. I only do it for my one Ls. I don’t do it for my upper level students because again, there are limitations and trade offs.

So the idea of being able to do this at multiple points for my one Ls to be able to do it even more for the one Ls who are struggling to be able to do it for my upper level students. I mean, I think we can just substantially improve the quality of the education we’re giving. Because this goes back to something I think many lawyers realize. A lot of law school education is reading and then going to class and listening and maybe saying a few things, and yet, what are the primary skills we want you to develop? It’s actually being able to write and reason on your own over complex problems, which is related to but very much distinct from those things that you’re being trained to do. And so the idea that we’re training our law students in the way we are, I mean, clearly it is substantially driven by the interests of law professors and law schools and the limitations we have.

And I’m not saying that that should be irrelevant. It’s clearly that should be relevant. Have we struck the balance correctly? I don’t think so. Certainly not at more elite schools where mostly the way that we’re competing to get students is frankly, unfortunately not by the quality of our education, but by our US news ranking, which is affected by who knows what. And to be frank, it’s even more important at schools where folks come in without the advantages and the skills that my students typically have. There are a lot of law schools where folks are struggling to pass theBar exam where folks don’t know how to write well when they come to law school, where folks don’t have the structured analytical thinking. And in those settings, it’s especially important to be providing consistent, repeated feedback. And yet it happens in some places, but in many, many places it does not happen.

So I really think that the capacity of AI to transform legal education, and also actually even I mentioned, alluded this earlier, even for young lawyers at law firms, I know many folks who are like, I stumble around and I figure it out, but I’m not getting feedback. I dunno if what I’m doing is good or not. I hand it in and then I see something slightly different and I make inferences about what the partner wanted or whatever else, but it’s all very inefficient. So I think the capacity of AI to be used is a tool to provide feedback to law students and young lawyers about how to improve their skills is really, really significant.

Victor Li:

Gotcha. Alright. So let’s take another quick break for a word from our sponsor and we’re back. So let’s look forward a little bit. So where do you think this technology is heading and how do you think law schools should integrate it into their curriculum as far as teaching students how to use it and how to work alongside it?

Professor Daniel Schwarcz:

Yeah, so this is a topic I’ve been thinking a lot about. I actually have currently a project that’s underway that is trying to understand this better. So to take a step back in terms of how AI is going to affect legal education, for me, the real question is to what extent does relying on AI undermine the ability of people to learn? And to what extent does relying on AI undermine the ability of lawyers to perform their job when they’re not using ai? And I think these are really consequential important questions in my mind, they’re the most important questions in figuring out how AI ultimately will impact the legal profession, at least until we get to a point where AI can be relied on to autonomously, independently do legal work. And I think we’re so far away from that, right? I don’t think anytime soon, both for technological reasons and frankly practical and regulatory ones, we’re going to entirely offload legal tasks or legal decision making to ai.

So given that baseline assumption, the question is, okay, not only what are the productivity gains from ai, which we’ve discussed, but what are the risks? And I do think there are real risks that we don’t fully understand in terms of undermining the development of legal thinking and undermining the understanding of legal concepts. So to first give you an answer to your question and then sort of explain the research I’m trying to do, I’m still a firm believer that in the first year of law school, law students should not be using AI at all. And I am a firm believer that you need to have exams that are locked down or you are technologically and physically making it not possible or at least incredibly difficult to use ai. And the reason for that, it goes back to something we were discussing earlier, which is AI is an incredibly powerful tool if you’re a good lawyer, but if you’re not a good lawyer, you can use AI in ways and make terrible mistakes that you don’t even realize.

So I think it is incumbent upon us as legal educators to first get our students to the point where they have the capacity to think like a lawyer, that elusive idea. And I still think a lot of the techniques that we’ve used in our classroom, at least in the first year, work reasonably well for that, especially if we supplement it with the instructor using AI to provide feedback. Now I admit that’s a hard thing to sell, right? To tell students, I’m going to use AI to give you feedback, but you do not use AI yourself. And so I think navigating that terrain is difficult once you get later in your legal education experience. And here, I can’t tell you, is it two L years? Is it three L years? Is it the first year of was? I’m still trying to figure that out myself as so many people are.

Then I think it becomes important to really train students on how to use AI and then have them experiment. And one of the things I’ve realized was I’ve done a lot of trainings for law firms and for in-house legal departments and whatnot on how to use AI is there’s some things you can tell there’s an important, I don’t know, hour or two you can do on giving tips and tricks and perspective. But at the end of the day, once people have that, the real learning is using it yourself directly and experimenting with it. And I think that’s in part because there are no clear rules for using ai given the nature of the technology. Really the way you should use AI is the way you use a human lawyer in many ways, a human assistant lawyer. And so different people are going to have different strategies for that.

I mean, imagine sort of giving a training on here’s how you should use your associates if you’re a senior associate, there’s some things you can probably say, but a lot of it’s experience and it’s a lot of, it’s going to depend on the associate. Some associates, some senior associates might use junior associates in one way that works very well for them, and some associates may use it in a different way. So that’s sort of the bottom line answer to your question, which is you limit AI use initially in learning, and then you encourage folks to be using ai. You give them some level of training, but at the end of the day, you are perpetually encouraging them to be experimenting themselves. And so what I’ve adapted to doing over the years that I’ve been doing these trainings is I give my little spiel, I talk about some tips and tricks, and then we do some work.

We say, okay, let’s take this legal problem. Let’s first try using it without ai. Let’s use ai, let’s reflect on it. I think that’s the way you’ve got to do it. Now in terms of the research angle and better understanding the risks here. So I currently have a project ongoing, I was just working on it before we started recording, where we’re trying to empirically measure what are the risks of using AI to human legal reasoning. And the basic approach we use is again, a randomized control trial. And this I’m working on with my colleagues at University of Minnesota, Nick Bednar, David Cleveland and Alan Ibsen. And what our approach has been is we had a randomized control trial. We recruited about a hundred law students and we split them up into two groups, and we had them do a number of tasks that were interrelated. So sort of task one was sort of reading a very complex dense legal materials and sort of distilling out the key rules.

Task two was taking a sort of objective exam on it. Task three was then we introduced a factual hypothetical and we had them apply the law. They had distilled to that factual hypothetical, and we randomized them. And the only thing that varied is whether they had AI use in stage one when they were summarizing. No one had AI use in stage two on the objective exam or stage three, which was the application. So going in consistent with some of the concerns we had and some of the concerns I’ve actually raised in my own writing, my hypothesis, our hypothesis was that of course the AI group was going to outperform the non-AI group in the summary task task one, consistent with prior research, but that the results would switch in tasks two and three, and that the students who used AI in task one would actually, because they didn’t really understand the material at the same level of depth as the students who were forced to work through it without AI, would actually suffer on both the objective exam portion and the application portion. And so this was both sort of based on some supposition as well as research in other areas. There’s research, I think philanthropic just came out with research similar in the software area, and there’s research in other areas as well. And our results so far do not support our conclusion. So our results so far

Actually are quite inconsistent with it. We found that there was no difference in terms of objective understanding between the students who used AI in stage one and those who didn’t. And then most surprisingly, we actually found that on a number of measures, the folks who used AI in stage one to summarize and distill the legal materials outperformed on the application task, those who didn’t use AI in stage one. There are a variety of reasons why this might be, and I think probably the most promising reason is, look, if you actually use the AI to really sort of distill out the key elements and to organize them well, it’s much easier to apply the law versus if you have a fuzzy understanding of the law and you without AI have distilled out certain concepts that are not quite right or that are less well organized or don’t appreciate the ways that the elements of the rule fit together, then your application’s going to suffer as a result. But nonetheless, one, these are preliminary results, and two, there are still a lot of questions left unanswered, but to me, it complicates the question. We went in with the hypothesis

That we were going to see a diminishment in the capacity of students who used AI to work without ai, and we in a sense designed the experiment and the specific tasks to try to produce that, right? The application task really requires a deep understanding of the source materials, and as I said, that’s not what we found. So I think we need more research like this, but at least right now, to me, that makes me, it moves the needle for me some thinking that we should be discouraging use of AI because it can crack out human reasoning. I think in some ways, maybe AI can actually enhance it by making the rules clearer for us and distilling down what’s important. So to be continued, but that’s sort of how I’m thinking about it right now.

Victor Li:

Yeah, that’d be interesting to see when that comes out. Just let us know about that. I wouldn’t have thought that either, based on how you were framing it, I was like, oh, that’s unexpected. But I guess, is that a sign of just how far the technology has progressed that it’s able to do something like that?

Professor Daniel Schwarcz:

Well, it’s hard to say because really what we were testing, I mean, what we were most interested in is how these two groups of participants performed against each other when neither had ai. So in the application task, when they’re both sort of thinking through how to apply these rules to facts, neither of them had ai. And so it’s not clear to me that the results depend on how advanced the AI is. You might think that if the AI is super advanced, then actually that’s going to mean that humans think even less about the underlying. They’re like they plug it in, they get the perfect answer, and it’s done, and they don’t think about it at all. They have this great work product, but then once you take the AI away from them, they’re really in trouble. And again, that’s not what we found. What we found is that the folks who used AI in the initial stage were outperforming the folks who didn’t use AI even when we took AI away from them. And so I do not know how that finding interacts with the level of ai. The ai, the specific AI we were testing in that experiment was a pretty advanced one. Gemini 2.5 Pro. I mean, one feature of these experiments is you’re always testing outdated technology because of how quickly the technology moves, but because what we were testing wasn’t really about how much does the technology enhance your work when you’re using it, it was what are the downstream effects of using the technology? It is hard for me to say.

Victor Li:

Gotcha. Alright. Finally, if our listeners want to get in touch with you to I ask you questions about this or ask you questions about your research or even just about insurance law, what’s the best way for them to do that?

Professor Daniel Schwarcz:

Email me. So Schwartz at.edu, and the thing that’s very tricky is Schwartz is spelled in a very weird way. So instead of having the usual T towards the end, it has a C, but if you just remember that S-C-H-W-A-R-C as in Cat Z at.edu, you can email me or you can just Google me, Daniel Schwartz. I’m the only one with that spelling. And I’ll come up and I love talking to folks about it. I love going out and giving presentations to lawyers about this and trainings. I love hearing people’s own anecdotal experiences on any of this stuff. I mean, it’s part of how I learn. And I think one thing I’ve been trying to think about ways to do for a while is understand how this is working in the actual real world of law, how law firms are using it. It’s just very tricky because most law firms are willing to tell you things at a high level of generality, but then you say, well, give me access to your data. All of a sudden they’re less interested. And so doing some real world empirical testing on how the technology is affecting trends is difficult. But for that reason, just hearing, having conversations is incredibly useful for me. Yeah.

Victor Li:

Alright. That sounds great, and it was good catching up with you, and thanks again for doing this. I appreciate it.

Professor Daniel Schwarcz:

Yeah, absolutely. Thanks for inviting me.

Victor Li:

If you enjoyed this podcast and would like to hear more, please go to your favorite app and check out some other titles from Legal Talk Network. In the meantime, I’m Victor Lee and I’ll see you next time on the a b ABA Journal Legal Rebels podcast.

Announcer:

If you’d like more information about today’s show, please visit legal rebels.com. Legal talk network.com, subscribe via iTunes and RSS, find both the a b ABA Journal and Legal Talk Network on Twitter, Facebook, and LinkedIn, or download the free apps from a b ABA Journal and Legal Talk Network in Google Play and iTunes. The views expressed by the participants of this program are their own and do not represent the views of nor are they endorsed by Legal Talk Network is officers, directors, employees, agents, representatives, shareholders, and subsidiaries. None of the content should be considered legal advice. As always, consult a.

Continue Reading