What generative AI means for the future of predictive analytics

Episode Notes

Transcript

Announcer:

Welcome to the ABA Journal Legal Rebels podcast where we talk to men and women who are remaking the legal profession, changing the way the law is practiced, and setting standards that will guide us into the future.

Victor Li:

Lawyers, especially litigators like to say that they never ask a question they don’t already know the answer to, but there’s plenty of unknowns out there, especially when it comes to how a case might turn out or how much it’ll cost. Predictive judicial and law firm analytics take some of that guesswork out of the equation. Guessing how a judge might rule on a certain motion in a certain type of case, or forecasting how much time and resources a certain matter might take up based on similar matters in the past, allows lawyers to make more informed decisions when it comes to pricing, budgeting, and setting fees, and brings a higher level of certainty to their practice. Thanks to generative artificial intelligence, predictive analytics tools are easier to use and deploy now giving lawyers even more incentive to use them. My name is Victor Li and I’m assistant managing editor of the A BA Journal. My guest on today’s episode of the Legal Rebels podcast is Nicole Clark. Nicole is founder and CEO of Trellis, a legal research and analytics platform. She’s here today to talk about legal analytics and generative AI and how lawyers can utilize them to enhance the way they practice law and run their firms. Welcome to the show, Nicole. Thanks for joining us.

Nicole Clark:

Thanks so much. Great to be here.

Victor Li:

So obviously I just gave the very, very quick version of your bio. Can you tell me a little bit more about yourself and your background?

Nicole Clark:

Absolutely. So I grew up in Los Angeles. I ultimately decided to go to law school in New Jersey when I started law school was in 2008 right before the markets collapsed. So it was an interesting journey post law school, but I always stayed in litigation and ultimately did a lot of civil litigation and a lot of employment, litigation, class action, wage and hour work. I found myself really in trial court constantly as employment litigation claims are really heavily litigated in the state courts and I really just couldn’t believe that it was so difficult to access state court information here. It was the largest court system in the world and yet each court was fragmented with its own separate website and database with really no ability to search and find out information about the judge that you were appearing before or more information about your opposing counsel and their other cases.

And so really that’s sort of where Trellis was born with my own frustration in trying to access what I felt was practical info that was important to our cases. So we started really in California, which is where I was litigating at the time, and we started collecting data from courts where I was appearing and it was really just going to be my own secret weapon. I collected rulings by judges and put a search interface on top of it where I could search judges’ rulings that I was appearing before and see how they had ruled in similar emotions or similar issues. That ended up being such a game changer for me that really we launched Trellis started collecting far more than just rulings. And so at a core that’s really what Trellis is. It’s a single source of truth for the state trial court system and this is where you can understand your judge, you can understand opposing counsel, you can understand the way legal issues are being litigated, you can see case outcomes. So there’s so many ways to really gain insights at the state trial court level. If you think about Lexus or Westlaw as being this great view into court of appeals and Supreme Court cases, we really focus on the practical court system, which is where 99% of cases don’t ever even make it to trial, let alone to appeal. And so that’s where we live in the state trial court system.

Victor Li:

Gotcha. Well because it seems like everybody I talked to and having worked on some of the stuff myself, everybody hates pacer. Everyone says it’s the worst thing in the world. Everyone rules the day that they ever heard of it, but when it comes to state court level, most of ’em will kill for something like that to be in place. Why is it so hard for state courts to have something like that in place?

Nicole Clark:

I think it’s a difficulty when you think about a true democracy here, you have it at every state level, at every county level is really maintaining their data separately, hosting it separately. And I think the fragmented nature of that system where there’s not resources in place at a sort of a collective unified level the way that even PACER has, which of course could be improved greatly, but there’s not even that connectivity tissue between different counties, let alone states across the nation. And so you have a system where they have limited resources, their only incentive is really to make the minimum available that is publicly necessary and they don’t have the same incentives to make this information easily accessible, to provide insights on top of it. And you have really fun the government level, you have the lowest bidders being bid for some of these court websites and so it’s really a hassle and we certainly have the incentive to go in and make it a better experience for everyone.

Victor Li:

Gotcha. So lemme ask you about, and I’ve spoken to people in the analytics field and whatnot, and I always ask this question, so how do you take some of the unpredictability out of it? Because at the end of the day, judges are human, they may have certain tendencies, but you can never account for that one case where they might see things differently or something might happen that you don’t anticipate and results in a ruling that maybe is 180 degrees from where they’ve been in the past. How do you account for that? Or is that just part of the game?

Nicole Clark:

Well, I think it’s a big part of what we try and do. So I like to think of court of appeals traditional research that lawyers do as sort of your academic research. What should the lobby, practically speaking, just like you mentioned, judges are human, they’re fallible and they make very different decisions. The exact same issue before different judges is often decided differently. We can see it in the data. And so what we’re trying to do really is shine a light on what actually happens on a practical level for your judge for this motion type and a particular legal issue in general, how have they ruled on that? How long do they take on deciding these cases? How much experience do they have in a particular practice area? Now is that to say that you look and you can see that your judge always grant summary judgment motions on a particular issue?

Does that mean he’s absolutely going to grant it for yours? Not necessarily. Right? There can always be judges, like we said, can be swayed by different pieces of information, but to have the understanding early on in the case this, let’s say it’s a case where you need to win a summary judgment motion, you’re a defense and that’s really the successful outcome for your client would be that well to know early on that this is going to be a possible a possibility or absolutely not. This judge is just not going to grant a summary judgment motion. You can really change the way you think about litigation strategy and you can certainly change the way that you set a client’s expectations on what success is going to look like and not have to manage expectations really on the backend when things don’t go as you hoped or as you planned, bringing the client in on saying, look, this is the data and this is what we’re working with and we’re going to do our best, but instead of bringing that motion, maybe we want to think about what it takes to get all the way to trial in this case or what have you.

So you can continuously sort of navigate your litigation strategy based on some of the insights that you can get at the court level. And that goes not only for your judge, the judge is a huge piece of this, but also understanding what your opposing counsel is likely to argue based on similar cases they’ve had in the past based on motions that they’ve written actually looking at their prior motions. I mean, that allows you to be proactive in a really different and strategic way from the start of the case instead of having to really be reactive once they file papers and you see what they’re trying to argue.

Victor Li:

And also I would assume that the time would also be an issue if you have a judge that takes two years to rule on a certain motion as opposed to average of one year. That would also be very important information for lawyers and clients to have as far as talking about fee setting or budgeting and things like that. Right.

Nicole Clark:

That’s exactly right. How long are you going to be dealing with this matter? And I think that goes in a couple different ways. One is, okay, let’s say we have to make it all the way to trial. What does that look like from start to finish? What’s the earliest we’re going to get to trial? But then there’s also other timing considerations for this particular practice area, for this matter type, what’s the average of any potential outcome, whether it’s a settlement or going to trial. But then also let’s say that the case is dismissed following a dispositive motion. Well what’s the earliest in the case that potential outcome could even be an off-ramp for the case. And so having those different timing milestones in mind to understand staffing, to set client expectations, and really to think through what it looks like in terms of the amount of cases that a particular attorney or associate could be handling at any given time and how long this is going to be on their plate. Those are all really important aspects.

Victor Li:

Gotcha. And so talking about judicial analytics as opposed to law firm analytics. So how would you differentiate those two?

Nicole Clark:

So judicial analytics is looking at your judge, right? That is a complete analysis based on every case that the judge has had before them. So you’re looking at their practice area matter types, you’re looking at their timing, you’re looking at the outcomes based on practice area and matter type for similar cases before them. And of course you’re looking at how they actually rule on important pretrial motions really across the board. And for that, an important thing is not just the judge in a vacuum and understanding that, but it’s really to understand where the judge fits in with other judges in the county and the state. How does their grant or denial rates differ? Are they an outlier or this is just the way this county rules in general judges sometimes cluster rule. And that’s an important thing to understand as well. So knowing where your judges and outlier can be an important piece of understanding is the judge part of what’s going to be the obstacle or is it this case type, this county, this venue?

That’s the obstacle. And so judge analytics, I’d like to think of as really important at the start of the case. In places like California for instance, you can actually request assignment to a different judge if you do it within the first 15 days of the case. You have to do it timely, but that can change the entire trajectory of a case. So having a hard data right then is huge. But then also throughout the lifespan of the case, when you’re thinking about what motions to bring, what’s going to be the best use of your client’s funds, what might lead to the best outcomes, and being able to see that you can really shift your litigation strategy, think about the motions that are going to have the most impact for your client. Now for law firms, this is where we step away from looking at the judge and making some of our strategic litigation decisions there.

We move to two ways to think about law firm. First of, I’ll talk about from the litigation strategy perspective, which is let’s look at this particular law firm and let’s look at all of their cases really across in the state trial court system. Something that’s never been possible before. And one of the reasons that’s never been possible is because beyond the fragmented system of thousands and thousands of different counties and thousands of ways that attorney’s names can be said, but also depending on the county, they’ll list either an individual attorney or a law firm as the appearing counsel in the case. But that makes it really difficult to look at a law firm level because what you end up needing to do and what we needed to do is really match attorneys to the law firms that they worked at at that time, which took a lot of work from extracting data out from motions that they filed and doing just a massive mapping.

So it’s a huge effort that goes into that. But this is where at a law firm level, you can gain some insights on them, you can see some of the same information that you’d look at for a judge, I want to understand their motion grant rates, I want to understand the types of cases that they have the most experience in. I want to see the judges that they appear before, most often are they very experienced before this judge? That’s useful. How about their trial outcomes? Do they actually take cases to trial? What’s happened in those cases? So there’s so many ways to look at this information strategically on the sort of opposing counsel side, but on the law firm analytics, there’s a couple other ways that were heavily used as well. So we talked a little bit about litigation strategy, but for the law firm analytics side, there’s also the corporate side, which is corporations use these law firm analytics, law firm intelligence to evaluate their panel council to have some hard data on how the firms are actually performing across the nation in a way that has been impossible for them to do on their own really.

And so this allows them not only to see, okay, which panel council is really helping to provide the most value, finishing cases faster, getting to better outcomes, but also is there a different firm that, let’s say I’m Uber that I should try because we have a random case in Kansas and our panel council doesn’t have experience there. Well, I can actually look and see what are the law firms that do have experience in that county before that judge with similar cases and really choose the right attorney for the right matter. So that’s another way that corporations tend to use the law firm intelligence. And then the final side is business development for law firms, particularly large firms that are looking for corporate clients to be able to dig in and look at a particular corporation’s litigation history, where are they getting sued? By whom do particular plaintiff’s firms target them on what types of cases?

So that law firms when they go in, they can really position themselves to a particular business, they understand their business needs, they understand who their former counsel was, they can look at how those cases turned out, and they can do more with analytics in order actually show visualizations to a potential client that demonstrate why they might be a better choice than other firms. So we’re also heavily utilized in the marketing and business development aside as well for looking at again, this court system that has been so opaque and so difficult to glean information from for so long.

Victor Li:

Yeah. So how many states are you guys in right now?

Nicole Clark:

We are now 46 states. We’re over 3000 courts. So we have been working really hard to really take as much of the nation as possible. And I can tell you the couple states and counties that are left are very small. So really we have the majority of the nation at this point and we’re very excited to offer that.

Victor Li:

Gotcha. Alright, so let’s take a quick break for a word from our sponsor and we’re back. Alright, so now let’s talk generative artificial intelligence. How has that changed things for you guys at Trellis and how have you integrated into your holdings?

Nicole Clark:

So I think generative AI is such an interesting piece and I genuinely believe that it is going to change the way that lawyers practice. And I think that it’s already starting now. I think there’s some areas that are easier to tackle than others on the corporate transactional side. I think in contracts there’s a lot that can be done there in a much easier format. I’d say the litigation side is actually much harder and it’s harder for the same reason that it’s been difficult for us to build this business, which is you’re talking about massive amounts of data that is unstructured. And we’ve done so much work in this data to structure it to classify all of the metadata of a case from a party to a judge, to the outcomes to the docket events, to the motions to the outcomes of the motions. Everything is classified in order for us to do the analytics work that we do.

And what that leaves trellis with is this incredible database of structured data that contains hundreds of millions of really attorney’s motions and briefs, their work product. So if you think about the entire trial court system as, and all of the motions filed in it as being this incredible database of work product that really no one has in the way that we have it at scale, we’re actually uniquely positioned to be able to provide generative AI tools specifically for litigators. It’s the court system that we know the best and we have all of this work product to give as samples to build models on as well as structured data to be able to tell models what it’s looking at. It really needs that pieces of structure to make sense. And we’ve done so much of that work. And so for Trellis, what we’re really focusing on is not requiring attorneys to become prompt engineers.

There’s a lot of work that needs to be done in order to create prompts and there’s a lot of inconsistency depending on the prompting that different attorneys can do or different legal support professionals. And so what we’ve really worked towards is how do we create one click work product for attorneys where they don’t have to be prompt engineers, but they can take the data at the trial court level really in their trial court litigation and they can understand a case from the very start. They can summarize all of the important emotions, they can pull out the important legal analysis for particular of a complaint and what likely defenses would be based on others, thousands of other cases in the nation, or they can look at a motion that needs to be drafted. And through our system they can see generated content that represents what thousands of other attorneys have drafted that have ultimately been successful on very similar fact patterns.

And so there’s so much that we have in our database that really unlocks a different level of work product and the ability to help litigators in their practice than I think anything else that is on the market right now. But I have no doubt that generative AI is going to slowly and then quickly change the way that all lawyers practice. I think we have a timing question, which is does this happen very quickly? Does it happen over a few years over more? But I think that I personally believe it will change the fabric of the way that lawyers litigate.

Victor Li:

Okay, you say you don’t want them to become profit engineers, but they would still have to come up with some kind of way to interact with the software. Show me Judge Smith’s decisions on motions to dismiss and employment law cases in this district and whatnot. I mean, they still have to know how to do that, right?

Nicole Clark:

The way that we’re thinking about it’s actually creating tools for them. So if you want to understand a judge better, that would be your judge analysis tool. If you want to create a timeline, that’s a timeline tool where all they do is really click the button for the work product that they’re looking for. And then of course, we’ve done all the work with our thousands of attorneys testing and testing. It takes so much work and iteration to be able to create the output that is really usable by attorneys and that’s what we’re working towards. So it’s not what we’re not thinking of creating sort of a chat interface. I think there’s a lot of great uses of that, but we’re looking to create really work product with adding more and more tools as we continue on.

Victor Li:

Okay, interesting. All right. And then how would that work for law firm analytics then? Would it be something similar or would you still have to have some kind of prompt there?

Nicole Clark:

It’s a good question. I think at this point it would be, the idea would be a tool. The idea would not be because there’s so many different ways you can think about asking different law firm information. That being said, because we have the whole database, because we have all of the analytics, there’s easy ways in to look at more and more information if they want to get more specific or target a particular motion type. But some of that can just be this is the law firm, this is the case. I can upload my complaint, tell me what I need to know about this law firm that’s going to be helpful for my case.

Victor Li:

So if I’m a longtime trellis user or a customer and I’m using the traditional tools without the generative AI capabilities, how much time would I save by using the gen AI tools that you’re putting in the trellis?

Nicole Clark:

I think time savings is massive. Now it depends on the tool. How long does it take you to take all of the documents, discovery complaint in a case and create a timeline of that? Okay, that’s one thing. But what about motion drafting, which I think we all know is both the most difficult, most time consuming and requires the most effort, let’s say. So for that, we’re talking days saved. One of the things that we’re really working on is how do we take sort of facts of a case and a complaint and then use that to look across thousands of other cases with the motions drafted in those cases and those arguments. And then how do we recommend the best arguments and that which would otherwise be hours and hours of manual research on our side of actually looking at motions and trying to understand and seeing if they were granted, right?

There’s so many pieces and it’s part of manual research that lawyers do on our site right now because the corpus of data is actually there, but the amount of hours that we can save in creating that first draft for them with the arguments that have been successful in other cases, the case law, and then of course it’s a draft, right? We’re not saying this is an N word product, but what we are saying is we can create the draft work product of this motion in a way that no other legal tech company out there right now is capable of doing for this court system.

Victor Li:

So I mean, we talked about the advantages that gen AI and whatnot. Let’s talk about maybe some of the drawbacks or some of the things you need to be worried about. So are hallucinations a problem in your area? Is that something that you guys have have to be concerned about?

Nicole Clark:

I think hallucinations are a problem across the board. I don’t think we’re necessarily immune to that, but I can tell you why we do better than some of the other products that are out there, which is we have more of a closed universe. We’re not asking it to come up with overall public information and return, but we’re asking it to look at our corpus of data within these documents across these cases and to pull from those. So when a model has information to pull from, it is far less likely to hallucinate than when you’re asking open-ended questions where it doesn’t have an answer. That’s when output comes up with hallucinations is when it’s not sure of the answer. And so it fills some stuff in because that’s what it’s supposed to do. Now we really have, I’m proud of the output that we have right now, and I am able to be proud of it because of the amount of testing that we have done on this output. But in general, we have less issues with hallucination simply because we’re actually pointing it to data instead of just sort of open-ended questions.

Victor Li:

Alright, so let’s take another quick break for a word from our sponsor and we’re back. So let’s look into the future a little bit. So what is next for you guys? Are you mainly concerned about trying to get coverage in all 50 states or are there other things that you guys are working on?

Nicole Clark:

So coverage for us has always been integral to the business that we have built, and I think we’ve done an incredible job and around five years we’ve taken most of the nation when everyone said it was impossible. So we’re really proud we have the best coverage that’s out there for the state trial court system bar done. That being said, sure there’s a couple of places left. We want to have everywhere simply because we want folks to feel confident and safe that if something’s out there we have it. We struggle with some of those counties that are entirely offline. So there’s still work to do there on the manual side, but I think at this point we continue to go after coverage, but we’ve come so far there that what we do is maintain coverage, data quality, and then we continue to build upon the insights that we offer.

There are so many ways that we can continue to build up. There are so many AI tools that we can continue to build. And so at this point, the question isn’t necessarily the data because we have the data. The question is what are the insights and value that we can continue to offer on top of that data? And I really think we’re just getting started there. We have some incredible things coming that I’m super excited about and I know our users will be excited about as well. But this is still very early and what models are capable of right now and the insights that we can layer on in addition to what we have so far, we have a long way to go to really be the end all, be all for the legal tool that trial court litigators need in their everyday practice. That’s really how many different workflows can we become a part of?

Can we change the way that lawyers think that they have to start from scratch? Can we help them get there faster? And then can we offer them insights that make them confident in the strategic decisions that they’re making as well? So we’re still early in what we’re offering. I know what we’ve done is incredible and I’m really proud, but I think we’ve still got so far to go in terms of being just indispensable part of attorney’s workflows. What I will say is we are also going after some additional data sets there We are at our core have always been really a data company. So we are also taking on PACER and we’ll be releasing the federal aspect of our site very shortly. We already have it available via API and I have no doubt that additional data sets are to come, including really we think of it almost as a data distillery.

There’s so many insights that are still within the data. Let’s say verdicts for example. We have verdict data, we have all of the verdicts, the documents that are filed, the jury instructions, but the amount of work that it takes to extract those numbers out so you can be able to classify and layer analytics and insights on top of them massive efforts. But that’s some of the work that we’ll be doing in the future is how do we extract out these pieces from this massive data set that we have to continue to offer really important insights to attorneys.

Victor Li:

Gotcha. Well, lucky for you, theBar is really low when it comes to Pacer. So as long as just improve a little bit, then people will love you.

Nicole Clark:

Well good, I’m glad that we can help with Pacer. It’s so funny because PACER is so much better than the state trial court system, but yes, it just shows you theBar that we’ve conquered already. Pacer shouldn’t be too tough for

Victor Li:

Us. So lemme ask you general in general about generative ai. Where do you kind of see that heading and what do you think it’ll be capable of down the road?

Nicole Clark:

I think the models are going to continue to get just much better. I think they’ll get less expensive and I think that what we imagine it’s capable of now, the work quality and product will continue to get better and better. I do not see a situation where litigators are ever replaced or not the architects of these tools in the first place. I think that there are so much nuance that the attorneys and the litigators really need to be at the forefront of looking at all of this data, looking at this output and deciding what makes sense and what to use strategically and what not to. And that’s really a human decision at the end of the day. But I think that the tools and the output and the drafts that attorneys are going to have to work with are going to continue to get better and better and better and change some of the work eventually that lawyers do from some of the rote menial tasks that need to get done on cases to really being the sophisticated aspects of cases now, I think that might change a little bit of the way law firms are made up and I think it may change the way they bill.

On the other hand, I’m really not worried about large law firms. We work with many of them about them figuring out a way to bill clients and still be profitable. I think they’re going to make it just fine, but I think there’s going to be incredible tools out there that are going to be more and more useful for attorneys and get past what we have now is a sort of hiccup of confidence where we’ve seen models hallucinate and we’ve seen attorneys get sanctioned and we need build up that level of confidence. And the way to build it up is for attorneys to work with it to see the levels that they’re at right now to use what makes sense now to always check, but at what point does it become where you have a higher level of confidence in the output in general. And that’s when true time savings will change practice when lawyers are far more confident in the output instead of having to double check everything at this point.

Victor Li:

Gotcha. You talked about big firms and whatnot, so for small firms, solos and whatnot, let’s say I’m a brand new lawyer, I hanging out my shingle, I’m looking to stop my own practice and whatnot. So how would you recommend that I utilize not just Trellis, but technology in general to help me become productive and also still be able to make a living while taking on some of these bigger firms?

Nicole Clark:

In a way, some of these smaller firms are really positioned to be able to embrace technology in a way that some of the larger firms have trouble with simply because they’re such large organizations and it takes a lot of work. You can’t be as agile when you’re an organization that large and change workflows constantly and really embrace technology in the same way I think that smaller firms have the ability to really embrace this technology. And what I mean by that, what I would say for them is at a core level they should be playing, testing, learning, and using generative AI right now, I would even recommend starting with just getting some familiarity with some of the large models, right? Creating an account for on open ai, creating an account on Anthropic and using Claude and start to play because that’s going to give them the knowledge, first of all to have a base output of an understanding of what do these models do right now, which they can then use when they’re looking at technology to say, is this something I could just do straight on Che GBT right now?

Or have they done some incredible work here, this legal tech company to make this even more usable for me? So having that baseline allows them to discern what technology is going to be right, and the more comfortable they get with it, the more they’re going to be able to see areas of opportunity to move quickly on them to create workflows around them. And they will ultimately have an opportunity now to have really a leveling of the playing field where they can get so much more done with technology that maybe larger firms had a massive advantage before. And I think larger firms that they don’t move faster need to watch out. For smaller firms that are more willing to embrace technology can move much faster ultimately, but getting comfortable with the technology from the start is just key.

Victor Li:

Gotcha. If our listeners wish keep up to date with the latest developments at Trellis, what’s the best way for them to do so?

Nicole Clark:

Well, the best way is really to go onto our website. That’s Trellis Law. If they’d like to start searching right away, get into the data Trellis law slash search. You can also follow us on LinkedIn and I am more than happy to answer any questions for folks. You can reach out to me directly at Nicole at Trellis Law.

Victor Li:

Great. Thanks again for joining us, Nicole. I appreciate it.

Nicole Clark:

Such a pleasure. Thanks for having me.

Victor Li:

Of course. If you enjoyed this podcast and would like to hear more, please go to your favorite app and check out some other titles from Legal Talk Network. In the meantime, I’m Victor Li and I’ll see you next time on the ABA Journal Legal Rebels podcast.

Announcer:

If you’d like more information about today’s show, please visit legal rebels.com. Legal talk network.com, subscribe via iTunes and RSS, find both the ABA, Journal and Legal Talk Network on Twitter, Facebook, and LinkedIn, or download the free apps from ABA, Journal and Legal Talk Network in Google Play and iTunes. The views expressed by the participants of this program are their own and do not represent the views of nor are they endorsed by Legal Talk Network is officers, directors, employees, agents, representatives, shareholders, and subsidiaries. None of the content should be considered legal advice.

Continue Reading