Building Three AI Chatbots: One Organization’s Approach

Episode Notes

Transcript

Ashley Oborn:

As long as we keep in mind that AI is just another tool in our technology tool belt, then we can definitely see some great successes in advancing our impact.

Announcer:

Equal access to justice is a core American value. In each episode of Talk Justice and LSC Podcast, we’ll explore ways to expand access to justice and illustrate why it is important to the legal community, business, government, and the general public. Talk Justice is sponsored by the Leaders Council of the Legal Services Corporation.

Cat Moon:

Hello and welcome to Talk Justice. I’m Cat Moon, your host for this episode. Today, we’ll be talking about a chatbot project at a Texas Legal Services organization where they’ve been working on not one, but three AI chatbots. We have some folks here from Lone Star Legal Aid to tell us what they’ve accomplished and learned along the way. Lone Star has 11 offices serving 76 counties. They’ve handled over 25,000 cases in 2024. So you can see why they might be looking to technology to build capacity and remove some pain points in their workflow. Their AI chatbot project is supported by funding from LSC’s Technology Initiative Grant Program. I’m excited to dig in and hear about what y’all have done. So without further ado, I’ll introduce our guests. Joining me today, we have Ashley Oborn, who is the director of data analytics at Lone Star Legal Aid and Sourav Mohan data analyst at Lone Star Legal Aid and the lead developer on the chatbot project.

Welcome, welcome. I’m so glad to talk with y’all today.

Ashley Oborn:

Yeah, thank you guys so very much for having us.

Cat Moon:

Yes. So let’s jump right in because I have a lot of questions. I’m so curious. So let’s start here. You started working on this project more than three years ago, right? So what was the initial spark of the idea that got you started down this path?

Ashley Oborn:

Honestly, I was just watching the ChatGPT boom happen right before my eyes and I was like, “We could solve so many problems with this if we just tried. Let’s just do something with this. ” So I knew Sue was very good at Python coding, a little bit of a passion of his. So I wanted to give him something to fuel that fire inside and pet that passion. And instead of just the regular monotony of just doing the same reporting over and over again, I wanted to give him a little bit of a passion project on the side that would also potentially help us out as well. And that’s where our first chatbot Jiras came from. So Juris is our proof of concept, we like to call him. And we really wanted to use Juris as a way of not only giving Suru something else to do outside of the regular roles and responsibilities that he has, but try to use this technology while it was hot and live and see what we could do with it.

Cat Moon:

So I love the idea of giving time for a passion project and you were also trying to solve some problems, right?

Announcer:

Yeah.

Cat Moon:

So tell us a little bit about the problems you were trying to solve specifically.

Ashley Oborn:

One of the problems that everybody and really any community, but especially in our community, we’re always trying to figure out how we can get more bang for our buck. How can we increase our productivity? How can we solve for better efficiency? I think that’s something we’re already always trying to solve for here at LSLA, and especially in our team, is how can we make things more efficient? And for us, time. I don’t know about you guys, but I don’t seem to have enough time in any day, no matter what year it is. So one thing I know that a lot of people spend a lot of time on is finding different sources to help them figure out the ins and outs of what they need to be doing to provide services to their clients. So I wanted to figure out a way of centralizing all of that information because another passion of mine in particular is centralizing information and getting rid of all these different fragmented sources to create that efficiency and create that productivity for everybody.

Cat Moon:

So I absolutely can agree with not having enough time. I’m waiting for AI to create more time. Same. That hasn’t seemed to happen, but anyhow, we continue to be hopeful there. Well, you started to hint at this a little bit, but I’d like to build on this. So why you decided to build these chatbots yourself? So question of resources, you just wanted to be scrappy and get your hands dirty. Well, it sounds like you had the human resources in house to do this work. Talk to us a little bit about that.

Ashley Oborn:

I think it’s always a question of resources in this community. How can we best use what we have already to do the most with it? I knew Soro had a penchant for Python coding. I knew he was particularly skilled in it. We just hadn’t fully explored that avenue yet. And this was a great way of using what was booming in technology and wrapping it into something that could actually make some impact at work. So we’re always willing to do something and be a certain level of scrapping, get our hands dirty, especially if you’re trying to make a difference. I

Cat Moon:

Love that. Let’s jump into what these chatbots are and do. So can you break down for us at a high level, the three different chatbots and the superpowers that each of them have, what do they do for you?

Ashley Oborn:

As I already mentioned, Jurisy was our first. Juris is a legal research chatbot, and Juris wasn’t created to replace other platforms, but mostly curate the way our staff conducts legal research, curated specifically to them in our clientele. So gathering up secondary sources and pairing them with black letter law that’s already available on platforms such as Westlaw that we have access to is really the key there, but also verifying those sources and grounding them in subject matter experts expertise and what we really want our advocates to have access to and be referencing time and time again. So that’s Juris. And then I’m sure Sugar will go more into the different intricacies of all these different bots, like how we build trust in all of them. But we also have LSLA Ask, and LSL Ask is what I like to call our administrative assistant for everybody. It’s the workplace knowledge that everybody should have access to that’s been fragmented over the SharePoint drives, the S drive, the PDFs that someone’s hiding in an email box somewhere over there, the managers that have it only on print on their desk.

And then also those cross-departmental FAQs that have the same answer on repeat that you give time and time and time again. It’s a way of centralizing all of those resources with all of our internal forms and policies and procedures so that anybody can access that information that works at LSLA, of course, and really get that broken down in their own vernacular. Because I know I, in particular, this one was a pet peeve of mine. I did not understand our leave policy for way too long. This was for that purpose in general, in my head at least.

Cat Moon:

Solving a known problem. Yeah.

Ashley Oborn:

I think that technology should be used to solve a problem, but just one. Yes. Just one problem at a time. We also then have NAVI, and NAVI is our only external facing bots, and it is what I like to call a resource hub for information and referrals and resources for civil legal issues for Texans. And what we’ve done is we’ve gathered all these subject matter experts to help us generate what I’m calling lifecycles for each one of these legal issues. So from the very start to the very, very end of the journey, one could go on one of these legal issues, outlining that and putting it down in plain language so that somebody could come and interact at no matter what point they’re at in their legal journey and get information, get access to resources, maybe even get a referral to a legal aid or to a shelter or whatever social services are in their area because there’s also a geographical element that we’ve put onto this chatbot and really centralize all that fragmented knowledge again that’s scoured across the internet that law libraries have, that legal aids have and et cetera, and put it all together and be able to direct all of that traffic out to anybody that needs it without the hassle of knowing legal jargon.

Cat Moon:

Yeah. Centralizing that is really powerful, I think for users. And so you have built bots for internal use and for external use, but the idea is the same, right? You are trying to gather and centralize that data so it’s easy for folks to find what they need and get accurate information, even if they don’t know exactly how to frame what they’re asking about. And I think that’s one of the powers of this technology. It enables us to do that. But before you can create the bot, you have to gather all the information that the chatbot’s going to use. So what was the approach that you all took to gather all the data you needed to create the dataset or data sets I’m sure that the AI is pulling from to make these bots work?

Ashley Oborn:

So having our background, working in data, we have this saying and it’s garbage in, garbage out. So one of the things that I knew from the very beginning that we wanted to solve is we wanted to really curate and control the information going into these chatbots so that the information going in wasn’t garbage so that we eliminated how much garbage was going to come out. I think that logic kind of follows if most people are thinking of it that way. So really our thought process was gather the experts in their fields, get their input, get the people that regardless if we’re talking about legal experts or administrative experts, when we’re looking at different chatbots, gather all the experts’ expertise because they’re the ones that truly know these areas and listen to them, gather it, put it together, centralize this information, stop this chaos of trying to figure out where is everything and put it into one bin and make sure it’s that true source of vetted information first before you put any technology to it and then add some prompting techniques after that.

But really gathering up the vetted sourced information from the experts themselves, I think was our way of making sure we didn’t have any garbage going in.

Cat Moon:

Well, absolutely. The foundation that you’re building, the dataset you’re building on is critical to having good output, right? Exactly. And this definitely touches on something I think a lot of folks are curious about because many people across really the spectrum of the legal profession still have a lot of hesitancies when it comes to jumping into this kind of project. And so I think your project is really a role model for folks and one

Ashley Oborn:

Way- Oh wow.

Cat Moon:

Yeah, I think so. And I think one way helping people understand how to proceed appropriately, assessing the risk. And again, you’ve touched on this, right? You start with really good data, you get the expert input you need, make sure no garbage in. Is there anything in particular you all did with respect to gathering and creating the data sets for the internal facing chatbots versus the external? I think there’s maybe a little more anxiety or concern when it comes to creating this kind of tool for people who don’t have legal expertise, right? Absolutely. So they might not be as capable. They probably aren’t as capable of judging whether the output is actually good or not. They don’t have those filters that we have as trained legal professionals. So were there any differences there?

Ashley Oborn:

I think that my logic for the external bot definitely had a lot more steps behind it before I could get to this point, but it still is backed by the same thing I just went over. It’s making sure you have that expert input right from the beginning and really mapping out everything from A to Z. So you have all of these legal issues, all 99 of these legal issues, and finding experts for all of that was no small task. And sourcing all of that information, I think that’s part of the hesitancy a lot of people have with chatbots nowadays, is that they don’t know if they can trust the information. And I think that’s why we started out at this place of making sure we’re sourcing it from the true experts that are working in this day by day and are considered experts by others in their field on these very specific legal issues and starting with their input, dragging in all these other bits of expertise that are published all over the place and combining that into a true source because nobody really, truly knows where the true source is nowadays.

I think that’s really the key here. And that’s really where I was going to hopefully build trust in these kinds of bots and the accuracy of the bots and what people are getting out of them, because you can go back it up. It’s going to bring you to the true source of the resource you want to go to or refer you to the correct legal aid in your area based on the county you’re putting in. And this information is being sourced by the experts in the area. So building that trust from the get- go, I think is going to help with that hesitancy. And also we’ve also put in some prompting techniques that Cure can actually talk about.

Cat Moon:

Yeah. I would love to know because you have the trusted source as the data set, but then what you do on top of that is critical. So what does that look like?

Sourav Mohan:

The main thing to understand about large language models is that it wants to give you an answer no matter what, and a lot of times it wants to agree with you. So those were the things we wanted to combat. The main thing is prompting is very important. We give the chatbot an Out. If you cannot answer based on the sources that we give it, then don’t answer at all. Say, “Hey, I don’t know. The sources you give me aren’t good enough, so I can’t answer this question.” The other thing we do for our internal chatbots is we have citation that shows up right under the answer and they can now not only see the chapter, the section, the page number that it comes from, but also an excerpt that shows the actual source that used to answer that particular question. So that means a lawyer, when they ask a question, they just don’t have to blindly believe what they see in front of them.

They can go back and look right underneath and see, okay, so this is where the source is coming from and this is what the source is saying. I think that is absolutely the most important part along with the prompting. Prompting, we make it very specific. If you can’t answer, don’t answer. And we have to put it multiple times just to make sure that it doesn’t overstep that bound. So the whole point of everything we do is just to constrain the large language model. That’s all we’re trying to do.

Cat Moon:

Yeah. It occurs to me you’re giving the model humility, right? You’re giving it permission to say, I don’t know the answer to that question, which sometimes I wonder if lawyers might benefit a little bit from a little bit of that humility as well. But I can say that I am a lawyer. I can say that. Well, critical. So you’re building in safeguards and you are showing in the response the trusted source. And so that builds trust in users. Fantastic. Seeing how you architected that, I think is very helpful or hearing how you architected that is very helpful. So here’s something else that I’m confident that folks listening would like to know that would be helpful. So Ashley, yeah, you both have other major job functions. You’re doing a lot of things and you’re responsible for all these things while you’re working on this, essentially a side project based on how you’ve described it.

So I’m curious, how did you find the time to do all this? How did you make this happen and not only make it happen in terms of the work that you all chose to do, but then getting buy-in from the folks in your organizations and leadership and the attorneys and the staff who you wanted to use these platforms. What did that look like?

Ashley Oborn:

Yeah, we’re curious too, actually. No. I think really it goes back to how we started this whole project off that really helped us shape the buy-in from the beginning. So starting with that proof of concept, starting with Juris, debuting it at ITC, which is the Innovations and Technology Conference for whomever doesn’t know that’s listening in. That was our huge win for us strategically and planning this project and getting our work accepted by leadership within our own organization and for others in that matter. We at the very beginning were flooded with a lot of positive responses to this project. And this was our first national presentation. We were definitely feeling it. And honestly, that positive response, it hasn’t stopped. It’s been pretty overwhelming. As far as getting staff to buy in from LSLA to use it, I think that having those subject matter experts, a lot of them are internal to LSLA.

We’re a very large organization, at least in the legal aid community. And a lot of those subject matter experts are big hitters and they’re well respected at LSLA. So that definitely helps with that buy-in from those other peoples, that respect that they’ve already earned for themselves. Finding the time for this, I definitely don’t want to play around and act like this hasn’t been a hard one because I’m pretty sure I’ve had to add a lot more work hours to my day than we’re already there. But really, I think our method here was to stretch out the time length of this project so we could take on that extra capacity. We made this project a two-year long project after we presented Jiras and that was on purpose. We have full-time jobs and everybody else on this project has full-time jobs. So if we had all had all of our time to spend just on this project, it would’ve been done at least a year ago, but we wanted to be realistic.

We have to be realistic and especially if we want to make this project a thing, we had to be realistic from the get- go. And I think that’s really why we spread it out so much.

Cat Moon:

That makes complete sense. So I’m wondering if there are any secret tips you can offer to the folks who are in your domain, particularly in other organizations about how you fit this in with the rest of your day job.

Sourav Mohan:

I would say it was maybe not as tough just because my boss already allows me to do it, which is Ashley. So I didn’t have to really worry too much about if I wanted to allocate time to this, as long as I got my other duties done. So it was just more of a balancing act of just getting things done earlier than I usually do, just so I had more time to work on a project like this. So luckily for me, it wasn’t too bad on my end.

Cat Moon:

Well, I want to highlight some things I heard from both of you. One, Ashley, you highlighted how the opportunity to present this at ITC really made a difference in terms of building momentum and acceptance, and I think that’s important to identify. And then you both mentioned that really the support of your organization and the buy-in and recognizing that it was going to take time and giving you that time and space. So I think those are a couple of helpful takeaways for everyone listening who are wondering how they can make something like this happen.

Ashley Oborn:

Oh, absolutely. And then building an extra six months. We’ll

Cat Moon:

Say that. It always takes longer than you think it’s going to take. Absolutely.That is perennial advice there. Well, I’m a big believer in the potential of technology and specifically artificial intelligence to help us expand access to justice by scaling in a lot of ways. And I think your trio of chatbots is a great example of this, but I would love to hear from both of you your perspective on the potential here, because you’re both embedded in legal services. And so what do you see as a vision for the future with this technology and how it can improve how your organization works to help clients?

Ashley Oborn:

I have a mantra of as that as long as we keep in mind that AI is just another tool in our technology tool belt, then we can definitely see some great successes in advancing our impact. AI isn’t scary. It’s not over here to overtake all of everybody’s jobs. It’s a technology tool. And I think that’s something that a lot of people forget from the get- go and something that I really am passionate about making sure is at the forefront of everybody’s eyes whenever they’re thinking about a chatbot or using AI as some sort of tool to help their staff or make another impact of some sort. I think the community in general would be doing the service to our communities that we serve by not trying to use every tool that we have in our tool belt to help them get this access to Justice.

I mean, there’s, what do we have? A 92% access to justice gap. Just the other day I was doing a paper for a panel that I have coming up and an LA service area alone for us to meet the need of eligible clients or just to meet all eligible clients. For every one LSLA attorney, we would have to serve 17,000 clients for our service area alone. There’s no way we’re going to come close to that without using technology.

Cat Moon:

I agree. I agree. Well, I would love to hear, Sue, from you, from a technological standpoint, do you have anything to add on the potential for this technology to start closing some of these gaps?

Sourav Mohan:

I think the important thing to know about large language models is that it cannot take away the human elements. So it’s always there to just make your life easier, essentially. I have corrected so many things that large language model has given me because it’s been wrong a lot of the time. And so the idea that it can replace a human person in the whole transaction is, I think, a little overblown. Maybe I just don’t know enough. But from what I’ve seen is that what large language models really are is just predicting words. And I don’t think that’s enough to have a knowledge to answer the hard questions or do very hard tasks. But at the same time, it can be used to make everyone’s life easier so we can serve more people and improve our service area and stuff. So I think that’s what everyone should be focusing on because it is a very useful tool, but it is also limited in some capacity.

Cat Moon:

That’s an excellent point. And I think that builds on what Ashley shared, that it’s a tool and part of what we’ve got to figure out as we scale it is scaling it in the ways that truly help and not in the directions that don’t and not in a way that displaces the important human aspects of the work that we do, which some folks would view that as a challenge. I view it as an opportunity. And again, I think the project that you all are sharing with us, the three projects, the three bots serve as an excellent example of how you work within the constraints that technology gives you to address what I call actually the 92% failure rate. I put a very fine point on it. I’m working on that one. I think we have to name it what it is, and that should be a lot of motivation for us to figure out how we close that gap and bring down that failure rate.

All right. Well, this has been a fantastic conversation and I know that there is so much more we could dig into and our time to wrap up is approaching, but I want to make sure that the folks who are interested know where they can go to learn more about the project. I imagine that this is inspiring to many people. So I know that your ITC presentation is available on LSC’s YouTube channel, so we can point folks there. And you also shared a presentation and materials on the LSCAI Peer Learning Lab hub, and that is publicly available at lsc.gov/learninglab. Is there anywhere else that we should send folks to learn more?

Ashley Oborn:

I’ve got two more places for you. So you can go to justticebench.org, and that is Stanford’s Legal Design Labs R&D platform. So not only do they have case studies on our work, but there’s many others there, as well as our website. So we have not only a project page, but a blog that’s giving updates as regularly as we can. We’re trying to do monthly, but you can go on there and see our project updates, see where we’re at with everything. And just a little bit of Insider Scoop. This week, we are launching our two internal bots with all of our staff. Actually, Sue will be doing that presentation tomorrow.

Cat Moon:

That’s so exciting. Congratulations. Congratulations.

Sourav Mohan:

Thank you.

Cat Moon:

Well, this is amazing. Thank you both, Ashley and Ciri for giving us a look into your process and for sharing your takeaways that I know many folks can learn from about your chatbot projects. It’s been really, really fascinating and thanks to all the listeners out there for tuning in to this episode of Talk Justice. Be sure to subscribe so you don’t miss an episode.

Announcer:

Podcast guest speakers views, thoughts, and opinions are solely their own, and do not necessarily represent the legal services corporation’s views, thoughts, or opinions. The information and guidance discussed in this podcast are provided for informational purposes only and should not be construed as legal advice. You should not make decisions based on this podcast content without seeking legal or other professional advice.