Evaluating Legal Technology Applications

Episode Notes

Transcript

Law Technology Now

Evaluating Legal Technology Applications

03/18/2020

 

[Music]

 

Daniel Linna: Hello, this is Dan Linna. Welcome to Law Technology Now on the Legal Talk Network. My guests today are Marc Lauritsen and Quinten Steenhuis.

 

Marc is President of Capstone Practice Systems. Capstone builds custom document drafting systems for law firms, legal departments and other organizations, delivers training for users and developers and advises organizations on software selection and project design.

 

Quinten is a senior housing attorney and network administrator at Greater Boston Legal Services. Quinten practices law and represents low income tenants while also being responsible for IT systems administrations at Greater Boston Legal Services.

 

Quinten also develops open-source user facing legal technology and in addition to developing legal technology, Marc and Quinten last year drafted a research paper about evaluating the legal advice provided by legal technology tools entitled ‘Substantive Legal Software Quality: A Gathering Storm’ and that’s the topic for today’s show Evaluating Legal Technology Applications.

 

Marc and Quinten welcome to the show.

 

Marc Lauritsen: Thank you.

 

Quinten Steenhuis: Good morning, thank you.

 

Daniel Linna: Well, before we get started, we want to thank our sponsors. Thank you to our sponsor Logikcull, instant discovery software from modern legal teams. Logikcull offers perfectly predictable pricing at just $250 per matter per month. Create your free account anytime at logikcull.com/ltn. That’s logikcull.com/ltn.

 

And we also want to thank our sponsor Headnote. Headnote helps lawyers get paid faster, with their compliant e-payments and account receivables automation platform. To learn how to get paid quicker and more efficiently visit them at headnote.com.

 

Okay, well, Marc and Quinten, before we jump in, I just want to give you each a little bit of an opportunity to tell us a little bit about yourself and your background.

 

Marc, do you want to go ahead and please kick things off?

 

Marc Lauritsen: Sure, happy to do so, Dan. I have a checkered past, I’ve been a Massachusetts lawyer for over 40 years, but most of that time I’ve not been practicing, I’ve been doing other crazy things. I began life as a clinical teacher and a legal aid lawyer, and then I got diverted into technology way back in the 1980s.

 

So that’s been my main focus in the last couple of decades is building software, helping people learn how to do that, and in recent years I’ve been also moonlighting as a law professor teaching courses in which students actually build these kinds of applications as part of their coursework.

 

Daniel Linna: Great Quinten.

 

Quinten Steenhuis: So I’m one of those computer programmers who went to law school. I got a degree in Logic and Computation from Carnegie Mellon University, but my real passion at that time was Social Justice, and I probably started about a million different or was a member of a million different social justice organizations and that prompted me to head to law school, that was what I thought was the best way to further my goals in that area and I started my legal career at Greater Boston Legal Services.

 

So I fell back into software development kind of by accident after that. In the financial crisis, I volunteered to start spinning in our IT team, I still kept my legal hat as well and kind of added that second one and it took a few years for me to see the connection between the IT work that I was doing, the software development I was doing to kind of help make those systems more efficient and/or client-facing work. When I did, it really just makes a lot of sense. Expert systems can kind of fill that gap in legal aid. We’re turning away two-thirds of our clients who can’t get help, who actually come into our office and request it but there are a lot of people who can’t even make it to our offices because they don’t have money to pay for the bus fare, they don’t have ability to take time off from work or they don’t have childcare or they have disabilities that make it hard for them to reach just when technology is a way that we can reach those people that really answer directly.

 

Daniel Linna: Well, I’m excited to have you both as my guests today. You’re both doing some really interesting work and we’re going to try to focus, narrow the focus just a little bit although it’s a really broad area to evaluating legal technology applications, and I mentioned the article that you wrote, Marc and Quinten, and Marc, maybe you want to kick us off and just talk about, what was it that prompted you to write this article about the quality of legal applications?

 

Marc Lauritsen: Sure, I think it was really the intersection of the things Quinten just mentioned, which is the possibilities we have nowadays to express legal knowledge in software and the yawning gap between what people need in the world for legal help and what they get, and so there’s this organization called the International Association for Artificial Intelligence and Law that’s been operative for over 30 years. They have a conference every two years and it’s really the main gathering place where folks who were into that field trade ideas and discuss their latest research. So it occurred to us that we should really inject this topic into that community.

 

(00:05:00)

 

How can we assess and evaluate these kinds of tools that folks like us are inflicting on the world in terms of their quality, in terms of their substantive quality, their correctness, and it was meant as a way to organize our thoughts, cover the landscape of what’s happening in the world and to try to enlist the community of AI and law researchers in this quest to improve the quality of that kind of software. So that was our referent. We were lucky enough to get the paper accepted and Quinten presented it at the conference up in Montreal last summer.

 

Daniel Linna: How did that go, Quin, what about for you? Why did you think it was important to write this and what was the reception when you presented this paper?

 

Quinten Steenhuis: It was a really good reception, I think. People were very interested in the topic. It’s a group of a lot of computer science researchers who I think don’t always get to see the application of their work. So they were really excited to see us talk about how we were using these apps to solve real-world problems for people.

 

People had a lot of interesting ideas. I mean, I certainly learned a lot and I was kind of fascinated to see how many big ideas I hadn’t run into as a practitioner were bubbling around in computer science and the AI community. I hope to have more synergy there and take more of it back into practice.

 

Daniel Linna: Yeah, that’s exciting to see more-and-more that’s happening. Now just to be clear to what we’re talking about in the article, you’re talking mostly about the quality of rules driven systems like document automation and expert systems. Quinten, do you want to kind of give us some examples of the software tools that you’re talking about in this article and in this area/

 

Quinten Steenhuis: There are dozens of these tools that build these apps and a lot of them are aimed at people like us who are lawyers who kind of have decided to — I want to automate some part of my own work. So they’re unique and that a lot of them are built around kind of low code expectations.

 

HotDocs is a major player in this area and it has been for a long time Contract Express on the more expert system side of things than Document Automation, there’s Neota Logic, and Docassemble is an open-source platform that I use a lot in my work. Those are kind of some of the big players.

 

Daniel Linna: Okay, well, Marc, I know that you’re also doing work in the legal aid space with your class, but then also for law firms and legal departments and one of the things I’d like to see more in these discussions is sometimes we kind of silo legal aid projects versus law firm and legal department. I mean, to me it seems like more-and-more of these questions are the same ones, right, and so if we have principles for quality, and engineering these, and creating these applications can’t we come up with some generalized principles at least for starting point, that would really kind of apply across this whole space that we could be working together on?

 

Marc Lauritsen: Yeah, I think so, and that’s definitely part of my motivation because like you say I do sort of straddle the worlds of private practice in law firms and departments and their technologies and the kind of public-facing legal aid nonprofit world and the challenges are similar, I mean, and again, there’s a whole raft of different criteria by which you can judge these things in terms of their accessibility and usability.

 

This article really tried to focus on the narrower question of correctness, of substantive quality. It’s the knowledge we’re communicating, it’s the knowhow a pickle application is expressing true. Is it correct, is it reasonably accurate about how the legal world works, and that applies whether you’re in a law firm or whether you’re a self-represented litigant in California trying to defend against an eviction, so the issue is cross over.

 

As you mentioned, Dan, there are metrics that are used in other legal software, other kinds of software, Recall, Precision, etcetera. Those are critical but those really go to how well a particular piece of software does its job. For us that the focus of this article was how well does the particular piece of software communicate the know-how that the user needs in a way they understand, and that is a vast problem in my experience and it’s a problem both in public sectors and in private practice contexts.

 

Daniel Linna: Yeah, well, I think that, I mean, this is something I hear more-and-more now when I talk about technology and its ability to prove access to legal services for everyone and particularly to narrow the justice gap. I sometimes — well people will say, well, wait a second. How do you know that this technology tool is actually helping people? And it’s an important question, but yet at the same time, how do we know when lawyers are doing work, that work like how do we assess the quality of that work, and I mean, it raises a question of what’s the right baseline? Well, there are two questions there in my mind, I guess. Do we need to also start doing a better job of evaluating the work that human lawyers do and then what’s the right baseline for these tools?

 

Marc Lauritsen: Well, yeah, I think there’s — the good and bad thing about technology is that we’re automating something so we can see how it performs in the same circumstances again-and-again, and that can be either really well or really poorly. So you can really replicate a bad outcome over-and-over or you could have a really good outcome that comes out that way.

 

(00:10:01)

 

And lawyers of course are a lot more variable. From day-to-day the advice that they give might be very different separately as time changes, and if you are looking at a large pool of lawyers, you’re going to feel that there are a lot of different answers to the same question.

 

A benefit of a system like an expert system is you can look at the whole system at once and get a sense of what it’s doing, it’s really hard, a lot harder to do that with a live person, and hopefully you can apply that kind of analysis to see is it actually giving the right advice, is it helping someone on the right way?

 

Daniel Linna: Yeah, well, I think a really important point is that these tools allow us, we can run tests, right? We can do the exact thing you’re advocating for, even when it does get into the realm of using neural networks and things like that, people will complain frequently, well, it’s a black box, we don’t know what it’s doing, but it’s like there are ways you can still even test these black boxes and run a bunch of counterfactuals through them, things like that.

 

I do think though that one of the challenges we run into in this space is this kind of raises the point that if you want to create some of these tools you need to try to figure out, if the lawyers are doing it in a hundred different ways, then you want to try to put a Document Automation tool in place or create an expert system. I mean, how do you go through this process of kind of getting people on the same page on like this is a defensible way to handle these matters consistently?

 

I mean, Marc, I’m sure you’ve dealt with that many times and in many different types of organizations?

 

Marc Lauritsen: Yeah, it’s a real, it’s a perennial challenge in private law firms very often you have to lock people in a room for days in order to get agreement about where to put the commas and the spaces and the tabs in documents. And I think we presently have so much diversity in how people practice even within the same organization and when you try to substantively automate it, build an expert system or a document assembly system it kind of expresses the best practices of a group. You surface all those variabilities, and it’s the sociological challenge as well as a technical one. That was not much of an answer, but I think I’m just nodding to you, Dan to say that that’s a key challenge, we face both in the private practice world and in the legal aid public-facing access to justice world.

 

Daniel Linna: Yeah, to me the key point of recognizing that we can assess things for their quality, we can assess software, we can also assess the things humans do, well, then that means that we can look at all that variants and we can start asking and gathering data, it’s an empirical question which of these activities produce are high quality, produce value, produce better results, things like that, right? We can test these things just like we can test the software that we’re developing in tandem.

 

Marc Lauritsen: Yeah, I think you’re right. These things are assessable. I mean, in some ways they’re more easily assessable in eDiscovery, in context we have a discrete task and you have well understood metrics.

 

In the world of substantive software where you’re trying to communicate knowledge about very complex legal situations there are at least two problems. One is, you don’t know the impact it’s going to have in the actual real world when the litigant took them where it goes forth and does something, and secondly, there’s such a combinatorial explosion of possibilities in the software as to what it can do under different answer scenarios that it’s almost impossible to exhaustively check everything, and that was really one of our — one of our themes was how do you wrestle with that problem? You got this billions of possibilities about a way Pickler software can respond to a user, how can we be sure that in most of those situations if not all it actually does the right thing?

 

Daniel Linna: Yeah. Well, so having read your article and then we chatted a little bit beforehand, I guess what I’d like to do is try to as much as we can, and in the short time we have together help our listeners start thinking about a framework for how they might go through this process, and why don’t we start about, I mean, Quinten, if you can talk with us a little bit about, I mean, how much you classify the different types of errors that we think we might see in legal applications, because I think the first thing we have to do is think about what are the bad outcomes, where could the errors be and then we can start thinking about what are the countermeasures to try to prevent them?

 

Quinten Steenhuis: Sure, so we’ve kind of found that there was a framework we could use to talk about substantive errors, because obviously there are errors that have to do with bugs, your software application crashes, it doesn’t do anything and there’s all kinds of substantive errors that you can run into, and I think that first of all, there’s just being plain old wrong, giving the wrong advice that doesn’t actually fit what the law says.

 

It can be unclear, your application might be used in the wrong way, wrong circumstance, so it’s right advice for different circumstances that users are actually really has, and it can be obsolete, and that’s a problem with maintenance.

 

And one of the things that we realized and really thought through as we’re working on this paper, I think your own experience is that that lack of clarity is a problem you have to zoom in a little bit because something can be unclear both because it just is hard to read or hard to understand and it’s what is on the page and sometimes it’s what’s not on the page that can create some misimpressions with a user.

 

(00:14:57)

 

They might think that your system can do a lot more than it really can do, they don’t understand the scope and the limits of what it’s doing, and that can be done in the wrong direction.

 

Marc Lauritsen: Yeah, I think it’s useful to think at least in terms of two main categories; one is the interface that’s presented to the user, what questions are being asked, what guidance is being given, and secondly, the outputs, what documents are being generated, what customized advice might be emitted at the end and both of those can be problematic. It can fail to ask a question which it needs to know or a human lawyer would need to know in order to give a good answer or it could ask a question that’s irrelevant and convey this impression to the user.

 

So it’s got these kind of layers, the core of what is the quality of the content being communicated, and then secondarily, what is the likelihood of the user actually properly understanding that that content, so that it ends up in their head in a way that gets them to where they want to go and it’s — I keep saying this, but it’s a really numbingly complex problem that I think we only sort of touched on the outer limits of at the moment.

 

Daniel Linna: And so we should just probably clarify again too that this narrow focus on substantive quality, you’re not really talking about like the UX design for example, although that could have some interplay. So just a little bit more maybe just briefly on that about talking about that other kind of design that goes into putting these together and how that could affect substantive quality.

 

Quinten Steenhuis: Sure. So I think the app that I spent the most time building at Greater Boston Legal Services is covers about probably almost 300 questions, so I found a lot in the analytics that I’ve learned looking at where people kind of stop and give up in the interview. I can see where they — which screen they ended on if they didn’t finish it.

 

I think that the order of questions that’s one part of UX, but what’s the process that you’re guiding someone through, where you are presenting questions and what time, that’s something that makes a big difference on how successful someone else in using it. And definitely we would classify those kind of user experience questions as part of the substantive quality of the application.

 

Marc Lauritsen: Yeah, I think, I mean, the user experience aspects are critical for all kinds of reasons. Appropriately we focus on design. How do we construct an interactive experience for a user that is pleasant and effective, but in some ways this whole question of correctness is orthogonal to that. How do we — not only do that, which is hard to know, but make sure that what we’re saying in the application, what we’re implying by what we ask and how we ask, it also is correct, sort of this double-headed challenge to do both and neither are more important than the other but they are both critical.

 

Daniel Linna: Well, this is all really interesting Marc and Quinten, what we’re going to do is we’re going to take a quick break to hear a message from our sponsors and then when we come back I want to talk a little bit about best practices for building high quality software, a little bit about lawyer ethics as well and then we’ll kind of talk about how we need to be training lawyers for the 21st Century. So let’s hear a message from our sponsor.

 

[Music]

 

Advertiser: Hey law firms, getting paid is fantastic, but dealing with accounts receivable is such a pain, what if there was a better way? Enter Headnote, an industry-leading compliant e-payments and AR automation system. Their unique blend of features cuts through the noise and helps you get paid 70% faster. Skip the paper checks, spreadsheets and awkward calls due to overdue clients. Get paid faster with less effort. Visit headnote.com for more information.

 

[Music]

 

Advertiser: 10 years ago, eDiscovery meant lawyers packed into a basement, fumbling with complex slow software wondering where their lives had gone wrong. Today much of that frustration remains, but fortunately there is Logikcull, not eDiscovery, but instant discovery. Logikcull’s intuitive cloud-based software makes document search and review easy, fast and affordable. It’s time to get out of the basement. Create a free account instantly any time of day at logikcull.com/ltn.

 

[Music]

 

Daniel Linna: And we’re back. Thank you for joining us. We’re with Marc Lauritsen, President of Capstone Practice Systems, and Quinten Steenhuis, Senior Housing Attorney and Network Administrator at Greater Boston Legal Services.

 

So, Quinten and Marc, we have been talking a little bit about your paper, about legal applications and substantive quality of legal applications. Before we go any further, Marc, why don’t you tell us where our listeners could find that paper, just so we make sure we have that in the show?

 

Marc Lauritsen: Well, you can find it in two places. One, you can find it in the Proceedings of the International Conference of AI and Law, but if you don’t want to buy that or get it, you can go to the capstonepractice.com website and there’s a section called PDFs, and I believe it’s at the bottom of the list of PDFs you can download for free, and read our timeless wisdom.

 

(00:19:58)

 

Daniel Linna: All right, so let’s jump in now and talk a little bit, Quinten and Marc, and we could devote a whole day if not much more to this, but I mean, what would you tell our audience, what are the some of the things that you’ve learned, and as you’re developing this body of literature, what do you think ought to be our best practices so that we can improve the quality of the legal applications that we’re building?

 

Quinten Steenhuis: So I think one of the biggest areas of interest for me is this idea of translating law to code and back. So you have your expert that you’re working with closely, you might not even have a developer, sometimes those are the same person, sometimes they’re different people. You have to have some systems in place to really be able to do that effectively.

 

One approach that I think is interesting is to write out explicit decision tables, those are kind of a first-class object and things like Neota Logic, you can use them with Docassemble as well. But even if you don’t use those explicitly and they’re not actually the logic engine of your software tool, if you’ve taken that process and you have like a kind of an algorithm for going back and forth, that’s something that might be helpful.

 

There are other techniques along those lines too, but thinking about that is the key issue I think is really helpful and how am I doing that translation process.

 

Marc Lauritsen: I would add I think as a kind of a key best practices, is regarding these efforts, these projects to build these tools as genuine projects that need to be managed and you begin with the end in mind, involve people who have had prior experience doing them because as Quinten says you often have this mixture of domain experts who don’t know much about technology and perhaps some technology experts who don’t know much about the domain of law. And getting them in the same context working together is always a challenge. Sometimes you just need somebody who really knocks heads together from a project management point of view to make the system come out reasonably well.

 

Quinten Steenhuis: One kind of interesting model of the development only side is in Python there’s this living document called PEP 8, which kind of lays out the standards and techniques for developing a good Python application, one that looks good and it’s reading the bowline. When you start to follow and develop standards like that, it’s easier for someone to jump in and help with a project that might take it over or you as the developer to know what you were doing three months ago when you wrote the section that you completely forgotten.

 

So having standards is really important and I think we have to start to develop those in the legal software developer community, and we can learn a lot from the lessons a wider software industry has given us.

 

Marc Lauritsen: I think a critical point is, which I think is we find another software development applications is finding some method to externalize the content, the knowledge you want to express in your software. So don’t entirely put it into code but had some external representation, some sort of a decision tree or a table or some way to capture the rules that you’re trying to effectuate in your code. External to the code you’re building so you can test it against that framework.

 

Daniel Linna: You guys make what I think is a good point is that we ought to be developing some best practices and principles for the legal space but as you also noted there — I mean a lot of this is just stuff for good practices of software development, what part of this do you think is maybe somewhat of a unique question for laws but particularly thinking about the substantive question? I mean, what kind of additional principles do we need and how should we be going about developing them, how do we get more people engaged in this conversation?

 

Quinten Steenhuis: So we kind of have talked about some of the mitigation strategies that are important in the paper and we had kind of reprised at the innovations and technology conference as well, got some really great expertise from people in Michigan and Neota Logic to add into that, and I’m just looking at that list now and I think some of the things that are literally unique to substantive quality have to do with having clear scope limits. And then there’s this also this other aspect which is when someone is using an expert system, they treat it a lot different than they do something like a form book, right? They are going to read a form and they know that there’s some limits that they are understanding of what they’re reading and then they try to apply it to their situation. They know that there can be flaws in that process, but when something is interactive and it’s flying back and forth then that’s a little harder to see. So I think having some sense of attribution, discussions, and disclaimers, and cautions that help the user to understand that this is not an Oracle, right? I mean, there’s some limits to what the computer is doing, can help set expectations correctly. That’s a little unique to the substantive context.

 

Daniel Linna: Marc, anything you’d like to add to that?

 

Marc Lauritsen: Yeah, I mean, I think a basic point is we’re in this new phase of opportunity and danger, where we have enormous possibilities of communicating legal knowledge and helping people deal with legal situations whether they’re self-representing litigants or associates at a big firm, but there’s the correlative danger of the content being over-relied upon and accepted non-critically.

 

(00:24:57)

 

So you asked how do we improve this generally, I think getting more people to get involved in the various events and activities in this space, there’s international conferences, there’s the innovations in technology, conference, the legal services corporation produces each year, there’s the SubTech Conference that’s happening in Nashville this summer. So there’s plenty of opportunities, and of course, there’s the Twitter Watering Hole, I mean, anybody who’s interested in this space can probably quickly find fellow travelers, and there’s a vast literature that’s emerging. I just think it’s a matter of getting engaged and joining enthusiastically in this new world.

 

Daniel Linna: Yeah, well, it’s great to see that the work you guys are already doing and I know I’m sure you probably plan to bring some of this to SubTech this summer. I know this is some of the stuff that we’re doing here in North Western in connection with our law and technology initiative. So there’s a lot of work I sense to be done in this space. So I’m excited to see the work that you’re doing.

 

And let’s transition just a little bit in the thinking about what about lawyer ethics in AI, right? I mean, what are the things that are kind of top of mind for you when you’re thinking about — I mean we’re not just developing software and even thinking about the getting it right in the substantive side but in the background we have duty of competence, duty to supervise, duty to communicate to the client, confidentiality, in the private world thinking about reasonable fee questions and being sure we use technology when appropriate, I mean, what would you say maybe are kind of top of mind as far as thinking, gee, okay, well, we also have to think about the lawyer ethics questions here as well.

 

Marc Lauritsen: That’s a big topic, Dan, and I’m sure Quinten is scratching his head like I am. I mean, my first reaction is, we need to be sensitive to the fact that some of the stuff happens in the context of practice where a lawyer is engaged professionally representing clients, and the whole panoply of ethical expectations apply; confidentiality, zealousness, et cetera, and I think in that world there’s no question, but that an effective lawyer in the modern economy needs to understand the possibilities and the dangers of technology, and unless they do, they run the risk of practicing in a malpracticing way.

 

But there’s also this side world where lawyers may be involved like us in building applications that are not meant to be part of a practice of law. They’re explicitly meant as a non-law practice service delivery mode. And that’s a challenging world of its own, some Bar Associations would claim it’s unauthorized practice, some First Amendment absolutists like me would say, no, this is a freedom of expression thing, and there the rules are not so much what are the ethical constraints, but what’s really morally proper, what’s good for the world for us to do to help people get legal needs solved where there’s not necessarily confidentiality or the usual structure of a professional relationship.

 

Quinten Steenhuis: And I agree with everything that Marc just said and I think that the other side that you maybe we’re hinting at Dan is should lawyers be using these tools and lawyers that aren’t using them, are they doing an ethical disservice to their clients by not gaining the efficiencies that can come from technology.

 

I think that’s a very interesting question, I think every lawyer should be considering if there’s a way that their client could save their billable time by entering information at a form on their own and that’s what the lawyer is getting paid hundreds of dollars an hour to do instead. I think that they probably should be doing that.

 

Daniel Linna: Yeah, I’m starting to see more discussion about that question, right, is that the duty of competence then under Rule 1.1 and then tying it to Rule 1.5 and asking if technology tools are available that you could do something much more efficiently and then which goes directly to what you guys are talking about the substantive quality of the tools, right, I think sometimes we focus just on efficiency but these tools can also improve quality and outcomes then would it be unreasonable to do it in the manual way and charge your client for that.

 

Marc Lauritsen: Exactly, I mean, I’ve had many, many experiences where I’ll go into a law firm that’s been practicing in a certain area for a long time, developing municipal finance documentation, for instance, and they’ll hand us a collection of documents they’ve historically generated saying, hey, use these as examples and we’ll find errors, we’ll find problems where the wrong declaration is attached or the wrong municipality is named in a form and so in those situations substantive software can often reduce the errors and highly improve quality.

 

So I think that’s where you get into this question of — it’s not really a matter of so much only of efficiency but of quality assurance and deployed properly these tools can improve quality assurance but they can also introduce errors that maybe get ignored.

 

Daniel Linna: Can you talk just briefly about kind of like best practices, I’m thinking even like in a classroom setting but anyone developing them like how might you describe the quality assurance and the testing process?

 

(00:30:05)

 

What would be like maybe just some examples of the kind of things you think you’re kind of done building the product, now ideally you’ve had users engage the whole time, but what sort of quality assurance or testing might you suggest that a developer would go through to ensure that we have high quality from the substantive side?

 

Quinten Steenhuis: So I think that the more you can kind of isolate these legal roles and test them against facts, that’s something you can automate if you’ve done it correctly. So you build like a rule in Python or any platform, and you can just run different scenarios at it and see does it give the output of it, supposed to; that’s one strategy and that could be something that you’re automating on the command line, it can be automation of a web browser, there’s lots of tools that do that and almost all of the platforms I mentioned do work on the web. There’s really no excuse to not do that kind of automated testing.

 

At the same time that’s not going to capture everything, it’s only going to capture the things that you’ve thought of, so you have to do manual walkthroughs of your application, take different paths, and then you have to recognize the limits of that. Even when you’ve thrown hundreds of testers at your application, they’re not going to find all of the bugs. So you have to make sure there’s a feedback loop at the end where your users can come back to you and tell you this isn’t working right and then you have a mechanism to fix that.

 

Marc Lauritsen: Yeah, I think the idea is to comprehensively test throughout the process and as a rule of thumb allocate at least as much time for testing as for development if not more and just be humble enough to recognize you’re never going to find all the errors. So just this ethos of testing confirmation throughout the development process and once a system is ready to be tested externally throw as much at it as you possibly can try to imagine the most creative users out there who can do all kinds of unexpected things and constantly monitor and be ready to remediate when you discover problems you just haven’t noticed.

 

Daniel Linna: All right, so those are all some really good tips on kind of how to keep your users engaged and keep testing through the process, getting feedback, having avenues for users to give feedback when they encounter problems.

 

So given everything that we’re talking about what should we be thinking about? We see the increasing use of technology to deliver legal services, how should we be training law students and there’s kind of this debate should lawyers code, not code, things like that. I mean, I think you guys give an example of where lawyers can really be integrated in this process of developing these tools, but I mean, what are maybe even some of the other things that, I mean coding is — that’s a really simplistic even just to talk about that, how should we be training law students today for the future?

 

Marc Lauritsen: I think to start with there’s no question but law schools need to expose their students to these developing situations of technology doing legal work whether or not you were involved in learning how to build them you should know that they exist and that you’re going to be exposed to them as an emerging practitioner.

 

Secondly, I think it is — Quinten and I are both very enthusiastic about courses in which students not only are exposed but are engaged in building of making applications and — we’re not saying that lawyers need to code, some lawyers will learn by coding but by having some interaction with the craft of wrestling with software and building these tools, I think law students are well-exposed to how they work and how they can fail to work. So those two dimensions, exposure to the software that’s happening and some opportunity to be engaged in the process of building and fixing these things to me seem to be absolutely central to current legal education.

 

Quinten Steenhuis: The only thing I would add is that writing a brief is actually not that different from writing an application except that you need a much higher level of rigor. So I think that coding can be a way to help law students learn the law in a way that’s really deep and rigorous that really gives them the full picture, you need to understand all the different scenarios and with it you don’t necessarily need to when you’re writing just one legal brief on one topic.

 

Daniel Linna: We’re coming nearly to the end of our time but we’ve had such a broad ranging discussion I did want to give each of you kind of a chance just to give some closing thoughts about the importance of thinking about means of improving, the substantive quality of legal applications and just any kind of advice you want to give to our listeners about thinking about how they can contribute to this and either this movement generally or how they can build better software themselves, and Quinten, you want to start off?

 

Quinten Steenhuis: I would just encourage people to give this a try. It’s actually a really fun process that’s part of why we’re doing this and I think we’re bringing in this idea of making sure that you do it right, but if you haven’t even started trying to build an app, I encourage you to do that.

 

Daniel Linna: Great. Great. Marc.

 

Marc Lauritsen: Yeah, I would say that the threshold is lower than it’s ever been for folks who want to just dip their toe into this world to do so. There are tools like Community.lawyer and Documate nowadays that for very little money can get you into the space of building interactive questionnaires. So, if you’re at all interested, go forth and try it.

 

(00:35:09)

 

We’re in this era I think of law where we’re going to see increasingly active roles for technology and if you’re a lawyer or a would-be lawyer or a soon-to-be lawyer, there’s no excuse for not embracing this understanding and the extent it gets you interested actually doing it.

 

Daniel Linna: Well, great. So now before we wrap up I was hoping that you can each tell our listeners how to follow your work and how to get in contact with you. So sharing your Twitter handle and other avenues to get connected would be great? Quinten?

 

Quinten Steenhuis: I’m at @QSteenhuis or you can find me on my website, which is nonprofittechy.com.

 

Daniel Linna: Great. Marc?

 

Marc Lauritsen: Yeah, on Twitter @MarcLauritsen, and you can also reach me at [email protected].

 

Daniel Linna: Well, this has been great, guys. I really appreciate the work that you’re doing in this space. I hope we’ll have some opportunities to collaborate on this. This is really interesting. I think it’s important work to be thinking about the quality of lawyer work, the quality of legal software, the value that’s added with these tools, so thank you for all that you’re doing.

 

Quinten Steenhuis: Thank you.

 

Marc Lauritsen: Thank you Dan.

 

Daniel Linna: This has been another edition of Law Technology Now on the Legal Talk Network. Please take a minute to subscribe and rate us in Apple Podcasts and Google Podcasts. You can find me on Twitter @DanLinna.

 

Please follow me, re-tweet links this episode and join the legal innovation and technology discussion online. And join us next time for another edition of Law Technology Now. I’m Dan Linna, signing off.

 

[Music]

 

Outro: If you would like more information about what you have heard today, please visit legaltalknetwork.com. Subscribe via iTunes and RSS. Find us on Twitter and Facebook or download our free Legal Talk Network app in Google Play and iTunes.

 

The views expressed by the participants of this program are their own and do not represent the views of, nor are they endorsed by Legal Talk Network, its officers, directors, employees, agents, representatives, shareholders, and subsidiaries. None of the content should be considered legal advice. As always, consult a lawyer.

 

[Music]

Continue Reading