The Battle Against COVID-19: How eDiscovery Software is Playing a Role

Episode Notes

Transcript

Digital Detectives

The Battle Against COVID-19: How eDiscovery Software is Playing a Role

07/16/2020

[Music]

Intro: Welcome to Digital Detectives, reports from the battlefront. We will discuss computer forensics, electronic discovery and information security issues and what’s really happening in the trenches; not theory, but practical information that you can use in your law practice, right here on the Legal Talk Network.

[Music]

Sharon D. Nelson: Welcome to the 117^th edition of Digital Detectives. We’re glad to have you with us. I’m Sharon Nelson, President of Sensei Enterprises, a digital forensics cybersecurity and information technology firm in Fairfax, Virginia.

John W. Simek: And I’m John Simek, Vice-President of Sensei Enterprises. Today on Digital Detectives our topic is “The Battle Against COVID-19: How eDiscovery Software is Playing a Role.”

Sharon D. Nelson: Before we get started, I’d like to thank our sponsors. Thanks to our sponsor Logikcull, instant discovery software for modern legal teams. Logikcull offers perfectly predictable pricing at just $250 per matter per month. Create your free account at any time at logikcull.com/ltn, that’s logikcull.com/ltn.

We would like to thank our sponsor, PInow.com. If you need a private investigator you can trust, visit pinow.com to learn more.

John W. Simek: Today our guest is Maura R. Grossman, J.D., Ph.D. She is a research professor in the School of Computer Science at the University of Waterloo and an adjunct professor at Osgoode Hall Law School of York University, both in Ontario, Canada.

She is also a Principal at Maura Grossman Law, an eDiscovery law and consulting firm in Buffalo, New York. Maura is most well known for her scholarly work on Technology Assisted Review (“TAR”) and for her role as a Special Master in multiple high-profile Federal and state court cases.

It’s great to have you with us today Maura.

Maura R. Grossman: Thank you and thanks Sharon for having me on your show in these rather trying times. It’s great to be here and I hope you and your listeners are doing well and staying healthy and safe.

Sharon D. Nelson: Thank you. I think we all wish the same for all of our listeners and for another.

Maura, we’re pretty certain that the title of this podcast is mysterious to most of our listeners, can you tell us how eDiscovery software is playing a role in fighting COVID-19?

Maura R. Grossman: Sure. In the past decade or so, eDiscovery software that’s known as Technology Assisted Review or TAR has been used in large litigation matters to help lawyers to distinguish between relevant and irrelevant information more effectively and more efficiently than they could do through a linear manual review if they were doing just that.

Maura R. Grossman: My research collaborator and my partner Gordon Cormack and I thought we could apply the same approach to separate studies on COVID-19 treatments from studies on other things to help medical researchers to find the information they needed to reach sound conclusions about what works and what doesn’t work in treating this virus.

John W. Simek: Well Maura, we’ve known you a long time and you’re so focused on this, the eDiscovery software, but how did you become involved in using eDiscovery software for health-related research?

Maura R. Grossman: Well, in this particular instance Gordon and I were approached by a team of medical researchers from the Knowledge Synthesis Team at St. Michael’s Hospital in Toronto, and they were working in conjunction with several Canadian public health agencies including Health Canada and the Canadian Frailty Network to identify relevant research both studies that had been peer-reviewed and published in medical journals, but also those that were un-vetted preprints of studies that were in various repositories all over the web.

And one of the team members happily enough turned out to be a graduate of the University of Waterloo and was generally familiar with our work using machine learning and eDiscovery so he reached out to see there — if there was any way we could help their team and he didn’t know at the time, we’d actually already dabbled in using TAR for healthcare research in the past and I had recently been granted a fellowship from the University to do further research on the application of AI to healthcare. So it all came together quite nicely.

Sharon D. Nelson: Just amazing stuff. This blows my mind what you’re doing and I think it’s so great. What kind of feedback have you received so far from the medical researchers and how much faster does the research go when using your software?

Maura R. Grossman: It’s really been great working with the research team. They’re actually very happy with the results. What they did in the past was they would apply keywords to the abstracts of studies contained in these multiple huge databases to find potentially relevant studies. Then they would have a medical researcher review the abstracts of each study to determine if the study was potentially relevant, and if so they pulled the entire article which usually encourage some kind of feat and then the doctor would read the whole study to determine whether it met certain criteria and should be included in the meta-analysis.

(00:05:13)

This task is typically called Systematic Review and it normally can take a year or more to do, because the keyword searches are very over-inclusive, the databases are constantly being updated with new studies every day and not all of the studies are in English, but using TAR Gordon and I were able to assist the team in doing what’s called the rapid systematic review that took us about two weeks and so as you can imagine they were thrilled to be able to do the same work at a higher quality in a fraction of the time.

John W. Simek: You know Maura it’s — it’s interesting what you just described there sounds frighteningly like the early days of electronic evidence review, doesn’t it?

Maura R. Grossman: Yep, absolutely.

John W. Simek: So what role do you think that, that accelerating the identification of the medical research can play in finding a cure for COVID-19?

Maura R. Grossman: John, in a systematic review you’re not looking at a single study to reach a conclusion about the effectiveness of a particular treatment, you’re looking at many, many studies for a trend.

So imagine if you have 20 different studies and each of them show a very negligible treatment effect for a particular drug, each of those studies the results are not statistically significant. So if you looked at each study in isolation you’d conclude that that treatment isn’t particularly helpful, but if you looked at all 20 studies together and saw a modest treatment effect, you might come to a very different conclusion, that in fact might be strong evidence that the treatment has some positive effect. And that’s the purpose of a systematic review, it’s a meta-analysis of all of the studies you can find on a particular treatment.

So the systematic reviews of COVID-19 helped us to gather the studies from all over the world, some of the which had gone through actual peer review channels and publication, but because of the crisis a lot of them had not and we were trying to help the team look for trends and the results so that they could reach faster and empirically based conclusions about various treatments for this virus.

Sharon D. Nelson: Maura, I understand that the eDiscovery company Relativity is also engaging in what seems to be a similar effort, is it much the same as what you are doing or are there differences between the two efforts?

Maura R. Grossman: So my understanding Sharon is that after the White House Office of Science and Technology released a massive data set of COVID-19 medical research, they issued a call to the technology community for help in developing data mining techniques to help scientists be able to search that data for answers to questions about COVID-19. And what Relativity did was respond by offering the use of its eDiscovery technology to help facilitate that review. So in essence Relativity was offering the scientists the use of its software.

What Gordon and I did was a little different. We actually went out and found the potentially relevant studies from the multiple databases in multiple languages and forward the results of our searches to the researchers to actually just review and included in their meta-analysis or not. So they didn’t have to do any searching themselves.

We also redid our search everyday because we weren’t working with a single static database like this White House collection, our sources were located different places on the web, we’re constantly being updated.

So while the two efforts were similar in some respects they weren’t exactly the same.

Sharon D. Nelson: Thank you, that really makes it clear. I didn’t understand the difference when I was reading about it, so thank you for the explanation.

John W. Simek: Well before we move on to our next segment, let’s take a quick commercial break.

[Music]

Advertiser: Trying to cut costs, you are not alone. In today’s climate a five-figure eDiscovery bill per month is steep, don’t pay that. Use Logikcull to reduce expense and control your discovery process. Get started today for only $250 per matter and they will waive migration costs from competing platforms. For more information, visit logikcull.com/ltn. That’s logikcull.com/ltn.

[Music]

Advertiser: Does your law firm need an investigator for a background check, civil investigation or other type of investigation? PInow.com is a one of a kind resource for locating investigators anywhere in the US and worldwide. The professionals listed on PInow understand the legal constraints of an investigation, are up-to-date on the latest technology, and have extensive experience in many types of investigation, including workers’ compensation and surveillance. Find a prescreened private investigator today. Visit www.pinow.com.

[Music]

(00:10:08)

Sharon D. Nelson: Welcome back to Digital Detectives on the Legal Talk Network. Today our topic is ‘The Battle Against COVID-19: How eDiscovery Software is Playing a Role’.

Today our guest is Maura R. Grossman, J.D., Ph.D., a research professor in the School of Computer Science at the University of Waterloo and an adjunct professor at Osgoode Hall Law School of York University, both in Ontario, Canada. She is also Principal at Maura Grossman Law, an eDiscovery law and consulting firm in Buffalo, New York.

John W. Simek: Maura, before the break we’ve talked a little bit about Relativity and what they are doing, but I noted that they’ve listed four ways in which its technology could help with the fight against COVID-19, could you tell our listeners a little bit about those four ways and does your software do the same things as well?

Maura R. Grossman: So my understanding is that Relativity’s software performs four distinct tasks; the first thing they did was to deduplicate or to eliminate exact duplicates from the database, because researchers were complaining that they were coming across the same studies over and over and that’s a problem as you know we also see in eDiscovery where the same email comes up time and time again.

The second thing Relativity did was to tag the studies by language because the data set included studies from around the world and obviously you need people who speak the language to read the medical studies in that language.

The third thing they did was to provide users with the ability to search the database using concepts instead of just keywords. So if a study was about the same topic as another study but happened to use different terms, different keywords, it would still be found through a search.

And the last thing they did was that the researchers in that particular effort were very interested in finding and summarizing articles from Spanish journals that involved pediatric patients that did not have symptoms, and Relatively assisted the searchers in identifying those studies by focusing the search engine on the ages of the study participants so that they could zero in on those studies very quickly.

Our process was a little bit different because we weren’t again offering the researchers software per se, we were offering them a service that happened to use our TAR software. So we were able to deduplicate, we were certainly able to identify studies by language, but our task wasn’t to provide the researchers with a search tool to use for themselves, but rather to actually go out and go all over the web and to find all the studies that met very specific criteria that they had given to us so in one case it was, please find us anything on methods for preventing the transmission of COVID-19 in older adults living in long-term care.

Sharon D. Nelson: This is just fascinating stuff.

John W. Simek: Sure it is.

Sharon D. Nelson: I’m sure the listeners are going to be really immersed in this, especially since we’re also impacted by it. You know you’re using Technology Assisted Review or TAR to assist the medical researchers, how does the way you’re using the software now, how does it differ from the standard use of TAR in eDiscovery?

Maura R. Grossman: So, typically you need discovery and as was the case for Relativity’s efforts, you’re dealing with the single fixed data collection that you upload to your tool and then you search that repository for relevant information related to your case, maybe it’s updated periodically but what’s called a rolling collection. You have a three new people that you didn’t know about it and you add their data, but it isn’t going to be located in multiple places on the web and it certainly isn’t updating by the hour.

So we were actually searching not only different public data sources on the web, but they were dynamic, they certainly weren’t set up for eDiscovery purposes so it presented a different set of challenges than what you usually see in eDiscovery.

Sharon D. Nelson: Well I can see you met the challenge.

John W. Simek: Well, I think attorneys are really slow in adopting TAR as a — eDiscovery in the early days and do you think the medical research folks are going to adopt the technology at a faster pace to speed up their work?

Maura R. Grossman: I do John. I think the incentive structures are very different in the two fields. As you know as well as I do lawyers live by the billable hour, and they’re not necessarily as motivated to reduce the time and therefore the cost to clients for eDiscovery services and eDiscovery efforts also take place in an adversarial context, there’s a winner and there’s a loser and that depends heavily on what’s found as a result of the search effort.

(00:15:07)

So obviously the producing party wants to produce as little as possible and the requesting party wants to get as much as they can. Medical researchers on the other hand are highly incentivized to reduce the time and the cost to do these systematic reviews and to complete their meta analyses as quickly as possible, because that saves lives.

So their review effort doesn’t take place in the same adversarial context. There’s no good and bad evidence, it’s just a matter of finding everything that’s out there that meets the criteria they’ve specified for inclusion in the systematic review, and there the biggest risk is not finding a study that you should have included.

So unlike lawyers who are slow to adopt new technologies, the medical researchers are actually very excited to try new tools that can expedite their work and Gordon and I are pretty confident that they’re going to begin using machine learning for this task more and more frequently as they move forward.

Sharon D. Nelson: Well the faster they can go the better for all of us. So thank you for all your efforts in that direction. And I have a question just based on sheer ignorance, are you doing any other research in the health area?

Maura R. Grossman: Actually I am Sharon. One of my graduate students and I are working with a small team at the University of Waterloo on a really interesting project and we’re using machine learning technologies very much like TAR to identify health misinformation on the web.

So as you know, all of us turn to the Internet, we experience the symptom, we would want to figure out, oh no, what might it be, what disease do I have, how am I supposed to treat it. I mean we don’t want to go to the doctor unless we have to.

So health misinformation is rampant on the web and it can be very, very, very harmful. Take for example, the suggestion of ingesting disinfectants that might cure the corona virus or cinnamon is a cure for if you Googled it, you would find cinnamon is a cure for diabetes, can help prevent Alzheimer’s, HIV, multiple sclerosis and cardiovascular disease.

So what our team is experimenting with is using TAR methods to rapidly and effectively identify this misinformation so that it can either be removed or at least tagged on social media.

Sharon D. Nelson: Well I know that there is nobody listening who doesn’t know what a problem that is and it’s very difficult sometimes to know that you’re looking at something which is really not true, so that’s another great contribution.

John W. Simek: Yeah, I think that is true, and I think it’s funny more when you said that because that’s exactly what Sharon does. She talks about her medical degree as she searches the Internet.

Maura R. Grossman: Yes, all of us do.

Sharon D. Nelson: But my doctor tells me my diagnosis is always correct. But there are reputable sites with reputable information and I think that’s what’s key, but that’s a whole another issue.

John W. Simek: Yes, yes, yes. I think stick to law dear. So some more, do you have any final thoughts and please tell our listeners where they can find out more about your efforts in this area.

Maura R. Grossman: Thank you again for having me. Basically it’s been great to be able to help out the community even, even in this sort of tiny way, it’s very difficult especially up in Canada where we’re watching a fire raging out of control and you feel like there’s nothing you can do.

There is a short article — I’m not going to give you the exact site because it’s a really long URL, but if you Google Waterloo and computer science and Technology Assisted Review and COVID-19 you will definitely come across it, and it describes our work. It has links to several media reports about it including Robert Ambrogi’s piece on Above the Law. We meaning Gordon and I in the medical research team have submitted the results of the rapid reviews for publication and certainly if anybody wants more information, they should feel free to reach out to me at [email protected], and thanks again.

Sharon D. Nelson: Well we want to thank you Maura for being our guest today. Also want to extend our thanks for to Gordon for the research that you all are doing and how you’re helping in this effort. I mean it’s a horrible time, but then when you run into people like you who are doing everything possible to make this better and bring this to a close, it’s really very inspirational.

So, I know you’re busy and thank you for taking the time to be with us today.

Maura R. Grossman: My pleasure.

John W. Simek: That does it for this edition of Digital Detectives and remember you can subscribe to all the editions of this podcast at legaltalknetwork.com or on Apple podcast. And if you enjoyed our podcast please rate us on the Apple podcast.

Sharon D. Nelson: And you can find out more about Sensei’s Digital Forensics Technology and cybersecurity services at senseient.com. We’ll see you next time on Digital Detectives.

(00:20:04)

Outro: Thanks for listening to Digital Detectives on the Legal Talk Network. Check out some of our other podcasts on legaltalknetwork.com and in iTunes.

[Music]