Small models as paralegals: LexisNexis distills models to build AI assistant

When legal research company LexisNexis created its AI assistant Protégé, it wanted to figure out the best way to leverage its expertise without deploying a large model. 

Protégé aims to help lawyers, associates and paralegals write and proof legal documents and ensure that anything they cite in complaints and briefs is accurate. However, LexisNexis didn’t want a general legal AI assistant; they wanted to build one that learns a firm’s workflow and is more customizable. 

LexisNexis saw the opportunity to bring the power of large language models (LLMs) from Anthropic and Mistral and find the best models that answer user questions the best, Jeff Riehl, CTO of LexisNexis Legal and Professional, told VentureBeat.

“We use the best model for the specific use case as part of our multi-model approach. We use the model that provides the best result with the fastest response time,” Riehl said. “For some use cases, that will be a small language model like Mistral or we perform distillation to improve performance and reduce cost.”

While LLMs still provide value in building AI applications, some organizations turn to using small language models (SLMs) or distilling LLMs to become small versions of the same model. 

Distillation, where an LLM “teaches” a smaller model, has become a popular method for many organizations. 

Small models often work best for apps like chatbots or simple code completion, which is what LexisNexis wanted to use for Protégé. 

This is not the first time LexisNexis built AI applications, even before launching its legal research hub LexisNexis + AI in July 2024.

“We have used a lot of AI in the past, which was more around natural language processing, some deep learning and machine learning,” Riehl said. “That really changed in November 2022 when ChatGPT was launched, because prior to that, a lot of the AI capabilities were kind of behind the scenes. But once ChatGPT came out, the generative capabilities, the conversational capabilities of it was very, very intriguing to us.”

Small, fine-tuned models and model routing 

Riehl said LexisNexis uses different models from most of the major model providers when building its AI platforms. LexisNexis + AI used Claude models from Anthropic, OpenAI’s GPT models and a model from Mistral. 

This multimodal approach helped break down each task users wanted to perform on the platform. To do this, LexisNexis had to architect its platform to switch between models. 

“We would break down whatever task was being performed into individual components, and then we would identify the best large language model to support that component. One example of that is we will use Mistral to assess the query that the user entered in,” Riehl said. 

For Protégé, the company wanted faster response times and models more fine-tuned for legal use cases. So it turned to what Riehl calls “fine-tuned” versions of models, essentially smaller weight versions of LLMs or distilled models. 

“You don’t need GPT-4o to do the assessment of a query, so we use it for more sophisticated work, and we switch models out,” he said. 

When a user asks Protégé a question about a specific case, the first model it pings is a fine-tuned Mistral “for assessing the query, then determining what the purpose and intent of that query is” before switching to the model best suited to complete the task. Riehl said the next model could be an LLM that generates new queries for the search engine or another model that summarizes results. 

Right now, LexisNexis mostly relies on a fine-tuned Mistral model though Riehl said it used a fine-tuned version of Claude “when it first came out; we are not using it in the product today but in other ways.” LexisNexis is also interested in using other OpenAI models especially since the company came out with new reinforcement fine-tuning capabilities last year. LexisNexis is in the process of evaluating OpenAI’s reasoning models including o3 for its platforms. 

Riehl added that it may also look at using Gemini models from Google. 

LexisNexis backs all of its AI platforms with its own knowledge graph to perform retrieval augmented generation (RAG) capabilities, especially as Protégé could help launch agentic processes later. 

The AI legal suite

Even before the advent of generative AI, LexisNexis tested the possibility of putting chatbots to work in the legal industry. In 2017, the company tested an AI assistant that would compete with IBM’s Watson-powered Ross and Protégé sits in the company’s LexisNexis + AI platform, which brings together the AI services of LexisNexis. 

Protégé helps law firms with tasks that paralegals or associates tend to do. It helps write legal briefs and complaints that are grounded in firms’ documents and data, suggest legal workflow next steps, suggest new prompts to refine searches, draft questions for depositions and discovery, link quotes in filings for accuracy, generate timelines and, of course, summarize complex legal documents. 

“We see Protégé as the initial step in personalization and agentic capabilities,” Riehl said. “Think about the different types of lawyers: M&A, litigators, real estate. It’s going to continue to get more and more personalized based on the specific task you do. Our vision is that every legal professional will have a personal assistant to help them do their job based on what they do, not what other lawyers do.”

Protégé now competes against other legal research and technology platforms. Thomson Reuters customized OpenAI’s o1-mini-model for its CoCounsel legal assistant. Harvey, which raised $300 million from investors including LexisNexis, also has a legal AI assistant.