Customizing Contextualized Language Models for Legal Document Reviews

Shohreh Shaghaghian, Luna Yue Feng, Borna Jafarpour, Nicolai Pogrebnyakov

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

110 Downloads (Pure)


Inspired by the inductive transfer learning on computer vision, many efforts have been made to train contextualized language models that boost the performance of natural language processing tasks. These models are mostly trained on large general-domain corpora such as news, books, or Wikipedia. Although these pre-trained generic language models well perceive the semantic and syntactic essence of a language structure, exploiting them in a real-world domain-specific scenario still needs some practical considerations to be taken into account such as token distribution shifts, inference time, memory, and their simultaneous proficiency in multiple tasks. In this paper, we focus on the legal domain and present how different language models trained on general-domain corpora can be best customized for multiple legal document reviewing tasks. We compare their efficiencies with respect to task performances and present practical considerations.
Original languageEnglish
Title of host publicationProceedings - 2020 IEEE International Conference on Big Data. Big Data 2020
EditorsXintao Wu, Chris Jermaine, Li Xiong, Xiaohua Hu, Olivera Kotevska, Siyuan Lu, Weija Xu, Srinivas Aluru, Chengxiang Zhai, Eyhab Al-Masri, Zhiyuan Chen, Jeff Saltz
Number of pages10
Place of PublicationLos Alamitos, CA
Publication date2020
Article number9378201
ISBN (Print)9781728162522
ISBN (Electronic)9781728162515
Publication statusPublished - 2020
EventEighth IEEE International Conference on Big Data. IEEE BigData 2020 - Virtual Event
Duration: 10 Dec 202013 Dec 2020
Conference number: 8


ConferenceEighth IEEE International Conference on Big Data. IEEE BigData 2020
LocationVirtual Event
Internet address


  • Adaptation models
  • Law
  • Computational modeling
  • Big Data
  • Natural language processing
  • Task analysis
  • Context modeling

Cite this