Measuring Corporate Governance with Large Language Models
News originally published by
The Harvard Law School Forum on Corporate GovernanceCorporate governance research today would not look the same without commercial datasets. While much information on publicly traded corporations is available on the SEC’s EDGAR platform, commercial dataset providers translate this information into a format that researchers can use in empirical analyses. Many of the most influential studies of the last decades have relied on this data. Of course, researchers can also code the information by hand, and many studies have done so. But doing so is costly and usually limits the scale of feasible investigations.
While commercial corporate governance data has enabled important work, it has also shaped and constrained the field. Commercial datasets made some companies and governance arrangements much easier to study than others. Figuratively speaking: Where those datasets cast light, researchers tended to gather. Other questions, equally worthy of attention, might have been too costly to pursue.
While it’s hard to imagine all the questions researchers might have studied had more comprehensive data been available, a few recent studies using alternative data sources illustrate what research based on commercial datasets can miss. Kobi Kastiel and Yaron Nili’s 2022 study illustrates the implications of limited firm coverage. They found that the corporate governance regimes of companies at the bottom of the Russell 3000 index differ markedly from those of the larger companies typically covered by commercial datasets and included in empirical studies. Emiliano Catan and Marcel Kahan’s 2019 paper shows why it matters which provisions researchers measure, even at well-covered firms. They found that the rights to call a special meeting or act by written consent, both tracked in commercial datasets, became more widespread. Yet looking beyond those datasets, they found that other governance provisions prevented shareholders at many companies from using those rights to change the board’s composition between annual meetings. The expansion of these rights therefore often did little to strengthen shareholders’ hand. And in my own work, I have shown that the adoption of fee-shifting provisions collapsed nationwide after Delaware’s legislative ban, even in states where companies remained free to adopt them. This pattern may have gone unnoticed in part because commonly used governance datasets do not track fee-shifting provisions.
Now, of course, the limits of commercial datasets are no reason not to use them in empirical analyses. When used well, some data is surely better than no data. But what if new technology allowed researchers to obtain corporate governance data automatically from source documents at a fraction of the cost of manual coding and on a much larger scale? We should certainly take that opportunity.
My new paper, Measuring Corporate Governance with Large Language Models, argues that large language models (LLMs) provide us with this opportunity. Compared with earlier natural language processing tools (computational techniques that allow computers to capture the “meaning” encoded in ordinary, unstructured language), these models are more powerful and flexible, allowing them to translate the text of legal documents into the structured information that has made commercial datasets such a useful resource for researchers.
To test this proposition, I systematically examine how well LLMs can extract legally relevant information from charters and bylaws, arguably the most important sources of information on a corporation’s governance regime. I use a benchmark dataset I assembled in companion work that combines 300 charters and 150 bylaws obtained from EDGAR with around 25 hand-coded yes-or-no labels per document. These labels capture important governance information encoded in the documents, such as whether the company has a staggered board, whether it vests shareholders with the authority to call a special meeting, and whether the chairperson or board has unconditional authority to adjourn a meeting. Overall, this benchmark dataset contains 11,550 individual extraction tasks. I also construct several pipelines through which LLMs can recover these variables from the source documents. The pipelines use frontier models available earlier this year from three AI labs.
The most important result is that all the main pipelines tested in this paper perform remarkably well. Measured against the labels that most likely represent ground truth, the pipelines achieve an accuracy of around 98%, with each pipeline returning the correct label for at least 11,300 of the 11,550 extraction tasks. The paper also finds that an easy way to improve the accuracy of LLM coding is to combine labels generated by individual coding routines using a simple majority vote, since the pipelines tend to make different mistakes. The resulting ensemble labels agree with the same reference labels 98.8% of the time.
While this accuracy rate may seem high, more context is needed to understand what it means. In principle, it could be that most labels are trivially easy to extract and that the extraction routines fail on many of the more challenging decisions. To evaluate LLM coding, the paper therefore compares the accuracy of the coding routines with that of the human coders who helped assemble the benchmark datasets. Human coders also make mistakes when extracting governance data. This is evident, for example, from my earlier coauthored work, which shows that the IRRC dataset, compiled at a time when hand coding was the only practical option, contains substantial errors. For researchers, the practical question, then, is how the accuracy of LLM coding compares with that of the available alternatives, most importantly the hand coding they would otherwise use to build a new dataset from source documents.
To ensure that human coders and LLMs are compared on fair terms, I blindly recoded all cases in which the LLMs and the original human coders disagreed. Measured against the resulting reference labels, the original human labels were correct 97.1% of the time. Put differently, the human coders made 337 errors and the ensemble 142. And where the two disagreed, my blind recoding sided with the ensemble about three-quarters of the time. Statistical tests indicate that the ensemble’s superior performance is systematic and not a result of chance.
That said, LLMs do not perform equally well on all variables. This suggests that their superior performance cannot be taken as a given, and researchers should validate extraction routines before putting them to work in a research project.
The ability to generate corporate governance data at scale has profound implications for the questions researchers can investigate. With this data, it will be possible to extend analyses beyond the sample of companies customarily included in corporate governance datasets and shine light on a segment of the market that has largely been invisible to the empirical corporate governance literature. It will be possible to conduct large-scale analyses of less common governance arrangements that these datasets omit. And it will be possible to better understand what the arrangements captured by coarse dummy variables in these datasets actually imply.
To illustrate what this makes possible, I replicate and extend Catan and Kahan’s 2019 study of shareholder venue rights using automatically generated governance data and ask whether the trends they documented hold outside the S&P 500. One of their results is that effective venue rights increased less than formal venue rights among S&P 500 companies. Among the Delaware-incorporated S&P 500 firms covered by both studies, my automatically generated measures of special meeting and written consent rights closely track their hand-collected data. Extending the analysis to Delaware-incorporated companies outside the S&P 500, I find a different picture (see figure). Between 2005 and 2025, special meeting rights became slightly more common there, while written consent rights declined substantially. The share of companies with any formal venue right fell overall. Effective venue rights declined more sharply still, a finding that has, to my knowledge, gone undocumented in the literature.
Figure: Shareholder venue rights among Delaware-incorporated companies, 2005–2025
Of course, questions remain on the way to reliable automatically generated governance data. One main obstacle is that there is currently no easy way to obtain a comprehensive sample of charters and bylaws from which researchers could extract governance variables using the techniques proposed in this paper. However, I have shown in previous work that assembling such a collection is feasible. Others have already made collections of other EDGAR documents, such as merger agreements, available to researchers. I am hoping to make a similar collection of charters and bylaws public, too.
Finally, this paper leaves open whether LLM-generated labels can match or even exceed the accuracy of labels in commercial datasets. The answer matters for researchers deciding how to obtain data for new studies when commercial data is available. It also matters for how we read the studies we already have. By recreating governance variables from source documents at scale, researchers could examine more systematically where the commercial datasets underlying much empirical corporate governance research are reliable and where their accuracy deserves a closer look.
Legal Disclaimer:
EIN Presswire provides this news content "as is" without warranty of any kind. We do not accept any responsibility or liability for the accuracy, content, images, videos, licenses, completeness, legality, or reliability of the information contained in this article. If you have any complaints or copyright issues related to this article, kindly contact the author above.
