I got 99 problems — and the law is one: how can we measure legislative efficacy?

CDS’s Andreu Casas and colleagues use NLP to analyze the evolution of 104,005 non-enrolled bills in the US Congress

In 2014, The Washington Post published an article summarizing the career of retired US legislator Robert E. Andrews under a damning headline: “Andrews proposed 646 bills, passed 0: worst record of past 20 years.”

Ouch. The statistic appears to suggest that Andrews’s career was a flop — but as CDS’s Data Science Fellow Andreu Casas explained at a recent Moore Sloan Research Lunch Seminar, new conclusions arise once we consider how the legislative system functions as a whole, and use a more nuanced approach to analyzing the data about successful and unsuccessful legal bills.

Presently, researchers use a metric named the Legal Effectiveness Score (LES) to analyze the efficacy of bills and legislators. The LES scoring system measures how far a particular bill advances within the complex multi-step legislative process.

“But,” as Casas reminded us, “a bill is a vehicle for policy ideas and not necessarily a policy idea itself.” What LES does not account for is that the main ideas of several bills are usually extracted and then inserted into other larger bills that do eventually become the law. Moreover, the text, meaning, and intention of the ideas often remain intact when incorporated into larger bills.

With this in mind, analyzing the evolution of these ‘hitchhiker’ bills, as Casas and co-authors Matthew Denny and John Wilkerson called them, instead of simply counting how many bills passed and failed, would be a more accurate way of measuring legislative efficiency. The question is, how can this be done?

After compiling a dataset of 104,005 versions of non-enrolled bills and 4,073 enrolled bills from the 103rd to 113th Congress between the years of 1993 to 2014, Casas and colleagues tracked the insertion of non-enrolled bills into laws using an ensemble of NLP algorithms (that boast a 95% accuracy rate!). Essentially, these algorithms first pre-process the text of bills, reducing them to their core expression, and then evaluate the extent to which the full meaning of each non-enrolled bill has been inserted into a bill that became law in that same Congress.

Their investigation yielded some revealing conclusions. For example, not only do more senate bills become law as hitchhikers on house laws (1,118) than when enacted on their own (1,037), but that they often become law when included in a bill that concerns a different topic. The key role that hitchhiker bills play in forming larger bills suggest, Casas concluded, that the legislative system is more decentralized and less partisan than we think. When taking these hitchhikers into account,we see that legislation is shaped by more viewpoints, interests, and people.

An open question, however, is whether we have the right people in Congress. Will data science one day have the power to identify the ill-intentioned from the heroes? Well, it’s not a reality yet — but one can certainly dream.

by Cherrie Kwok