Using Natural Language Processing and Machine Learning to Forecast Legislative Outcomes

Transcription

1 White Paper Using Natural Language Processing and Machine Learning to Forecast Legislative Outcomes 1 Thomas Circle, 8th Floor, Washington DC contact@fiscalnote.com

2 As anyone who has watched Schoolhouse Rock can attest, the process of creating legislation is complicated. Understanding the official process of creating law is hard enough, but in order to get below the surface and understand the nuance we enlist the help of a machine. At FiscalNote, each introduced bill is analyzed using sophisticated learning algorithms which allow us to forecast its likelihood of reaching the floor and eventually being passed in each legislative chamber (unless you happen to be Nebraska or DC). The technical details of the natural language processing and machine learning algorithms powering our platform are important to understand in order to rely on the predictive analytics they generate. The complexity of the legislative process How could we possibly know all the different paths a bill may take? Sponsors come and go, committee assignments change, redrafts occur, amendments are proposed and all before lunch. We certainly can t anticipate every scenario with certainty at the outset. We can, however, estimate the probability of a particular outcome using a substantial amount of information housed in our platform. As new actions take place, the forecast is dynamically adjusted to include new information. For example, the addition of an influential sponsor or favorable recommendation from a committee is likely to increase the probability of a bill passing. On the other hand, assignment to an ineffective committee or a harsh amendment makes the bill less likely to pass. The key point is that these probability adjustments are fully automated. Pre-floor vs floor predictions A bill must first reach the floor before it can be voted on. It turns out that these are two fundamentally different problems and must be modeled separately. What does it take for a measure to make it to the floor? The process varies 1

3 from state to state, but usually it involves assignment to one or more committees for consideration where a subset of the chamber is ultimately responsible for the process of pushing it through. In other words, the decision rests with the sponsors and the committee membership. Committees A bill may go through multiple committees, wherein hearings may be called or amendments proposed. Oftentimes, committees may even propose legislation themselves. To model this process we rely heavily on the legislator properties of the sponsors, the content of the bill, and various committee measures we compute. We look at the sponsors specifically from the perspective of their success in the pre-floor stage. Finally, the committees are evaluated from an effectiveness standpoint, as well as the committee membership composition and how they are associated with the sponsors. Bill Text 2

4 The informative signals (or features ) we extract from the bill content encode attributes such as how often certain language, or underlying issues, are associated with legislation that passed or failed to make it to the floor. Individually, these features provide a small amount of information; but as they compound, so do the insights we glean. Extracting and processing this text requires sophisticated natural language processing (NLP) algorithms. Next, machine learning algorithms can incorporate the entire set of features to provide a very accurate prediction of a bill s passage through a legislative chamber. The training data for these models comes from up to 20 years of historical data from each state. In any session, many bills are introduced, but few make it to the floor. That is, most bills die in committee. Thus the prior probability of making it to the floor is very low. This means that the training set usually has many more instances of bills that failed to reach the floor than those that did. However, when a bill does make it to the floor, the prior probability of passage is now quite high meaning that if a bill made it to the floor, chances are good it ll pass a vote. A successful floor passage depends on how each legislator, regardless of their affiliation with this bill or a committee, will vote. While the content of the bill certainly matters, the relationships between legislators now play a much greater role than in the pre-floor forecast. Legislator Histories But how can we know how someone s going to vote when we haven t even met them? Well, it turns out that we can learn a whole lot from how they have acted in the past. Thus, you can think of the model for the floor prediction as a network of legislator influence, with nodes representing legislators and edges showing their relationships. The main aspect in the floor model is how we determine the relationships for any given bill, which depends on the content of the bill, metrics computed from the legislators, and background knowledge on the composition of the chamber. Using machine learning, we can now predict how a particular legislator is likely to vote on a particular bill. These voting predictions for legislators can now be used to estimate the probability that a bill will pass. Freshman Legislators 3

5 Hold on, you might say! What about newly elected legislators that you ve never seen before? It may seem that we don t know anything about someone when they first come in. Or do we? We actually know quite a bit. We know how the typical freshman Republican legislator in Idaho s lower chamber will vote, so when a newly elected representative enters the legislature, we can use a backoff model. This model abstracts away from the specific individual to a collection of individuals that share some commonality. The main point is that we have a solid place to start, and from there we update quickly based on subsequent actions he or she takes. This allows us to move away from the backoff model and to one specific legislator. This also can be used to capture any change in long-serving legislators. For example, they may gain or lose power, care more about certain issues than others, and so on. By constantly updating our models with the most recent data, we maintain an accurate estimate of a legislator s likelihood of voting on a bill. State Models Of course, freshman legislators from one state to the next may act differently. For this and a variety of reasons, we have separate models for each chamber in each state. This is done to allow the models to accurately represent the unique characteristics of each state and chamber, which can differ substantially across the country. The models make their predictions using the same computational logic and information signals or features. However, each model is trained only on the data that pertains to that model (i.e., only the bills that appear in the Texas House are used to train the Texas House prediction model). As a result, the parameters of each model are typically different from other models since they must adapt to best fit the statistical properties of that particular setting. For both of these cases, we have different models for different types of bills, across each chamber, and across each state. The models look at the same set of underlying patterns, but adapt the parameters associated with each 4

6 feature to best fit the specific circumstances of the setting. To measure our own performance we hold out a portion of our training data from the model, and treat it as new, unseen bills. We know whether this training data has passed or failed, so when the model is run on them, we ll see its accuracy. Interpreting forecasts The pre-floor and floor models each produce a probability associated with successful passage, but what does that probability actually mean? Although it s a straightforward question, we ve now walked into a controversy in statistics about how to interpret probability. One interpretation, held by frequentists, is that a probability of passage of 0.90 means that if we simulate this bill being introduced 100 times, it will pass 90 times. Thus, a frequentist interpretation of probability involves the notion of repeated trials. The alternative view, held by bayesians, is that probability is a measure of our belief or confidence. Under this interpretation, the probability is correlated with our confidence in the outcome. A forecast probability of 0.96 indicates a much higher confidence than a forecast probability of 0.70, which 5

7 indicates a higher level of uncertainty in the prediction. Similarly, we re fairly certain that a 0.04 forecast will fail, while a 0.25 means that we are less certain of failure. Whichever interpretation you prefer, the exact point estimates that we compute are just that estimates. They reflect our best guess of what the probability is, but additional information is very useful to place the estimate in context. You are probably already familiar with margins of error in polling estimates. Another way of thinking about the same concept is a confidence interval a range of values we are reasonably certain contains the true probability of a bill s passage. To ensure that the above interpretations are meaningful, we calibrate our models. This step ensures that, in a group of 10 bills, each given a passage probability of 0.90, nine of those bills should actually pass. 6

8 Comparing a bill with its peers FiscalNote allows you to look at both forecasts associated with a particular bill, which is informative in itself. However, it is useful to place this bill in the context of its peers other bills being considered by the same legislative body. Placing this bill in the context of a distribution over all bills indicates its relative chance of passing, as well as an overview of how effective the chamber is overall. Explaining the forecasts While the forecast itself is meaningful, it is natural to wonder about the exact computation that went into a particular forecast. Since the model combines a large number of information sources or features, and there are multiple models, it s not an easy task to break it open and uncover what is driving the prediction. This problem is not novel to our setting. The field of machine learning has typically been concerned with building the best predictive models, which are measured by how well they actually perform. That is very different from building explanatory models, which try to explain the underlying phenomenon in human-interpretable terms. At FiscalNote we choose predictive models but we also provide justifications for each forecast. These are derived from the model s application on each particular bill. Justifications indicate which features were quantitatively the most salient for prediction, and if they are positively or negatively affecting passage. We break justifications into five main areas for the pre-floor model: 7

9 Primary sponsorship explains factors such as the FiscalNote Effectiveness score and relative effectiveness of the sponsor. Co-sponsorship does the same for individual cosponsors, as well as grouping their combined effect. Committees relay how effective the assigned committee is at getting bills to the floor, and any interaction of the sponsor and the committee, such as a sponsor being on the committee. Bill content refers to the actual text of the bill. Other considers chamber composition, bipartisanship effects, along with other miscellaneous elements. The floor model justifications are very similar, but without the committee section. In addition to the justifications and forecast for the floor, we calculate the likelihood of each legislator in the chamber to vote for the bill. In the floor model, these individual votes are taken into consideration. Conclusion If these statistical processes were in place when Schoolhouse Rock wrote Just a Bill, perhaps the song would have spoke more about probability and network relationships. Today, however, legislative professionals are armed with these tools to greater understand policy-making and the impact small changes may have on predictive outcomes. Sign up for our newsletter to get more tips: or contact@fiscalnote.com for more information. 8