Tobacco policies at the state, county, and city jurisdiction levels had similar degrees and patterns of co-occurrence among policies. For example, comprehensive clean-air laws for bars and comprehensive clean-air laws for restaurants frequently co-occurred at the state, county, and city levels . Most policy measures were positively correlated, but we also found pockets of negative correlations. For example, country-years with child tax credits tended not to have child tax allowances . The heat maps also revealed groups of co-occurring and independent policies. For example, labor policies requiring licensing for different professions frequently co-occurred, but this set was relatively independent of policies regarding collective bargaining and minimum wages .Most of the variability in policy measures across jurisdictions and times was explained by the other policies in the same database. Figure 4 displays the distributions of R2 values: the higher the R2, the less unique variation there is for an individual policy, to a maximum of 1.0. The impacts of policy co-occurrence on identifiability were generally substantial: of all 502 policies considered, 65% had R2 values greater than 0.90 when regressed on other policies in the same database. Child benefits had the lowest R2 distribution, with a median of 0.19; policies on poverty and social welfare, family leave, fertility/immigration, firearms, cannabis, alcohol, state tobacco control, and county tobacco control had R2 distributions with medians of approximately 0.9 or greater. In some cases,rolling benches correlations between predictor policy variables were so strong that the statistical software forced certain variables from the model .
Policy co-occurrence substantially reduced the precision of possible effect estimates in all cases . Across policy measures, databases, and simulation iterations, policy co-occurrence effectively increased the variance of effect estimates by a median of 57-fold. Across policies, the lowest degree of variance inflation observed was 7% for country child tax rebates. For other policies, particularly family leave, variance inflation was so substantial as to render estimates effectively indeterminate. Again, some predictors were dropped from models due to strong correlations with other predictors .We analyzed 13 social policy databases drawn from contemporary research in top epidemiology, clinical, and social science journals. These exemplar databases represented diverse policy domains, geographies, and times to describe the pervasiveness and impacts of policy co-occurrence on estimation of health effects. We found that high degrees of co-occurrence were the norm rather than the exception. For a majority of policies, greater than 90% of the variation across jurisdictions and times was explained by other related policies in the same database. Unbiased studies attempting to isolate individual policy effects must control for these related policies, so for many applications, there may be little independent variation left with which to study the policy of interest. Consistent with this, we found that adequate control for co-occurring policies is also likely to substantially reduce the precision of estimated effects, often so dramatically that informative effect estimates are unlikely to be derived.Several factors make the pervasiveness and consequences of policy co-occurrence likely to be even greater than we have estimated. First, we only examined policy cooccurrence within domain-specific databases.
Yet social policy changes may happen in multiple domains simultaneously. For health outcomes affected by diverse types of policies , researchers must consider policy co-occurrence across domains, which likely will indicate even more severe co-occurrence. Second, each policy database we considered included only 1 jurisdictional level, but true policy environments involve complex overlays of national, state or province, county, municipal, employer, and/or school policies. Third, we did not incorporate lagged effects or nonlinear relationships between variables. Fourth, policy variables that perfectly or near-perfectly predicted one another were dropped from the regression models. Finally, we did not consider the multitude of social, economic, political, or societal factors that may also co-occur with policies of primary interest, including changes in social norms, implementation, or enforcement that can be conflated with policy changes. Some such confounders can be controlled with jurisdiction or time fixed effects; measured confounders that are jurisdiction-specific and time-varying could be evaluated using the same methods illustrated here. This is a formidable task; data sharing efforts would facilitate its assessment and handling. We found that the overall degree of policy co-occurrence varied across databases, ranging from very high for state level recreational cannabis policies to low for country-level sexual minority rights policies. Several different factors may drive this variation. Our finding that tobacco policies at the state, county, and city levels had similar degrees and patterns of co-occurrence among similar sets of policies suggests that co-occurrence may be a characteristic of the domain. Political polarization may result in greater co-occurrence for certain policy domains versus others . Databases with rarer policies, fewer umbrella policies , or more nested policies also tended to have more co-occurrence. Databases with more unique policies also generally had more cooccurrence; with a fixed number of jurisdictions and times of observation, considering more policies creates more opportunities for alignment.
Importantly, these patterns highlight that the measured degree of co-occurrence depends not only on the policies themselves but also on the investigator’s choices of policy measures. Furthermore, policies that could be considered alternatives rather than complements co-occurred less frequently and may offer the opportunity for more robust studies of causal impacts. Differences in the ways that policies are adopted across different political systems and different jurisdictional levels may also matter. In our examples, country-level policies appeared to co-occur less frequently than state-level policies, implying that estimating causal effects of country-level policies may be more feasible. Similar considerations apply to the temporal scale of analyses as well: The feasibility of estimating health effects may depend on whether analyses are conducted at the level of the year, month, or even election cycle. Our analysis could not determine which of these factors drives variation in policy cooccurrence; this would be a fruitful area for future research.Several other limitations of this study must be noted. Certain policy domains were not covered, either because no social policy studies for that domain were sampled or because no corresponding policy database was identified or accessed. We did not review all potentially relevant journals. Our results may not generalize to policy domains or journals not examined. Our approach also assumes that all the policies in each domain-specific database are relevant to the health outcome of interest; this is plausible for social interventions that likely affect a broad range of health outcomes, but for some outcomes, only a subset of the policies in a database may need to be controlled to isolate the effect of the index policy. In addition, our approach is only relevant when a database of the relevant policies exists or can be constructed. Developing policy databases is often an arduous task requiring systematic review of legal language. We did not consider the quality of the underlying databases. Our selections serve to illustrate the policy co-occurrence problem, but for applied researchers, the optimal policy database may differ from the one used here. The problem of correlated exposures arises in many domains, including environmental health, and although social policies are distinct in important regards, methods in other domains may nonetheless prove helpful. Furthermore, our analysis did not examine the distinctions between policy adoption, implementation, promulgation, or changes in norms that precede or follow from policy changes,grow tray but these considerations are essential in applied policy research. Finally, data sparsity arising from co-occurring policies can lead to bias, not just imprecision. Our simulations did not incorporate this because this type of bias is less relevant to studies of the health effects of social policies and is highly context specific. Simulation results on the magnitude of bias from positivity violations are therefore unlikely to be generalizable. Specifically, bias arising from positivity problems depends on the estimation method. For methods that rely on modeling the outcome , positivity-related bias arises from model-based extrapolation. For methods that involve modeling the exposure mechanism , bias can result from disproportionate reliance on the experiences of a just a few units or on the absence of certain confounder strata . Because our simulations were based on outcome regressions—the most common approach for differences-in differences, panel fixed effects, and related designs—bias would only arise from model-based extrapolation. However, for the vast majority of policies identified in this study, measures were binary, and thus extrapolation cannot occur. For continuous policy measures , model-based extrapolation is possible but application dependent. Thus, simulating the potential degrees of bias resulting from model-based extrapolation requires either tenuous generalizations or substantive assumptions about each policy area. We suspect that extremely nonlinear relationships that would lead to large extrapolation bias are rare for policy effects, but this remains an open question.Researchers should be cautious when seeking to make causal inferences about the health effects of single social policies using methodological approaches premised on arbitrary or quasi-random variation in policies across jurisdictions and time. Not every policy change offers a valid differences-in-differences or panel fixed-effects study design. These methods are most compelling when policy implementation is staggered across jurisdictions and dates independently from other policies and for plausibly like-random or arbitrary reasons. For example, there could be differing timing of elections, legislative sessions, crises that provoke specific policy changes, or lottery-type roll outs. In these cases, such research can be very persuasive, or at least constrain the set of co-occurring policies.
Our finding of pervasive policy co-occurrence across numerous databases suggests that many policies do not fit this criterion. Inadequate control for co-occurring policies or differences in the set of policies controlled may explain surprising or conflicting results in previous studies. Investigators should base interpretations of social policy research on the plausibility that policy adoption is distributed arbitrarily with respect to other uncontrolled policies or social changes, a phenomenon that, in reality, may be rare. This evaluation should be based on deep content knowledge of law, politics,and society—a compelling argument for involving policymakers in the design and interpretation of studies.We illustrate an approach for researchers to assess whether the effects of individual policies can be estimated. Although other simulation-based methods for assessing positivity exist , the approach we propose is tailored to the policy co-occurrence problem and facilitates examining how a full set of policies substantively occur together. For a given application, if the heat map indicates high correlations, and estimated R2 values and variance inflation are high, it may be necessary to alter the research question and corresponding analytic approach. Researchers have applied numerous analytic approaches to address the challenge of highly co-occurring policies, ranging from machine-learning algorithms that identify policy measures most strongly related to an outcome of interest to methods that characterize overall policy environments based on expert panels. The second article in this issue on policy co-occurrence provides a systematic assessment of available methods. We briefly review 3 promising analytic options here, and refer the reader to the other article for more detail. One approach is to focus on outcomes that are affected by the index policy of interest but not the co-occurring policies. For example, changes in state Earned Income Tax Credits co-occur with changes in other social welfare policies . Rehkopf et al. took advantage of seasonality in the disbursement of EITC cash benefits versus benefits without the same seasonal dispersal pattern, to examine the association of EITC with health using a differences-in differences design. By comparing health outcomes that can change monthly for EITC-eligible versus non-eligible individuals in months of income supplementation versus non-supplementation, the authors measured potential short-term impacts of EITC independent of other social welfare policies. Another approach is to move beyond binary measures of policy adoption to more detailed characterizations . These measures may co-occur less frequently with related policies or provide opportunities to examine dose– response effects among jurisdictions adopting the policy. For example, the adoption of certain unemployment benefit policies co-occurs frequently with other social welfare policies across state-years. Researchers have successfully assessed these policies’ health impacts by comparing varying levels of unemployment benefit generosity—measured as the total maximum allowable benefit per bout of unemployment—across states and years . Heatmaps like those presented in this study may help researchers identify specific policy measures that are more independent from related policies. A final option is to conceptualize policy clusters, instead of individual policies, as exposures. This is promising if policies are typically adopted as a group, as is the case with the large omnibus bills that are increasingly common at the state and federal levels.