Lukas Vashold

Five Minute Summary: Treatment Effects in Interactive Fixed Effects Models

2022-02-16T00:00:00+00:00

[Sonia Karami](https://sites.temple.edu/soniakarami/) and I just had our paper [Treatment Effects in Interactive Fixed Effects Models with a Small Number of Time Periods](https://arxiv.org/abs/2006.15780) accepted at *Journal of Econometrics*. One of the things that I have been very interested in over the past couple of years is trying to identify treatment effect parameters when (i) parallel trends assumptions are violated and (ii) the number of time periods is "small". Parallel trends assumptions are very closely related to the following model for untreated potential outcomes: $$ Y_{it}(0) = \theta_t + \eta_i + U_{it} $$ where $\theta_t$ is a time fixed effect, $\eta_i$ is an individual fixed effect, and $U_{it}$ are idiosyncratic time varying unobservables. But the additive separability between the time-period and unit fixed effects is important here. **cite other IFE papers** **cite manski, roth, and try to connect motivation** ## Connection between Difference in Differences and Interactive Fixed Effects Models A lot of my research has involved identifying treatment effect parameters in a difference in differences (DID) framework. For DID, the main identifying assumption is the parallel trends assumption: **Parallel Trends Assumption** $$ \E[\Delta Y_t(0) | D=1] = \E[\Delta Y_t(0) | D=0] $$ As discussed above, parallel trends assumptions are very closely related to the following model for untreated potential outcomes: $$ Y_{it}(0) = \theta_t + \eta_i + U_{it} $$ where $\theta_t$ is a time fixed effect, $\eta_i$ is an individual fixed effect, and $U_{it}$ are idiosyncratic time varying unobservables. An extended version of the parallel trends assumption is the following *conditional parallel trends assumption* **Conditional Parallel Trends Assumption** $$ \E[\Delta Y_t(0) | \tilde{Z}, D=1] = \E[\Delta Y_t(0) | \tilde{Z}, D=0] $$ which says that parallel trends holds after conditioning on $\tilde{Z}$. To give an example, the application in our paper is about job displacement and the outcome is an individual's earnings. It seems likely that the path of untreated potential outcomes (how outcomes would change over time if an individual were not displaced from their job) likely depends on a person's education, demographic characteristics, etc. If these are distributed differently across displaced workers and non-displaced workers (which is also likely), then conditioning on these sorts of variables before invoking parallel trends will be important. This conditional parallel trends assumption is closely related to the following model for untreated potential outcomes $$ Y_{it}(0) = g_t(\tilde{Z}_i) + \eta_i + U_{it} $$ where $g_t$ is a nonparametric, time-varying function of $\tilde{Z}$, and $\eta_i$ and $U_{it}$ are the same as before (the important thing here is the additive separability of $\eta_i$ which allows for it to be differenced out). It's common to impose linearity for $g_t$ to get to $$ Y_{it}(0) = \tilde{Z}_i'\tilde{\delta}_t + \eta_i + U_{it} $$ where we take $\tilde{Z}_i$ to include an intercept so that the time fixed effect is absorbed into $\tilde{Z}_i'\tilde{\delta}_{t}$ from here on out. I have been a bit purposely vague about $\tilde{Z}$ above. What variables need to be conditioned on though is largely a theoretical exercise. The variable that I mentioned above (education and/or demographic characteristics) are commonly observed in many datasets, but one might also think that parallel trends only holds after additionally conditioning on "ability" which is unlikely to be observed in most data. Let's partition $\tilde{Z} = (Z,\lambda)$ where $Z$ corresponds to the observed components of $\tilde{Z}$ and $\lambda$ corresponds to the unobserved components of $\tilde{Z}$. Similarly, let's partition $\tilde{\delta}_t = (\delta_t, F_t)$ where $\delta_t$ corresponds to the elements in $Z$ and $F_t$ corresponds to the elements in $\lambda$. Plugging this back into the model for untreated potential outcomes above yields $$ Y_{it}(0) = Z_i'\delta_t + \lambda_i'F_t + \eta_i + U_{it} $$ ~~**This is an interactive fixed effects model for untreated potential outcomes!**~~ ## Identification Challenges Even if we like the interactive fixed effects model for untreated potential outcomes, it is still not clear if we can recover any causal effect parameters of interest under palatable identifying assumptions. For one thing, like DID, we'd like to identify causal effect parameters when the number of time periods is small, and much of the interactive fixed effects literature involves arguments where the number of time periods goes to infinity. To make things concrete, let's consider the case with 3 time periods: $t^*$, $t^*-1$, and $t^*-2$. And let's suppose that no one is treated until the last period. We also define $D_i$ as a variable that it is equal to one for individuals in the treated group (i.e., that become treated in the last period) and is equal to 0 otherwise. Like most of the literature on treatment effects with panel data, we'll target identifying the average treatment effect on the treated (ATT) which is given by $$ ATT = \E[Y_{t^*}(1) - Y_{t^*}(0) | D=1] $$ which is the difference between treated and untreated potential outcomes on average among individuals in the treated group. Our main identification challenge is therefore to recover $\E[Y_{t^*}(0)|D=1]$. Towards this end, we use a "quasi-differencing" approach to difference out $\eta_i$ and $\lambda_i$ (see, for example, Holtz-Eakin, Newey, and Rosen (1988) and Ahn, Lee, and Schmidt (2013)); that is, $$ Y_{it^*-1}(0) - Y_{it^*-2}(0) = \lambda_i \big(F_{t^*-1} - F_{t^*-2}\big) + Z_i' \big(\delta_{t^*-1} - \delta_{t^*-2} \big) + U_{it^*-1} - U_{it^*-2} $$ which implies $$ \lambda_i = \Big( \big( Y_{it^*-1}(0) - Y_{it^*-2}(0) \big) - Z_i' \big(\delta_{t^*-1} - \delta_{t^*-2}\big) - \big(U_{it^*-1}-U_{it^*-2}\big) \Big) \Big/ (F_{t^*-1} - F_{t^*-2}) $$ Similarly, $$ \begin{aligned} Y_{it}(0) - Y_{it^*-2}(0) &= \lambda_i(F_t - F_{t^*-2}) + Z_i'(\delta_t - \delta_{t^*-2}) + U_{it} - U_{it^*-2} \nonumber \\ &= Z_i'\delta^*_t + F^*_t (Y_{it^*-1} - Y_{it^*-2}) + V_{it} \end{aligned} $$ ## Our Idea ## The Rest of the Paper... We have a number of extensions to these kind of results in the paper. * Pretty much the same arguments apply in cases where there are more than one interactive fixed effect. The main additional requirement is that, for each interactive fixed effect, we need at least one covariate whose effect on untreated potential outcomes is time invariant * We spend a lot of time thinking about practical issues such as weak instruments, not enough covariates with time invariant effects, and tests for covariates actually having time invariant effects. Except in one or two very pernicious cases, we think that our approach should either work or that one would be able to successfully detect that it is not working.

Ife

2022-02-16T00:00:00+00:00

Five Minute Summary: Treatment Effects in Interactive Fixed Effects Models

Sonia Karami and I just had our paper Treatment Effects in Interactive Fixed Effects Models with a Small Number of Time Periods accepted at Journal of Econometrics.

One of the things that I have been very interested in over the past couple of years is trying to identify treatment effect parameters when (i) parallel trends assumptions are violated and (ii) the number of time periods is “small”.

Parallel trends assumptions are very closely related to the following model for untreated potential outcomes:

\[ Y_{it}(0) = \theta_t + \eta_i + U_{it} \] where $\theta_t$ is a time fixed effect, $\eta_i$ is an individual fixed effect, and $U_{it}$ are idiosyncratic time varying unobservables.

But the additive separability between the time-period and unit fixed effects is important here.

cite other IFE papers

cite manski, roth, and try to connect motivation

Connection between Difference in Differences and Interactive Fixed Effects Models

A lot of my research has involved identifying treatment effect parameters in a difference in differences (DID) framework. For DID, the main identifying assumption is the parallel trends assumption:

Parallel Trends Assumption \[ \mathbb{E}[\Delta Y_t(0) | D=1] = \mathbb{E}[\Delta Y_t(0) | D=0] \]

As discussed above, parallel trends assumptions are very closely related to the following model for untreated potential outcomes:

\[ Y_{it}(0) = \theta_t + \eta_i + U_{it} \] where $\theta_t$ is a time fixed effect, $\eta_i$ is an individual fixed effect, and $U_{it}$ are idiosyncratic time varying unobservables.

An extended version of the parallel trends assumption is the following conditional parallel trends assumption

Conditional Parallel Trends Assumption \[ \mathbb{E}[\Delta Y_t(0) | \tilde{Z}, D=1] = \mathbb{E}[\Delta Y_t(0) | \tilde{Z}, D=0] \] which says that parallel trends holds after conditioning on $\tilde{Z}$. To give an example, the application in our paper is about job displacement and the outcome is an individual’s earnings. It seems likely that the path of untreated potential outcomes (how outcomes would change over time if an individual were not displaced from their job) likely depends on a person’s education, demographic characteristics, etc. If these are distributed differently across displaced workers and non-displaced workers (which is also likely), then conditioning on these sorts of variables before invoking parallel trends will be important.

This conditional parallel trends assumption is closely related to the following model for untreated potential outcomes \[ Y_{it}(0) = g_t(\tilde{Z}_i) + \eta_i + U_{it} \] where $g_t$ is a nonparametric, time-varying function of $\tilde{Z}$, and $\eta_i$ and $U_{it}$ are the same as before (the important thing here is the additive separability of $\eta_i$ which allows for it to be differenced out). It’s common to impose linearity for $g_t$ to get to \[ Y_{it}(0) = \tilde{Z}_i'\tilde{\delta}_t + \eta_i + U_{it} \] where we take $\tilde{Z}_i$ to include an intercept so that the time fixed effect is absorbed into $\tilde{Z}_i'\tilde{\delta}_{t}$ from here on out.

I have been a bit purposely vague about $\tilde{Z}$ above. What variables need to be conditioned on though is largely a theoretical exercise. The variable that I mentioned above (education and/or demographic characteristics) are commonly observed in many datasets, but one might also think that parallel trends only holds after additionally conditioning on “ability” which is unlikely to be observed in most data.

Let’s partition $\tilde{Z} = (Z,\lambda)$ where $Z$ corresponds to the observed components of $\tilde{Z}$ and $\lambda$ corresponds to the unobserved components of $\tilde{Z}$. Similarly, let’s partition $\tilde{\delta}_t = (\delta_t, F_t)$ where $\delta_t$ corresponds to the elements in $Z$ and $F_t$ corresponds to the elements in $\lambda$. Plugging this back into the model for untreated potential outcomes above yields \[ Y_{it}(0) = Z_i'\delta_t + \lambda_i'F_t + \eta_i + U_{it} \] ~~This is an interactive fixed effects model for untreated potential outcomes!~~

Identification Challenges

Even if we like the interactive fixed effects model for untreated potential outcomes, it is still not clear if we can recover any causal effect parameters of interest under palatable identifying assumptions.

For one thing, like DID, we’d like to identify causal effect parameters when the number of time periods is small, and much of the interactive fixed effects literature involves arguments where the number of time periods goes to infinity.

To make things concrete, let’s consider the case with 3 time periods: $t^*$, $t^*-1$, and $t^*-2$. And let’s suppose that no one is treated until the last period. We also define $D_i$ as a variable that it is equal to one for individuals in the treated group (i.e., that become treated in the last period) and is equal to 0 otherwise. Like most of the literature on treatment effects with panel data, we’ll target identifying the average treatment effect on the treated (ATT) which is given by \[ ATT = \mathbb{E}[Y_{t^*}(1) - Y_{t^*}(0) | D=1] \] which is the difference between treated and untreated potential outcomes on average among individuals in the treated group. Our main identification challenge is therefore to recover $\mathbb{E}[Y_{t^*}(0)|D=1]$.

Towards this end, we use a “quasi-differencing” approach to difference out $\eta_i$ and $\lambda_i$ (see, for example, Holtz-Eakin, Newey, and Rosen (1988) and Ahn, Lee, and Schmidt (2013)); that is,

\[ Y_{it^*-1}(0) - Y_{it^*-2}(0) = \lambda_i \big(F_{t^*-1} - F_{t^*-2}\big) + Z_i' \big(\delta_{t^*-1} - \delta_{t^*-2} \big) + U_{it^*-1} - U_{it^*-2} \]

which implies

\[ \lambda_i = \Big( \big( Y_{it^*-1}(0) - Y_{it^*-2}(0) \big) - Z_i' \big(\delta_{t^*-1} - \delta_{t^*-2}\big) - \big(U_{it^*-1}-U_{it^*-2}\big) \Big) \Big/ (F_{t^*-1} - F_{t^*-2}) \]

Similarly, \[ \begin{aligned} Y_{it}(0) - Y_{it^*-2}(0) &= \lambda_i(F_t - F_{t^*-2}) + Z_i'(\delta_t - \delta_{t^*-2}) + U_{it} - U_{it^*-2} \nonumber \\ &= Z_i'\delta^*_t + F^*_t (Y_{it^*-1} - Y_{it^*-2}) + V_{it} \end{aligned} \]

Our Idea

The Rest of the Paper…

We have a number of extensions to these kind of results in the paper.

Pretty much the same arguments apply in cases where there are more than one interactive fixed effect. The main additional requirement is that, for each interactive fixed effect, we need at least one covariate whose effect on untreated potential outcomes is time invariant
We spend a lot of time thinking about practical issues such as weak instruments, not enough covariates with time invariant effects, and tests for covariates actually having time invariant effects. Except in one or two very pernicious cases, we think that our approach should either work or that one would be able to successfully detect that it is not working.

Five Minute Summary: Difference in differences with time-varying covariates

2022-02-08T00:00:00+00:00

$$\newcommand{\E}{\mathbb{E}}$$ My colleague [Carol Caetano](http://carolinacaetano.net/), and two UGA grad students [Hugo Sant'Anna Rodrigues](https://entr.uga.edu/directory/people/hugo-rodrigues) and Stroud Payne just posted [a new working paper](https://arxiv.org/abs/2202.02903) about DID with time varying covariates. This is a topic that has been in the back of my mind for a long time --- and I get a lot of [questions](https://github.com/bcallaway11/did/issues/71) about time varying covariates from people using the [`did` package](https://bcallaway11.github.io/did/). My sense has been there are perhaps a number of limitations to the sorts of two way fixed effects (TWFE) regressions that include covariates that are very common in applied work. And, in particular, that there could be distinct issues from those that show up in the literature on TWFE regressions with multiple periods and variation in treatment timing (e.g., [Goodman-Bacon (2021)](https://doi.org/10.1016/j.jeconom.2021.03.014), [de Chaisemartin and d'Haultfoeuille (2020)](https://doi.org/10.1257/aer.20181169), and [Borusyak, Jaravel, and Spiess (2021)](https://arxiv.org/abs/2108.12419)) In this paper, we have worked out a lot of these issues -- particularly, in the case with exactly two time periods (which is a case where TWFE regressions work well under unconditional parallel trends). We also provide alternative strategies that (i) are able to get around these issues and (ii) are only slightly more complicated to implement than TWFE regressions. **TWFE Regressions** To fix ideas, let me write down how most researchers implement DID identification strategies when they think that the underlying parallel trends assumption ought to be conditional on some covariates: $$ \begin{aligned} Y_{it} = \theta_t + \eta_i + \alpha D_{it} + X_{it}'\beta + v_{it} \end{aligned} $$ where $$Y_{it}$$ is the outcome of interest (for unit $$i$$ in time period $$t$$), $$\theta_t$$ is a time fixed effect, $$\eta_i$$ is an individual fixed effect, $$D_{it}$$ is a treatment dummy variable, $$\alpha$$ is what will be reported as *the* causal effect of the treatment (or, maybe loosely as some kind of average causal effect), $$X_{it}$$ are the time-varying covariates, and $$v_{it}$$ are idiosyncratic time-varying unobservables. We show that there are a number of potential limitations with using this two-way fixed effects (TWFE) regression: 1. In cases with multiple periods and variation in treatment timing, this sort of TWFE regression uses already treated units as the comparison group, and therefore suffers from all well-known weaknesses in this case. Both Goodman-Bacon (2021) and de Chaisemartin and d'Haultfoeuille (2020) already have results along these lines, so I'm going to only talk about the case with two time periods below (which is a case where, at least in the case of unconditional parallel trends, TWFE regressions work fine). 2. TWFE regressions won't work well if the time-varying covariates are affected by the treatment. This issue is often referred to as a "bad control" problem. It seems to be standard practice just not to include covariates that are potentially affected by the treatment. I agree that it's a bad idea to include a time-varying covariate that is itself affected by the treatment, but I am less sure that a good solution is to just not include it. For example, suppose that a labor economist is studying the effect of a treatment on a person's earnings and thinks parallel trends holds after conditioning on a person's occupation, but occupation is potentially affected by the treatment (there is tons of work in labor economics that would be concerned with this issue). Both ignoring occupation and including occupation run into issues. One helpful way to think about this is to try to condition on *untreated potential occupation* --- that is, what occupation would have occurred if a person had not been treated. TWFE regressions don't naturally accommodate this, but we propose some solutions for this case (I'll come back to this below). 3. TWFE regressions like this are *highly* sensitive to the functional form. Since we are considering the case with two periods (and, like the "textbook" version of DID, where no units are treated yet in the first period), we can write $$ \begin{aligned} \Delta Y_{it^*} = (\theta_{t^*} - \theta_{t^*-1}) + \alpha D_i + \Delta X_{it^*} \beta + \Delta v_{it^*} \end{aligned} $$ where $$t^*$$ indicates the second time period. You can see from this specification that, due to our linear functional form for the levels, we are effectively only controlling for the change in covariates over time. We show that TWFE regressions implicitly rely on (i) the conditional parallel trends assumption only depending on the change in the time-varying covariates over time, and (ii) similarly, that TWFE regressions rely on conditional ATTs only depending on changes in time-covariates over time. Thus, TWFE can perform poorly in cases where, for example, the path of untreated potential outcomes also depends on the *level* of the time-varying covariate. Let me give you a concrete example where only controlling for changes in covariates seems undesirable. Suppose you are using county-level data and are studying the effect of a treatment in Georgia using counties from Tennessee as the comparison group, and that you think that parallel trends holds after you condition on county population. I live in Oconee County, Georgia. From 2010 to 2021 (which were the dates I could most easily find for county population), Oconee County grew from about 33,000 to about 42,000. In Tennessee, the county with the most similar population change was Sevier County which increased from about 90,000 to about 99,000. But Sevier County is more than twice as big as Oconee County. This is probably not what we had in mind when we said we wanted to condition on county population. Maybe this is just bad luck, let's check the county with the next most similar population change. It is Shelby County --- this is Memphis! --- which increased from 928,500 to 938,800. I don't think anyone would think that comparing paths of outcomes for Shelby County and Oconee County is what any researcher has in mind for DID conditioning on county population. As a side-comment, if you switch to, say, the change in log population over time, you do not do much better either --- in that case, the closest match is Montgomery County, TN which has over 5 times the population of Oconee County. Perhaps somewhat surprisingly TWFE regressions also require strong functional form assumptions on the propensity score (see paper for details). 4. Similarly, we show that TWFE regressions are not robust to parallel trends assumptions and conditional ATTs depending on *time-invariant covariates*. However, conditioning on time-invariant covariates in the parallel trends assumption is important in many applications. For example, if you are a labor economist studying the effect of some treatment on people's earnings, the most important covariates to condition on in the parallel trends assumption are all likely to be time invariant --- e.g., demographics, education, etc. 5. $$\alpha$$ is hard to interpret in the presence of treatment effect heterogeneity. Even if none of the issues above apply in a particular application, if treatment effects are heterogeneous (particularly, if they can vary across different values of the covariates), (under some additional conditions) $$\alpha$$ will be equal to a weighted average of conditional ATT parameters but they will suffer from the "weight reversal" property pointed out in [Sloczynski (2020)](https://doi.org/10.1162/rest_a_00953) in a different context --- conditional ATTs for values of the covariates that are uncommon for the treated group relative to the untreated group get lots of weight, and the opposite happens for relatively common values of the covariates. If a researcher is fortunate enough that none of these issues apply in their application, then a TWFE regression would recover the ATT. **Existing work in econometrics** Most work on DID under conditional parallel trends (e.g., [Abadie (2005)](https://doi.org/10.1111/0034-6527.00321), [Sant'Anna and Zhao (2020)](https://doi.org/10.1016/j.jeconom.2020.06.003), and [Chang (2020)](https://doi.org/10.1093/ectj/utaa001)) considers the case with time-invariant covariates or uses "pre-treatment" values of time-varying covariates (which effectively just makes time-varying covariates time invariant by using their value in the pre-treatment period). This already solves most of the above issues: they can be adapted to handle cases multiple periods and variation in treatment timing in (1), they do not require the same strong functional form assumptions as in (3), they solve (4) above because they include time-invariant covariates, and they recover the overall ATT directly rather than a hard-to-interpret weighted average of conditional ATTs as in (5). **What's new in our paper** 1. First, in order to address (2), where the time-varying covariates could themselves be affected by the treatment, we provide specific conditions under which it is sufficient to condition on pre-treatment values of the time-varying covariates as is common in the econometrics literature. In particular, the condition that rationalizes conditioning on pre-treatment covariates is $$ \begin{aligned} X_{t^*}(0) \perp D | X_{t^*-1}, Z \end{aligned} $$ where $$X_{t^*}(0)$$ is the value that $$X$$ would take in time period $$t^*$$ if the treatment had not occurred and $$Z$$ is the vector of time-invariant covariates in the parallel trends assumption. This is an unconfoundedness assumption, but for time-varying covariates rather than the outcome. In words, it says that covariates are evolving similarly among treated and untreated units that have the same pre-treatment characteristics $$X_{t^*-1}$$ and time-invariant covariates $$Z$$. This condition may or may not be reasonable in particular applications, but it is the sort of thing that reseachers ought to think about. It is also "pre-testable" (i.e., you can look at data in pre-treatment periods and potentially find evidence for or against it). In cases where this assumption does not hold, the strategy of just conditioning on pre-treatment covariates does not generally work. But we consider a number of other possible assumptions that can lead to alternative identification arguments in the paper. A big part of the paper is about these cases, but it is perhaps best just to consult the paper itself on this front as these arguments are somewhat more complicated. 2. Another important case is when a researcher is confident that covariates are evolving exogenously from the treatment; a simple version of this is just where $$X_{it^*}(1) = X_{it^*}(0)$$ for all units (that is, the value of the covariates is the same under the treatment as without the treatment). Ignoring the issue of time-invariant covariates, the main issue with TWFE in this case are the functional form issues pointed out in (3) above. In this case, we provide a doubly robust expression for that ATT that does not rely on those sorts of functional form assumptions. These expressions involve outcome regressions and propensity scores that depend on both $$X_{t^*}$$ and $$X_{t^*-1}$$ --- these can be challenging to estimate well because $$X_{t^*}$$ and $$X_{t^*-1}$$ are likely to be highly collinear in many applications. However, the doubly robust expression for the ATT allows us to connect to the literature on DID with machine learning (Chang (2020)) which provides an attractive way to try to estimate these functions. 3. Finally, in cases where these kinds of doubly robust / machine learning approaches are more complicated than a researcher actually wants to implement, we provide strategies for all of the cases discussed above that can be implemented using just regressions and averaging. Relative to the previous two points, these approaches require additional linearity assumptions (though substantially less restrictive than the issues discussed earlier for TWFE regressions), but have the benefit of being easier to implement; these ideas build on the ideas of regression adjustment and imputation that have shown up recently in the DID literature ([Liu, Wang, and Xu (2021)](https://arxiv.org/abs/2107.00856), [Gardner (2021)](https://jrgcmu.github.io/2sdd_current.pdf), Borusyak, Jaravel, and Spiess (2021)). Let me just give the example of what we propose to do in cases where the time-varying covariates evolve exogenously. Similar to the "imputation" literature, we can exploit the connection between parallel trends assumptions and a model for untreated potential outcomes: $$ \begin{aligned} Y_{it}(0) = Z_i'\delta_t + \eta_i + X_{it}(0) \beta_t + v_{it} \end{aligned} $$ where we take $$Z$$ to include an intercept. The $$\beta_t$$ is perhaps non-standard (see discussion in next paragraph). Taking the difference over time implies $$ \begin{aligned} \Delta Y_{it^*}(0) = Z_i'\delta^*_{t^*} + \Delta X_{it^*}(0) \beta_{t^*} + X_{it^*-1}(0) \beta^*_{t^*} + \Delta v_{it^*} \end{aligned} $$ where we define $$\delta^*_{t^*} := (\delta_{t^*} - \delta_{t^*-1})$$ and $$\beta^*_{t^*} := (\beta_{t^*} - \beta_{t^*-1})$$. In my view, this is particularly attractive specification for untreated potential outcomes in terms of time-varying covariates. It includes both the initial level of the covariates (which is similar to including the "pre-treatment" value of the covariate) as well as the change in covariates over time. And, for example, (up to the parametric assumptions) this expression would avoid the issues of comparing counties with similar changes in population over time but very dissimilar overall populations. Moreover, since we observe untreated potential outcomes and covariates for the untreated group, we can recover all of the parameters from the regression of $$\Delta Y_{t^*}$$ on $$Z$$, $$\Delta X_{t^*}$$, and $$X_{t^*-1}$$ using the untreated group. Next, notice that $$ \begin{aligned} ATT &= \E[\Delta Y_{t^*} | D=1] - \E[\Delta Y_{t^*}(0) | D=1] \\ &= \E[\Delta Y_{t^*} | D=1] - \Big(\E[Z|D=1]'\delta^*_{t^*} + \E[\Delta X_{t^*}(0) | D=1] \beta_{t^*} + \E[X_{t^*-1}|D=1]\beta^*_{t^*} \Big) \\ \end{aligned} $$ where the second equality holds by plugging in the expression for $$\Delta Y_{t^*}(0)$$ from the previous display. Everything is identified in the last line except for $$\E[\Delta X_{t^*}(0) | D=1]$$. If we believe that covariates evolve exogenously though, it means that this term is equal to $$\E[\Delta X_{t^*} | D=1]$$ which is identified. We consider 5 additional scenarios for recovering $$\E[\Delta X_{t^*}(0) | D=1]$$ in the paper. To summarize, this suggests a simple two-step estimation procedure: (i) estimate a regression using untreated observations and recover the estimates of the parameters in the model for untreated potential outcomes, (ii) combine these with estimates of the averages of the change in outcomes over time and averages of covariates for the treated group (as in the previous display) to compute the ATT. **Conclusion** In my view, the sorts of TWFE regressions that show up in many applications in economics have a number of limitations -- when these TWFE regressions include time-varying covariates, we are arguing that they are likely to have a number of disadvantages even in "textbook" cases with only two time periods. Fortunately, it is quite straightforward to use other approaches (that are not much more complicated) that can essentially avoid all of these issues. We don't have code yet, but we are working on it. If you have comments/questions, please feel free to get in touch. **References** * Abadie, Alberto. "Semiparametric difference-in-differences estimators." The Review of Economic Studies 72.1 (2005): 1-19. * Borusyak, Kirill, Xavier Jaravel, and Jann Spiess. "Revisiting event study designs: Robust and efficient estimation." arXiv preprint arXiv:2108.12419 (2021). * Chang, Neng-Chieh. "Double/debiased machine learning for difference-in-differences models." The Econometrics Journal 23.2 (2020): 177-191. * de Chaisemartin, Clément, and Xavier d'Haultfoeuille. "Two-way fixed effects estimators with heterogeneous treatment effects." American Economic Review 110.9 (2020): 2964-96. * Gardner, John. "Two-stage differences in differences." (2021). * Goodman-Bacon, Andrew. "Difference-in-differences with variation in treatment timing." Journal of Econometrics (2021). * Liu, Licheng, Ye Wang, and Yiqing Xu. "A practical guide to counterfactual estimators for causal inference with time-series cross-sectional data." arXiv preprint arXiv:2107.00856 (2021). * Sant’Anna, Pedro HC, and Jun Zhao. "Doubly robust difference-in-differences estimators." Journal of Econometrics 219.1 (2020): 101-122. * Słoczyński, Tymon. "Interpreting ols estimands when treatment effects are heterogeneous: Smaller groups get larger weights." The Review of Economics and Statistics (2020): 1-27.

Five Minute Summary: Difference in differences with time-varying covariates

2022-02-08T00:00:00+00:00

\[\newcommand{\E}{\mathbb{E}}\]

My colleague Carol Caetano, and two UGA grad students Hugo Sant’Anna Rodrigues and Stroud Payne just posted a new working paper about DID with time varying covariates. This is a topic that has been in the back of my mind for a long time — and I get a lot of questions about time varying covariates from people using the did package.

My sense has been there are perhaps a number of limitations to the sorts of two way fixed effects (TWFE) regressions that include covariates that are very common in applied work. And, in particular, that there could be distinct issues from those that show up in the literature on TWFE regressions with multiple periods and variation in treatment timing (e.g., Goodman-Bacon (2021), de Chaisemartin and d’Haultfoeuille (2020), and Borusyak, Jaravel, and Spiess (2021))

In this paper, we have worked out a lot of these issues – particularly, in the case with exactly two time periods (which is a case where TWFE regressions work well under unconditional parallel trends). We also provide alternative strategies that (i) are able to get around these issues and (ii) are only slightly more complicated to implement than TWFE regressions.

TWFE Regressions

To fix ideas, let me write down how most researchers implement DID identification strategies when they think that the underlying parallel trends assumption ought to be conditional on some covariates:

\[\begin{aligned} Y_{it} = \theta_t + \eta_i + \alpha D_{it} + X_{it}'\beta + v_{it} \end{aligned}\]

where $Y_{it}$ is the outcome of interest (for unit $i$ in time period $t$), $\theta_t$ is a time fixed effect, $\eta_i$ is an individual fixed effect, $D_{it}$ is a treatment dummy variable, $\alpha$ is what will be reported as the causal effect of the treatment (or, maybe loosely as some kind of average causal effect), $X_{it}$ are the time-varying covariates, and $v_{it}$ are idiosyncratic time-varying unobservables.

We show that there are a number of potential limitations with using this two-way fixed effects (TWFE) regression:

In cases with multiple periods and variation in treatment timing, this sort of TWFE regression uses already treated units as the comparison group, and therefore suffers from all well-known weaknesses in this case. Both Goodman-Bacon (2021) and de Chaisemartin and d’Haultfoeuille (2020) already have results along these lines, so I’m going to only talk about the case with two time periods below (which is a case where, at least in the case of unconditional parallel trends, TWFE regressions work fine).
TWFE regressions won’t work well if the time-varying covariates are affected by the treatment. This issue is often referred to as a “bad control” problem. It seems to be standard practice just not to include covariates that are potentially affected by the treatment. I agree that it’s a bad idea to include a time-varying covariate that is itself affected by the treatment, but I am less sure that a good solution is to just not include it.

For example, suppose that a labor economist is studying the effect of a treatment on a person’s earnings and thinks parallel trends holds after conditioning on a person’s occupation, but occupation is potentially affected by the treatment (there is tons of work in labor economics that would be concerned with this issue). Both ignoring occupation and including occupation run into issues. One helpful way to think about this is to try to condition on untreated potential occupation — that is, what occupation would have occurred if a person had not been treated. TWFE regressions don’t naturally accommodate this, but we propose some solutions for this case (I’ll come back to this below).
TWFE regressions like this are highly sensitive to the functional form. Since we are considering the case with two periods (and, like the “textbook” version of DID, where no units are treated yet in the first period), we can write
\[\begin{aligned} \Delta Y_{it^*} = (\theta_{t^*} - \theta_{t^*-1}) + \alpha D_i + \Delta X_{it^*} \beta + \Delta v_{it^*} \end{aligned}\]
where $t^*$ indicates the second time period. You can see from this specification that, due to our linear functional form for the levels, we are effectively only controlling for the change in covariates over time.

We show that TWFE regressions implicitly rely on (i) the conditional parallel trends assumption only depending on the change in the time-varying covariates over time, and (ii) similarly, that TWFE regressions rely on conditional ATTs only depending on changes in time-covariates over time. Thus, TWFE can perform poorly in cases where, for example, the path of untreated potential outcomes also depends on the level of the time-varying covariate.

Let me give you a concrete example where only controlling for changes in covariates seems undesirable. Suppose you are using county-level data and are studying the effect of a treatment in Georgia using counties from Tennessee as the comparison group, and that you think that parallel trends holds after you condition on county population. I live in Oconee County, Georgia. From 2010 to 2021 (which were the dates I could most easily find for county population), Oconee County grew from about 33,000 to about 42,000. In Tennessee, the county with the most similar population change was Sevier County which increased from about 90,000 to about 99,000. But Sevier County is more than twice as big as Oconee County. This is probably not what we had in mind when we said we wanted to condition on county population. Maybe this is just bad luck, let’s check the county with the next most similar population change. It is Shelby County — this is Memphis! — which increased from 928,500 to 938,800. I don’t think anyone would think that comparing paths of outcomes for Shelby County and Oconee County is what any researcher has in mind for DID conditioning on county population. As a side-comment, if you switch to, say, the change in log population over time, you do not do much better either — in that case, the closest match is Montgomery County, TN which has over 5 times the population of Oconee County.

Perhaps somewhat surprisingly TWFE regressions also require strong functional form assumptions on the propensity score (see paper for details).
Similarly, we show that TWFE regressions are not robust to parallel trends assumptions and conditional ATTs depending on time-invariant covariates. However, conditioning on time-invariant covariates in the parallel trends assumption is important in many applications. For example, if you are a labor economist studying the effect of some treatment on people’s earnings, the most important covariates to condition on in the parallel trends assumption are all likely to be time invariant — e.g., demographics, education, etc.
$\alpha$ is hard to interpret in the presence of treatment effect heterogeneity. Even if none of the issues above apply in a particular application, if treatment effects are heterogeneous (particularly, if they can vary across different values of the covariates), (under some additional conditions) $\alpha$ will be equal to a weighted average of conditional ATT parameters but they will suffer from the “weight reversal” property pointed out in Sloczynski (2020) in a different context — conditional ATTs for values of the covariates that are uncommon for the treated group relative to the untreated group get lots of weight, and the opposite happens for relatively common values of the covariates.

If a researcher is fortunate enough that none of these issues apply in their application, then a TWFE regression would recover the ATT.

Existing work in econometrics

Most work on DID under conditional parallel trends (e.g., Abadie (2005), Sant’Anna and Zhao (2020), and Chang (2020)) considers the case with time-invariant covariates or uses “pre-treatment” values of time-varying covariates (which effectively just makes time-varying covariates time invariant by using their value in the pre-treatment period). This already solves most of the above issues: they can be adapted to handle cases multiple periods and variation in treatment timing in (1), they do not require the same strong functional form assumptions as in (3), they solve (4) above because they include time-invariant covariates, and they recover the overall ATT directly rather than a hard-to-interpret weighted average of conditional ATTs as in (5).

What’s new in our paper

First, in order to address (2), where the time-varying covariates could themselves be affected by the treatment, we provide specific conditions under which it is sufficient to condition on pre-treatment values of the time-varying covariates as is common in the econometrics literature. In particular, the condition that rationalizes conditioning on pre-treatment covariates is
\[\begin{aligned} X_{t^*}(0) \perp D | X_{t^*-1}, Z \end{aligned}\]
where $X_{t^*}(0)$ is the value that $X$ would take in time period $t^*$ if the treatment had not occurred and $Z$ is the vector of time-invariant covariates in the parallel trends assumption. This is an unconfoundedness assumption, but for time-varying covariates rather than the outcome. In words, it says that covariates are evolving similarly among treated and untreated units that have the same pre-treatment characteristics $X_{t^*-1}$ and time-invariant covariates $Z$.

This condition may or may not be reasonable in particular applications, but it is the sort of thing that reseachers ought to think about. It is also “pre-testable” (i.e., you can look at data in pre-treatment periods and potentially find evidence for or against it).

In cases where this assumption does not hold, the strategy of just conditioning on pre-treatment covariates does not generally work. But we consider a number of other possible assumptions that can lead to alternative identification arguments in the paper. A big part of the paper is about these cases, but it is perhaps best just to consult the paper itself on this front as these arguments are somewhat more complicated.
Another important case is when a researcher is confident that covariates are evolving exogenously from the treatment; a simple version of this is just where $X_{it^*}(1) = X_{it^*}(0)$ for all units (that is, the value of the covariates is the same under the treatment as without the treatment). Ignoring the issue of time-invariant covariates, the main issue with TWFE in this case are the functional form issues pointed out in (3) above. In this case, we provide a doubly robust expression for that ATT that does not rely on those sorts of functional form assumptions. These expressions involve outcome regressions and propensity scores that depend on both $X_{t^*}$ and $X_{t^*-1}$ — these can be challenging to estimate well because $X_{t^*}$ and $X_{t^*-1}$ are likely to be highly collinear in many applications. However, the doubly robust expression for the ATT allows us to connect to the literature on DID with machine learning (Chang (2020)) which provides an attractive way to try to estimate these functions.
Finally, in cases where these kinds of doubly robust / machine learning approaches are more complicated than a researcher actually wants to implement, we provide strategies for all of the cases discussed above that can be implemented using just regressions and averaging. Relative to the previous two points, these approaches require additional linearity assumptions (though substantially less restrictive than the issues discussed earlier for TWFE regressions), but have the benefit of being easier to implement; these ideas build on the ideas of regression adjustment and imputation that have shown up recently in the DID literature (Liu, Wang, and Xu (2021), Gardner (2021), Borusyak, Jaravel, and Spiess (2021)).

Let me just give the example of what we propose to do in cases where the time-varying covariates evolve exogenously. Similar to the “imputation” literature, we can exploit the connection between parallel trends assumptions and a model for untreated potential outcomes:
\[\begin{aligned} Y_{it}(0) = Z_i'\delta_t + \eta_i + X_{it}(0) \beta_t + v_{it} \end{aligned}\]

where we take $Z$ to include an intercept. The $\beta_t$ is perhaps non-standard (see discussion in next paragraph). Taking the difference over time implies

\[\begin{aligned} \Delta Y_{it^*}(0) = Z_i'\delta^*_{t^*} + \Delta X_{it^*}(0) \beta_{t^*} + X_{it^*-1}(0) \beta^*_{t^*} + \Delta v_{it^*} \end{aligned}\]

where we define $\delta^*_{t^*} := (\delta_{t^*} - \delta_{t^*-1})$ and $\beta^*_{t^*} := (\beta_{t^*} - \beta_{t^*-1})$. In my view, this is particularly attractive specification for untreated potential outcomes in terms of time-varying covariates. It includes both the initial level of the covariates (which is similar to including the “pre-treatment” value of the covariate) as well as the change in covariates over time. And, for example, (up to the parametric assumptions) this expression would avoid the issues of comparing counties with similar changes in population over time but very dissimilar overall populations.

Moreover, since we observe untreated potential outcomes and covariates for the untreated group, we can recover all of the parameters from the regression of $\Delta Y_{t^*}$ on $Z$, $\Delta X_{t^*}$, and $X_{t^*-1}$ using the untreated group. Next, notice that

\[\begin{aligned} ATT &= \E[\Delta Y_{t^*} | D=1] - \E[\Delta Y_{t^*}(0) | D=1] \\ &= \E[\Delta Y_{t^*} | D=1] - \Big(\E[Z|D=1]'\delta^*_{t^*} + \E[\Delta X_{t^*}(0) | D=1] \beta_{t^*} + \E[X_{t^*-1}|D=1]\beta^*_{t^*} \Big) \\ \end{aligned}\]

where the second equality holds by plugging in the expression for $\Delta Y_{t^*}(0)$ from the previous display. Everything is identified in the last line except for $\E[\Delta X_{t^*}(0) | D=1]$. If we believe that covariates evolve exogenously though, it means that this term is equal to $\E[\Delta X_{t^*} | D=1]$ which is identified. We consider 5 additional scenarios for recovering $\E[\Delta X_{t^*}(0) | D=1]$ in the paper.

To summarize, this suggests a simple two-step estimation procedure: (i) estimate a regression using untreated observations and recover the estimates of the parameters in the model for untreated potential outcomes, (ii) combine these with estimates of the averages of the change in outcomes over time and averages of covariates for the treated group (as in the previous display) to compute the ATT.

Conclusion

In my view, the sorts of TWFE regressions that show up in many applications in economics have a number of limitations – when these TWFE regressions include time-varying covariates, we are arguing that they are likely to have a number of disadvantages even in “textbook” cases with only two time periods. Fortunately, it is quite straightforward to use other approaches (that are not much more complicated) that can essentially avoid all of these issues.

We don’t have code yet, but we are working on it. If you have comments/questions, please feel free to get in touch.

References

Abadie, Alberto. “Semiparametric difference-in-differences estimators.” The Review of Economic Studies 72.1 (2005): 1-19.
Borusyak, Kirill, Xavier Jaravel, and Jann Spiess. “Revisiting event study designs: Robust and efficient estimation.” arXiv preprint arXiv:2108.12419 (2021).
Chang, Neng-Chieh. “Double/debiased machine learning for difference-in-differences models.” The Econometrics Journal 23.2 (2020): 177-191.
de Chaisemartin, Clément, and Xavier d’Haultfoeuille. “Two-way fixed effects estimators with heterogeneous treatment effects.” American Economic Review 110.9 (2020): 2964-96.
Gardner, John. “Two-stage differences in differences.” (2021).
Goodman-Bacon, Andrew. “Difference-in-differences with variation in treatment timing.” Journal of Econometrics (2021).
Liu, Licheng, Ye Wang, and Yiqing Xu. “A practical guide to counterfactual estimators for causal inference with time-series cross-sectional data.” arXiv preprint arXiv:2107.00856 (2021).
Sant’Anna, Pedro HC, and Jun Zhao. “Doubly robust difference-in-differences estimators.” Journal of Econometrics 219.1 (2020): 101-122.
Słoczyński, Tymon. “Interpreting ols estimands when treatment effects are heterogeneous: Smaller groups get larger weights.” The Review of Economics and Statistics (2020): 1-27.

Faster did Code

2021-12-10T00:00:00+00:00

```{r echo=FALSE} base.dir <- "~/Dropbox/website/" base.url <- "/" fig.path <- "files/figures/" # set knitr parameters knitr::opts_knit$set(base.dir = base.dir, base.url = base.url) knitr::opts_chunk$set(fig.path = fig.path) ``` We just put out a big update to the `did` package (version 2.1). It is on GitHub and should be on CRAN in a day or two. This version contains a lot of behind-the-scenes improvement such as better error handling and substantially more thorough testing. It also allows users to [use a universal base period in event studies](/posts/event-study-universal-v-varying-base-period). This post though is about speed improvements in the new version of the `did` package. These are almost all coming from improved performance of our bootstrap procedure. As a side-comment: we have always used the multiplier bootstrap, which is much, much faster than the empirical bootstrap to begin with, but we implemented the new version in C++ which results in notable improvements in computation time. Using the multiplier bootstrap is what allows the `did` package to report, for example, uniform confidence bands for group-time average treatment effects and event studies; that is, they tend to be somewhat wider than pointwise confidence intervals but are robust to the multiple testing arising in both of these cases. | Type | \# units | \# time periods | rows of data | old computation time | new computation time | percentage reduction | | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Balanced Panel | 5000 | 8 | 40,000 | 12.5 | 3.5 | 72% | | Unbalanced Panel | 5000 | 8 | 39,999 | 283.4 | 3.4 | 99% | | Repeated Cross Sections | 5000 | 8 | 5000 | 9.3 | 3.4 | 63% | | Larger Panel Data | 20,000 | 20 | 400,000 | 328.5 | 97.2 | 70% | For the full code for this post, see: [https://github.com/bcallaway11/did/blob/master/tests/code_profiling.Rmd](https://github.com/bcallaway11/did/blob/master/tests/code_profiling.Rmd)

Universal vs. Varying Base Period in Event Studies

2021-11-09T00:00:00+00:00

Introduction

One of the main ways that researchers use our did package is to plot event studies. These are quite useful in order to think about (i) dynamic effects of participating in the treatment and (ii) to “pre-test” the parallel trends assumption.

You can find an extended discussion about event studies, limitations of event study regressions in a number of relevant cases, etc. here.

This post isn’t about criticizing event study regressions; instead, what I want to talk about is the choice of the “base period” in event studies.

Types of Base Periods

Event study regressions typically have a universal base period. This means that all differences are relative to a particular period, and, most commonly, it is set to be the period immediately before the treatment starts.

In the did package, our default is to use a varying base period. In pre-treatment periods, the base period is the immediately preceding period; e.g., if period 4 is pre-treatment, then the base period for this period will be period 3.

If there are violations of parallel trends in pre-treatment periods, then the interpretation of reported “effects” in pre-treatment periods in an event study differs depending on whether one uses a varying or universal base period. Here is the difference:

With a varying base period, the reported effects are pseudo-ATTs. They are what we would have estimated effect of participating in the treatment to be (on impact) if the treatment had occurred in that period (instead of when it actually occurred).
With a universal base period, event study estimates in pre-treatment periods are not themselves treatment effect parameters, but they are useful for showing how outcomes are trending over time.

In the newest version (version 2.1) of did, we have added a new argument, base_period, to att_gt to give users the option to choose either a varying (the default) or universal base period.

A couple of other things that are also worth mentioning:

In post-treatment periods, the base period is the period immediately before treatment both cases $\implies$ the only place where this difference matters is in pre-treatment periods.
In pre-treatment periods, either case is just a linear combination of the other, so they essentially are just alternative ways of reporting the same information. That is, choosing between a varying or universal base period is more related to how to the “style” of presenting results and shouldn’t change conclusions about whether parallel trends is violated in pre-treatment periods, etc.

Comments/Opinions

My sense is that providing results using a varying base period tends to work better when (i) the researcher is primarily concerned with treatment effect anticipation, and/or (ii) the number of pre-treatment periods is relatively small. And that using a universal base period tends to work better when (i) the researcher thinks that there are long-term differences in trends across groups, and/or (ii) the number of pre-treatment periods is relatively large.

Finally, although using a universal based period is relatively more common in applications, it seems to me that this is mainly because it is easier to implement this when you are running an event study regression. For researchers that are directly computing averages of paths of outcomes at different lengths of exposure to the treatment (as we do in the did package), reporting the results in using either type of base period is easy to do.

Some Examples

Example 1: No violations of parallel trends

Let’s start with the simplest case where there are no violations of parallel trends in pre-treatment periods.

library(did) # need to load version 2.1 of package

Below is some code to generate data where parallel trends holds in all periods, and the average effect of participating in the treatment is equal to 1 (reset.sim and build_sim_dataset are functions in the did package for generating simulated data).

# create data with no pre-trends
time.periods <- 5
sp <- reset.sim(time.periods=time.periods)
sp$te <- 1
data <- build_sim_dataset(sp)
data <- subset(data, G==time.periods | G==0)

# varying base period
res1_varying <- att_gt(yname="Y", xformla=~X, data=data, tname="period",
              idname="id",
              control_group="nevertreated",
              gname="G", est_method="dr")
dynamic1_varying <- aggte(res1_varying, type="dynamic")
p1_varying <- ggdid(dynamic1_varying, ylim=c(-2,2))

# universal base period
res1_universal <- att_gt(yname="Y", xformla=~X, data=data, tname="period",
              idname="id",
              control_group="nevertreated",
              gname="G", est_method="dr", base_period="universal"
              )
dynamic1_universal <- aggte(res1_universal, type="dynamic")
p1_universal <- ggdid(dynamic1_universal, ylim=c(-2,2))

ggpubr::ggarrange(p1_varying, p1_universal, nrow=1)

The plot on the left uses a varying base period while the plot on the right uses a universal base period. The estimated treatment effects when $e=0$ are numerically identical. The pre-treatment estimates are not numerically identical (they are based on different paths of outcomes in pre-treatment periods for the treated group relative to the untreated group), but (as expected) neither provides any evidence against parallel trends. Finally, notice that using a varying base period provides an estimate when $e=0$, but does not provide an estimate when $e=-4$; using a universal base period provides an estimate when $e=-4$ but not when $e=-1$.

Example 2: Anticipation Effects

Next, we generate data where there anticipation effects. What is happening here is that there is a group that becomes treated in the last period and a group that never participates in the treatment (in order to not clutter the post with code, let me just point you to the complete code for this post…it is very similar to the code above). Parallel trends holds in all periods except the period right before treatment when the treated group experiences a negative “anticipation” effect of participating in the treatment.

As before, the results using a varying base period are in the panel on the left, and the results using a universal base period are on the right. As before, the post-treatment estimated effects are exactly the same. To me, it seems much clearer to interpret the figure on the left (recall that there are anticipation effects that are equal to -1 in the pre-treatment period). For me, the figure on the right is hard to interpret.

Example 3: Longer Run Linear Trends

Finally, let’s consider the case where there are longish-run linear trend differences between the treated group and untreated group (and, thus, parallel trends is violated in pre-treatment periods). That is, we are in the case where, on average, outcomes are increasing by one in the treated group relative to the untreated group across all periods (both pre-treatment and post-treatment).

As in the earlier two cases, the panel on the left contains results using a varying base period, and the panel on the right contains results using a universal base period; likewise, the post-treatment estimates are numerically identical. In this case, to me, it seems easier to notice the linear difference in trends in the right panel. If you are careful, you can still interpret the results using a varying base period. Particularly, in every pre-treatment period, we would have over-estimated the effect of participating in the treatment (if the treatment had started in that period) – this happpens because of the linear violations of parallel trends in all periods.

Event_study_universal_v_local_base_period

2021-11-09T00:00:00+00:00

## ----echo=FALSE------------------------------------------------------------------------- base.dir <- "~/Dropbox/website/" base.url <- "/" fig.path <- "files/figures/event-study-universal-v-varying-base-period/" # set knitr parameters knitr::opts_knit$set(base.dir = base.dir, base.url = base.url) knitr::opts_chunk$set(fig.path = fig.path) ## ----echo=FALSE, message=FALSE---------------------------------------------------------- devtools::load_all("~/Dropbox/did") ## ----eval=FALSE------------------------------------------------------------------------- ## library(did) # need to load version 2.1 of package ## ---- fig.align='center', fig.width=10, fig.height=5------------------------------------ # create data with no pre-trends time.periods <- 5 sp <- reset.sim(time.periods=time.periods) sp$te <- 1 data <- build_sim_dataset(sp) data <- subset(data, G==time.periods | G==0) # varying base period res1_varying <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr") dynamic1_varying <- aggte(res1_varying, type="dynamic") p1_varying <- ggdid(dynamic1_varying, ylim=c(-2,2)) # universal base period res1_universal <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr", base_period="universal" ) dynamic1_universal <- aggte(res1_universal, type="dynamic") p1_universal <- ggdid(dynamic1_universal, ylim=c(-2,2)) ggpubr::ggarrange(p1_varying, p1_universal, nrow=1) ## ----fig.align='center', fig.width=10, fig.height=5, echo=FALSE------------------------- # treatment anticipation effects time.periods <- 5 sp <- reset.sim(time.periods=time.periods) sp$te <- 0 sp$te.e <- c(-1,1) data <- build_sim_dataset(sp) # add anticipation data <- subset(data, G==(time.periods-1) | G==0) data$G[data$G > 0] <- 5 # varying base period res2_varying <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr") dynamic2_varying <- aggte(res2_varying, type="dynamic") p2_varying <- ggdid(dynamic2_varying, ylim=c(-2,3)) # universal base period res2_universal <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr", base_period="universal" ) dynamic2_universal <- aggte(res2_universal, type="dynamic") p2_universal <- ggdid(dynamic2_universal, ylim=c(-2,3)) ggpubr::ggarrange(p2_varying, p2_universal, nrow=1) ## ----fig.align='center', fig.width=10, fig.height=5, echo=FALSE------------------------- # linear trends time.periods <- 5 sp <- reset.sim(time.periods=time.periods) sp$te <- 0 sp$te.e <- 1:(time.periods-1)#-1:(time.periods-2) data <- build_sim_dataset(sp) # add anticipation data <- subset(data, G==2 | G==0) data$G[data$G > 0] <- 5 # varying base period res3_varying <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr") dynamic3_varying <- aggte(res3_varying, type="dynamic") p3_varying <- ggdid(dynamic3_varying, ylim=c(-5,2)) # universal base period res3_universal <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr", base_period="universal" ) dynamic3_universal <- aggte(res3_universal, type="dynamic") p3_universal <- ggdid(dynamic3_universal, ylim=c(-5,2)) ggpubr::ggarrange(p3_varying, p3_universal, nrow=1)

Universal vs. Varying Base Period in Event Studies

2021-11-09T00:00:00+00:00

```{r echo=FALSE} base.dir <- "~/Dropbox/website/" base.url <- "/" fig.path <- "files/figures/event-study-universal-v-varying-base-period/" # set knitr parameters knitr::opts_knit$set(base.dir = base.dir, base.url = base.url) knitr::opts_chunk$set(fig.path = fig.path) ``` ## Introduction One of the main ways that researchers use our `did` package is to plot event studies. These are quite useful in order to think about (i) dynamic effects of participating in the treatment and (ii) to "pre-test" the parallel trends assumption. You can find an extended discussion about event studies, limitations of event study regressions in a number of relevant cases, etc. [here](https://bcallaway11.github.io/did/articles/TWFE.html). This post isn't about criticizing event study regressions; instead, what I want to talk about is the choice of the "base period" in event studies. ## Types of Base Periods Event study regressions typically have a **universal base period**. This means that all differences are relative to a particular period, and, most commonly, it is set to be the period immediately before the treatment starts. In the `did` package, our default is to use a **varying base period**. In pre-treatment periods, the base period is the immediately preceding period; e.g., if period 4 is pre-treatment, then the base period for this period will be period 3. If there are violations of parallel trends in pre-treatment periods, then the interpretation of reported "effects" in pre-treatment periods in an event study differs depending on whether one uses a varying or universal base period. Here is the difference: * With a varying base period, the reported effects are **pseudo-ATTs**. They are what we would have estimated effect of participating in the treatment to be (on impact) if the treatment had occurred in that period (instead of when it actually occurred). * With a universal base period, event study estimates in pre-treatment periods are not themselves treatment effect parameters, but they are useful for showing how outcomes are trending over time. In the newest version (version 2.1) of `did`, we have added a new argument, `base_period`, to `att_gt` to give users the option to choose either a `varying` (the default) or `universal` base period. A couple of other things that are also worth mentioning: * In post-treatment periods, the base period is the period immediately before treatment both cases $$\implies$$ the only place where this difference matters is in pre-treatment periods. * In pre-treatment periods, either case is just a linear combination of the other, so they essentially are just alternative ways of reporting the same information. That is, choosing between a varying or universal base period is more related to how to the "style" of presenting results and shouldn't change conclusions about whether parallel trends is violated in pre-treatment periods, etc. ## Comments/Opinions My sense is that providing results using a varying base period tends to work better when (i) the researcher is primarily concerned with treatment effect anticipation, and/or (ii) the number of pre-treatment periods is relatively small. And that using a universal base period tends to work better when (i) the researcher thinks that there are long-term differences in trends across groups, and/or (ii) the number of pre-treatment periods is relatively large. Finally, although using a universal based period is relatively more common in applications, it seems to me that this is mainly because it is easier to implement this when you are running an event study regression. For researchers that are directly computing averages of paths of outcomes at different lengths of exposure to the treatment (as we do in the `did` package), reporting the results in using either type of base period is easy to do. ## Some Examples ```{r echo=FALSE, message=FALSE} devtools::load_all("~/Dropbox/did") ``` **Example 1: No violations of parallel trends** Let's start with the simplest case where there are no violations of parallel trends in pre-treatment periods. ```{r eval=FALSE} library(did) # need to load version 2.1 of package ``` Below is some code to generate data where parallel trends holds in all periods, and the average effect of participating in the treatment is equal to 1 (`reset.sim` and `build_sim_dataset` are functions in the `did` package for generating simulated data). ```{r, fig.align='center', fig.width=10, fig.height=5} # create data with no pre-trends time.periods <- 5 sp <- reset.sim(time.periods=time.periods) sp$te <- 1 data <- build_sim_dataset(sp) data <- subset(data, G==time.periods | G==0) # varying base period res1_varying <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr") dynamic1_varying <- aggte(res1_varying, type="dynamic") p1_varying <- ggdid(dynamic1_varying, ylim=c(-2,2)) # universal base period res1_universal <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr", base_period="universal" ) dynamic1_universal <- aggte(res1_universal, type="dynamic") p1_universal <- ggdid(dynamic1_universal, ylim=c(-2,2)) ggpubr::ggarrange(p1_varying, p1_universal, nrow=1) ``` The plot on the left uses a varying base period while the plot on the right uses a universal base period. The estimated treatment effects when $$e=0$$ are numerically identical. The pre-treatment estimates are not numerically identical (they are based on different paths of outcomes in pre-treatment periods for the treated group relative to the untreated group), but (as expected) neither provides any evidence against parallel trends. Finally, notice that using a varying base period provides an estimate when $$e=0$$, but does not provide an estimate when $$e=-4$$; using a universal base period provides an estimate when $$e=-4$$ but not when $$e=-1$$. **Example 2: Anticipation Effects** Next, we generate data where there anticipation effects. What is happening here is that there is a group that becomes treated in the last period and a group that never participates in the treatment (in order to not clutter the post with code, let me just point you to the [complete code for this post](/files/2021-11-09-event_study_universal_v_local_base_period.R)...it is very similar to the code above). Parallel trends holds in all periods except the period right before treatment when the treated group experiences a negative "anticipation" effect of participating in the treatment. ```{r fig.align='center', fig.width=10, fig.height=5, echo=FALSE} # treatment anticipation effects time.periods <- 5 sp <- reset.sim(time.periods=time.periods) sp$te <- 0 sp$te.e <- c(-1,1) data <- build_sim_dataset(sp) # add anticipation data <- subset(data, G==(time.periods-1) | G==0) data$G[data$G > 0] <- 5 # varying base period res2_varying <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr") dynamic2_varying <- aggte(res2_varying, type="dynamic") p2_varying <- ggdid(dynamic2_varying, ylim=c(-2,3)) # universal base period res2_universal <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr", base_period="universal" ) dynamic2_universal <- aggte(res2_universal, type="dynamic") p2_universal <- ggdid(dynamic2_universal, ylim=c(-2,3)) ggpubr::ggarrange(p2_varying, p2_universal, nrow=1) ``` As before, the results using a varying base period are in the panel on the left, and the results using a universal base period are on the right. As before, the post-treatment estimated effects are exactly the same. To me, it seems much clearer to interpret the figure on the left (recall that there are anticipation effects that are equal to -1 in the pre-treatment period). For me, the figure on the right is hard to interpret. **Example 3: Longer Run Linear Trends** Finally, let's consider the case where there are longish-run linear trend differences between the treated group and untreated group (and, thus, parallel trends is violated in pre-treatment periods). That is, we are in the case where, on average, outcomes are increasing by one in the treated group relative to the untreated group across all periods (both pre-treatment and post-treatment). ```{r fig.align='center', fig.width=10, fig.height=5, echo=FALSE} # linear trends time.periods <- 5 sp <- reset.sim(time.periods=time.periods) sp$te <- 0 sp$te.e <- 1:(time.periods-1)#-1:(time.periods-2) data <- build_sim_dataset(sp) # add anticipation data <- subset(data, G==2 | G==0) data$G[data$G > 0] <- 5 # varying base period res3_varying <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr") dynamic3_varying <- aggte(res3_varying, type="dynamic") p3_varying <- ggdid(dynamic3_varying, ylim=c(-5,2)) # universal base period res3_universal <- att_gt(yname="Y", xformla=~X, data=data, tname="period", idname="id", control_group="nevertreated", gname="G", est_method="dr", base_period="universal" ) dynamic3_universal <- aggte(res3_universal, type="dynamic") p3_universal <- ggdid(dynamic3_universal, ylim=c(-5,2)) ggpubr::ggarrange(p3_varying, p3_universal, nrow=1) ``` As in the earlier two cases, the panel on the left contains results using a varying base period, and the panel on the right contains results using a universal base period; likewise, the post-treatment estimates are numerically identical. In this case, to me, it seems easier to notice the linear difference in trends in the right panel. If you are careful, you can still interpret the results using a varying base period. Particularly, in every pre-treatment period, we would have over-estimated the effect of participating in the treatment (if the treatment had started in that period) -- this happens because of the linear violations of parallel trends in all periods.

Universal vs. Varying Base Period in Event Studies

2021-11-09T00:00:00+00:00

Introduction

You can find an extended discussion about event studies, limitations of event study regressions in a number of relevant cases, etc. here.

This post isn’t about criticizing event study regressions; instead, what I want to talk about is the choice of the “base period” in event studies.

Types of Base Periods

With a varying base period, the reported effects are pseudo-ATTs. They are what we would have estimated effect of participating in the treatment to be (on impact) if the treatment had occurred in that period (instead of when it actually occurred).
With a universal base period, event study estimates in pre-treatment periods are not themselves treatment effect parameters, but they are useful for showing how outcomes are trending over time.

In the newest version (version 2.1) of did, we have added a new argument, base_period, to att_gt to give users the option to choose either a varying (the default) or universal base period.

A couple of other things that are also worth mentioning:

In post-treatment periods, the base period is the period immediately before treatment both cases $\implies$ the only place where this difference matters is in pre-treatment periods.
In pre-treatment periods, either case is just a linear combination of the other, so they essentially are just alternative ways of reporting the same information. That is, choosing between a varying or universal base period is more related to how to the “style” of presenting results and shouldn’t change conclusions about whether parallel trends is violated in pre-treatment periods, etc.

Comments/Opinions

Some Examples

Example 1: No violations of parallel trends

Let’s start with the simplest case where there are no violations of parallel trends in pre-treatment periods.

library(did) # need to load version 2.1 of package

# create data with no pre-trends
time.periods <- 5
sp <- reset.sim(time.periods=time.periods)
sp$te <- 1
data <- build_sim_dataset(sp)
data <- subset(data, G==time.periods | G==0)

# varying base period
res1_varying <- att_gt(yname="Y", xformla=~X, data=data, tname="period",
              idname="id",
              control_group="nevertreated",
              gname="G", est_method="dr")
dynamic1_varying <- aggte(res1_varying, type="dynamic")
p1_varying <- ggdid(dynamic1_varying, ylim=c(-2,2))

# universal base period
res1_universal <- att_gt(yname="Y", xformla=~X, data=data, tname="period",
              idname="id",
              control_group="nevertreated",
              gname="G", est_method="dr", base_period="universal"
              )
dynamic1_universal <- aggte(res1_universal, type="dynamic")
p1_universal <- ggdid(dynamic1_universal, ylim=c(-2,2))

ggpubr::ggarrange(p1_varying, p1_universal, nrow=1)

The plot on the left uses a varying base period while the plot on the right uses a universal base period. The estimated treatment effects when $e=0$ are numerically identical. The pre-treatment estimates are not numerically identical (they are based on different paths of outcomes in pre-treatment periods for the treated group relative to the untreated group), but (as expected) neither provides any evidence against parallel trends. Finally, notice that using a varying base period provides an estimate when $e=0$, but does not provide an estimate when $e=-4$; using a universal base period provides an estimate when $e=-4$ but not when $e=-1$.

Example 2: Anticipation Effects

Example 3: Longer Run Linear Trends

As in the earlier two cases, the panel on the left contains results using a varying base period, and the panel on the right contains results using a universal base period; likewise, the post-treatment estimates are numerically identical. In this case, to me, it seems easier to notice the linear difference in trends in the right panel. If you are careful, you can still interpret the results using a varying base period. Particularly, in every pre-treatment period, we would have over-estimated the effect of participating in the treatment (if the treatment had started in that period) – this happens because of the linear violations of parallel trends in all periods.

Five Minute Summary: Treatment Effects in Interactive Fixed Effects Models

2021-08-15T00:00:00+00:00

\newcommand{\E}{\mathbb{E}}

Sonia Karami and I just posted a new version of our paper Treatment Effects in Interactive Fixed Effects Models.

Connection between Difference in Differences and Interactive Fixed Effects Models

A lot of my research has involved identifying treatment effect parameters in a difference in differences (DID) framework. For DID, the main identifying assumption is the parallel trends assumption:

Parallel Trends Assumption $\mathbb{E}[\Delta Y_t(0) | D=1] = \mathbb{E}[\Delta Y_t(0) | D=0]$

Parallel trends assumptions are very closely related to the following model for untreated potential outcomes:

$Y_{it}(0) = \theta_t + \eta_i + U_{it}$ where $\theta_t$ is a time fixed effect, $\eta_i$ is an individual fixed effect, and $U_{it}$ are idiosyncratic time varying unobservables.

Lukas Vashold

Five Minute Summary: Treatment Effects in Interactive Fixed Effects Models

Ife

Five Minute Summary: Treatment Effects in Interactive Fixed Effects Models

brant

2022-02-16

Connection between Difference in Differences and Interactive Fixed Effects Models

Identification Challenges

Our Idea

The Rest of the Paper…

Five Minute Summary: Difference in differences with time-varying covariates

Five Minute Summary: Difference in differences with time-varying covariates

Faster did Code

Universal vs. Varying Base Period in Event Studies

Introduction

Types of Base Periods

Comments/Opinions

Some Examples

Event_study_universal_v_local_base_period

Universal vs. Varying Base Period in Event Studies

Universal vs. Varying Base Period in Event Studies

Introduction

Types of Base Periods

Comments/Opinions

Some Examples

Five Minute Summary: Treatment Effects in Interactive Fixed Effects Models

Connection between Difference in Differences and Interactive Fixed Effects Models

Identification Challenges

Our Idea

The Rest of the Paper…