Research Design 1: Fixed Effects and Instrumental Variables

Stijn Masschelein

Did we not cover that already?

• Yes, but briefly

• Yes, but starting from the perspective of a regression (and the code)

The regression perspective is not bad. It means that we can see that more advanced regression techniques can be implemented in our linear regression framework. It’s also how most researchers in accounting and finance have been thought to think about research methods. However, there is a shift coming from economics where the focus is more on the research design.

Research Design

The focus is on Research Design

Important

1. Which data should we use?
2. Which comparison identifies the effect that we are interested in?

Is there sufficient variation that can identify the effect. - See also the pitching document - A specific example is the identification of performance effects

• For instance Alcohol and Mortality, Chapter 5 in Huntington-Klein (2021).

• Is there sufficient variation in the treatment and the outcome?

• Are we reasonably sure that there are no confounders or only a few and we can measure them?

Prevously, we used models and assumptions to identify effects

Mathematical models

\[ V = T^{\alpha_T} \Bigl( \frac{K}{\alpha_K} \Bigl)^{\alpha_K} \Bigl( \frac{L}{\alpha_L} \Bigl)^{\alpha_L} \] \[ \alpha_T + \alpha_K + \alpha_L = 1 \]

• \(V =\) The value of the firm
• \(K =\) Capital of the firm
• \(L =\) Labour of the firm
• \(T =\) CEO talent/skills/ability/experience

DAGs

Just focus on a setting where we are confident in the assumptions

Actual random assignment

Speedboat racing, game shows, Vietnam draft

Natural experiments

See Gippel, Smith, and Zhu (2015), Chapter 19 Instrumental Variables in Huntington-Klein (2021)

Policy Changes

Chapter 18, Difference-in-Difference in Huntington-Klein (2021)

Discrete cutoffs

e.g. WAM > 75, Chapter 20 Regression Continuity Design in Huntington-Klein (2021)

Unexpected news

Chapter 17 Event Studies in Huntington-Klein (2021)

Natural experiments is not the best terminology because most of these instances are not natural nor real experiments. Nevertheless, I still prefer the name over an instrumental variable approach. In too many proposals, I read an off hand comment that the student proposes to use a robustness test where they are going to use an instrumental variable approach. My answer to that is (1) if you have a natural experiment where you can exploit an instrumental variable, this should be the main analysis and (2) instrumental variables need to be defended as a research design based on your understanding of the setting. Calling the design a natural experiment forces you to think more about the experiment (i.e. the research design).

Look for these designs!

• Based on your understanding of the industry and setting or the Data Generating Process
• When you read good papers for this unit and other units.

This is one of the main reasons that I want you to read broadly. It is unlikely that you will find a paper with a good research design exactly for the research question that you are interested in. However, you might find inspiration in similar or related fields that help you to design a better study for the research question that you are interested in.

What is this effect?

What effect can we identify?

• Average Treatment Effect

• Average Treatment on the Treated

• Average Treatment on the Untreated

• Local Average Treatment Effect

• Weigthed Average Treatment Effect

Chapter 10, Treatment Effects in Huntington-Klein (2021)

• Do you have an example of an effect that we might be interested in in Accounting and Finance?
• Average implies that not all firms will respond the same to the treatment. This is the source of a lot trouble.
• Average over which population?
• How would you put these different effects in your own words?
• WATE is evil and I am going to largely ignore it.

It all depends on where the variation is coming from.

Warning

Different firms react differently and are differently represented in the control group and the treatment group.

Note

• With actual random assignment, you probably have an ATE for the population that received the assignment.
• If you can use a control group because that is what the treated group would look like if they were not treated, you probably have an ATT.
• If you use a natural experiment to identify part of the variation, you probably have a LATE.

Chapter 10, Treatment Effects in Huntington-Klein (2021)

Why do we care?

Research Design

There is a deep connection between the variation in your research design and the effect you can identify.

Policy Implications

Whether your study has implications for “regulators and investors” depends heavily on the type of effect you can identify.

Chapter 10, Treatment Effects in Huntington-Klein (2021)

That is the setting of your data determines which research design you can use. The research design determines which effect you can identify. The effect you can identify determines which conclusions you can draw.

A simulated example

Generate the Data

N <- 1000
rd1 <- tibble(
  firm = 1:N,
  high_performance = rbinom(N, 1, 0.5),
  noise = rnorm(N, 0, 3)
) %>%
  mutate(
    donation = high_performance,
    performance = ifelse(high_performance == 1, 4, 1),
    payoff_donation = 4 - 8 / performance + noise,
    payoff_no_donation = 1 + noise
  )
glimpse(rd1) 

Rows: 1,000
Columns: 7
$ firm               <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, …
$ high_performance   <int> 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1…
$ noise              <dbl> 0.8634427, 3.9991062, -2.208085…
$ donation           <int> 1, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1…
$ performance        <dbl> 4, 1, 4, 1, 1, 1, 1, 4, 4, 4, 4…
$ payoff_donation    <dbl> 2.863443e+00, -8.938289e-04, -2…
$ payoff_no_donation <dbl> 1.8634427, 4.9991062, -1.208085…

• What is the effect that we are we interested in?
• What are the policy implications?

Have a look at the data

We will talk more about the pivot_wider and pivot_longer functions in week 7.

Have a second look at the data

Real data does not have the counterfactuals. We only observe blue!

Note

The actual sample determines which comparisons we can make.

Why does this work? What effect are we identifying and how.

Let’s redo the simulated example with averages

rd1 %>%
  mutate(causal_effect = payoff_donation - payoff_no_donation) %>%
  summarise(M_causal = mean(causal_effect),
            sd_causal = sd(causal_effect),
            N = n()) %>%
  knitr::kable(format = "markdown", digits = 2)

M_causal	sd_causal	N
-2.14	3	1000

• The causal effect of donating for each firm is difference in payoff between donating and not donating.
• What effect are we estimating here?

Let’s redo the simulated example with averages

M_causal	sd_causal	N
-2.14	3	1000

rd1 %>%
  mutate(causal_effect = payoff_donation - payoff_no_donation) %>%
  group_by(donation) %>%
  summarise(M_causal = mean(causal_effect),
            sd_causal = sd(causal_effect),
            N = n()) %>%
  knitr::kable(format = "markdown", digits = 2)

donation	M_causal	sd_causal	N
0	-5	0	523
1	1	0	477

Let’s redo the regression with averages

summary_data  <- rd1 %>%
  group_by(donation) %>%
  summarise(M_payoff_donation = mean(payoff_donation),
            M_payoff_no_donation = mean(payoff_no_donation))
knitr::kable(summary_data, format = "markdown", digits = 2)

donation	M_payoff_donation	M_payoff_no_donation
0	-4.17	0.83
1	1.76	0.76

causal_effect_true <-
  summary_data$M_payoff_donation[summary_data$donation == 1] -
  summary_data$M_payoff_no_donation[summary_data$donation == 1]
causal_effect_reg <-
  summary_data$M_payoff_donation[summary_data$donation == 1] -
  summary_data$M_payoff_no_donation[summary_data$donation == 0]

Note

• The true ATT is 1
• The effect estimated by the regression is 0.926

If you do not believe me, here is the regression

rd1 <- mutate(rd1, actual_payoff =
       ifelse(donation, payoff_donation, payoff_no_donation))
ols <- feols(actual_payoff ~ donation, data = rd1)

	(1)
(Intercept)	0.833***
	(0.131)
donation	0.926***
	(0.190)
* p < 0.1, ** p < 0.05, *** p < 0.01

What could possibly go wrong?

rd2 <- tibble(
  high_performance = rbinom(N, 1, 0.5),
  noise = rnorm(N, 0, 3)) %>%
  mutate(
    donation = high_performance,
    performance = ifelse(high_performance == 1, 4, 1),
    payoff_donation = 4 - 8 / performance + noise,
    payoff_no_donation = ifelse(high_performance == 1, 1, 2) + noise
  )

Causal Effect Estimates with a Confounder

summary_data  <- rd2 %>%
  group_by(donation) %>%
  summarise(M_payoff_donation = mean(payoff_donation),
            M_payoff_no_donation = mean(payoff_no_donation))
knitr::kable(summary_data, format = "markdown", digits = 2)

donation	M_payoff_donation	M_payoff_no_donation
0	-4.01	1.99
1	1.97	0.97

causal_effect_true <-
  summary_data$M_payoff_donation[summary_data$donation == 1] -
  summary_data$M_payoff_no_donation[summary_data$donation == 1]
causal_effect_reg <-
  summary_data$M_payoff_donation[summary_data$donation == 1] -
  summary_data$M_payoff_no_donation[summary_data$donation == 0]

• The true ATT is 1
• The effect estimated by the regression is -0.024

A Simulated Example of Panel Data

Note

• We want to use the counterfactual as the control group
• Panel data + fixed effects is the next best thing

Where is the variation coming from?

We need firms that make mistakes

• Firms that should donate but do not always do it.
• Firms that should not donate but sometimes donate.

Panel Data Simulation (100 firms)

N <- 100
rd_firm <- tibble(
  firm = 1:N,
  high_performance = rbinom(N, 1, 0.5),
  other_payoff = rnorm(N, 0, 3)) %>%
  mutate(
    donation = high_performance,
    performance = ifelse(high_performance == 1, 4, 1),
    payoff_no_donation = ifelse(high_performance == 1, 1, 2) + other_payoff,
    payoff_donation = 4 - 8/performance + other_payoff
  )
summary_data  <- rd_firm %>%
  group_by(donation) %>%
  summarise(M_payoff_donation = mean(payoff_donation),
            M_payoff_no_donation = mean(payoff_no_donation))
knitr::kable(summary_data, digits = 1)

donation	M_payoff_donation	M_payoff_no_donation
0	-3.9	2.1
1	1.8	0.8

Panel Data Simulation (10 time periods)

The variation comes from high performers not donating some years

T <- 10
rd_panel_forget <- tibble(
  firm = rep(1:N, each = T),
  year = rep(1:T, times = N)) %>%
  left_join(rd_firm, by = "firm") %>%
  mutate(forget_donation = rbinom(N * T, 1, plogis(-other_payoff)),
         actual_donation = (1 - forget_donation) * donation,
         actual_payoff = ifelse(actual_donation == 1,
                                payoff_donation, payoff_no_donation))

• The way we simulate the data reflects the firm fixed effects and the time varying effects.
• Which effect are we identifying with this sample?

The New Assignment

1. Run a fixed effect model and interpret the result
2. Create a new dataset where all firms make mistakes
3. Run a fixed effect model and interpret the result

Instrumental Variable Approach

Causal Diagram

Causal Diagram

See Instrumental Variables, Chapter 19 in Huntington-Klein (2021).

Mechanically, there are two regressions. (2-stage-least-squares) 1. Use the IV to estimate the randomly generated variation in X -> fitted(X) 2. Use fitted(X) to estimate the effect of random variation in X on Y

Simulation and Implementation with fixest

#|label: simulation-iv
d <- tibble(
  iv = rnorm(N, 0, 1),
  confounder = rnorm(N, 0, 1)) %>%
  mutate(
    x = rnorm(N, .6 * iv - .6 * confounder, .6),
    y = rnorm(N, .6 * x + .6 * confounder, .6),
    survival = if_else(y > 0, 1, 0)
  )
surv <- filter(d, survival == 1)
lm1 <- lm(y ~ x, d)
lm2 <- lm(y ~ x + confounder, d)
lm3 <- lm(y ~ x, surv)
lm4 <- lm(y ~ x + confounder, surv)
iv1 <- feols(y ~ 1 | 0 | x ~ iv, data = d)
iv2 <- feols(y ~ 1 | 0 | x ~ iv, data = surv)

All the exogenous variable are in the tibble statement, all the endogenous variables are in the mutate statement. That is not a coincidence. It also highlights the value and tight link between being able to simulate your theory and understanding it.

Note, the collider bias is the biggest problem if the selection bias is on both x and y because then the collider bias effects the first stage regressions.

Simulation results with a real effect of 0.6

msummary(list("confounded" = lm1, "with control" = lm2, "collider" = lm3, "collider" = lm4,
              "iv no collider" = iv1, "iv with collider" = iv2),
         gof_omit = gof_omit, stars = stars)

	confounded	with control	collider	collider	iv no collider	iv with collider
(Intercept)	−0.082	−0.082	0.591***	0.442***	−0.095	0.546***
	(0.082)	(0.061)	(0.072)	(0.089)	(0.104)	(0.082)
x	0.257***	0.595***	0.103	0.293***
	(0.081)	(0.070)	(0.067)	(0.097)
confounder		0.644***		0.248**
		(0.070)		(0.095)
fit_x					0.874***	0.271**
					(0.195)	(0.129)
* p < 0.1, ** p < 0.05, *** p < 0.01

This is not strictly a collider because there is no effect of x on survival. However, it already shows that there are problems with “simple” selection bias.

Simulation without an effect

d <- tibble(
  iv = rnorm(N, 0, 1),
  confounder = rnorm(N, 0, 1)) %>%
  mutate(
    x = rnorm(N, .6 * iv - .6 * confounder, .6),
    y = rnorm(N, .6 * confounder, .6),
    survival = if_else(y > 0, 1, 0)
  )
surv <- filter(d, survival == 1)
lm1 <- lm(y ~ x, d)
lm2 <- lm(y ~ x + confounder, d)
lm3 <- lm(y ~ x, surv)
lm4 <- lm(y ~ x + confounder, surv)
iv1 <- feols(y ~ 1 | 0 | x ~ iv, data = d)
iv2 <- feols(y ~ 1 | 0 | x ~ iv, data = surv)

Simulation without an effect

msummary(list("confounded" = lm1, "with control" = lm2, "collider" = lm3, "collider" = lm4,
              "iv no collider" = iv1, "iv with collider" = iv2),
         gof_omit = gof_omit, stars = stars)

	confounded	with control	collider	collider	iv no collider	iv with collider
(Intercept)	0.017	0.005	0.696***	0.483***	0.057	0.714***
	(0.075)	(0.058)	(0.083)	(0.077)	(0.083)	(0.087)
x	−0.125*	0.151**	−0.020	0.118*
	(0.069)	(0.064)	(0.080)	(0.068)
confounder		0.619***		0.484***
		(0.077)		(0.093)
fit_x					0.181	0.035
					(0.112)	(0.113)
* p < 0.1, ** p < 0.05, *** p < 0.01

Example:Sitting Duck Governors by Falk and Shelton (2018)

Note

• Research Question: Does political uncertainty effect investment?
• More uncertainty in a state when governor does not come up for reelection.
• State level laws with term limits (~ Random)

An exercise to be run in class

Data

library(readit)
duck <- readit(here("data", "LameDuckData.dta")) %>%
  select(-starts_with("nstate"), -starts_with("stdum"),
         -starts_with("yd_alt")) %>%
  group_by(statename) %>%
  arrange(year) %>%
  mutate(log_I_1 = lag(log_I), log_I_2 = lag(log_I, 2),
         log_Y_1 = lag(log_Y), log_Y_2 = lag(log_Y, 2),
         log_real_GDP_1 = lag(log_real_GDP),
         log_real_GDP_2 = lag(log_real_GDP, 2)) %>%
  ungroup() %>%
  arrange(statename) %>%
  filter(year >= 1967, year <= 2004)

Reduced Form

form_red <- formula(
  log_I ~ gov_exogenous_middling + log_I_1 + log_I_2 +
  log_Y + log_Y_1 + log_Y_2 + log_real_GDP + log_real_GDP_1 +
  log_real_GDP_2 | statename
  )
red_reg <- feols(form_red, data = duck)

2 Stage Least Squares (2SLS)

form_iv <- formula(log_I ~ log_I_1 + log_I_2 +
  log_Y + log_Y_1 + log_Y_2 + log_real_GDP + log_real_GDP_1 +
  log_real_GDP_2
  # fixed effects
  | statename
  # 1st regression
  | uncertainty_continuous ~ gov_exogenous_middling
  )
iv_reg <- feols(form_iv, data = duck)

Results

coef_map = c("gov_exogenous_middling" = "lame duck governor",
             "fit_uncertainty_continuous" = "uncertainty")
msummary(list("reduced" = red_reg,
              "first stage iv" = summary(iv_reg, stage = 1),
              "second stage iv" = iv_reg),
         gof_omit = gof_omit, stars = stars,
         coef_map = coef_map)

	reduced	first stage iv	second stage iv
lame duck governor	−0.049**	1.801***
	(0.021)	(0.112)
uncertainty			−0.027**
			(0.012)
* p < 0.1, ** p < 0.05, *** p < 0.01

Diagnostics: Test for endogeneity (Durbin-Wu-Hausmann)

Note

Is the IV result different from the OLS result?

summ_iv <- summary(iv_reg)
summ_1st <- summary(iv_reg, stage = 1)
summ_iv$iv_wh$stat  # iv wu hausmann

[1] 3.829033

summ_iv$iv_wh$p     # iv wu hausmann

[1] 0.05054677

Instrumental Variables, Chapter 19 in Huntington-Klein (2021)

Diagnostics: Test for weak instrument

Note

Is the instrument predicting the variable we want it to predict?

fitstat(iv_reg, type = "ivf")

F-test (1st stage), uncertainty_continuous: stat = 331.0, p < 2.2e-16, on 1 and 1,640 DoF.

New Assignment

Let’s assume that firms are less likely to donate when there is a local election

N <- 5000
rd_iv_el <- tibble(
  high_performance = rbinom(N, 1, .5),
  extra_payoff = rnorm(N, 0, 3),
  local_election = rbinom(N, 1, .33)) %>%
  mutate(
    actual_donation = ifelse(high_performance == 1, 1 - local_election, 0),
    payoff_donation = ifelse(high_performance == 1, 2, - 4) + extra_payoff,
    payoff_no_donation = ifelse(high_performance == 1, 1, 2) + extra_payoff,
    actual_payoff = ifelse(actual_donation == 1,
                           payoff_donation, payoff_no_donation))

• Which effect can we identify with this data?
• Run the instrumental variable analyses and interpret the results.

A Published IV example

How Do Quasi-Random Option Grants Affect CEO Risk-Taking? by Shue and Townsend (2017) in The Journal of Finance

This paper is a finished product, your pitch, proposal, or dissertation is not.

We are grateful to Michael Roberts (the Editor), the Associate Editor, two anonymous referees, Marianne Bertrand, Ing-Haw Cheng, Ken French, Ed Glaeser, Todd Gormley, Ben Iverson (discus- sant), Steve Kaplan, Borja Larrain (discussant), Jonathan Lewellen, Katharina Lewellen, David Matsa (discussant), David Metzger (discussant), Toby Moskowitz, Candice Prendergast, Enrichetta Ravina (discussant), Amit Seru, and Wei Wang (discussant) for helpful suggestions. We thank seminar participants at AFA, BYU, CICF Conference, Depaul, Duke, Gerzensee ESSFM, Harvard, HKUST Finance Symposium, McGill Todai Conference, Finance UC Chile, Helsinki, IDC Herzliya Finance Conference, NBER Corporate Finance and Personnel Meetings, SEC, Simon Fraser Uni- versity, Stanford, Stockholm School of Economics, University of Amsterdam, UC Berkeley, UCLA, and Wharton for helpful comments. We thank David Yermack for his generosity in sharing data. We thank Matt Turner at Pearl Meyer, Don Delves at the Delves Group, and Stephen O’Byrne at Shareholder Value Advisors for helping us understand the intricacies of executive stock option plans. Menaka Hampole provided excellent research assistance. We acknowledge financial support from the Initiative on Global Markets.

On the one hand, we do not expect you to come up with a design like this. On the other hand, why not use these hard won insights.

This paper has 1 (one!) research question. This is a good thing!

It’s not necessarily advantageous to have too many hypotheses. You want to answer one question well.

Do increases in option grants increase risk taking?

Example of annoyances: Risk averse CEOs might take less risks and therefore receive more option grants.

IV 1: Scheduled Discrete Increases in Fixed-Value Option Grants

For our first instrument, we use fixed-value firms, for which option grants can increase only at regularly prescheduled intervals (i.e., when new cycles start). For example, consider a fixed-value firm on regular three-year cycles. Other time-varying factors may drive trends in risk for this firm. However, these trends are unlikely to coincide exactly with the timing of when new cycles are scheduled to start.

Basically saying the beginning of a cycle effect on option grants is not affected by the annoyances.

IV 2: Within Cycle Grant Increases due to Industry Shocks in Fixed-Number Option Grants

For our second instrument, we focus on fixed-number firms. The value of options granted in any particular year varies with aggregate returns within a fixed-number cycle. This means that the timing of increases in option pay within a cycle will be random in the sense that the increases are driven in part by industry shocks that are beyond the control of the firm and are largely unpredictable. To account for the possibility that aggregate returns can directly affect risk, we use fixed-value firms as a control group because their option compensation must remain fixed despite changes in aggregate returns.

The identifying assumption is that fixed-number vs fixed-value might be a part of the annoyances. So might the industry shocks. However, the IV assumes that the industry shocks are not different except in how they effect the option grant value.

The authors know their setting!

Our identification strategy builds on Hall’s (1999)) observation that firms often award options according to multiyear plans. Two types of plans are commonly used: fixed-number and fixed-value. Under a fixed-number plan, an executive receives the same number of options each year within a cycle. Under a fixed-value plan, an executive receives the same value of options each year within a cycle.

Our conversations with leading compensation consultants suggest that multiyear plans are used to minimize contracting costs, as option compensation only has to be set once every few years. Hall (1999, p. 97) argues that firms sort into the two types of plans somewhat arbitrarily, observing that “Boards seem to substitute one plan for another without much analysis or understanding of their differences.”

• Read qualitative studies and descriptions of actual practice!
• We are looking at “slightly suboptimal” decision making to get variation.

Key Assumption 1 - Relevance: IV is related to Option Grants

We find that the first-year indicator corresponds to a 15% larger increase in the Black-Scholes value of new option grants than in other years.

All estimates are highly significant, with F-statistics greatly exceeding 10, the rule of thumb threshold for concerns related to weak instruments (Staiger and Stock (1997). (III A.)

Chapter 19 Instrumental Variables in Huntington-Klein (2021)

Key Assumption 2 - Exclusion (or validity): Only path from IV to Risk Taking is through Option Grants.

One might be concerned that predicted first years provide exogenously timed but potentially anticipated increases in option compensation. However, this is not an issue for our empirical strategy. […] He would have no incentive to increase risk prior to an anticipated increase in the value of his option compensation next period.

In addition, we directly examine whether fixed-value cycles appear to be correlated with other firm cycles […]

Chapter 19 Instrumental Variables in Huntington-Klein (2021)

The key for the exclusion assumption is that anticipation would have an impact on the risk taking prior to the new cycle. This than would have an impact on the actual measure, i.e. the change in risk.

One Criticism

First, option compensation tends to follow an increasing step function for executives on fixed-value plans. This is because compensation tends to drift upward over time, yet executives on fixed-value plans cannot experience an upward drift within a cycle.

While these two stylized facts do not hold in all cases—as can also be seen in Figure 1—our identification strategy only requires that they hold on average.

Some more terminology

• Compliers
• Always-takers/never-takers
• Defiers

Chapter 19 Instrumental Variables in Huntington-Klein (2021)

The LATE is identified for the compliers. IV assumes that there are no defiers because now our estimated effect becomes an average of the defiers and compliers. One solution is to just remove the defiers if you can (which they do in the paper as a robustness check).

References

Falk, Nathan, and Cameron A. Shelton. 2018. “Fleeing a Lame Duck: Policy Uncertainty and Manufacturing Investment in US States.” American Economic Journal: Economic Policy 10 (4): 135–52. https://doi.org/10.1257/pol.20160365.

Gippel, Jennifer, Tom Smith, and Yushu Zhu. 2015. “Endogeneity in Accounting and Finance Research: Natural Experiments as a State-of-the-Art Solution.” Abacus 51 (2): n/a–. https://doi.org/10.1111/abac.12048.

Huntington-Klein, Nick. 2021. The Effect: An Introduction to Research Design and Causality. First. Boca Raton: Chapman and Hall/CRC. https://doi.org/10.1201/9781003226055.

Shue, Kelly, and Richard R. Townsend. 2017. “How Do Quasi-Random Option Grants Affect CEO Risk-Taking?” The Journal of Finance 72 (6): 2551–88. https://doi.org/10.1111/jofi.12545.