I have been haunted by weights when applying matching methods to deal with causal inference problem. Things become more and more difficult when I have to deal with sampling weights and matching weights at the same time. So I had a small Q&A session with Professor Jennifer Hill, whom we look up for as an expert for causal inference. Here is our dialogue. (If you feel there is any wrong, please do comment on this entry. And all faults are on mine short memory that I cannot transcribe 100% what Professor Hill said!)
1. Does matching destroy the data structure thus make inference less-general to the population because of discarding un-matched units?
Yes and No!
(1) No: when your treated units are representative to your population (treated united is sampled with equal probably within your sampling strata) and you do NOT discard any treated unit, then the causal inference (treatment effects) using matching is still generalizables.
(2) Yes: If your treated units are not representative to your population (sampled with unequal probability) or you discard some of your treated units after matching, the treatment effects is not generalizables.
2. Is it still make sense to use multilevel modeling after matching given that we drop some unmatched units (control units)? Or how to do matching with multilevel data structure?
(1) We can still do multilevel modeling after matching and make general inference if our treated units are intact.
(2) One way to do is to do two-stage matching. First matching in the group level and matching in the individual level!!
3. Do we need to incorporate sampling weights into matching?
Yes, because our treated units are not representative to the whole population. If we want to make general causal inference, we have to take care of sampling weights.
4. How to do matching with sampling weights? Do we need to use weights in creating propensity scores?
(Should be, but not sure). One easier way to do it is to include the variable that was used to construct weights into the model.
5. What if we don't know which variable was used to construct weights?
One creative way is to use weights variable directly. Treat them as strata as categorical. So if there are many strata, we can merge some strata to make thing easier. (This is actually a point suggested by Professor Andrew Gelman)
1 hour ago