A plethora of weights in Matching methods

I have been haunted by weights when applying matching methods to deal with causal inference problem. Things become more and more difficult when I have to deal with sampling weights and matching weights at the same time. So I had a small Q&A session with Professor Jennifer Hill, whom we look up for as an expert for causal inference. Here is our dialogue. (If you feel there is any wrong, please do comment on this entry. And all faults are on mine short memory that I cannot transcribe 100% what Professor Hill said!)

1. Does matching destroy the data structure thus make inference less-general to the population because of discarding un-matched units?
Yes and No!
(1) No: when your treated units are representative to your population (treated united is sampled with equal probably within your sampling strata) and you do NOT discard any treated unit, then the causal inference (treatment effects) using matching is still generalizables.
(2) Yes: If your treated units are not representative to your population (sampled with unequal probability) or you discard some of your treated units after matching, the treatment effects is not generalizables.

2. Is it still make sense to use multilevel modeling after matching given that we drop some unmatched units (control units)? Or how to do matching with multilevel data structure?
(1) We can still do multilevel modeling after matching and make general inference if our treated units are intact.
(2) One way to do is to do two-stage matching. First matching in the group level and matching in the individual level!!

3. Do we need to incorporate sampling weights into matching?
Yes, because our treated units are not representative to the whole population. If we want to make general causal inference, we have to take care of sampling weights.

4. How to do matching with sampling weights? Do we need to use weights in creating propensity scores?
(Should be, but not sure). One easier way to do it is to include the variable that was used to construct weights into the model.

5. What if we don't know which variable was used to construct weights?
One creative way is to use weights variable directly. Treat them as strata as categorical. So if there are many strata, we can merge some strata to make thing easier. (This is actually a point suggested by Professor Andrew Gelman)

My love affair with BART

Professor Jennifer Hill gave a vivid talk at NYU today discounting the matching methods. In the middle of the talk, I was wondering that maybe it is Professor Donald Rubin's great tutorship that his students, Professor Andrew Gelman & Professor Jennifer Hill, whom I am lucky to work with, are both great presenters.

I am on Jennifer's side that I have switched my support from matching methods to BART (Bayesian Additive Regression Tree) an year ago.

BART 1.0-0
I might be the first student around Columbia University who uses BART. I recall that the first time Jennifer introduced BART in the Quantitative Research Seminar at Columbia, I was so excited to know there was an alternative exists. I was so frustrated by how different matching methods could possibly yield different estimates. So after Jennifer's talk, I tried BART right away. I was disappointed with BARTbecause it failed on a n=40 fake data, which the true treatment effect is 4. BART gave me the treatment effect of 1.8. I did not know what's going on and I did not know if I should ask Jennifer. She was always cool, sitting in Andy's multilevel modeling class. I did not know she was such a great tutor like Andy.

BART 2.0-0
Jeronimo Cortina is a great friend of mine. He just got his Ph.D at Columbia this April. A year ago, he told me he was working with Jennifer. So I asked him if he knew what's going on with BART. He did not know and he was not a fan of BART then. But he was upset with matching methods too. I told him he should try BART and I shared with him how to use BART. I was actually hoping that he could use BART in this work. This way, I could know the feedback of the real applied case of BART.

I was actually disappointed to know that Andy's suggestion for using BART is to use it as a robust check. But anyway, I had a chance to talk to Jeninfer through Jeronimo. I knew then that BART 1.0-0 was buggy when n<200.>

Becoming a fan of BART
Now I had a BART friend-Jeronimo and a BART instructor-Jennifer. So I began to use BART in my work. Whenever I had a new idea of making BART graphs, I share it with Jeronimo. Whenever I had a question about BART I ask Jennifer. I was so excited when I finished a work of a trichotomous treatments analysis using BART. This can be a bit challenging using matching methods because it involves using multinomial logit model to get propensity scores.

Does BART have its future?
During today's talk, someone asked Jennifer a very tricky question. The question I also had in my mind for a long time. He asked that if she is the journal reviewer of a causal inference paper using BART, how she is going to judge if the guy is doing the right thing.

Andy was skeptical about BART because he does not like the black-box-like procedure of BART. We learned that BART does outperform other methods. But there is no way now to know what's really going on inside. What variables are used? What are the interaction? I am worried about the future of BART if we can not show reader these stuffs. It might need times for BART to build up its fame that everyone is convinced the superiority of BART and takes the result of BART for granted.

Jennifer complains about how economic journal reviewers always give her hard time when she uses matching method in her paper (They don't trust the claim of the ignorability using matching). Those reviewers might have the same feeling over BART. Although I am a fan of BART, I am worried about the future of BART. In the meantime, maybe I will only use BART as a robust check to other methods.