Is it valid to iterate over every permutation of a regression specification and compute an “average...
$begingroup$
I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.
Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.
Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?
regression least-squares
asked 6 hours ago
ParseltongueParseltongue
267114
$endgroup$
add a comment |
$begingroup$
I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.
Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.
Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?
regression least-squares
asked 6 hours ago
ParseltongueParseltongue
267114
$endgroup$
$begingroup$
I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
$endgroup$
– duckmayr
1 hour ago
add a comment |
$begingroup$
I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.
Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.
Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?
regression least-squares
asked 6 hours ago
ParseltongueParseltongue
267114
$endgroup$
I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.
Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.
Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?
regression least-squares
regression least-squares
asked 6 hours ago
ParseltongueParseltongue
267114
asked 6 hours ago
ParseltongueParseltongue
267114
asked 6 hours ago
ParseltongueParseltongue
267114
asked 6 hours ago
ParseltongueParseltongue
267114
asked 6 hours ago
ParseltongueParseltongue
267114
267114
$begingroup$
I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
$endgroup$
– duckmayr
1 hour ago
add a comment |
$begingroup$
I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
$endgroup$
– duckmayr
1 hour ago
$begingroup$
I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
$endgroup$
– duckmayr
1 hour ago
$begingroup$
I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
$endgroup$
– duckmayr
1 hour ago
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.
https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf
I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.
https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419
answered 6 hours ago
eric_kernfelderic_kernfeld
2,9921726
$endgroup$
add a comment |
$begingroup$
Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.
In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.
answered 6 hours ago
Bryan KrauseBryan Krause
495210
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f387998%2fis-it-valid-to-iterate-over-every-permutation-of-a-regression-specification-and%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.
https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf
I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.
https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419
answered 6 hours ago
eric_kernfelderic_kernfeld
2,9921726
$endgroup$
add a comment |
$begingroup$
One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.
https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf
I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.
https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419
answered 6 hours ago
eric_kernfelderic_kernfeld
2,9921726
$endgroup$
add a comment |
$begingroup$
One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.
https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf
I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.
https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419
answered 6 hours ago
eric_kernfelderic_kernfeld
2,9921726
$endgroup$
One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.
https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf
I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.
https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419
answered 6 hours ago
eric_kernfelderic_kernfeld
2,9921726
answered 6 hours ago
eric_kernfelderic_kernfeld
2,9921726
answered 6 hours ago
eric_kernfelderic_kernfeld
2,9921726
answered 6 hours ago
eric_kernfelderic_kernfeld
2,9921726
2,9921726
add a comment |
add a comment |
$begingroup$
Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.
In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.
answered 6 hours ago
Bryan KrauseBryan Krause
495210
$endgroup$
add a comment |
$begingroup$
Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.
In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.
answered 6 hours ago
Bryan KrauseBryan Krause
495210
$endgroup$
add a comment |
$begingroup$
Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.
In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.
answered 6 hours ago
Bryan KrauseBryan Krause
495210
$endgroup$
Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.
In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.
answered 6 hours ago
Bryan KrauseBryan Krause
495210
answered 6 hours ago
Bryan KrauseBryan Krause
495210
answered 6 hours ago
Bryan KrauseBryan Krause
495210
answered 6 hours ago
Bryan KrauseBryan Krause
495210
495210
add a comment |
add a comment |
Thanks for contributing an answer to Cross Validated!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f387998%2fis-it-valid-to-iterate-over-every-permutation-of-a-regression-specification-and%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
$endgroup$
– duckmayr
1 hour ago