Is it valid to iterate over every permutation of a regression specification and compute an “average...












1












$begingroup$


I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






regression least-squares







share|cite|improve this question









$endgroup$












  • $begingroup$
    I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
    $endgroup$
    – duckmayr
    1 hour ago
















1












$begingroup$


I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






regression least-squares







share|cite|improve this question









$endgroup$












  • $begingroup$
    I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
    $endgroup$
    – duckmayr
    1 hour ago














1












1








1


1



$begingroup$


I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






regression least-squares







share|cite|improve this question









$endgroup$




I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner.



Let's say I run an OLS model and the results come back significant. There is discussion as to whether the positive results hold up given a different set of control variables. However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.



Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each. We can report the proportion of models where this coefficient is significant, but is there some kind of meta-test that can be done showing that the test statistic is significant in a significant number of alternative specifications? Is this ever done?






regression least-squares




regression least-squares






share|cite|improve this question













share|cite|improve this question











share|cite|improve this question




share|cite|improve this question










asked 6 hours ago









ParseltongueParseltongue

267114




267114












  • $begingroup$
    I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
    $endgroup$
    – duckmayr
    1 hour ago


















  • $begingroup$
    I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
    $endgroup$
    – duckmayr
    1 hour ago
















$begingroup$
I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
$endgroup$
– duckmayr
1 hour ago




$begingroup$
I don't have the time to write up a full explanation right now, but you may want to look into Leamer bounds
$endgroup$
– duckmayr
1 hour ago










2 Answers
2






active

oldest

votes


















4












$begingroup$

One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






share|cite|improve this answer









$endgroup$





















    0












    $begingroup$

    Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



    In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






    share|cite|improve this answer









    $endgroup$













      Your Answer





      StackExchange.ifUsing("editor", function () {
      return StackExchange.using("mathjaxEditing", function () {
      StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
      StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
      });
      });
      }, "mathjax-editing");

      StackExchange.ready(function() {
      var channelOptions = {
      tags: "".split(" "),
      id: "65"
      };
      initTagRenderer("".split(" "), "".split(" "), channelOptions);

      StackExchange.using("externalEditor", function() {
      // Have to fire editor after snippets, if snippets enabled
      if (StackExchange.settings.snippets.snippetsEnabled) {
      StackExchange.using("snippets", function() {
      createEditor();
      });
      }
      else {
      createEditor();
      }
      });

      function createEditor() {
      StackExchange.prepareEditor({
      heartbeatType: 'answer',
      autoActivateHeartbeat: false,
      convertImagesToLinks: false,
      noModals: true,
      showLowRepImageUploadWarning: true,
      reputationToPostImages: null,
      bindNavPrevention: true,
      postfix: "",
      imageUploader: {
      brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
      contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
      allowUrls: true
      },
      onDemand: true,
      discardSelector: ".discard-answer"
      ,immediatelyShowMarkdownHelp:true
      });


      }
      });














      draft saved

      draft discarded


















      StackExchange.ready(
      function () {
      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f387998%2fis-it-valid-to-iterate-over-every-permutation-of-a-regression-specification-and%23new-answer', 'question_page');
      }
      );

      Post as a guest















      Required, but never shown

























      2 Answers
      2






      active

      oldest

      votes








      2 Answers
      2






      active

      oldest

      votes









      active

      oldest

      votes






      active

      oldest

      votes









      4












      $begingroup$

      One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



      https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



      I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



      https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






      share|cite|improve this answer









      $endgroup$


















        4












        $begingroup$

        One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



        https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



        I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



        https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






        share|cite|improve this answer









        $endgroup$
















          4












          4








          4





          $begingroup$

          One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



          https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



          I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



          https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419






          share|cite|improve this answer









          $endgroup$



          One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.



          https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf



          I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.



          https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419







          share|cite|improve this answer












          share|cite|improve this answer



          share|cite|improve this answer










          answered 6 hours ago









          eric_kernfelderic_kernfeld

          2,9921726




          2,9921726

























              0












              $begingroup$

              Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



              In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






              share|cite|improve this answer









              $endgroup$


















                0












                $begingroup$

                Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






                share|cite|improve this answer









                $endgroup$
















                  0












                  0








                  0





                  $begingroup$

                  Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                  In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.






                  share|cite|improve this answer









                  $endgroup$



                  Bayesian Model Averaging (BMA) is a principled way to do something like what you are describing.



                  In short, with BMA, you have a collection of models and weight them according to the likelihood of each model.







                  share|cite|improve this answer












                  share|cite|improve this answer



                  share|cite|improve this answer










                  answered 6 hours ago









                  Bryan KrauseBryan Krause

                  495210




                  495210






























                      draft saved

                      draft discarded




















































                      Thanks for contributing an answer to Cross Validated!


                      • Please be sure to answer the question. Provide details and share your research!

                      But avoid …



                      • Asking for help, clarification, or responding to other answers.

                      • Making statements based on opinion; back them up with references or personal experience.


                      Use MathJax to format equations. MathJax reference.


                      To learn more, see our tips on writing great answers.




                      draft saved


                      draft discarded














                      StackExchange.ready(
                      function () {
                      StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f387998%2fis-it-valid-to-iterate-over-every-permutation-of-a-regression-specification-and%23new-answer', 'question_page');
                      }
                      );

                      Post as a guest















                      Required, but never shown





















































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown

































                      Required, but never shown














                      Required, but never shown












                      Required, but never shown







                      Required, but never shown







                      -

                      Popular posts from this blog

                      Loup dans la culture

                      Image
                      Loups sur une enluminure du Bestiaire d'Aberdeen Le loup est l'animal le plus emblématique de l'histoire de l'Eurasie, il était à l'honneur durant l'Antiquité chez la totalité des anciens peuples européens [ 1 ] . Sommaire 1 Le loup dans la mythologie 2 Les loups dans le folklore 2.1 L'évolution des mentalités 2.2 Le loup dans la psychanalyse 3 Le loup dans les œuvres culturelles 3.1 Le loup dans la littérature 3.1.1 Fables 3.1.2 Poésie 3.1.3 Romans 3.1.4 Littérature jeunesse 3.1.4.1 Quelques titres d'albums contemporains 3.1.5 Bande dessinée 3.2 Le loup à l'écran 3.2.1 Le cinéma et la télévision 3.2.2 Documentaires 3.2.3 Dessins animés 3.2.4 Jeux vidéo 3.3 Le loup en peinture et en sculpture 3.4 Le loup dans les œuvres musicales 4 Le loup et les noms propres 5 Bibliographie 6 Notes et références 7 Voir aussi 7.1 Articles connexes
                      Read more

                      How to solve the problem of ntp “Unable to contact time server” from KDE?

                      Image
                      1 0 I want to set date and time automatically but this error appears when I try to .. I'm using openSUSE, I tried different servers but it didn't work either, help please! It doesn't seem to be a network problem: $ ping pool.ntp.org 64 bytes from 41.78.128.17: icmp_seq=1 ttl=41 time=260 ms To the question of whether ntpd is installed and running: $ ps -C ntpd PID TTY TIME CMD i.e. there's no ntpd process running. $ rpm -qa | grep ntp yast2-ntp-client-3.1.12-1.7.noarch ntp-4.2.6p5-25.2.1.i586 Further information from the comments: $ sudo ntpdate pool.ntp.org 25 Sep 19:41:51 ntpdate[3162]: no server suitable for synchronization found networking kde opensuse ntp ntpd sh
                      Read more

                      ASUS Zenbook UX433/UX333 — Configure Touchpad-embedded numpad on Linux

                      Image
                      0 1 I have an ASUS Zenbook UX433,on which ASUS decided to innovate by embedding a Numpad inside the touchpad (see picture 1). Toggling between classic touchpad mode and numpad (also turns the touchpad LEDs on) can be done through the golden button in the top-right corner of the touchpad. However, on my Manjaro arch Linux config, this behavior does not work by default and I can only use the touchpad, no numpad. Has anyone been able to configure the numpad to work on Linux, or have any idea how this could be achieved? EDIT1: I have had a look at the official Windows drivers for the UX433FA TouchPad, but was not able to make anything of it. I tried installing them on Linux using tools like NDISWrapper, but they do not detect the TouchPad/Numpad hardware, and my understanding of drivers is too limited.
                      Read more