Distribution of Maximum Likelihood Estimator
$begingroup$
Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)
Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$
Then after taking sample of size n
$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$
And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)
Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$
where
$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$
Taking the derivative with respect to $theta$
$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$
$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?
probability distributions normal-distribution estimation sampling
asked 3 hours ago
Colin HicksColin Hicks
1353
New contributor
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)
Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$
Then after taking sample of size n
$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$
And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)
Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$
where
$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$
Taking the derivative with respect to $theta$
$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$
$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?
probability distributions normal-distribution estimation sampling
asked 3 hours ago
Colin HicksColin Hicks
1353
New contributor
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
add a comment |
$begingroup$
Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)
Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$
Then after taking sample of size n
$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$
And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)
Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$
where
$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$
Taking the derivative with respect to $theta$
$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$
$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?
probability distributions normal-distribution estimation sampling
asked 3 hours ago
Colin HicksColin Hicks
1353
New contributor
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
Why is the Maximum Likelihood Estimator Normally distributed? I can't figure out why it is true for large n in general. My attempt (for single parameter)
Let $L(theta)$ be the maximum likelihood function for the distribution $f(x;theta)$
Then after taking sample of size n
$$L(theta)=f(x_1;theta)cdot f(x_2theta)...f(x_n;theta)$$
And we want to find $theta_{max}$ such that $L(theta)$ is maximized and $theta_{max}$ is our estimate (once a sample has actually been selected)
Since $theta_{max}$ maximizes $L(theta)$ it also maximizes $ln(L(theta))$
where
$$ln(L(theta))=ln(f(x_1;theta))+ln(f(x_2;theta))...+ln(f(x_n;theta))$$
Taking the derivative with respect to $theta$
$$frac{f'(x_1;theta)}{f(x_1;theta)}+frac{f'(x_2;theta)}{f(x_2;theta)}...+frac{f'(x_n;theta)}{f(x_n;theta)}$$
$theta_{max}$ would be the solution of the above when set to 0 (after selecting values for all $x_1,x_2...x_n$) but why is it normally distributed and how do I show that it's true for large n?
probability distributions normal-distribution estimation sampling
probability distributions normal-distribution estimation sampling
asked 3 hours ago
Colin HicksColin Hicks
1353
New contributor
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 3 hours ago
Colin HicksColin Hicks
1353
New contributor
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 3 hours ago
Colin HicksColin Hicks
1353
New contributor
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 3 hours ago
Colin HicksColin Hicks
1353
asked 3 hours ago
Colin HicksColin Hicks
1353
1353
New contributor
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Colin Hicks is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.
answered 3 hours ago
dlnBdlnB
3917
New contributor
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
$frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
$endgroup$
– Colin Hicks
3 hours ago
$begingroup$
You're welcome :)
$endgroup$
– dlnB
3 hours ago
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f397619%2fdistribution-of-maximum-likelihood-estimator%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.
answered 3 hours ago
dlnBdlnB
3917
New contributor
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
$frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
$endgroup$
– Colin Hicks
3 hours ago
$begingroup$
You're welcome :)
$endgroup$
– dlnB
3 hours ago
add a comment |
$begingroup$
MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.
answered 3 hours ago
dlnBdlnB
3917
New contributor
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
$begingroup$
$frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
$endgroup$
– Colin Hicks
3 hours ago
$begingroup$
You're welcome :)
$endgroup$
– dlnB
3 hours ago
add a comment |
$begingroup$
MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.
answered 3 hours ago
dlnBdlnB
3917
New contributor
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$endgroup$
MLE requires $$frac{partial ln L(theta)}{partial theta} = sum_{i=1}^n frac{ f'(x_i;theta)}{f(x_i;theta)},$$
where $f'(x_i;theta)$ could denote a gradient (allowing for the multivariate case, but still sticking to your notation). Define a new function $g(x;theta)=frac{ f'(x;theta)}{f(x;theta)}.$ Then ${g(x_i;theta)}_{i=1}^n$ is a new iid sequence of random variables, with $Eg(x_1;theta)=0$. If $Eg(x_1;theta)g(x_1;theta)'<infty$, CLT implies,
$$sqrt{n}(bar{g}_n(theta)-Eg(x_1;theta))=sqrt{n}bar{g}_n(theta) rightarrow_D N(0,E(g(x;theta)g(x;theta)'),$$
where $bar{g}_n(theta)=frac{1}{n} sum_{i=1}^n g(x_i;theta).$ The ML estimator solves the equation
$$bar{g}_n(theta)=0.$$
It follows that the ML estimator is given by
$$hat{theta}=bar{g}_n^{-1}(0).$$
So long as the set of discontinuity points of $bar{g}_n^{-1}(z)$, i.e. the set of all values of $z$ such that $bar{g}_n^{-1}(z)$ is not continuous, occur with probability zero, the continuous mapping theorem gives us asymptotic normality of $theta$.
answered 3 hours ago
dlnBdlnB
3917
New contributor
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited 2 hours ago
answered 3 hours ago
dlnBdlnB
3917
New contributor
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
answered 3 hours ago
dlnBdlnB
3917
answered 3 hours ago
dlnBdlnB
3917
3917
New contributor
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
dlnB is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
$begingroup$
$frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
$endgroup$
– Colin Hicks
3 hours ago
$begingroup$
You're welcome :)
$endgroup$
– dlnB
3 hours ago
add a comment |
$begingroup$
$frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
$endgroup$
– Colin Hicks
3 hours ago
$begingroup$
You're welcome :)
$endgroup$
– dlnB
3 hours ago
$begingroup$
$frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
$endgroup$
– Colin Hicks
3 hours ago
$begingroup$
$frac{f'(x)}{f(x)}$ is just another function of x so central limit theorem applies thank you for that
$endgroup$
– Colin Hicks
3 hours ago
$begingroup$
You're welcome :)
$endgroup$
– dlnB
3 hours ago
$begingroup$
You're welcome :)
$endgroup$
– dlnB
3 hours ago
add a comment |
Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.
Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.
Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.
Colin Hicks is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Cross Validated!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f397619%2fdistribution-of-maximum-likelihood-estimator%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
