sufficiency: Definition, Synonyms from Answers.com

In statistics, a sufficient statistic is a statistic which has the property of sufficiency with respect to a statistical model and its associated unknown parameter, meaning that "no other statistic which can be calculated from the same sample provides any additional information as to the value of the parameter".^[1] More generally, the "unknown parameter" may represent a vector of unknown quantities or may represent everything about the model that is unknown or not fully specified. A statistic is sufficient for a family of probability distributions if the sample from which it is calculated gives no additional information than does the statistic, as to which of those probability distributions is that of the population from which the sample was taken.

The concept, due to Sir Ronald Fisher, is equivalent to the statement that, conditional on the value of a sufficient statistic for a parameter, the joint probability distribution of the data is not a function of that parameter. Both the statistic and the underlying parameter can be vectors.

The concept of sufficiency has fallen out of favor in descriptive statistics because of the strong dependence on an assumption of the distributional form (see Pitman–Koopman–Darmois theorem below), but remains very important in theoretical work.^[2]

1 Mathematical definition
- 1.1 Example
2 Fisher–Neyman factorization theorem
- 2.1 Likelihood Principle Interpretation
- 2.2 Proof
3 Minimal sufficiency
4 Examples
5 Rao–Blackwell theorem
6 Exponential family
7 See also
8 Notes
9 References

Mathematical definition

The concept is most general when defined as follows: a statistic T(X) is sufficient for underlying parameter θ precisely if the conditional probability distribution of the data X, given the statistic T(X), is not a function of the parameter θ,^[3] i.e.

$\Pr(X=x|T(X)=t,\theta) = \Pr(X=x|T(X)=t), \,$

or in shorthand

$\Pr(x|t,\theta) = \Pr(x|t).\,$

Example

As an example, the sample mean is sufficient for the mean (μ) of a normal distribution with known variance. Once the sample mean is known, no further information about μ can be obtained from the sample itself.

Fisher–Neyman factorization theorem

Fisher's factorization theorem or factorization criterion provides a convenient characterization of a sufficient statistic. If the probability density function is ƒ_θ(x), then T is sufficient for θ if and only if functions g and h can be found such that

$f_\theta(x)=h(x) \, g_\theta(T(x)), \,\!$

i.e. the density ƒ can be factored into a product such that one factor, h, does not depend on θ and the other factor, which does depend on θ, depends on x only through T(x).

Likelihood Principle Interpretation

An implication of the theorem is that when using likelihood-based inference, two sets of data yielding the same value for the sufficient statistic T(X) will always yield the same inferences about θ. By the factorization criterion, the likelihood's dependence on θ is only in conjunction with T(X). As this is the same in both cases, the dependence on θ will be the same as well, leading to identical inferences.

Proof

Due to Hogg and Craig ^[4]. Let $X_1, X_2, \ldots, X_n$ , denote a random sample from a distribution having the pdf f(x,θ) for γ < θ < δ. Let Y = u(X₁, X₂, ..., X_n) be a statistic whose pdf is g(y;θ). Then Y = u(X₁, X₂, ..., X_n) is a sufficient statistic for θ if and only if, for some function H,

$\prod_{i=1}^{n} f(x_i; \theta) = g \left[u(x_1, x_2, \dots, x_n); \theta \right] H(x_1, x_2, \dots, x_n). \,\!$

First, suppose that

$\prod_{i=1}^{n} f(x_i; \theta) = g \left[u(x_1, x_2, \dots, x_n); \theta \right] H(x_1, x_2, \dots, x_n). \,\!$

We shall make the transformation y_i = u_i(x₁, x₂, ..., x_n), for i = 1, ..., n, having inverse functions x_i = w_i(y₁, y₂, ..., y_n), for i = 1, ..., n, and Jacobian $J = \left[w_i/y_j \right]$ . Thus,

$\prod_{i=1}^{n} f \left[ w_i(y_1, y_2, \dots, y_n); \theta \right] = |J| g(y; \theta) H \left[ w_1(y_1, y_2, \dots, y_n), \dots, w_n(y_1, y_2, \dots, y_n) \right].$

The left-hand member is the joint pdf g(y₁, y₂, ..., y_n; θ) of Y₁ = u₁(X₁, ..., X_n), ..., Y_n = u_n(X₁, ..., X_n). In the right-hand member, $g(y_1,\dots,y_n;\theta)$ is the pdf of $Y 1$ , so that $H[ w_1, \dots , w_n] |J|$ is the quotient of $g(y_1,\dots,y_n;\theta)$ and $g 1 (y 1;θ)$ ; that is, it is the conditional pdf $h(y_2, \dots, y_n | y_1; \theta)$ of $Y_2,\dots,Y_n$ given $Y 1 = y 1$ .

But $H(x_1,x_2,\dots,x_n)$ , and thus $H\left[w_1(y_1,\dots,y_n), \dots, w_n(y_1, \dots, y_n))\right]$ , was given not to depend upon $θ$ . Since $θ$ was not introduced in the transformation and accordingly not in the Jacobian $J$ , it follows that $h(y_2, \dots, y_n | y_1; \theta)$ does not depend upon $θ$ and that $Y 1$ is a sufficient statistics for $θ$ .

The converse is proven by taking:

$g(y_1,\dots,y_n;\theta)=g_1(y_1; \theta) h(y_2, \dots, y_n | y_1),\,$

where $h(y_2, \dots, y_n | y_1)$ does not depend upon $θ$ because $Y 2 ... Y n$ depend only upon $X 1 ... X n$ which are independent on $Θ$ when conditioned by $Y 1$ , a sufficient statistics by hypothesis. Now divide both members by the absolute value of the non-vanishing Jacobian $J$ , and replace $y_1, \dots, y_n$ by the functions $u_1(x_1, \dots, x_n), \dots, u_n(x_1,\dots, x_n)$ in $x_1,\dots, x_n$ . This yields

$\frac{g\left[ u_1(x_1, \dots, x_n), \dots, u_n(x_1, \dots, x_n); \theta \right]}{|J*|}=g_1\left[u_1(x_1,\dots,x_n); \theta\right] \frac{h(u_2, \dots, u_n | u_1)}{|J*|}$

where $J *$ is the Jacobian with $y_1,\dots,y_n$ replaced by their value in terms $x_1, \dots, x_n$ . The left-hand member is necessarily the joint pdf $f(x_1;\theta)\cdots f(x_n;\theta)$ of $X_1,\dots,X_n$ . Since $h(y_2,\dots,y_n|y_1)$ , and thus $h(u_2,\dots,u_n|u_1)$ , does not depend upon $θ$ , then

$H(x_1,\dots,x_2)=\frac{h(u_2,\dots,u_n|u_1)}{|J*|}$

is a function that does not depend upon $θ$ .

We use the shorthand notation to denote the joint probability of $(X, T (X))$ by $f θ (x, t)$ . Since $T$ is a function of $X$ , we have $f θ (x, t) = f θ (x)$ and thus:

f θ (x) = f θ (x, t) = f θ | t (x) f θ (t)

with the last equality being true by the definition of conditional probability distributions. Thus $f θ (x) = a (x) b θ (t)$ with $a (x) = f θ | t (x)$ and $b (x) = f θ (t)$ .

Reciprocally, if $f θ (x) = a (x) b θ (t)$ , we have

$\begin{align} f_\theta(t) & = \sum _{x : T(x) = t} f_\theta(x, t) \\ & = \sum _{x : T(x) = t} f_\theta(x) \\ & = \sum _{x : T(x) = t} a(x) b_\theta(t) \\ & = \left( \sum _{x : T(x) = t} a(x) \right) b_\theta(t). \end{align}$

With the first equality by the definition of pdf for multiple variables, the second by the remark above, the third by hypothesis, and the fourth because the summation is not over $t$ .

Thus, the conditional probability distribution is:

$\begin{align} f_{\theta|t}(x) & = \frac{f_\theta(x, t)}{f_\theta(t)} \\ & = \frac{f_\theta(x)}{f_\theta(t)} \\ & = \frac{a(x) b_\theta(t)}{\left( \sum _{x : T(x) = t} a(x) \right) b_\theta(t)} \\ & = \frac{a(x)}{\sum _{x : T(x) = t} a(x)}. \end{align}$

With the first equality by definition of conditional probability density, the second by the remark above, the third by the equality proven above, and the fourth by simplification. This expression does not depend on $θ$ and thus $T$ is a sufficient statistic.^[5]

Minimal sufficiency

A sufficient statistic is minimal sufficient if it can be represented as a function of any other sufficient statistic. In other words, S(X) is minimal sufficient if and only if

S(X) is sufficient, and
if T(X) is sufficient, then there exists a function f such that S(X) = f(T(X)).

Intuitively, a minimal sufficient statistic most efficiently captures all possible information about the parameter θ.

A useful characterization of minimal sufficiency is that when the density f_θ exists, S(X) is minimal sufficient if and only if

$\frac{f_\theta(x)}{f_\theta(y)}$ is independent of θ : $\Longleftrightarrow$ S(x) = S(y)

This follows as a direct consequence from Fisher's factorization theorem stated above.

A case in which there is no minimal sufficient statistic was shown by Bahadur, 1957^[6] . However, under mild conditions, a minimal sufficient statistic does always exist. In particular, in Euclidean space, these conditions always hold if the random variables (associated with Pθ ) are all discrete or are all continuous.

If there exists a minimal sufficient statistic, and this is usually the case, then every complete sufficient statistic is necessarily minimal sufficient^[7](note that this statement does not exclude the option of a pathological case in which a complete sufficient exists while there is no minimal sufficient statistic). While it is hard to find cases in which a minimal sufficient statistic does not exist, it is not so hard to find cases in which there is no complete statistic.

The collection of likelihood ratios $\left\{\frac{L(\theta_1|X)}{L(\theta_2|X)}\right\}$ is a minimal sufficient statistic if $P (X | θ)$ is discrete or has a density function.

Examples

Bernoulli distribution

If X₁, ...., X_n are independent Bernoulli-distributed random variables with expected value p, then the sum T(X) = X₁ + ... + X_n is a sufficient statistic for p (here 'success' corresponds to X_i = 1 and 'failure' to X_i = 0; so T is the total number of successes)

This is seen by considering the joint probability distribution:

$\Pr\{X=x\}=\Pr\{X_1=x_1,X_2=x_2,\ldots,X_n=x_n\}.$

Because the observations are independent, this can be written as

$p^{x_1}(1-p)^{1-x_1} p^{x_2}(1-p)^{1-x_2}\cdots p^{x_n}(1-p)^{1-x_n} \,\!$

and, collecting powers of p and 1 − p, gives

$p^{\sum x_i}(1-p)^{n-\sum x_i}=p^{T(x)}(1-p)^{n-T(x)} \,\!$

which satisfies the factorization criterion, with h(x) = 1 being just a constant.

Note the crucial feature: the unknown parameter p interacts with the data x only via the statistic T(x) = Σ x_i.

Uniform distribution

Poisson distribution

If X₁, ...., X_n are independent and have a Poisson distribution with parameter λ, then the sum T(X) = X₁ + ... + X_n is a sufficient statistic for λ.

To see this, consider the joint probability distribution:

$\Pr(X=x)=P(X_1=x_1,X_2=x_2,\ldots,X_n=x_n). \,$

Because the observations are independent, this can be written as

${e^{-\lambda} \lambda^{x_1} \over x_1 !} \cdot {e^{-\lambda} \lambda^{x_2} \over x_2 !} \cdot\,\cdots\,\cdot {e^{-\lambda} \lambda^{x_n} \over x_n !} \,\!$

which may be written as

$e^{-n\lambda} \lambda^{(x_1+x_2+\cdots+x_n)} \cdot {1 \over x_1 ! x_2 !\cdots x_n ! } \,\!$

which shows that the factorization criterion is satisfied, where h(x) is the reciprocal of the product of the factorials. Note the parameter λ interacts with the data only through its sum T(X).

Rao–Blackwell theorem

Sufficiency finds a useful application in the Rao–Blackwell theorem. It states that if g(X) is any kind of estimator of θ, then typically the conditional expectation of g(X) given T(X) is a better estimator of θ, and is never worse. Sometimes one can very easily construct a very crude estimator g(X), and then evaluate that conditional expected value to get an estimator that is in various senses optimal.

Exponential family

Main article: Exponential family

According to the Pitman–Koopman–Darmois theorem, among families of probability distributions whose domain does not vary with the parameter being estimated, only in exponential families is there a sufficient statistic whose dimension remains bounded as sample size increases. Less tersely, suppose $X_n, n = 1, 2, 3, \dots$ are independent identically distributed random variables whose distribution is known to be in some family of probability distributions. Only if that family is an exponential family is there a (possibly vector-valued) sufficient statistic $T(X_1, \dots, X_n)$ whose number of scalar components does not increase as the sample size n increases.

This theorem shows that sufficiency (or rather, the existence of a scalar or vector-valued of bounded dimension sufficient statistic) sharply restricts the possible forms of the distribution.

Notes

^ Fisher, R.A. (1922). "On the mathematical foundations of theoretical statistics". Philosophical Transactions of the Royal Society of London. Series A 222: 309–368. doi:10.1098/rsta.1922.0009. JSTOR 91208. JFM 48.1280.02. http://digital.library.adelaide.edu.au/dspace/handle/2440/15172.
^ Stigler, Stephen (December 1973). "Studies in the History of Probability and Statistics. XXXII: Laplace, Fisher and the Discovery of the Concept of Sufficiency". Biometrika 60 (3): 439–445. doi:10.1093/biomet/60.3.439. MR 0326872, JSTOR 2334992.
^ Casella, George; Berger, Roger L. (2002). Statistical Inference, 2nd ed. Duxbury Press.
^ Hogg, Robert V.; Craig, Allen T. (1995). Introduction to Mathematical Statistics. Prentice Hall. ISBN 9780023557224.
^ "The Fisher–Neyman Factorization Theorem". http://cnx.org/content/m11480/1.6/.
^ Lehmann and Casella
^ Lehmann and Casella , page 72

References

Kholevo, A.S. (2001), "Sufficient statistic", in Hazewinkel, Michiel, Encyclopaedia of Mathematics, Kluwer Academic Publishers, ISBN 978-1556080104, http://eom.springer.de/S/s091070.htm
Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation. Springer. pp. 2nd ed, ch. 4. ISBN 0-387-98502-6.

This entry is from Wikipedia, the leading user-contributed encyclopedia. It may not have been reviewed by professional editors (see full disclaimer)

Donate to Wikimedia

		Dictionary. The American Heritage® Dictionary of the English Language, Fourth Edition Copyright © 2007, 2000 by Houghton Mifflin Company. Updated in 2009. Published by Houghton Mifflin Company. All rights reserved. Read more
		Thesaurus. Roget's II: The New Thesaurus, Third Edition by the Editors of the American Heritage® Dictionary Copyright © 1995 by Houghton Mifflin Company. Published by Houghton Mifflin Company. All rights reserved. Read more
		WordNet. WordNet 1.7.1 Copyright © 2001 by Princeton University. All rights reserved. Read more
		Wikipedia. This article is licensed under the Creative Commons Attribution/Share-Alike License. It uses material from the Wikipedia article "Sufficient statistic". Read more

	Is Lycopodium-200 sufficient? Read answer...
	Is sufficient a verb? Read answer...
	What is the meaning of 'sufficient'? Read answer...

	What to do When there is sufficient water?
	What is sufficiency rating?
	Sufficiency of consideration?

sufficiency

Contents