Consistent estimator

This article may require copy-editing for grammar, style, cohesion, tone or spelling. You can assist by editing it now.  (July 2009)

A sequence of estimators {T₁, T₂, T₃, …} for parameter θ (true value of which is 4). This sequence is consistent: the estimators are getting more and more concentrated near the true value of θ. The limiting distribution of the sequence is a degenerate random variable which equals θ with probability 1.

In statistics, a sequence of estimators for parameter θ is said to be consistent (or asymptotically consistent) if this sequence converges in probability to θ. Intuitively, this means that estimators taken far enough in the sequence are more likely to be in the vicinity of the parameter being estimated, and in the limit they will be arbitrarily close to θ with probability one.

In practice one usually constructs a single estimator as a function of an available sample of size n, and then imagines being able to keep collecting data and expanding the sample ad infinitum. In this way one would obtain a sequence of estimators indexed by n and the notion of consistency will be understood as the sample size “tends to infinity”. If this sequence converges in probability to the true value of the parameter being estimated, we call it a consistent estimator; otherwise the estimator is said to be inconsistent.

This interpretation of sample size “growing to infinity” is pervasive in modern statistics, although it is also criticized as being too unrealistic. Indeed, in those rare cases when additional data can be collected, the statistician is usually forced to use more refined models as more data becomes available, meaning that the “true” parameter θ does not remain fixed and in fact its dimension expands.^[1]

Consistency with convergence in probability is sometimes referred to as weak consistency. The notion can be extended to other modes of convergence of random variables. In particular, a sequence is said to be strongly consistent if it converges almost surely to the true parameter.

Definition

Loosely speaking, an estimator T_n of θ is said to be consistent if ^[2]

where →^p denotes convergence in probability.

More precise definition takes into account the fact that θ is actually unknown, and thus the convergence above must take place for every possible value of θ: Let be a family of distributions, and let X^θ={X₁, X₂, … : X_i ~ p_θ} denote a sample from the distribution p_θ. Suppose {T_n(X^θ)} is a sequence of estimators for parameter g(θ). Usually T_n will be based on the first n observations of a sample. Then this sequence {T_n} is said to be (weakly) consistent on g(Θ) if ^[3]

or plugging in the definition of convergence in probability:

We use g(θ) in this definition instead of simply θ, because often one is interested in estimating only a certain subvector of the underlying parameter, and regard the rest of the components of θ as nuisance. In the next example we estimate the location parameter of the model, but not the scale:

Example: sample mean for normal random variables

Suppose one has a sequence of observations {X₁, X₂, …} from a N(μ,σ²) distribution. To estimate μ based on the first n observations we use the sample mean: T_n = (X₁ + … + X_n)/n. This defines a sequence of estimators, indexed by the sample size n.

Now, from the properties of the normal distribution, it is known that T_n will itself be normally distributed, with mean μ and variance σ²/n. Equivalently, has a standard normal distribution. Then

as n tends to infinity, for any fixed ε > 0. Therefore, the sequence T_n of sample means is consistent for the population mean μ.

Establishing consistency

The notion of asymptotic consistency is very close, almost synonymous to the notion of convergence in probability. As such, any theorem, lemma, or property which establishes convergence in probability may be used to prove the consistency. Many such tools exist:

• In order to demonstrate consistency directly from the definition one can use inequality ^[4]

  

  the most common choice for function h being either the absolute value (in which case it is known as Markov inequality), or the quadratic function (respectively Chebychev's inequality).
• Another useful result is the continuous mapping theorem: if T_n is consistent for θ and g(·) is a real-valued function continuous at point θ, then g(T_n) will be consistent for g(θ):^[5]

  

• Slutsky's theorem can be used to combine several different estimators, or an estimator with a non-random covergent sequence. If T_n →^pα, and S_n →^pβ, then ^[6]

  

• If estimator T_n is given by an explicit formula, then most likely the formula will employ sums of random variables, and then the law of large numbers can be invoked: for a sequence {X_n} of random variables and under suitable conditions,

  

• If estimator T_n is defined implicitly, for example as a value that maximizes certain objective function, then more complicated argument involving uniform convergence in probability or stochastic equicontinuity has to be used. ^[7]

See also

• Fisher consistency — alternative although rarely used concept of consistency for the estimators
• Consistent test — the notion of consistency in context of hypothesis testing

References

Further reading

• Amemiya, Takeshi (1985). Advanced Econometrics. Harvard University Press. ISBN 0-674-00560-0. 
• Le Cam, Lucien; Yang, Grace Lo (2000). Asymptotics in statistics (some basic concepts), 2nd ed.. Springer. ISBN 0-387-95036-2. 
• Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation, 2nd ed. Springer. ISBN 0-387-98502-6. 
• Newey, W.; McFadden, D. (1994). Large sample estimation and hypothesis testing. in Handbook of Econometrics, Vol. 4, Ch.36. Elsevier Science. ISBN 0-444-88766-0.