Kernel density estimation: Information from Answers.com

Kernel density estimation of 100 normally distributed random numbers using different smoothing bandwidths.

In statistics, kernel density estimation (or Parzen window method, named after Emanuel Parzen) is a non-parametric way of estimating the probability density function of a random variable. As an illustration, given some data about a sample of a population, kernel density estimation makes it possible to extrapolate the data to the entire population.

The Parzen window is also used in signal processing as a lag window, in such procedures as the Blackman-Tukey procedure.

A histogram can be thought of as a collection of point samples from a kernel density estimate for which the kernel is a uniform box the width of the histogram bin.

1 Definition
2 Intuition
3 Properties
4 Statistical implementation
5 Example of density estimation in R programming language
6 See also
7 External links
8 References

Definition

If x₁, x₂, ..., x_n ~ ƒ is an independent and identically-distributed sample of a random variable, then the kernel density approximation of its probability density function is

$\widehat{f}_h(x)=\frac{1}{nh}\sum_{i=1}^n K\left(\frac{x-x_i}{h}\right)$

where K is some kernel and h is a smoothing parameter called the bandwidth. Quite often K is taken to be a standard Gaussian function with mean zero and variance 1. Thus the variance is controlled indirectly through the parameter h:

$K\left(\frac{x-x_i}{h}\right) = {1 \over \sqrt{2\pi} }\,e^{-\frac{(x-x_i)^2}{2 h^2}}.$

Intuition

Although less smooth density estimators such as the histogram density estimator can be made to be asymptotically consistent, others are often either discontinuous or converge at slower rates than the kernel density estimator. Rather than grouping observations together in bins, the kernel density estimator can be thought to place small "bumps" at each observation, determined by the kernel function. The estimator consists of a "sum of bumps" and is clearly smoother as a result (see below image).

Six Gaussians (red) and their sum (blue). The Parzen window density estimate

f (x)

is obtained by dividing this sum by 6, the number of Gaussians. The variance of the Gaussians was set to 0.5. Note that where the points are denser the density estimate will have higher values.

This is closely related to the heat kernel (the fundamental solution to the heat equation) and can be seen as the amount of heat at a specific location $x$ if heat sources are placed into $x i$ . Similar methods are used to construct discrete Laplace operators on point clouds for manifold learning.

Properties

Let $\scriptstyle R(f,\hat f(x))$ be the L² risk function for ƒ. Under weak assumptions on ƒ and K,

$R(f,\hat f(x)) \approx \frac{1}{4}\sigma_k^4h^4\int(f''(x))^2\,dx + \frac{\int K^2(x)\,dx}{nh}\text{ where }\sigma_K^2 = \int x^2K(x)\,dx.$

By minimizing the theoretical risk function, it can be shown that the optimal bandwidth is

$h^* = \frac{c_1^{-2/5}c_2^{1/5}c_3^{-1/5}}{n^{1/5}}$

where

$c_1 = \int x^2K(x)\,dx$

$c_2 = \int K(x)^2\,dx$

$c_3 = \int (f''(x))^2\,dx.$

The values of $c 1$ and $c 2$ can be easily computed for some common choices of K. The value of $c 3$ depends on the distribution of $f$ and must be estimated, since $f$ itself is usually not known. A popular choice is the solve-the-equation plug-in algorithm. Other options to optimize the bandwidth include leave-one-out cross validation.^[1]

When the optimal choice of bandwidth is chosen, the risk function is $\scriptstyle R(f, \hat f(x))\, \approx\, \tfrac{c_4}{n^{4/5}}$ for some constant c₄ > 0. It can be shown that, under weak assumptions, there cannot exist a non-parametric estimator that converges at a faster rate than the kernel estimator^{[citation needed]}. Note that the n^−4/5 rate is slower than the typical n⁻¹ convergence rate of parametric methods.

Statistical implementation

In MATLAB, kernel density estimation is implemented through the ksdensity function, but this function does not provide automatic data-driven bandwidth. A free MATLAB software package which implements an automatic bandwidth selection method is available from the MATLAB Central File Exchange for 1 dimensional data and for 2 dimensional data.
An example of implementing kernel density estimation in Mathematica is available here.
In Stata, it is implemented through kdensity; for example histogram x, kdensity.
In R, it is implemented through the density function (univariate), the kde2d function in the MASS package (bivariate), and the npudens function in the np package (multivariate allowing for numeric and categorical data types with a range of kernel functions and automatic bandwidth selectors).
In SAS, proc kde can be used to estimate univariate and bivariate kernel densities.
In SciPy, scipy.stats.gaussian_kde can be used to perform gaussian kernel density estimation in arbitrary dimensions, including bandwidth estimation.
In Perl, an implementation can be found in the Statistics-KernelEstimation module [1]
In Java, the Weka (machine learning) package provides weka.estimators.KernelEstimator[2], among others.
In Gnuplot, kernel desity esimation is implemented by the smooth kdensity option, the datafile can contain weight and bandwidth for each point, or the bandwidth can be set automatically.
In ESRI products, kernel density mapping is managed out of the Spatial Analyst toolbox and uses the Silverman (1986) quadratic kernal function.
For Haskell, kernel density is implemented in the statistics package.

Example of density estimation in R programming language

This example is based on the Old Faithful Geyser, a tourist attraction located in Yellowstone National Park. This famous dataset containing 272 records consists of two variables, eruption duration (minutes) and waiting time until the next eruption (minutes). The data is taken from the datasets package of the R programming language.

We consider estimating the joint density of eruption duration and waiting times using the R np package that employs automatic (data-driven) bandwidth selection that is optimal for joint density estimates; see the np vignette for an introduction to the np package. The figure below shows the joint density estimate using a second order Gaussian kernel.

Estimated joint density.

Script for the example

The following commands of the R programming language use the npudens() function to deliver optimal smoothing and to create the figure given above. These commands can be entered at the command prompt by using cut and paste.

 library(np)
 library(datasets)
 data(faithful)
 
 f <- npudens(~eruptions+waiting,data=faithful)
 
 plot(f,view="fixed",neval=100,phi=30,main="",xtrim=-0.2)

External links

Introduction to kernel density estimation
Free MATLAB code for one and two dimensional kernel density estimation
Free Online Software (Calculator) computes the Kernel Density Estimation for any data series according to the following Kernels: Gaussian, Epanechnikov, Rectangular, Triangular, Biweight, Cosine, and Optcosine.
FIGTree is a fast library that can be used to compute Kernel Density Estimates using a Gaussian Kernel. MATLAB and C/C++ interfaces available.
Good summary of methods to choose bandwidth Bandwidth Selection in Kernel Density Estimation: A Review.
The np package An R package that provides a variety of nonparametric and semiparametric kernel methods that seamlessly handle a mix of continuous, unordered, and ordered factor data types.
Kernel Density Estimation Applet An online interactive example of kernel density estimation. Requires .NET 3.0 or later.
Mathematica KDE demo An interactive demo in Mathematica that can be explored using the free player.

References

^ Finn Årup Nielsen; Lars Kai Hansen (March 2002). "Modeling of activation data in the BrainMap^TM: Detection of outliers". Human Brain Mapping 15 (3): 146–156. doi:10.1002/hbm.10012. PMID 11835605. http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=197.

Li, Q. and Racine, J.S. Nonparametric Econometrics: Theory and Practice. Princeton University Press, 2007, ISBN 0691121613.
Parzen E. (1962). On estimation of a probability density function and mode, Ann. Math. Stat. 33, pp. 1065–1076.
Duda, R. and Hart, P. (1973). Pattern Classification and Scene Analysis. John Wiley & Sons. ISBN 0-471-22361-1.
Wasserman, L. (2005). All of Statistics: A Concise Course in Statistical Inference, Springer Texts in Statistics.

This entry is from Wikipedia, the leading user-contributed encyclopedia. It may not have been reviewed by professional editors (see full disclaimer)

Donate to Wikimedia

	A farmer planted a field of Bt 123 corn and wants to estimate the yield in terms of bushels per acre He counts 22 ears in 11000 of an acre He determines that each ear has about 700 kernels on avera? Read answer...
	What is a kernel? Read answer...
	Is monolithic kernel are macro kernels? Read answer...

	How can you compare Value at risk and Expected shortfall using the kernel density histogram?
	How do you estimate the average density of your entire body?
	Methods of estimation population density of animals?

Kernel density estimation

Contents