roughness penalty

When a sample of n observations is taken from an unknown continuous distribution, the maximum likelihood estimate of that distribution consists of n spikes of probability (of size 1/n). This is an unrealistically 'rough' distribution. A roughness penalty is a modification of the likelihood function that results in the selection of a smoother estimated distribution. See also penalized likelihood.