Foreword for "Least Squares Data Fitting with Applications" by P. C. Hansen,
V. Pereyra and G. Scherer. To be published by Johns Hopkins University Press, 2012.
Scientific computing is founded in models that capture the properties
of systems under investigation, be they engineering systems; systems
in natural sciences; financial, economic, and social systems; or
conceptual systems such as those that arise in machine learning or
speech processing. For models to be useful, they must be calibrated
against "real-world" systems and informed by data. The recent
explosion in availability of data opens up unprecendented
opportunities to increase the fidelity, resolution, and power of
models --- but only if we have access to algorithms for incorporating
this data into models, effectively and efficiently.
For this reason, least squares --- the first and best-known technique
for fitting models to data --- remains central to scientific
computing. This problem class remains a fascinating topic of study
from a variety of perspectives. Least-squares formulations can be
derived from statistical principles, as maximum-likelihood estimates
of models in which the model-data discrepancies are assumed to arise
from Gaussian white noise. In scientific computing, they provide the
vital link between model and data, the final ingredient in a model
that brings the other elements together. In their linear variants,
least-squares problems were a foundational problem in numerical linear
algebra, as this field grew rapidly in the 1960s and 1970s. From the
perspective of optimization, nonlinear least-squares has appealing
structure that can be exploited with great effectiveness in algorithm
design.
Least squares is foundational in another respect: It can be extended
in a variety of ways: to alternative loss functions that are more
robust to outliers in the observations, to one-sided "hinge loss"
functions, to regularized models that impose structure on the model
parameters in addition to fitting the data, and to "total least
squares" models in which errors appear in the model coefficients as
well as the observations.
This book surveys least-squares problems from all these
perspectives. It is both a comprehensive introduction to the subject
and a valuable resource to those already well versed in the area. It
covers statistical motivations along with thorough treatments of
direct and iterative methods for linear least squares and optimization
methods for nonlinear least squares. The later chapters contain
compelling case studies of both linear and nonlinear models, with
discussions of model validation as well as model construction and
interpretation. It conveys both the rich history of the subject and
its ongoing importance, and reflects the many contributions that the
authors have made to all aspects of the subject.
Stephen Wright
Madison, Wisconsin, USA