Characterizing the function space for Bayesian kernel models

Natesh Pillai, Qiang Wu, Feng Liang, Sayan Mukherjee and Robert L. Wolpert

Duke University

July 2006
rev: January 2007

Kernel methods have been very popular in the machine learning literature in the last ten years, often in the context of Tikhonov regularization algorithms. In this paper we study a coherent Bayesian kernel model based on an integral operator whose domain is a space of signed measures. Priors on the signed measures induce prior distributions on their image functions under the integral operator. We study several classes of signed measures and their images, and identify general classes of measures whose images are dense in the reproducing kernel Hilbert space (RKHS) induced by the kernel. This gives a function-theoretic foundation for some nonparametric prior specifications commonly-used in Bayesian modeling, including Gaussian processes and Dirichlet processes, and suggests generalizations. A general framework for the construction of priors on signed measures using Lévy processes is described. A computational approach for sampling from the posterior distributions is presented, and illustrated for a univariate regression and a high-dimensional classification problem.

Keywords: Reproducing Kernel Hilbert Space, non-parametric Bayesian methods, Lévy processes, Dirichlet processes, integral operator

The manuscript is available in PDF format (270kb).

Cite as:

@Article{Pill:Wu:Lian:Mukh:Wolp:2007,
      Author = "Natesh Pillai and Qiang Wu and Feng Liang and Sayan
                  Mukherjee and Robert L. Wolpert", 
       Title = "Characterizing the function space for {B}ayesian kernel
                  models",
     Journal = "Journal of Machine Learning Research",
      Volume = 8,
       Month = aug,
       Pages = "1769--1797",
        Year = 2007,
}