Treed-Gaussian Processes with Support Vector Machines as Nodes for Nonstationary Bayesian Optimization

A large family of black box methods rely on surrogates of the unknown, possibly non linear non convex reward function. While it is common to assume stationarity of the reward, many real-world problems satisfy this assumption only locally, hindering the spread application of such methods. This paper proposes a novel nonstationary regression model combining Decision Trees and Support Vector Machine (SVM) classification for a hierarchical non-axis-aligned partition of the input space. Gaussian Process (GP) regression is performed within each identified subregion. The resulting nonstationary regression model is the Treed Gaussian process with Support Vector Machine (SVMTGP), and we investigate the sampling efficiency from using our a model within a Bayesian optimization (BO) context. Empirically, we show how the resulting algorithm, SVMTGP-BO never underperforms BO when this is applied to an homogeneous Gaussian process, while it shows always better performance compared to the homogeneous model with nonlinear functions with complex landscapes.