Abstract

Strategies for creating deep ensembles often rely on simple heuristics, such as averaging predictions, yet the potential of evolving explicit fusion schemas remains underexplored. We propose OKAPI, a Genetic Programming algorithm that evolves tree-structured fusion schemas over frozen base-model predictions. Using a multi-objective approach and a stateful representation, OKAPI performs automated model selection and, when configured with a parsimony objective, explicitly minimizes the ensemble size. We evaluate OKAPI across 12 medical imaging datasets (MedMNISTv2), a hyperspectral remote-sensing methane segmentation task (STARCOP), and an additional classical tabular benchmark of 144 datasets (PMLB), comparing it against heuristic averages, logistic-regression stacking, NSGA-III, and AutoML methods. The results show that no single fusion method dominates across all regimes, and OKAPI is competitive with strong state-of-the-art algorithms, produces compact ensembles, and is most useful when the base-predictor pool contains complementary information and validation performance transfers reliably to the test set. Our study showed that under unreliable validation signals or saturated tasks, robust averaging heuristics can be difficult to beat. It clarifies the practical scope of the prediction-level fusion search: OKAPI provides an efficient, prediction-native alternative for automated ensemble design, while exposing validation reliability as a key constraint on ensemble optimization. OKAPI is open-source for reproducibility.