publications

2024

1. 

   Multimodal base distributions in conditional flow matching generative models

   Shane Josias, and Willie Brink

   British Machine Vision Conference, 2024

   Abs

   Normalising flows are a flexible class of generative models that provide exact likelihoods, and are often trained through maximum likelihood estimation. Recent work suggests that these models can assign undesirably high likelihood to out-of-distribution data, questioning their reliability for applications where likelihoods are important (e.g. outlier detection). We show that continuous normalising flows trained with the conditional flow matching objective, instead of maximum likelihood, also provide unreliable likelihoods. We then argue for and investigate the utility of incorporating multimodality in the base distribution, through a Gaussian mixture model (GMM) centred at the empirical means of a target distribution’s modes. The GMM has an additional benefit in that samples can be generated from specified modes. We find that the GMM base distribution leads to performance comparable to a standard (unimodal) base distribution for in- and out-of-distribution likelihoods, at little to no extra cost in training and inference times. Interestingly, samples generated by models that use a GMM base have higher precision but significantly lower recall compared to the standard base. We also find support for the hypothesis that continuous flows depend too strongly on pixel values, rather than semantic content.

2023

1. 

   Multimodal base distributions for continuous-time normalising flows

   Shane Josias, and Willie Brink

   The Symbiosis of Deep Learning and Differential Equations Workshop at NeurIPS, 2023

   Abs PDF

   We investigate the utility of a multimodal base distribution in continuous-time normalising flows. Multimodality is incorporated through a Gaussian mixture model (GMM) centred at the empirical means of a target distribution’s modes. In- and out-of-distribution likelihoods are reported for flows trained with a unimodal and multimodal base distribution. Our results show that the GMM base distribution leads to performance that is comparable to a standard (unimodal) Gaussian distribution for in-distribution likelihoods, but provides the ability to sample from a specific mode in the target distribution, yields generated samples of improved quality, and gives more reliable out-of-distribution likelihoods for low-dimensional input spaces. We conclude that a GMM base distribution is an attractive alternative to the standard base, whose inclusion incurs little to no cost and whose parameterisation may assist with more reliable out-of-distribution likelihoods.

2. 

   Deep learning approaches to landmark detection in tsetse wing images

   Dylan Geldenhuys, Shane Josias, Willie Brink, and 6 more authors

   PLOS Computational Biology, 2023

   Abs HTML

   Morphometric analysis of wings has been suggested for identifying and controlling isolated populations of tsetse (Glossina spp), vectors of human and animal trypanosomiasis in Africa. Single-wing images were captured from an extensive data set of field-collected tsetse wings of species Glossina pallidipes and G. m. morsitans. Morphometric analysis required locating 11 anatomical landmarks on each wing. The manual location of landmarks is time-consuming, prone to error, and infeasible for large data sets. We developed a two-tier method using deep learning architectures to classify images and make accurate landmark predictions. The first tier used a classification convolutional neural network to remove most wings that were missing landmarks. The second tier provided landmark coordinates for the remaining wings. We compared direct coordinate regression using a convolutional neural network and segmentation using a fully convolutional network for the second tier. For the resulting landmark predictions, we evaluate shape bias using Procrustes analysis. We pay particular attention to consistent labelling to improve model performance. For an image size of 1024 × 1280, data augmentation reduced the mean pixel distance error from 8.3 (95% confidence interval [4.4,10.3]) to 5.34 (95% confidence interval [3.0,7.0]) for the regression model. For the segmentation model, data augmentation did not alter the mean pixel distance error of 3.43 (95% confidence interval [1.9,4.4]). Segmentation had a higher computational complexity and some large outliers. Both models showed minimal shape bias. We deployed the regression model on the complete unannotated data consisting of 14,354 pairs of wing images since this model had a lower computational cost and more stable predictions than the segmentation model. The resulting landmark data set was provided for future morphometric analysis. The methods we have developed could provide a starting point to studying the wings of other insect species. All the code used in this study has been written in Python and open sourced.

2022

1. 

   Jacobian norm regularisation and conditioning in neural ODEs

   Shane Josias, and Willie Brink

   Communications in Computer and Information Science, 2022

   Abs HTML PDF Poster

   A recent line of work regularises the dynamics of neural ordinary differential equations (neural ODEs), in order to reduce the number of function evaluations needed by a numerical ODE solver during training. For instance, in the context of continuous normalising flows, the Frobenius norm of Jacobian matrices are regularised under the hypothesis that complex dynamics relate to an ill-conditioned ODE and require more function evaluations from the solver. Regularising the Jacobian norm also relates to sensitivity analysis in the broader neural network literature, where it is believed that regularised models should be more robust to random and adversarial perturbations in their input. We investigate the conditioning of neural ODEs under different Jacobian regularisation strategies, in a binary classification setting. Regularising the Jacobian norm indeed reduces the number of function evaluations required, but at a cost to generalisation. Moreover, naively regularising the Jacobian norm can make the ODE system more ill-conditioned, contrary to what is believed in the literature. As an alternative, we regularise the condition number of the Jacobian and observe a lower number of function evaluations without a significant decrease in generalisation performance. We also find that Jacobian regularisation does not guarantee adversarial robustness, but it can lead to larger margin classifiers.

2021

1. 

   Class-Selective Mini-Batching and Multitask Learning for Visual Relationship Recognition

   Shane Josias, and Willie Brink

   SAIEE Africa Research Journal, 2021

   Abs HTML PDF

   An image can be described by the objects within it, and interactions between those objects. A pair of object labels together with an interaction label is known as a visual relationship, and is represented as a triplet of the form (subject, predicate, object). Recognising visual relationships in images is a challenging task, owing to the combinatorially large number of possible relationship triplets, which leads to an extreme multiclass classification problem. In addition, the distribution of visual relationships in a dataset tends to be long-tailed, i.e. most triplets occur rarely compared to a small number of dominating triplets. Three strategies to address these issues are investigated. Firstly, instead of predicting the full triplet, models can be trained to predict each of the three elements separately. Secondly a multitask learning strategy is investigated, where shared network parameters are used to perform the three separate predictions. Thirdly, a class-selective mini-batch construction strategy is used to expose the network to more of the rare classes during training. Experiments demonstrate that class-selective mini-batch construction can improve performance on classes in the long tail of the data distribution, possibly at the expense of accuracy on the small number of dominating classes. It is also found that a multitask model neither improves nor impedes performance in any significant way, but that its smaller size may be beneficial. In an effort to better understand the behaviour of the various models, a novel evaluation approach for visual relationship recognition is introduced. We conclude that the use of semantics can be helpful in the modelling and evaluation process.

2020

1. PRASA

   Batch construction and multitask learning in visual relationship recognition

   Shane Josias, and Willie Brink

   International SAUPEC/RobMech/PRASA Conference, 2020

   Abs HTML PDF

   An image can be described by the objects within it, as well as interactions between those objects. A pair of object labels together with an interaction label is known as a visual relationship, and is represented as a triplet of the form (subject, predicate, object). Recognising visual relationships in a given image is a challenging task, owing to the combinatorially large number of possible relationship triplets, which leads to an extreme classification problem, as well as a very long tail found typically in the distribution of those possible triplets. We investigate the effects of three strategies that could potentially address these issues. Firstly, instead of predicting the full triplet we opt to predict each element separately. Secondly, we investigate the use of shared network parameters to perform these separate predictions in a multitask setting. Thirdly, we consider a class-selective batch construction strategy to expose the network to more of the many rare classes during mini-batch training. Our experiments demonstrate that batch construction can improve performance on the long tail, possibly at the expense of accuracy on the small number of dominating classes. We also find that a multitask model neither improves nor impedes performance in any significant way, but that its smaller size may be beneficial.