Out-Of-Distribution Detection in Federated Learning

Federated Learning

Federated learning (FL) is a technique to train a shared model with data from multiple parties without the data itself being shared. Several organisations might want to build an AI model for a specific task but don’t have enough training data on their own. Using FL, they can each train the model locally but aggregate the ‘learnings’ in a central model which is then redistributed to each participant. This way, there is no need to send potentially confidential data to each other.

This scheme works great as long as all parties have the same goal. If however, one (or more) participants have malicious intentions to manipulate or degrade the model, they can purposefully use bad data during their own training which will contaminate the central model shared by everyone. Such bad data is referred to as “out-of-distribution” (OOD) and usually consists of mislabeled or unrelated data. For example, if the shared model’s goal is to identify certain diseases in medical scam images, OOD data could have switched ‘healthy’ and ‘sick’ labels or be entirely off-topic like cat pictures.
Attacking a model by feeding it bad training data is called ‘poisoning’ and is especially problematic in a FL scenario.

Outside of FL, OOD detection is usually carried out by inspecting the data in some way before it is given to the model, avoiding its degrading impact. Due to the data privacy aspect of FL however, participants can only inspect their own data and not the data of others.

OOD Detection Without Data Access

Since classic solutions for model poisoning are difficult to apply under FL, we needed to find a different approach. A 2023 paper by Wilson et al. introduces so-called “Hyperdimensional Feature Fusion” (HDFF) which extracts feature maps from a deep neural network processing images, projecting these features into a hyperdimensional space and from there creates class-specific descriptor vectors. These descriptor vectors are then used to measure the distance of new data passing through the model to the established classes. If the distance is too large, the new data is judged to be OOD.

We adapted this method into the context of FL by creating the aforementioned feature maps and descriptor vectors not from the data passing through the network but from the model updates that each participant sends to the central, shared model. This was done as part of Master’s thesis project with Måns Oskarsson who created a simulated FL environment and adapted the algorithm to that end.

The results were very promising; the adapted method is able to detect not only participants that fully use OOD data but also those who mix proper data with OOD data. Malicious participants need to reduce their OOD percentage quite heavily (less than 50 images) to avoid detection, at which point the amount of OOD data is simply too small to substantially degrade the shared model.

A scientific journal article about our method is currently in the works