The challenge tested participants' use of the latest tracking, data fusion and machine learning techniques to detect, track and identify small unmanned aircraft systems (UAS). Participants in the challenge fused together several sources of data provided by the Agency to track the drones. The challenge focuses on Class I Unmanned Aircraft Systems (UAS), which includes systems with a mass lower than 150 kg. Hobby drones are a typical example of Class I UAS.
The competition was run on Kaggle, a popular platform for artificial intelligence and machine learning challenges.
"It was really impressive to see the great interest shown by the participants in the challenge. This was the first challenge organized by the Agency using Kaggle, in a public domain where data science enthusiasts and research groups equally compete on diverse topics relevant for our society today," said Dr Michael Street, Head of Innovation and Data Science at the NCI Agency.
After evaluating the submissions by comparing the test set with the ground truth data, the top four submissions were asked to share their approach and techniques in more depth with a wider community during the special session on C-UAS and Radio Frequency technologies at ICMCIS.
The top four teams were:
Centre For Research and Technology Hellas (CERTH) Information Technologies Institute (ITI) Virtual and Augmented Reality Lab (VARlab) Team:
The CERTH ITI VARlab Team proposed a tracker based on machine learning technologies, which is something new in the tracking domain. Although the team has not specifically addressed data association, track management or fusion of data from multiple sensors, aspects their solution scored relatively high.
Defence Science & Technology Laboratory (Dstl):
The Dstl team used well-studied tracking and data fusion techniques available under the open source project known as Stone Soup. They fused all data from two radar sensors and two RF direction finders and solved specific challenges such as data association, track filtering and track management. The Dstl jupyter notebook posted on Kaggle is a great example of how the C-UAS tracking and data fusion problem can be solved.
CERTH ITI Visual Computing Lab Team:
The CERTH ITI Visual Computing Lab team used a mix of well-known techniques such as the Hungarian algorithm to solve the data association problem and a machine learning solution for the filtering part.
Horizon Lab:
The Horizon team proposed an innovative solution based on machine learning, and although with less focus on challenges such as data association, track management or data fusion; their solution scored relatively high in the Mean Root Square Error (MRSE) parameter used in the ranking.
"The teams achieved very good results using classical tracking and data fusion techniques, as well as new machine learning approaches applied to solve the problem," said Dr Cristian Coman, Principal Scientist at the NCI Agency. "The NCI Agency will continue to periodically challenge researchers and technology enthusiasts with practical security challenges, linked to realistic datasets."
This challenge is part of a larger research and development effort at the NCI Agency aiming at developing effective remote sensing technologies suitable for detecting, tracking and identifying Class I UASs. The data released for this challenge was recorded in 2020 during a measurement campaign at the Dutch Ministry of Defence's Counter-UAS Nucleus, in the Netherlands.
NATO sponsored the measurement campaign through its Defence Against Terrorism Programme of Work, and the event was further supported by several government and industry partners.
Another C-UAS Technical Interoperability Exercise (TIE21) will be organized in November 2021 and the intention is to again collect sensor data that can be used in another challenge. Anyone interested in this area is encouraged to stay in touch with the Agency for future details.
Watch this video for more information about our work with unmanned systems.