Academia.eduAcademia.edu

Figure 5 – uploaded by Rasit Abay

See full PDFdownloadDownload figure
Table 4: Evaluation of team sesc’s approach, as additional steps were added and combinations tested. Loss evaluations on the competition’s public leaderboard included. Risk values clipped at -6.00001 (-6.001 in LRP).

Table 4 Evaluation of team sesc’s approach, as additional steps were added and combinations tested. Loss evaluations on the competition’s public leaderboard included. Risk values clipped at -6.00001 (-6.001 in LRP).

Related Figures (25)

ABSTRACT
ABSTRACT
Figure 2: Distribution of the distance between the closest object and Sentinel-3B, and a fitted Weibull distribution (fit skewed to represent the left tail with higher accuracy). Figure 1: Visualization of the density of Low Earth Orbit orbiting objects as of 2020-May-22 (data from www.space-track.org).
Figure 2: Distribution of the distance between the closest object and Sentinel-3B, and a fitted Weibull distribution (fit skewed to represent the left tail with higher accuracy). Figure 1: Visualization of the density of Low Earth Orbit orbiting objects as of 2020-May-22 (data from www.space-track.org).
Figure 3: Operational altitudes for the missions in LEO supported by ESA Space Debris Office, and the spatial density of objects with a cross section > 10cm.
Figure 3: Operational altitudes for the missions in LEO supported by ESA Space Debris Office, and the spatial density of objects with a cross section > 10cm.
A second step in the data preparation was the anonymization of the data. This involved transforming absolute time stamps and position/velocity values in relative values, respectively, in terms of time to TCA and state with respect to the target. The names of the target mission were also removed and a numerical mission identifier was introduced to group similar missions. A random event identifier was assigned to each event. The full list of the attributes extracted from the CDMs and released in the dataset can be found, together with their explanation, on the Kelvins competition website. Here, we briefly describe only a few attributes that are relevant for later discussions: Table 1: The database of conjunction events at a glance.
A second step in the data preparation was the anonymization of the data. This involved transforming absolute time stamps and position/velocity values in relative values, respectively, in terms of time to TCA and state with respect to the target. The names of the target mission were also removed and a numerical mission identifier was introduced to group similar missions. A random event identifier was assigned to each event. The full list of the attributes extracted from the CDMs and released in the dataset can be found, together with their explanation, on the Kelvins competition website. Here, we briefly describe only a few attributes that are relevant for later discussions: Table 1: The database of conjunction events at a glance.
Such a forecast could assist the decision of whether or not to perform an avoidance manoeuvre, by providing a better estimate of the future collision risk, before further CDMs are released. Forecasting competitions[18] are widely recognized as an effective mean to find good predictive models and solutions for a given problem. To design such competitions successfully, requires to find a good balance between the desire to create an interesting and fair machine learning challenge, to motivate and involve the large community of data scientists worldwide, and to fulfil the objective of furthering the current understanding by answering a meaningful scientific question [19]. Figure 4: Histogram of the latest known risk value for the whole dataset (training and testing sets). Note that there are 9505 events with a final risk value of -30 or lower, that are not displayed on this figure.
Such a forecast could assist the decision of whether or not to perform an avoidance manoeuvre, by providing a better estimate of the future collision risk, before further CDMs are released. Forecasting competitions[18] are widely recognized as an effective mean to find good predictive models and solutions for a given problem. To design such competitions successfully, requires to find a good balance between the desire to create an interesting and fair machine learning challenge, to motivate and involve the large community of data scientists worldwide, and to fulfil the objective of furthering the current understanding by answering a meaningful scientific question [19]. Figure 4: Histogram of the latest known risk value for the whole dataset (training and testing sets). Note that there are 9505 events with a final risk value of -30 or lower, that are not displayed on this figure.
Figure 7: Distribution of the mission type for the test and training sets, for low-risk events.
Figure 7: Distribution of the mission type for the test and training sets, for low-risk events.
Figure 6: Projection of the original CDMs, from the test set, onto the first two principal components, colored according to mission_id.
Figure 6: Projection of the original CDMs, from the test set, onto the first two principal components, colored according to mission_id.
Figure 8: Distribution of the mission type for the test and training sets, for high-risk events.
Figure 8: Distribution of the mission type for the test and training sets, for high-risk events.
Table 2: Background of the participants, out of 22 respon- dents to the end of the competition questionnaire.
Table 2: Background of the participants, out of 22 respon- dents to the end of the competition questionnaire.
Table 3: Final rankings (from best to worst) evaluated on the test set, for the top 10 teams. The best results are displayed in bold.
Table 3: Final rankings (from best to worst) evaluated on the test set, for the top 10 teams. The best results are displayed in bold.
Figure 9: Evolution of the scores of the various top teams submissions.
Figure 9: Evolution of the scores of the various top teams submissions.
Figure 10: On the left, in (a), the f2 score and the MSEup are plotted for the top 10 teams. On the right, in (b), the fF’ scores is broken into two components: false negatives (out of 150 positive events) and false positives (out of 2017 negative events).
Figure 10: On the left, in (a), the f2 score and the MSEup are plotted for the top 10 teams. On the right, in (b), the fF’ scores is broken into two components: false negatives (out of 150 positive events) and false positives (out of 2017 negative events).
Figure 11: Diagram showing the Manhattan-LSTM architecture. The third ranked team was made of a single Space Situational Awareness (SSA) researcher and Machine Learning (ML) engineer. The team achieved its final score by leveraging Manhattan-LSTMs [26] (Figure 11), a siamese architecture based on recurrent neural networks. Team Magpies started with analysing the dataset and filtered the training data according to the test set requirements laid out in Section 4.2. A comprehensive exploratory data analysis was conducted and provided the following conclusions:
Figure 11: Diagram showing the Manhattan-LSTM architecture. The third ranked team was made of a single Space Situational Awareness (SSA) researcher and Machine Learning (ML) engineer. The team achieved its final score by leveraging Manhattan-LSTMs [26] (Figure 11), a siamese architecture based on recurrent neural networks. Team Magpies started with analysing the dataset and filtered the training data according to the test set requirements laid out in Section 4.2. A comprehensive exploratory data analysis was conducted and provided the following conclusions:
Figure 12: Diagram showing the number of anomalous and non-anomalous events from the train set, for the two cases: (i) left root split is low-to-high risk anomalies; (ii) right root split is high-to-low anomalies. Event counts (shown underneath the tree leaves) are based on filtering events as per test set specifications in Section 4.2, but allowing for final CDMs to lie within 2 days of the TCA.
Figure 12: Diagram showing the number of anomalous and non-anomalous events from the train set, for the two cases: (i) left root split is low-to-high risk anomalies; (ii) right root split is high-to-low anomalies. Event counts (shown underneath the tree leaves) are based on filtering events as per test set specifications in Section 4.2, but allowing for final CDMs to lie within 2 days of the TCA.
Figure 13: Events consistently misclassified by all the top 10 teams: (a) false negatives, (b) false positives. On the left panel, we show all the false negatives (11 out of 150 high risk events). Each event is represented as a line and the CDMs are marked with crosses. The evolution of risk between two CDMs is simply plotted as a linear interpolation. On the right panel, we show 10 randomly sampled events, out of 62 false positives in total. These events are particularly hard to classify due to the big jump in risk as we are getting closer to time of closest approach, going from low risk to high risk in (a) and vice-versa in (b).
Figure 13: Events consistently misclassified by all the top 10 teams: (a) false negatives, (b) false positives. On the left panel, we show all the false negatives (11 out of 150 high risk events). Each event is represented as a line and the CDMs are marked with crosses. The evolution of risk between two CDMs is simply plotted as a linear interpolation. On the right panel, we show 10 randomly sampled events, out of 62 false positives in total. These events are particularly hard to classify due to the big jump in risk as we are getting closer to time of closest approach, going from low risk to high risk in (a) and vice-versa in (b).
Figure 14: Evolution of the uncertainty in the radial velocity of the target spacecraft (t_sigma_rdot) over time, up until 2 days to time of closest approach. The evolution of the uncertainty of the 11 false negative events (see Figure 13) is shown in red. The evolution of the uncertainty of the 139 remaining true positive events is shown in black. close to TCA. In Figure 14, the evolution of the uncertainty in the radial velocity of the target spacecraft (t_sigma_rdoi is displayed, for the the 150 high-risk events in the test set.
Figure 14: Evolution of the uncertainty in the radial velocity of the target spacecraft (t_sigma_rdot) over time, up until 2 days to time of closest approach. The evolution of the uncertainty of the 11 false negative events (see Figure 13) is shown in red. The evolution of the uncertainty of the 139 remaining true positive events is shown in black. close to TCA. In Figure 14, the evolution of the uncertainty in the radial velocity of the target spacecraft (t_sigma_rdoi is displayed, for the the 150 high-risk events in the test set.
Figure 15: Performance levels achieved by 5000 gradient boosting regression models trained using the competition’s dataset split.
Figure 15: Performance levels achieved by 5000 gradient boosting regression models trained using the competition’s dataset split.
Figure 16: Extent to which different data splits impact ability to infer test set performance from the train set performance. Expected Spearman rank-order correlation coefficients between train and test set evaluations, as data sets vary in the fraction of events assigned to both (shown in the x-axis). Correlations measured in the MSEpp and 1/F> metrics, as well as their product.
Figure 16: Extent to which different data splits impact ability to infer test set performance from the train set performance. Expected Spearman rank-order correlation coefficients between train and test set evaluations, as data sets vary in the fraction of events assigned to both (shown in the x-axis). Correlations measured in the MSEpp and 1/F> metrics, as well as their product.
Figure 17: Expected percentage of machine learning models that will outperform the LRP baseline in both the train and test set, as data sets vary in the fraction of events assigned to both (shown in the z-axis). Performance measured in the MSEure and 1/F> metrics, as well as their product.
Figure 17: Expected percentage of machine learning models that will outperform the LRP baseline in both the train and test set, as data sets vary in the fraction of events assigned to both (shown in the z-axis). Performance measured in the MSEure and 1/F> metrics, as well as their product.
Figure 19: Expected 1/F> performance gain (%) over the LRP baseline as data splits vary in the fraction of events assigned to either the train or test set. Figure 18: Extent to which different data splits impact abi ity to infer test set performance from the train set performance, as seen through simultaneous evaluations of regression and classification metrics. Spearman rank- order correlation coefficients between train and test set evaluations of the three performance metrics, in 10000 different data splits, using different test size fractions. Red star corresponds to the official competition’s data split (Fig. 15).
Figure 19: Expected 1/F> performance gain (%) over the LRP baseline as data splits vary in the fraction of events assigned to either the train or test set. Figure 18: Extent to which different data splits impact abi ity to infer test set performance from the train set performance, as seen through simultaneous evaluations of regression and classification metrics. Spearman rank- order correlation coefficients between train and test set evaluations of the three performance metrics, in 10000 different data splits, using different test size fractions. Red star corresponds to the official competition’s data split (Fig. 15).
Figure 20: Variation in machine learning models’ train and test set performance (1/2) as a function of data availability. Performance normalized with respect to the LRP baseline’s performance in the same datasets.
Figure 20: Variation in machine learning models’ train and test set performance (1/2) as a function of data availability. Performance normalized with respect to the LRP baseline’s performance in the same datasets.
Table 6: Feature relevance estimates. In prediction of near-term changes in risk, percentage of the reduction in error attributable to the feature.
Table 6: Feature relevance estimates. In prediction of near-term changes in risk, percentage of the reduction in error attributable to the feature.
Related topics:
Machine LearningSpacecraft OperationsOrbital Mechanicssatellite collision riskcompetitive data science
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved