Academia.eduAcademia.edu

Figure 4 – uploaded by Nils Einecke

See full PDFdownloadDownload figure
Table 4 Evaluation of team sesc’s approach, as additional steps were added The training data were split using a three-fold cross- validation (eight events were selected in each validation fold, from the 23 anomalous events). Subsequently, {non-anomalous, non-anomalous}, and {non-anomalous, anomalous} pairs were generated for the siamese network to learn similar and dissimilar pairs, respectively. The training data were split using a three-fold cross-

Table 4 Evaluation of team sesc’s approach, as additional steps were added The training data were split using a three-fold cross- validation (eight events were selected in each validation fold, from the 23 anomalous events). Subsequently, {non-anomalous, non-anomalous}, and {non-anomalous, anomalous} pairs were generated for the siamese network to learn similar and dissimilar pairs, respectively. The training data were split using a three-fold cross-

Related Figures (23)

Fig. 2 Distribution of the distance between the closest object and Sentinel-3B, and a fitted Weibull distribution (fit skewed to represent the tail with higher accuracy). Fig. 1 Visualization of the density of objects orbiting the low Earth orbit as of 2020-May-22 (data from www.space- track.org).
Fig. 2 Distribution of the distance between the closest object and Sentinel-3B, and a fitted Weibull distribution (fit skewed to represent the tail with higher accuracy). Fig. 1 Visualization of the density of objects orbiting the low Earth orbit as of 2020-May-22 (data from www.space- track.org).
close to the origin. ‘lo obtain a first assessment of the risk posed to in active satellite operating, for example, in a Sun- synchronous orbit, we computed the closest distance of a Sun-synchronous satellite to the LEO population und its distribution at random epochs and within a two- year window. Figure 2 shows the results for Sentinel-3B. in most of the epochs, the satellite was far from other »bjects, but in some rare scenarios, the closest distance upproached values that were of concern. A Weibull listribution can be fitted to the obtained data, where ‘esults from extreme value statistics justify its use to make preliminary inferences on collision probabilities [11]. Such nferences are very sensitive to the Weibull distribution yarameters and, in particular, to the behavior of its tail close to the origin. inferences are very sensitive to the Weibull distribution
close to the origin. ‘lo obtain a first assessment of the risk posed to in active satellite operating, for example, in a Sun- synchronous orbit, we computed the closest distance of a Sun-synchronous satellite to the LEO population und its distribution at random epochs and within a two- year window. Figure 2 shows the results for Sentinel-3B. in most of the epochs, the satellite was far from other »bjects, but in some rare scenarios, the closest distance upproached values that were of concern. A Weibull listribution can be fitted to the obtained data, where ‘esults from extreme value statistics justify its use to make preliminary inferences on collision probabilities [11]. Such nferences are very sensitive to the Weibull distribution yarameters and, in particular, to the behavior of its tail close to the origin. inferences are very sensitive to the Weibull distribution
Fig. 3 Operational altitudes for the missions in LEO supported by ESA Space Debris Office, and the spatial density of objects with a cross-section of > 10 cm. The main source of information avoidance process at ESA is based data messages (CDMs). These are as and distributed by the United States of the collision on conjunction cit files produced based Combined Space Operations Center (CSpOC). Each conjunction contains information on one close ap monitored space object (the “target second space object (the “chaser satel proach between a satellite”) and a ite”). The CDMs contain multiple attributes of the approach, such as the The main source of information of the collision
Fig. 3 Operational altitudes for the missions in LEO supported by ESA Space Debris Office, and the spatial density of objects with a cross-section of > 10 cm. The main source of information avoidance process at ESA is based data messages (CDMs). These are as and distributed by the United States of the collision on conjunction cit files produced based Combined Space Operations Center (CSpOC). Each conjunction contains information on one close ap monitored space object (the “target second space object (the “chaser satel proach between a satellite”) and a ite”). The CDMs contain multiple attributes of the approach, such as the The main source of information of the collision
Table 1 Database of conjunction events at a glance
Table 1 Database of conjunction events at a glance
Fig. 4 Histogram of the latest known risk value (logarithmic scale) for the entire dataset (training and testing sets). Note that there are 9505 events with a final risk value of log,y r = —30 or lower, which are not displayed in this figure.
Fig. 4 Histogram of the latest known risk value (logarithmic scale) for the entire dataset (training and testing sets). Note that there are 9505 events with a final risk value of log,y r = —30 or lower, which are not displayed in this figure.
Fig. 5 Diagram depicting the raw CDMs time series for one event (top), and the same series if it was selected for the test set (bottom): only the CDMs prior to two days to TCA were made available (labeled as x) and the latest CDM was used as the target (labeled as y). diminished. After enforcing the three requirements
Fig. 5 Diagram depicting the raw CDMs time series for one event (top), and the same series if it was selected for the test set (bottom): only the CDMs prior to two days to TCA were made available (labeled as x) and the latest CDM was used as the target (labeled as y). diminished. After enforcing the three requirements
Fig. 6 Projection of the original CDMs, from the test set, onto the first two principal components, colored according to mission_id. data can still be distinguished and crudely clustered by In Fig. 6, the PCA projection of the original data is shown by maintaining only the first two principal components. While the first two principal components only account for 20% of the total variance, the projected data can still be distinguished and crudely clustered by
Fig. 6 Projection of the original CDMs, from the test set, onto the first two principal components, colored according to mission_id. data can still be distinguished and crudely clustered by In Fig. 6, the PCA projection of the original data is shown by maintaining only the first two principal components. While the first two principal components only account for 20% of the total variance, the projected data can still be distinguished and crudely clustered by
Table 2 Background of the participants, out of 22 respondents to the end of the competition questionnaire A total of 22 teams participated in the survey, including all top-ranked teams. Some questions from the survey were targeted to gather more information on the background of the participants. The questions were phrased as follows: “How would you describe your knowledge in space debris/astrodynamics?” <A similar approach was conducted for ML and data science. The possible answers were limited to “professional” , “studying”, and “amateur”. The answers are reported in Table 2, which shows that most participants had a background in ML and less in orbital mechanics. Note that the top three teams all identified themselves as ML professionals and two as studying orbital mechanics.
Table 2 Background of the participants, out of 22 respondents to the end of the competition questionnaire A total of 22 teams participated in the survey, including all top-ranked teams. Some questions from the survey were targeted to gather more information on the background of the participants. The questions were phrased as follows: “How would you describe your knowledge in space debris/astrodynamics?” <A similar approach was conducted for ML and data science. The possible answers were limited to “professional” , “studying”, and “amateur”. The answers are reported in Table 2, which shows that most participants had a background in ML and less in orbital mechanics. Note that the top three teams all identified themselves as ML professionals and two as studying orbital mechanics.
Fig. 7 Distribution of the mission type for the test and training sets for low-risk events. Iheretore, imbalances 1n mission type should not be created when splitting the data into training and test sets. Figure 7 shows that, for low-risk events, the missions are proportionally represented in both the training and test sets. However, when we examine only high-risk events (Fig. 8), we observe that the missions are not well distributed. In particular, mission_id: 2, mission_id: 15, and mission_id: 19 are over-represented in the test set, whereas mission_id: 1, mission_id: 6, and mission_id: 20 are under-represented. This is because the dataset was randomly split into training and testing, considering only the risk value and not the mission type. For future only the risk value and not the mission type. For future
Fig. 7 Distribution of the mission type for the test and training sets for low-risk events. Iheretore, imbalances 1n mission type should not be created when splitting the data into training and test sets. Figure 7 shows that, for low-risk events, the missions are proportionally represented in both the training and test sets. However, when we examine only high-risk events (Fig. 8), we observe that the missions are not well distributed. In particular, mission_id: 2, mission_id: 15, and mission_id: 19 are over-represented in the test set, whereas mission_id: 1, mission_id: 6, and mission_id: 20 are under-represented. This is because the dataset was randomly split into training and testing, considering only the risk value and not the mission type. For future only the risk value and not the mission type. For future
Fig. 8 Distribution of the mission type for the test and training sets for high-risk events.
Fig. 8 Distribution of the mission type for the test and training sets for high-risk events.
Table 3 Final rankings (from best to worst) evaluated on the test set, for the top ten teams. The best results are shown in bold To further investigate the differences between the F, score achieved by the teams and the LRP baseline solution, it is useful to examine the false positives and false negatives of each returned model (Fig. 10(b)). The Pareto front is very heterogeneous and consists of several teams: DunderMifflin, Valis, Magpies, DeCRA, dietmarw, vhrique, madks, and the baseline solution, denoted as Baseline. Although the baseline solution is in the Pareto front, we can observe that the resulting F2 score in Fig. 10(a) is dominated by several teams. This is because the F5 score places more emphasis on penalizing false
Table 3 Final rankings (from best to worst) evaluated on the test set, for the top ten teams. The best results are shown in bold To further investigate the differences between the F, score achieved by the teams and the LRP baseline solution, it is useful to examine the false positives and false negatives of each returned model (Fig. 10(b)). The Pareto front is very heterogeneous and consists of several teams: DunderMifflin, Valis, Magpies, DeCRA, dietmarw, vhrique, madks, and the baseline solution, denoted as Baseline. Although the baseline solution is in the Pareto front, we can observe that the resulting F2 score in Fig. 10(a) is dominated by several teams. This is because the F5 score places more emphasis on penalizing false
Fig. 9 Evolution of the scores of the submission of various top teams.
Fig. 9 Evolution of the scores of the submission of various top teams.
Fig. 10 On the top, in (a), the F2 score and the MSEur are plotted for the top ten teams. On the bottom, in (b), the F2 scores are broken into two components: false negatives (out of 150 positive events) and false positives (out of 2017 negative events).
Fig. 10 On the top, in (a), the F2 score and the MSEur are plotted for the top ten teams. On the bottom, in (b), the F2 scores are broken into two components: false negatives (out of 150 positive events) and false positives (out of 2017 negative events).
Fig. 11 Number of anomalous and non-anomalous events from the training set. by leveraging Manhattan-LSTMs [26] and a siamese architecture based on recurrent neural networks. Team Magpies began by analyzing the dataset and filtering the training data according to the test set requirements described in Section 4.2. Subsequently, they selected seven out of 103 features (time_to_tca, max_risk_estimate, maz_risk_scaling, mahalanobis_distance, miss_distance, c_position_covariance_det, and c_obs_used) by comparing the distribution difference of the non-anomalous event (last available collision risk is low and ends up low at close approach, and vice-versa for high-risk events) and anomalous events (last available collision risk is low and ends up high at the close approach, and vice-versa for high-risk events). Figure 11 shows the number of anomalous and non-anomalous scenarios. In addition to these seven attributes, three new features were included: the number of CDMs (number-CDMs) issued before two days, the mean (mean_risk_CDMs) and standard deviation (std_risk.CDMs) of the risk values of the CDMs. Hyperbolic tangents were used as activation functions, and Adam was used as the gradient descent optimizer [27].
Fig. 11 Number of anomalous and non-anomalous events from the training set. by leveraging Manhattan-LSTMs [26] and a siamese architecture based on recurrent neural networks. Team Magpies began by analyzing the dataset and filtering the training data according to the test set requirements described in Section 4.2. Subsequently, they selected seven out of 103 features (time_to_tca, max_risk_estimate, maz_risk_scaling, mahalanobis_distance, miss_distance, c_position_covariance_det, and c_obs_used) by comparing the distribution difference of the non-anomalous event (last available collision risk is low and ends up low at close approach, and vice-versa for high-risk events) and anomalous events (last available collision risk is low and ends up high at the close approach, and vice-versa for high-risk events). Figure 11 shows the number of anomalous and non-anomalous scenarios. In addition to these seven attributes, three new features were included: the number of CDMs (number-CDMs) issued before two days, the mean (mean_risk_CDMs) and standard deviation (std_risk.CDMs) of the risk values of the CDMs. Hyperbolic tangents were used as activation functions, and Adam was used as the gradient descent optimizer [27].
Fig. 12 Events consistently misclassified by all the top ten teams: (a) false negatives, (b) false positives. In the top panel, we show all false negatives (11 out of 150 high-risk events). Each event is represented as a line, and the CDMs are marked with crosses. The evolution of the risk between two CDMs is plotted as a linear interpolation. In the bottom panel, we show ten randomly sampled events out of 62 false positives in total. These events were particularly difficult to classify because of the big leap in risk closer to the TCA, ranging from low risk to high risk in (a) and vice-versa in (b).
Fig. 12 Events consistently misclassified by all the top ten teams: (a) false negatives, (b) false positives. In the top panel, we show all false negatives (11 out of 150 high-risk events). Each event is represented as a line, and the CDMs are marked with crosses. The evolution of the risk between two CDMs is plotted as a linear interpolation. In the bottom panel, we show ten randomly sampled events out of 62 false positives in total. These events were particularly difficult to classify because of the big leap in risk closer to the TCA, ranging from low risk to high risk in (a) and vice-versa in (b).
Fig. 13 Evolution of the uncertainty in the radial velocity of the target spacecraft (t_sigma_rdot) over time, up until two days to the TCA. The evolution of the uncertainty of the 11 false negative events (Fig. 12) is indicated in red. The evolution of the uncertainty of the 139 remaining true positive events is indicated in black.
Fig. 13 Evolution of the uncertainty in the radial velocity of the target spacecraft (t_sigma_rdot) over time, up until two days to the TCA. The evolution of the uncertainty of the 11 false negative events (Fig. 12) is indicated in red. The evolution of the uncertainty of the 139 remaining true positive events is indicated in black.
Fig. 14 Performance levels achieved by 5000 gradient boosting regression models trained using the competition’s datas split. the models on the training (a-axis) and test sets (y-axis). any focus on high-risk events during the training process. to address the existing class imbalance and the absence of measuring the MSEyr. We compared the performance of
Fig. 14 Performance levels achieved by 5000 gradient boosting regression models trained using the competition’s datas split. the models on the training (a-axis) and test sets (y-axis). any focus on high-risk events during the training process. to address the existing class imbalance and the absence of measuring the MSEyr. We compared the performance of
Fig. 16 Expected percentage of ML models that would outperform the LRP baseline in both the training and test set, as data sets varied in the fraction of events assigned to both (shown in the x-axis). Performance measured in the MSEyr anc 1/F> metrics, as well as their product. Fig. 15 Extent to which different data splits affected the ability to infer test set performance from the training set performance. Expected Spearman rank-order correlation coefficients between training and test set evaluations, as data sets vary in the fraction of events assigned to both (shown in the x-axis). Correlations measured in the MSEyr and 1/F2 metrics, as well as their product.
Fig. 16 Expected percentage of ML models that would outperform the LRP baseline in both the training and test set, as data sets varied in the fraction of events assigned to both (shown in the x-axis). Performance measured in the MSEyr anc 1/F> metrics, as well as their product. Fig. 15 Extent to which different data splits affected the ability to infer test set performance from the training set performance. Expected Spearman rank-order correlation coefficients between training and test set evaluations, as data sets vary in the fraction of events assigned to both (shown in the x-axis). Correlations measured in the MSEyr and 1/F2 metrics, as well as their product.
Spacecraft collision avoidance challenge: Design and results of a machine learning competition
Spacecraft collision avoidance challenge: Design and results of a machine learning competition
Fig. 17 Extent to which different data splits affected the ability to infer test set performance from the training set performance, as observed through simultaneous evaluations of regression and classification metrics. Spearman rank- order correlation coefficients between training and test set evaluations of the three performance metrics in 10,000 different data splits using different test size fractions. The red star corresponds to the data split of the official competition (Fig. 14).
Fig. 17 Extent to which different data splits affected the ability to infer test set performance from the training set performance, as observed through simultaneous evaluations of regression and classification metrics. Spearman rank- order correlation coefficients between training and test set evaluations of the three performance metrics in 10,000 different data splits using different test size fractions. The red star corresponds to the data split of the official competition (Fig. 14).
Fig. 18 Expected 1/F> performance gain (%) over the LRP baseline as data splits varied in the fraction of events assigned to either the training or test set.
Fig. 18 Expected 1/F> performance gain (%) over the LRP baseline as data splits varied in the fraction of events assigned to either the training or test set.
Fig. 19 Variation in the training and test set performance of the ML models (1/F2) as a function of data availability. Performance normalized with respect to LRP baseline performance in the same datasets. Spacecraft collision avoidance challenge: Design and results of a machine learning competition
Fig. 19 Variation in the training and test set performance of the ML models (1/F2) as a function of data availability. Performance normalized with respect to LRP baseline performance in the same datasets. Spacecraft collision avoidance challenge: Design and results of a machine learning competition
Table 5 Feature relevance estimates. In the prediction of near-term changes in risk, percentage of the reduction in error attributable to the feature. A description of the features is available on the Kelvins website of 80 of the 100 features. Note the absence of “target” features in Table 5. The relevance of “target” features summed to a total of 9.41%, while “chaser” features summed to 23.49%. If the models were to rely too much on the properties of the “target”, they would be learning mission-specific rules. Instead, we observed a greater reliance on properties of the “chaser”, and in features with relative values, thus enabling better generalization across missions.
Table 5 Feature relevance estimates. In the prediction of near-term changes in risk, percentage of the reduction in error attributable to the feature. A description of the features is available on the Kelvins website of 80 of the 100 features. Note the absence of “target” features in Table 5. The relevance of “target” features summed to a total of 9.41%, while “chaser” features summed to 23.49%. If the models were to rely too much on the properties of the “target”, they would be learning mission-specific rules. Instead, we observed a greater reliance on properties of the “chaser”, and in features with relative values, thus enabling better generalization across missions.
Related topics:
MathematicsComputer SciencePhilosophy of AgencyMachine LearningSpacecraft OperationsCollision AvoidanceOrbital Mechanicssatellite collision riskcompetitive data science
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved