A "black swan" event refers to an unexpected or rare event that has an outsized effect on the system the event occurs in. Because of their rare occurrence, a learning system may not prepare for the event, making the effect even more pronounced.
Example: How do you prepare for an earthquake? Looking at all previous earthquakes and preparing for the strongest of that set ignores the highly improbable (but still possible) event of an earthquake that exceeds previous records, thus rendering preparation null.
First explained by Nassim Taleb in his book of the same name, Black Swans pose a threat primarily because they are so unpredictable. Therefore, Taleb recommends practicing counterfactual reasoning.