Post-incident report

Regarding incidents:
DDOS attack on our operations provider June 24th
DDOS attack on our operations provider June 30th
Service interuption July 1st

This week and the weekend before, Litium have experienced multiple DDoS incidents that resulted in major outage for our customers. While no internal systems were affected by the incidents, all communication to and from our infrastructure was down. This meant that visitors to our customers websites could not connect.

All of us at Litium would like to sincerely apologize for the impact this caused to each and every one of you. With this postmortem we would like to explain what happened and how we have, and will, take steps to better ensure this doesn’t happen again.

What happened

All incidents were due to a major DDoS attacks toward our operations provider. None of these attacks were directed at Litium or any of our customers. But due to the extent of the attack, the central firewall cluster became inaccessible. This resulted in no internet traffic reaching our infrastructure.

Why did it happen

All connections to the internet are redundant and several operators are used. However, one of these operators needed to manually activate DDoS filtering.

Steps taken

Routing has been adjusted and all operators is now always filtering the traffic and no manual intervention is needed for this scenario.

We will continue to work with our operations provider to get even better routines and further improve protection against DDoS attacks.