Service interuption
Incident Report for Litium AB
Postmortem

Post-incident report

Regarding incidents:
DDOS attack on our operations provider June 24th
DDOS attack on our operations provider June 30th
Service interuption July 1st

This week and the weekend before, Litium have experienced multiple DDoS incidents that resulted in major outage for our customers. While no internal systems were affected by the incidents, all communication to and from our infrastructure was down. This meant that visitors to our customers websites could not connect.

All of us at Litium would like to sincerely apologize for the impact this caused to each and every one of you. With this postmortem we would like to explain what happened and how we have, and will, take steps to better ensure this doesn’t happen again.

What happened

All incidents were due to a major DDoS attacks toward our operations provider. None of these attacks were directed at Litium or any of our customers. But due to the extent of the attack, the central firewall cluster became inaccessible. This resulted in no internet traffic reaching our infrastructure.

Why did it happen

All connections to the internet are redundant and several operators are used. However, one of these operators needed to manually activate DDoS filtering.

Steps taken

Routing has been adjusted and all operators is now always filtering the traffic and no manual intervention is needed for this scenario.

We will continue to work with our operations provider to get even better routines and further improve protection against DDoS attacks.

Posted Jul 04, 2019 - 15:43 CEST

Resolved
At this time, we are considering this incident resolved. There has been no impact for an extended period. In case of further impact, we will open a new incident. We will update this incident with a post mortem as soon as we have all the information.
Posted Jul 02, 2019 - 11:07 CEST
Update
Our operations provider have made several configuration changes to adjust their DDoS protection. At this time both our internal and external telemetry shows full recovery. We will actively monitor the situation for the rest of the evening and night.
Posted Jul 01, 2019 - 21:25 CEST
Monitoring
Our operations provider is target of a DDoS attack. We are continuing to verify all our systems but at this time services should be back to normal. Updates will follow.
Posted Jul 01, 2019 - 18:55 CEST
Update
We are continuing to investigate this issue.
Posted Jul 01, 2019 - 18:30 CEST
Investigating
We are currently investigating this issue.
Posted Jul 01, 2019 - 18:30 CEST
This incident affected: Litium Cloud (Network & infrastructure).