Telia Latvija provides premium IaaS service to customers. The service is provided as a complete set of tools to build highly available customer workloads in a fully automated and self-service manner. The customer uses this toolset to create highly customisable IT workloads according to industry best practices.

Problem case

One of our customers has deployed a collaboration application on two Windows servers – terminal server and application/DB/file server. Users use remote desktop terminal sessions to work with the application and documents on file shares. For backup purposes a backup drive was attached to the file server and regular backup jobs scripted to perform daily disk to disk backups and some rotation/retention was in place. A backup drive was created in a different data centre and different storage system to the primary storage. In the case of primary storage failure or any other need, fast recovery would be available.

Recently one of the terminal power users had bad luck with phishing e-mails and the server was infected with a ransomware type virus. The trojan spread across all volumes including the backup volume and it encrypted all production data including backup copies. When the customer noticed the presence of the trojan, recovery from backups was no longer possible.

Solution

The customer had no more backup copies to restore production data and they asked for our help / advice. For its own purposes, Telia Latvija takes infrastructure level snapshots of all NetApp storage systems. These snapshots are not usually created for the customer’s needs, but in this case we were able to recover both server snapshots shortly before the moment of infection. All customer data were recovered with minimal loss.

The result

In this case the customer used the 3-1-0 backup strategy and it failed (“3” copies of data in “1” storage type with “0” copies offsite). Our recommendations would be to add the missing storage type and offsite copy. No backup strategy is 100% secure but the 3-2-1 rule is a very strong approach. For adding the missing storage type, we recommended to script backup copies to the Object Storage service provided by Telia Latvija. This service has a fully Amazon S3 compatible API interface and such economically affordable backup extension with proper retention and version control can be created and deployed in 1 hour. In this case with ransomware trojan backup copies in Object Storage service would not be affected as this service is not mounted as file share. In conclusion – the rule of the offsite copy is still required if you deploy company production data to any cloud service. For their own safety and reasonable risk mitigation the customers must set up an offsite copy to another cloud service or even to some storage space in their office.

“A proper backup strategy is fundamental for every company’s data safety; it can protect against data loss but not in cases when data is stolen. In today’s digital environment safety is challenging and one tool cannot guarantee complete protection. The issue must be mitigated before it impacts the infrastructure. In this case the weak point was endpoint protection. For a long time we believed in antivirus. But it can only protect against known threats and not customised attacks, zero day exploits and various modifications of crypto viruses (ransomware), as antivirus uses a signature database. Telia noticed this gap in security and in cooperation with Palo Alto Networks, offers a product called Traps. It is a lightweight endpoint protection application running on workstations and servers (Windows, Linux, Mac, Android). Traps do not use a signature database but analyse local processes on the platform ensuring efficient protection against possible security issues. It covers all the features provided by antivirus and offers extra protection and visibility while costing the same. Let’s use security tools for security and backup tools for infrastructure issues!” says Jānis Kuiva, Head of Data Transmission of Telia Latvija.