Development of a large-scale backup system for virtual machines. The implementation of this system reduced the cost of storing backup copies by up to 75%!
Objective:
The goal was to create an efficient backup system capable of reading data in real-time from the disks of virtual machines across several hundred servers and transferring it to a backup servers.
Researched:
- Client Side: A comprehensive analysis of the efficiency and performance of various open-source backup systems was undertaken. Restic emerged as the most optimal solution based on this research.
- Server Side: Different methods and filesystems were evaluated for their performance capabilities to establish an effective backend for the backup system.
Used technologies:
- Restic as highly efficient and feature-reach backup solution.
- ZFS with dRAID and rest-server used for backend. This setup allowed for efficient data storage and swift recovery in the event of disk failures, even with large-capacity disks.
- Kubernetes used for orchestration.
Outcomes:
This project successfully established a robust and scalable backup solution, enhancing data security and recovery capabilities across a large server network. The use of open-source tools like Restic and the ZFS file system, combined with the orchestration power of Kubernetes, resulted in a highly reliable and efficient backup system suitable for virtual machines backed by block storage.
The same approach was later adapted for the efficient backup of databases, demonstrating its versatility and scalability.
Additional links:
