{"id":726,"date":"2023-12-21T18:44:41","date_gmt":"2023-12-21T18:44:41","guid":{"rendered":"https:\/\/aenix.io\/?p=726"},"modified":"2023-12-24T23:57:14","modified_gmt":"2023-12-24T23:57:14","slug":"highly-efficient-backup-system-for-virtual-machines-and-databases","status":"publish","type":"post","link":"https:\/\/aenix.io\/highly-efficient-backup-system-for-virtual-machines-and-databases\/","title":{"rendered":"Highly efficient backup system for virtual machines and databases"},"content":{"rendered":"\n<p>Development of a large-scale backup system for virtual machines. The implementation of this system reduced the cost of storing backup copies by up to 75%!<\/p>\n\n\n\n<p><strong>Objective:<\/strong><\/p>\n\n\n\n<p>The goal was to create an efficient backup system capable of reading data in real-time from the disks of virtual machines across several hundred servers and transferring it to a backup servers.<\/p>\n\n\n\n<p><strong><strong>Researched<\/strong><\/strong>:<\/p>\n\n\n\n<ul>\n<li><strong>Client Side:<\/strong> A comprehensive analysis of the efficiency and performance of various open-source backup systems was undertaken. Restic emerged as the most optimal solution based on this research.<\/li>\n\n\n\n<li><strong>Server Side:<\/strong> Different methods and filesystems were evaluated for their performance capabilities to establish an effective backend for the backup system.<\/li>\n<\/ul>\n\n\n\n<p><strong>Used technologies:<\/strong><\/p>\n\n\n\n<ul>\n<li><strong>Restic<\/strong> as highly efficient and feature-reach backup solution.<\/li>\n\n\n\n<li><strong>ZFS<\/strong> with <strong>dRAID<\/strong> and <strong>rest-server<\/strong> used for backend. This setup allowed for efficient data storage and swift recovery in the event of disk failures, even with large-capacity disks.<\/li>\n\n\n\n<li><strong>Kubernetes<\/strong> used for orchestration.<\/li>\n<\/ul>\n\n\n\n<p><strong>Outcomes:<\/strong><\/p>\n\n\n\n<p>This project successfully established a robust and scalable backup solution, enhancing data security and recovery capabilities across a large server network. The use of open-source tools like Restic and the ZFS file system, combined with the orchestration power of Kubernetes, resulted in a highly reliable and efficient backup system suitable for virtual machines backed by block storage.<\/p>\n\n\n\n<p>The same approach was later adapted for the efficient backup of databases, demonstrating its versatility and scalability.<\/p>\n\n\n\n<p><strong>Additional links:<\/strong><\/p>\n\n\n<div class=\"vlp-link-container vlp-layout-quick wp-block-visual-link-preview-link\"><a href=\"https:\/\/itnext.io\/backup-storage-for-thousands-of-virtual-machines-using-free-tools-b3909004bef2\" class=\"vlp-link\" title=\"Backup storage for thousands of virtual machines using free tools\" rel=\"nofollow\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:1200\/0*A6SRoMPAkAf-RnRB.png\" style=\"max-width: 150px; max-height: 150px\" \/><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">Backup storage for thousands of virtual machines using free tools<\/div><div class=\"vlp-block-1 vlp-link-summary\">We needed a backup solution to handle thousands of backups each week and do it as fast and efficiently as possible.<\/div><\/div><\/div>\n\n<div class=\"vlp-link-container vlp-layout-quick wp-block-visual-link-preview-link\"><a href=\"https:\/\/kvaps.medium.com\/4bc1e8f083c1\" class=\"vlp-link\" title=\"Restic: Effective Backup from Stdin\" rel=\"nofollow\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:720\/0*uJ7As9uTvB-gvQLe.png\" style=\"max-width: 150px; max-height: 150px\" \/><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">Restic: Effective Backup from Stdin<\/div><div class=\"vlp-block-1 vlp-link-summary\">A ready-made solution for setting up effective backup from Stdin, with deduplication and automatic cleaning of the repository from old copie<\/div><\/div><\/div>","protected":false},"excerpt":{"rendered":"<p>Development of a large-scale backup system for virtual machines. The implementation of this system reduced the cost of storing backup copies by up to 75%!<\/p>\n","protected":false},"author":1,"featured_media":948,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"[\"content\",\"tags\",\"comments\"]","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","_ti_tpc_template_sync":false,"_ti_tpc_template_id":"","footnotes":""},"categories":[7],"tags":[],"_links":{"self":[{"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/posts\/726"}],"collection":[{"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/comments?post=726"}],"version-history":[{"count":24,"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/posts\/726\/revisions"}],"predecessor-version":[{"id":1132,"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/posts\/726\/revisions\/1132"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/media\/948"}],"wp:attachment":[{"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/media?parent=726"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/categories?post=726"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aenix.io\/wp-json\/wp\/v2\/tags?post=726"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}