{"id":748,"date":"2023-12-21T20:11:01","date_gmt":"2023-12-21T20:11:01","guid":{"rendered":"https:\/\/dev.aenix.io\/?p=748"},"modified":"2024-02-19T01:10:39","modified_gmt":"2024-02-19T01:10:39","slug":"kubernetes-in-kubernetes-and-pxe-bootable-server-farm","status":"publish","type":"post","link":"https:\/\/dev.aenix.io\/kubernetes-in-kubernetes-and-pxe-bootable-server-farm\/","title":{"rendered":"Kubernetes-in-Kubernetes and PXE bootable server farm"},"content":{"rendered":"<p>The comprehensive data center automation project, implementing infrastructure-as-code, stateless PXE-based server farm, software-defined storage, and declarative workload management.<\/p><p><strong>Objective:<\/strong><\/p><p>This project aimed to maximize the efficiency of WEDOS&#8217;s data centers, with a focus on utilizing oil cooling technology and automating server management processes. <\/p><p>At the time of implementation, WEDOS operated two data centers. The second data center, fully built with oil cooling, was designed primarily for the Top-load <strong>HPE Moonshot 1500 Chassis<\/strong>, each housing 45 <strong>HPE ProLiant m710x Server Blades<\/strong>. <\/p><figure class=\"wp-block-gallery has-nested-images columns-default is-cropped wp-block-gallery-1 is-layout-flex wp-block-gallery-is-layout-flex\"><figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/dev.aenix.io\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/wedos5.jpg&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-large&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-1020&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:1293,&quot;targetHeight&quot;:861,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-large wp-lightbox-container\"><img fetchpriority=\"high\" decoding=\"async\" width=\"1024\" height=\"682\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" data-id=\"1020\" src=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos5-1024x682.jpg\" alt=\"\" class=\"wp-image-1020\" srcset=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos5-1024x682.jpg 1024w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos5-300x200.jpg 300w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos5-768x511.jpg 768w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos5-930x620.jpg 930w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos5.jpg 1293w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/dev.aenix.io\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/wedos6-1.jpg&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-large&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-1021&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:2048,&quot;targetHeight&quot;:1152,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-large wp-lightbox-container\"><img decoding=\"async\" width=\"1024\" height=\"576\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" data-id=\"1021\" src=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos6-1-1024x576.jpg\" alt=\"\" class=\"wp-image-1021\" srcset=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos6-1-1024x576.jpg 1024w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos6-1-300x169.jpg 300w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos6-1-768x432.jpg 768w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos6-1-1536x864.jpg 1536w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos6-1.jpg 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure>\n\n<figure data-wp-context=\"{&quot;uploadedSrc&quot;:&quot;https:\\\/\\\/dev.aenix.io\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/wedos4.jpg&quot;,&quot;figureClassNames&quot;:&quot;wp-block-image size-large&quot;,&quot;figureStyles&quot;:null,&quot;imgClassNames&quot;:&quot;wp-image-1022&quot;,&quot;imgStyles&quot;:null,&quot;targetWidth&quot;:2048,&quot;targetHeight&quot;:1152,&quot;scaleAttr&quot;:false,&quot;ariaLabel&quot;:&quot;Enlarge image&quot;,&quot;alt&quot;:&quot;&quot;}\" data-wp-interactive=\"core\/image\" class=\"wp-block-image size-large wp-lightbox-container\"><img decoding=\"async\" width=\"1024\" height=\"576\" data-wp-init=\"callbacks.setButtonStyles\" data-wp-on-async--click=\"actions.showLightbox\" data-wp-on-async--load=\"callbacks.setButtonStyles\" data-wp-on-async-window--resize=\"callbacks.setButtonStyles\" data-id=\"1022\" src=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos4-1024x576.jpg\" alt=\"\" class=\"wp-image-1022\" srcset=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos4-1024x576.jpg 1024w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos4-300x169.jpg 300w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos4-768x432.jpg 768w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos4-1536x864.jpg 1536w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/wedos4.jpg 2048w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><button\n\t\t\tclass=\"lightbox-trigger\"\n\t\t\ttype=\"button\"\n\t\t\taria-haspopup=\"dialog\"\n\t\t\taria-label=\"Enlarge image\"\n\t\t\tdata-wp-init=\"callbacks.initTriggerButton\"\n\t\t\tdata-wp-on-async--click=\"actions.showLightbox\"\n\t\t\tdata-wp-style--right=\"context.imageButtonRight\"\n\t\t\tdata-wp-style--top=\"context.imageButtonTop\"\n\t\t>\n\t\t\t<svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"12\" height=\"12\" fill=\"none\" viewBox=\"0 0 12 12\">\n\t\t\t\t<path fill=\"#fff\" d=\"M2 0a2 2 0 0 0-2 2v2h1.5V2a.5.5 0 0 1 .5-.5h2V0H2Zm2 10.5H2a.5.5 0 0 1-.5-.5V8H0v2a2 2 0 0 0 2 2h2v-1.5ZM8 12v-1.5h2a.5.5 0 0 0 .5-.5V8H12v2a2 2 0 0 1-2 2H8Zm2-12a2 2 0 0 1 2 2v2h-1.5V2a.5.5 0 0 0-.5-.5H8V0h2Z\" \/>\n\t\t\t<\/svg>\n\t\t<\/button><\/figure><\/figure><p><\/p><p>The entire data center was constructed using these servers.<\/p><p><strong>Implementation:<\/strong><\/p><p> A distinctive feature of the implemented system was its stateless automatic provisioning. Servers booted from a network-boot server and connected to the assigned Kubernetes cluster, which then scheduled user workloads on them. Some servers were dedicated to cloud platform, while others managed hosting workloads like web servers, databases, and mail servers. Due to stateless nature the servers can be simple updated by reboot operation, during next-booting process they will fetch new images from the network-boot server.<\/p><figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/kubernetes.io\/images\/blog\/2021-12-22-kubernetes-in-kubernetes-and-pxe-bootable-server-farm\/scheme04.svg\" alt=\"\"\/><\/figure><p>To manage the entire infrastructure, the Kubefarm project was developed. It facilitated the launch of Kubernetes control planes for user servers, provided DHCP configuration, and operated a network boot server for node loading.<\/p><p><strong>Outcomes:<\/strong><\/p><p>The result was a fully managed infrastructure-as-a-code. System images for servers were compiled using Dockerfile too, allowing seamless cluster updates and convenient management through a specially developed utility, moonshotctl. This approach significantly enhanced the data center&#8217;s operational efficiency and resource management.<\/p><p>The project was open sourced, and the expertise was transferred as much as possible to the infrastructure team, where the project is used and developed to this day.<\/p><p><strong>Demonstration:<\/strong><\/p><div class=\"vlp-link-container vlp-layout-quick wp-block-visual-link-preview-link\"><a href=\"https:\/\/kvaps.github.io\/images\/posts\/Kubernetes-in-Kubernetes-and-PXE-bootable-servers-farm\/Argo_CD_Applications_kubefarm-network.html\" class=\"vlp-link\" title=\"Argo CD - Applications \/ kubefarm-foo\" rel=\"nofollow\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img loading=\"lazy\" decoding=\"async\" style=\"max-width: 150px;\" width=\"150\" height=\"118\" src=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/argocd02.png\" class=\"attachment-150x999 size-150x999\" alt=\"\" srcset=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/argocd02.png 1794w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/argocd02-300x236.png 300w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/argocd02-1024x807.png 1024w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/argocd02-768x605.png 768w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/argocd02-1536x1211.png 1536w\" sizes=\"(max-width: 150px) 100vw, 150px\" \/><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">Argo CD &#8211; Applications \/ kubefarm-foo<\/div><div class=\"vlp-block-1 vlp-link-summary\">Kubefarm cluster deployed using Argo CD<\/div><\/div><\/div><script async id=\"asciicast-258607\" src=\"https:\/\/asciinema.org\/a\/258607.js\"><\/script><p><strong>Additional Links:<\/strong><\/p><div class=\"vlp-link-container vlp-layout-quick wp-block-visual-link-preview-link\"><a href=\"https:\/\/kubernetes.io\/blog\/2021\/12\/22\/kubernetes-in-kubernetes-and-pxe-bootable-server-farm\/\" class=\"vlp-link\" title=\"Kubernetes-in-Kubernetes and the WEDOS PXE bootable server farm\" rel=\"nofollow\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img loading=\"lazy\" decoding=\"async\" style=\"max-width: 150px;\" width=\"150\" height=\"91\" src=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/Screenshot-2023-12-24-at-23-53-22-Kubernetes-in-Kubernetes-and-the-WEDOS-PXE-bootable-server-farm.png\" class=\"attachment-150x999 size-150x999\" alt=\"\" srcset=\"https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/Screenshot-2023-12-24-at-23-53-22-Kubernetes-in-Kubernetes-and-the-WEDOS-PXE-bootable-server-farm.png 435w, https:\/\/dev.aenix.io\/wp-content\/uploads\/2023\/12\/Screenshot-2023-12-24-at-23-53-22-Kubernetes-in-Kubernetes-and-the-WEDOS-PXE-bootable-server-farm-300x183.png 300w\" sizes=\"(max-width: 150px) 100vw, 150px\" \/><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">Kubernetes-in-Kubernetes and the WEDOS PXE bootable server farm<\/div><div class=\"vlp-block-1 vlp-link-summary\">When you own two data centers, thousands of physical servers, virtual machines and hosting for hundreds of thousands sites, Kubernetes can actually simplify the management of all these things.<\/div><\/div><\/div><div class=\"vlp-link-container vlp-layout-quick wp-block-visual-link-preview-link\"><a href=\"https:\/\/github.com\/kubefarm\/kubefarm\" class=\"vlp-link\" title=\"kubefarm: Automated Kubernetes deployment and the PXE-bootable servers farm\" rel=\"nofollow\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img decoding=\"async\" src=\"https:\/\/opengraph.githubassets.com\/05c317c5539bc7f062f37aeefa6cb066a1664c61469bbcc97ce9db935a46617f\/kubefarm\/kubefarm\" style=\"max-width: 150px; max-height: 150px\" \/><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">kubefarm: Automated Kubernetes deployment and the PXE-bootable servers farm<\/div><div class=\"vlp-block-1 vlp-link-summary\">Automated Kubernetes deployment and the PXE-bootable servers farm &#8211; GitHub &#8211; kubefarm\/kubefarm: Automated Kubernetes deployment and the PXE-bootable servers farm<\/div><\/div><\/div><div class=\"vlp-link-container vlp-layout-quick wp-block-visual-link-preview-link\"><a href=\"https:\/\/github.com\/kubefarm\/kubernetes-in-kubernetes\" class=\"vlp-link\" title=\"kubernetes-in-kubernetes: Deploy Kubernetes in Kubernetes using Helm\" rel=\"nofollow\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img decoding=\"async\" src=\"https:\/\/opengraph.githubassets.com\/629079abb01441c7d02d6c2e0624bd989fb316c7d31e4f95fe2c5c5beb2113a5\/kubefarm\/kubernetes-in-kubernetes\" style=\"max-width: 150px; max-height: 150px\" \/><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">kubernetes-in-kubernetes: Deploy Kubernetes in Kubernetes using Helm<\/div><div class=\"vlp-block-1 vlp-link-summary\">Deploy Kubernetes in Kubernetes using Helm. Contribute to kubefarm\/kubernetes-in-kubernetes development by creating an account on GitHub.<\/div><\/div><\/div><div class=\"vlp-link-container vlp-layout-quick wp-block-visual-link-preview-link\"><a href=\"https:\/\/itnext.io\/configuring-routing-for-metallb-in-l2-mode-7ea26e19219e\" class=\"vlp-link\" title=\"Configuring routing for MetalLB in L2 mode\" rel=\"nofollow\" target=\"_blank\"><\/a><div class=\"vlp-layout-zone-side\"><div class=\"vlp-block-2 vlp-link-image\"><img decoding=\"async\" src=\"https:\/\/miro.medium.com\/v2\/resize:fit:774\/0*wI1GLh4MrCzuwiwB.png\" style=\"max-width: 150px; max-height: 150px\" \/><\/div><\/div><div class=\"vlp-layout-zone-main\"><div class=\"vlp-block-0 vlp-link-title\">Configuring routing for MetalLB in L2 mode<\/div><div class=\"vlp-block-1 vlp-link-summary\">In this article I will show you how to configure source-based and policy-based routing for the external network on your cluster.<\/div><\/div><\/div><p><strong>Presentations:<\/strong><\/p><figure class=\"wp-block-embed is-type-video is-provider-youtube wp-block-embed-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<div class=\"nv-iframe-embed\"><iframe title=\"Case-Study: 600 Nodes on DRBD + LINSTOR for Kubernetes, OpenNebula and Proxmox\" width=\"1200\" height=\"675\" src=\"https:\/\/www.youtube.com\/embed\/kMU3JAsFXQk?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" allowfullscreen><\/iframe><\/div>\n<\/div><\/figure>","protected":false},"excerpt":{"rendered":"<p>The comprehensive data center automation project, implementing infrastructure-as-code, stateless PXE-based server farm, software-defined storage, and declarative workload management.<\/p>\n","protected":false},"author":1,"featured_media":942,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"postBodyMargin":[],"postBodyPadding":[],"postBodyBackground":{"backgroundType":"classic","gradient":""},"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"[\"content\",\"tags\",\"comments\"]","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","_ti_tpc_template_sync":false,"_ti_tpc_template_id":"","footnotes":""},"categories":[7,1],"tags":[],"class_list":["post-748","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-case-studies","category-featured"],"_links":{"self":[{"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/posts\/748"}],"collection":[{"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/comments?post=748"}],"version-history":[{"count":54,"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/posts\/748\/revisions"}],"predecessor-version":[{"id":1648,"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/posts\/748\/revisions\/1648"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/media\/942"}],"wp:attachment":[{"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/media?parent=748"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/categories?post=748"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dev.aenix.io\/wp-json\/wp\/v2\/tags?post=748"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}