Prepare a Kubernetes cluster for ELMA365

This article describes how to deploy a Kubernetes cluster using Deckhouse in a private environment with no access to the public container image registry (registry.deckhouse.io) or public deb/rpm packages repositories.

Deckhouse is a fully-featured platform based on Open Source components. Apart from “Vanilla Kubernetes”, it includes modules for monitoring, load balancing, autoscaling, secure access, etc. The modules are pre-configured, integrated with each other, and ready to use. All cluster and platform components are managed (and updated) in a fully automated fashion.

Deckhouse is certified by CNCF.

You will need:

1. A personal computer.

The computer from which the installation will be performed. It is only needed to run the Deckhouse installer and will not be part of the cluster.

Requirements:

• OS: Windows 10+, macOS 10.15+, Linux (e.g. Ubuntu 18.04+, Fedora 35+).
• Installed Docker to run the Deckhouse installer.
• Access to a proxy registry (read more about setting them up) or to a private container image registry containing Deckhouse images.
• SSH key access to the node, the primary node of the future cluster.

2. The primary node (nodes).

A server (physical server or virtual machine) that will be the primary node of the future cluster.

During the installation, the Deckhouse installer running on the personal computer will connect to that host via SSH, install the necessary packages, configure control plane Kubernetes, and deploy Deckhouse.

начало внимание

Note that installation directly on the primary node is not currently supported.

конец внимание

Requirements:

• At least 4 CPU cores.
• At least 8 GB of RAM.
• At least 40 GB of disk space for the cluster and etcd data.
• Supported OS.
• Access to a proxy registry or to a private container image registry containing Deckhouse images.

начало внимание

Note that Deckhouse only supports Bearer authentication for registries.

конец внимание

• Access to a proxy server for downloading deb/rpm OS packages (if necessary).
• SSH key access from the personal computer (section 1).
• Container runtime packages, such as containerd or docker, should not be installed on the node.

3. Additional nodes (not required).

Depending on the purpose of the cluster, you may need additional nodes, for example, worker nodes, dedicated nodes for monitoring, dedicated nodes for load balancers (frontend nodes), and others.

Below are the minimum requirements for any node (node group):

• Supported OS.
• Access to a proxy registry or to a private container image registry containing Deckhouse images.
• Access to a proxy server for downloading deb/rpm OS packages (if necessary).
• Container runtime packages, such as containerd or docker, should not be installed on the node.

Below are the minimum requirements for any node (node group) with the monitoring role:

• At least 4 CPU cores.
• At least 8 GB of RAM.

Read about configuring advanced scheduling in the Deckhouse documentation.

Uploading Deckhouse images to a private registry

On a computer with internet access, download the script that downloads the images, open the terminal, and run the following command:

curl -fsSL -o d8-pull.sh https://raw.githubusercontent.com/deckhouse/deckhouse/main/tools/release/d8-pull.sh
chmod 700 d8-pull.sh

Note that Docker, crane, and jq need to be installed on the host.

Example of a command for image download:

./d8-pull.sh --license YOUR_DECKHOUSE_LICENSE_KEY --output-dir /your/output-dir/

Load the directory with images you got at the previous step to the host that has access to the private registry. The host needs to have crane installed.

curl -fsSL -o d8-push.sh https://raw.githubusercontent.com/deckhouse/deckhouse/main/tools/release/d8-push.sh
chmod 700 d8-push.sh

Run the following command to upload the Deckhouse images to the private registry:

./d8-push.sh --source-dir /your/source-dir/ --path registry.example.com:443/images/deckhouse --username admin --password Harbor12345

Install Deckhouse

The personal computer used for installation needs to have SSH access to the node that will be the primary node of the future cluster.

Generate the public and private keys:

ssh-keygen

Pass the public key id_rsa.pub to the primary node:

ssh-copy-id user@example.com

To generate the configuration file, you can use the Getting started section on the Deckhouse official website.

Create a configuration file:

nano /$HOME/config.yml

Here is an example of a cluster configuration file, config.yml. It contains the installer parameters and the initial parameters of the cluster:

apiVersion: deckhouse.io/v1
kind: InitConfiguration
deckhouse:
  devBranch: v1.36.7
  configOverrides:
    global:
      modules:
        publicDomainTemplate: "%s.example.com"
    cniFlannelEnabled: true
    cniFlannel:
      podNetworkMode: VXLAN
  imagesRepo: registry.example.com:443/images/deckhouse
  registryDockerCfg: <YOUR_PRIVATE_ACCESS_STRING_IS_HERE>
  registryScheme: HTTPS
  registryCA: <REGISTRY_CA>
---
apiVersion: deckhouse.io/v1
kind: StaticClusterConfiguration
internalNetworkCIDRs:
- 192.168.0.0/24

Set the following parameters in the InitConfiguration resource:

• devBranch: if your air-gapped registry does not have images containing information about update channels, you need to use the exact tag of a Docker image to install the Deckhouse Platform. For example, if you want to install release v1.36.7, you have to use image registry.example.com/images/deckhouse/install:v1.36.7. You also have to use devBranch: v1.36.7 instead of releaseChannel: XXX in config.yml.
• publicDomainTemplate is used to generate system application domains in the cluster. For example, for template %s.example.com, Grafana will be available at grafana.example.com.
• imagesRepo: <PROXY_REGISTRY>/<DECKHOUSE_REPO_PATH>/<DECKHOUSE_REVISION> is the address of the Deckhouse image in the private registry, including the revision.
• registryDockerCfg: <BASE64> contains the access permissions to the private registry encrypted in the Base64 format.
• registryScheme: here you need to specify the protocol of the private registry (HTTP or HTTPS).
• registryCA is the root SSL certificate that can be used to check the private registry’s SSL certificate, for instance, if the registry’s certificate is self-signed.
• internalNetworkCIDRs is a list of internal networks of the cluster’s nodes (for example, '10.0.4.0/24'). It is used to connect Kubernetes components (kube-apiserver, kubelet, etc.) with each other.

Run the installer:

docker run --pull=always -it -v "/$HOME/config.yml:/config.yml" -v "$HOME/.ssh/:/tmp/.ssh/" example.com:443/images/deckhouse/install:v1.36.7 bash

dhctl bootstrap --ssh-user=user --ssh-host=example.com --ssh-agent-private-keys=/tmp/.ssh/id_rsa --config=/config.yml --ask-become-pass

Configuration

If one node is enough for you, allow Deckhouse components to work on the primary node. To do that, untaint the primary node by running the following command on it:

kubectl patch nodegroup master --type json -p '[{"op": "remove", "path": "/spec/nodeTemplate/taints"}]'

Install an Ingress controller

On the primary node, create a file called ingress-nginx-controller.yml containing the Ingress controller configuration:

nano /$HOME/ingress-nginx-controller.yml

# Section containing the parameters of the nginx ingress controller.
# Version of the Deckhouse API.
apiVersion: deckhouse.io/v1
kind: IngressNginxController
metadata:
  name: nginx
spec:
  # The name of the Ingress class to use with the Ingress nginx controller.
  ingressClass: nginx
  # Ingress version to use (use version 1.1 with Kubernetes 1.23+)
  controllerVersion: "1.1"
  # The way traffic goes to cluster from the outer network.
  inlet: HostPort
  hostPort:
    httpPort: 80
    httpsPort: 443
  # Describes which nodes the component will be located on.
  # You might consider changing them.
  nodeSelector:
    node-role.kubernetes.io/master: ""
  tolerations:
  - operator: Exists

Apply the configuration file by running the following command on the primary node:

kubectl create -f /$HOME/ingress-nginx-controller.yml

Install Helm

On the computer with internet access, go to the Helm releases page and download the helm-vX.Y.Z-linux-amd64.tar.gz archive for the version you need. To do that, open the terminal and run the following command:

wget https://get.helm.sh/helm-vX.Y.Z-linux-amd64.tar.gz

The archive you get needs to be copied to the server.

Unpack and move the archive:

tar -zxvf helm-vX.Y.Z-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm

Was this helpful?

Thanks for your feedback!

Please specify why:

Recommendations did not help meArticle is hard to understandDidn`t answer my questionContent does not match the topicOther

How we can improve it?

YesNo

Please specify whyRecommendations did not help meArticle is hard to understandDidn`t answer my questionContent does not match the topicOther

Found a typo? Highlight the text, press ctrl + enter and notify us