k8s armbian mini cluster - Part 1 Setup
Description
Below is the development ansible project to setup an arm64 kubernetes (k8s) cluster. For this example cluster setup i used 3x NanoPi Fire3, thought they have limited ram for k8s they do have 8 cpu cores and gigabit LAN!
Setup here is MAC OS X focused, apologies to purists!
Download Armbian
At time of writing i used Armbian image Armbian_20.02.1_Nanopifire3_buster_legacy_4.14.171_minimal.7z.
Go ahead and download the archive and extract. Compression is 7z so you will need to have 7zip installed.
brew install p7zipExtract the image for writing to the SD card.
$ 7z x Armbian_20.02.1_Nanopifire3_buster_legacy_4.14.171_minimal.7z
7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)
Scanning the drive for archives:
1 file, 152356566 bytes (146 MiB)
Extracting archive: Armbian_20.02.1_Nanopifire3_buster_legacy_4.14.171_minimal.7z
--
Path = Armbian_20.02.1_Nanopifire3_buster_legacy_4.14.171_minimal.7z
Type = 7z
Physical Size = 152356566
Headers Size = 289
Method = LZMA2:25
Solid = +
Blocks = 1
0%
Everything is Ok
Files: 4
Size: 645943389
Compressed: 152356566Flash the SD cards with armbian image
I have selected 64GB Application class SD cards for the nanopi3 operating system. Install etcher available here.
Etcher is easy to use. Insert the cards into your computer for writing, select the extracted image from the previous step and Flash!
Initial login and root password
With the SD cards flashed its time to boot into the operating system and configure static ip networking. If you have local DNS running setup your A and PTR records, if not add hostnames to /etc/hosts file for all your nanopi devices. How to login
I have DHCP on my network so i am going to just boot each nano pi up one by one and scan for its DHCP configured IP address on the local network with nmap.
Below shows 2 scans of local subnet, second after switching on the nanopi shows the ip at 182.
$ nmap -sP 192.168.2.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2020-03-07 19:19 GMT
Nmap scan report for 192.168.2.1
Host is up (0.0064s latency).
Nmap done: 256 IP addresses (1 hosts up) scanned in 2.26 seconds$ nmap -sP 192.168.2.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2020-03-07 19:21 GMT
Nmap scan report for 192.168.2.1
Host is up (0.0064s latency).
Nmap scan report for 192.168.2.182
Host is up (0.018s latency).
Nmap done: 256 IP addresses (2 hosts up) scanned in 2.67 secondsThe default root password is 1234 so login with SSH and set the new default password. I will also create an admin user.
$ ssh [email protected]
[email protected]'s password:
You are required to change your password immediately (administrator enforced)
_ _ ____ _ _____ _ _____
| \ | | _ \(_) | ___(_)_ __ ___|___ /
| \| | |_) | | | |_ | | '__/ _ \ |_ \
| |\ | __/| | | _| | | | | __/___) |
|_| \_|_| |_| |_| |_|_| \___|____/
Welcome to Armbian buster with Linux 4.14.171-s5p6818
System load: 0.00 0.07 0.06 Up time: 9 min
Memory usage: 7 % of 991MB IP: 192.168.2.182
CPU temp: 51°C
Usage of /: 1% of 58G
[ Menu-driven system configuration (beta): sudo apt update && sudo apt install armbian-config ]
Last login: Sat Mar 7 19:22:09 2020 from 192.168.4.3
Changing password for root.
Current password:
New password:
Retype new password:
Thank you for choosing Armbian! Support: www.armbian.com
Creating a new user account. Press <Ctrl-C> to abort
Please provide a username (eg. your forename): admin
Trying to add user admin
Adding user `admin' ...
Adding new group `admin' (1000) ...
Adding new user `admin' (1000) with group `admin' ...
Creating home directory `/home/admin' ...
Copying files from `/etc/skel' ...
New password:
Retype new password:
passwd: password updated successfully
Changing the user information for admin
Enter the new value, or press ENTER for the default
Full Name []:
Room Number []:
Work Phone []:
Home Phone []:
Other []:
Is the information correct? [Y/n]
Dear admin, your account admin has been created and is sudo enabled.
Please use this account for your daily work from now on.
root@nanopifire3:~# Static IPv4 Address configuration
Now to set the static IP address. There is a HOWTO guide here HOW TO SET FIXED IP.
I am going to assign each the following IP addresses.
- 192.168.2.21 nano1.home.lan
- 192.168.2.22 nano2.home.lan
- 192.168.2.23 nano3.home.lan
root@nanopifire3:~# nmcli con mod "Wired connection 1" \
> ipv4.addresses "192.168.2.21/24" \
> ipv4.gateway "192.168.2.1" \
> ipv4.dns "192.168.0.3" \
> ipv4.dns-search "home.lan" \
> ipv4.method "manual"
root@nanopifire3:~#
root@nanopifire3:~# nmcli con show
NAME UUID TYPE DEVICE
Wired connection 1 a2179b29-4e21-3008-aff0-c6357fe41772 ethernet eth0
root@nanopifire3:~# reboot
Connection to 192.168.2.182 closed by remote host.
Connection to 192.168.2.182 closed.Reboot is quick i can then log in over SSH to the new IP address and root password.
The authenticity of host '192.168.2.21 (192.168.2.21)' can't be established.
ECDSA key fingerprint is SHA256:+Af/QmT0RYpmaMg+sirtEbN4+p435Jt3PoCGRyvS2AE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.2.21' (ECDSA) to the list of known hosts.
[email protected]'s password:
_ _ ____ _ _____ _ _____
| \ | | _ \(_) | ___(_)_ __ ___|___ /
| \| | |_) | | | |_ | | '__/ _ \ |_ \
| |\ | __/| | | _| | | | | __/___) |
|_| \_|_| |_| |_| |_|_| \___|____/
Welcome to Armbian buster with Linux 4.14.171-s5p6818
System load: 0.24 0.13 0.05 Up time: 1 min
Memory usage: 7 % of 991MB IP: 192.168.2.21
CPU temp: 55°C
Usage of /: 1% of 58G
[ Menu-driven system configuration (beta): sudo apt update && sudo apt install armbian-config ]
Last login: Sat Mar 7 19:27:17 2020 from 192.168.4.3
root@nanopifire3:~# Set the hostname
root@nanopifire3:~# hostnamectl
Static hostname: nanopifire3
Icon name: computer
Machine ID: 25bb06f94489458aabb5147c4aef8708
Boot ID: 1a00a0cd659d4033991b6314c39bb4f8
Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.14.171-s5p6818
Architecture: arm64
root@nanopifire3:~# hostnamectl set-hostname nano1.home.lan
root@nanopifire3:~# logout
Connection to 192.168.2.21 closed.
Paul-Errington's-MacBook-Pro:~ paulerrington$ ssh [email protected]
_ _ ____ _ _____ _ _____
| \ | | _ \(_) | ___(_)_ __ ___|___ /
| \| | |_) | | | |_ | | '__/ _ \ |_ \
| |\ | __/| | | _| | | | | __/___) |
|_| \_|_| |_| |_| |_|_| \___|____/
Welcome to Armbian buster with Linux 4.14.171-s5p6818
System load: 0.00 0.25 2.56 Up time: 14:40 hours
Memory usage: 19 % of 991MB IP: 192.168.2.21
CPU temp: 63°C
Usage of /: 5% of 58G
[ Menu-driven system configuration (beta): sudo apt update && sudo apt install armbian-config ]
Last login: Wed Mar 11 11:25:25 2020 from 192.168.4.3
root@nano1:~# hostnamectl
Static hostname: nano1.home.lan
Icon name: computer
Machine ID: 25bb06f94489458aabb5147c4aef8708
Boot ID: 1a00a0cd659d4033991b6314c39bb4f8
Operating System: Debian GNU/Linux 10 (buster)
Kernel: Linux 4.14.171-s5p6818
Architecture: arm64
root@nano1:~# hostname
nano1.home.lanCheckpoint
Repeat the previous steps until all 3 nano pis have passwords set, static ip addresses and hostnames assigned.
Installing requirements.
Python is required as i am using ansible playbooks to provision software requirements for kubernetes.
apt-get install python python-dev python-pip libffi-dev -yprovision
before installing k8s, docker is required. below playbook has tasks to configure base requirements. For in depth build see the google documentation
ansible playbook for armbian to install docker and k8s
$ ansible-playbook docker.yml -i inventory –diff -u root -k
---
- hosts: all
become: yes
vars:
tasks:
- name: Set the max open file descriptors
sysctl: name=fs.file-max value=500000 state=present
- name: Add or modify hard nofile limits for wildcard domain
pam_limits:
domain: '*'
limit_type: hard
limit_item: nofile
value: 500000
- name: Add or modify soft nofile limits for wildcard domain
pam_limits:
domain: '*'
limit_type: soft
limit_item: nofile
value: 500000
# - selinux:
# state: disabled
- sysctl:
name: vm.swappiness
value: 0
state: present
- name: Remove swapfile from /etc/fstab
mount:
name: swap
fstype: swap
state: absent
- name: Disable swap
command: swapoff -a
- name: install requirements
apt:
name: "{{ packages }}"
state: present
vars:
packages:
- vim
- aptitude
- apt-transport-https
- ca-certificates
- curl
- gnupg2
- software-properties-common
- python
- python-dev
- python-pip
- libffi-dev
- libssl-dev
- build-essential
- python-setuptools
- iptables
- arptables
- ebtables
- name: switch to legacy versions (iptables)
alternatives:
name: iptables
path: /usr/sbin/iptables-legacy
- name: switch to legacy versions (ip6tables)
alternatives:
name: ip6tables
path: /usr/sbin/ip6tables-legacy
- name: switch to legacy versions (arptables)
alternatives:
name: arptables
path: /usr/sbin/arptables-legacy
- name: switch to legacy versions (ebtables)
alternatives:
name: ebtables
path: /usr/sbin/ebtables-legacy
- name: Add apt signing key for docker debian
apt_key:
url: https://download.docker.com/linux/debian/gpg
state: present
- name: Add docker repository
apt_repository:
repo: "deb [arch=arm64] https://download.docker.com/linux/debian buster stable"
state: present
tags: docker
- name: update cache
apt:
update_cache: yes
- name: Upgrade all packages to the latest version
apt:
name: "*"
state: latest
- name: install docker
apt:
name: "{{ packages }}"
vars:
packages:
- docker-ce=5:19.03.4~3-0~debian-buster
- docker-ce-cli=5:19.03.4~3-0~debian-buster
- containerd.io=1.2.10-3
- copy:
dest: /etc/docker/daemon.json
content: |
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
- name: reload systemd and restart docker
systemd:
state: restarted
daemon_reload: yes
name: docker
- copy:
dest: /etc/apt/preferences.d/docker-ce
content: |
Package: docker-ce
Pin: version 5:19.03.4*
Pin-Priority: 1000
- copy:
dest: /etc/apt/preferences.d/docker-ce-cli
content: |
Package: docker-ce-cli
Pin: version 5:19.03.4*
Pin-Priority: 1000
- copy:
dest: /etc/apt/preferences.d/containerd
content: |
Package: containerd.io
Pin: version 1.2.10-3*
Pin-Priority: 1000
- name: install docker-compose with pip
pip:
name:
- setuptools
- wheel
- virtualenv
- docker-compose
- name: Add google gpg key
apt_key:
url: https://packages.cloud.google.com/apt/doc/apt-key.gpg
state: present
- name: Add kube repository
apt_repository:
repo: "deb [arch=arm64] https://apt.kubernetes.io/ kubernetes-xenial main"
state: present
- name: update cache
apt:
update_cache: yes
#/etc/default/kubelet
#KUBELET_EXTRA_ARGS=--cgroup-driver=systemd
- name: install kubernets (1.17.3)
apt:
name: "{{ packages }}"
vars:
packages:
- kubelet=1.17.3-00
- kubeadm=1.17.3-00
- kubectl=1.17.3-00With the base OS installed, network configured and software requirments install. Next configuring a cluster.