Description

Below is the development ansible project to setup an arm64 kubernetes (k8s) cluster. For this example cluster setup i used 3x NanoPi Fire3, thought they have limited ram for k8s they do have 8 cpu cores and gigabit LAN!

Setup here is MAC OS X focused, apologies to purists!

Download Armbian

At time of writing i used Armbian image Armbian_20.02.1_Nanopifire3_buster_legacy_4.14.171_minimal.7z.

Go ahead and download the archive and extract. Compression is 7z so you will need to have 7zip installed.

brew install p7zip

Extract the image for writing to the SD card.

$ 7z x Armbian_20.02.1_Nanopifire3_buster_legacy_4.14.171_minimal.7z

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)

Scanning the drive for archives:
1 file, 152356566 bytes (146 MiB)

Extracting archive: Armbian_20.02.1_Nanopifire3_buster_legacy_4.14.171_minimal.7z
--
Path = Armbian_20.02.1_Nanopifire3_buster_legacy_4.14.171_minimal.7z
Type = 7z
Physical Size = 152356566
Headers Size = 289
Method = LZMA2:25
Solid = +
Blocks = 1

  0%
Everything is Ok                                                     

Files: 4
Size:       645943389
Compressed: 152356566

Flash the SD cards with armbian image

I have selected 64GB Application class SD cards for the nanopi3 operating system. Install etcher available here.

Etcher is easy to use. Insert the cards into your computer for writing, select the extracted image from the previous step and Flash!

More detail on flashing here

Flash flashing validating flash another

Initial login and root password

With the SD cards flashed its time to boot into the operating system and configure static ip networking. If you have local DNS running setup your A and PTR records, if not add hostnames to /etc/hosts file for all your nanopi devices. How to login

I have DHCP on my network so i am going to just boot each nano pi up one by one and scan for its DHCP configured IP address on the local network with nmap.

Below shows 2 scans of local subnet, second after switching on the nanopi shows the ip at 182.

$ nmap -sP 192.168.2.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2020-03-07 19:19 GMT
Nmap scan report for 192.168.2.1
Host is up (0.0064s latency).
Nmap done: 256 IP addresses (1 hosts up) scanned in 2.26 seconds
$ nmap -sP 192.168.2.0/24
Starting Nmap 7.80 ( https://nmap.org ) at 2020-03-07 19:21 GMT
Nmap scan report for 192.168.2.1
Host is up (0.0064s latency).
Nmap scan report for 192.168.2.182
Host is up (0.018s latency).
Nmap done: 256 IP addresses (2 hosts up) scanned in 2.67 seconds

The default root password is 1234 so login with SSH and set the new default password. I will also create an admin user.

$ ssh [email protected]
[email protected]'s password: 
You are required to change your password immediately (administrator enforced)
 _   _ ____  _   _____ _          _____ 
| \ | |  _ \(_) |  ___(_)_ __ ___|___ / 
|  \| | |_) | | | |_  | | '__/ _ \ |_ \ 
| |\  |  __/| | |  _| | | | |  __/___) |
|_| \_|_|   |_| |_|   |_|_|  \___|____/ 
                                        
Welcome to Armbian buster with Linux 4.14.171-s5p6818

System load:   0.00 0.07 0.06   Up time:       9 min    
Memory usage:  7 % of 991MB   IP:            192.168.2.182
CPU temp:      51°C             
Usage of /:    1% of 58G      

[ Menu-driven system configuration (beta): sudo apt update && sudo apt install armbian-config ]

Last login: Sat Mar  7 19:22:09 2020 from 192.168.4.3
Changing password for root.
Current password: 
New password: 
Retype new password: 


Thank you for choosing Armbian! Support: www.armbian.com

Creating a new user account. Press <Ctrl-C> to abort

Please provide a username (eg. your forename): admin
Trying to add user admin
Adding user `admin' ...
Adding new group `admin' (1000) ...
Adding new user `admin' (1000) with group `admin' ...
Creating home directory `/home/admin' ...
Copying files from `/etc/skel' ...
New password: 
Retype new password: 
passwd: password updated successfully
Changing the user information for admin
Enter the new value, or press ENTER for the default
  Full Name []: 
  Room Number []: 
  Work Phone []: 
  Home Phone []: 
  Other []: 
Is the information correct? [Y/n] 

Dear admin, your account admin has been created and is sudo enabled.
Please use this account for your daily work from now on.

[email protected]:~# 

Static IPv4 Address configuration

Now to set the static IP address. There is a HOWTO guide here HOW TO SET FIXED IP.

I am going to assign each the following IP addresses.

  1. 192.168.2.21 nano1.home.lan
  2. 192.168.2.22 nano2.home.lan
  3. 192.168.2.23 nano3.home.lan
[email protected]:~# nmcli con mod "Wired connection 1" \
>   ipv4.addresses "192.168.2.21/24" \
>   ipv4.gateway "192.168.2.1" \
>   ipv4.dns "192.168.0.3" \
>   ipv4.dns-search "home.lan" \
>   ipv4.method "manual"
[email protected]:~# 
[email protected]:~# nmcli con show
NAME                UUID                                  TYPE      DEVICE 
Wired connection 1  a2179b29-4e21-3008-aff0-c6357fe41772  ethernet  eth0   
[email protected]:~# reboot
Connection to 192.168.2.182 closed by remote host.
Connection to 192.168.2.182 closed.

Reboot is quick i can then log in over SSH to the new IP address and root password.

The authenticity of host '192.168.2.21 (192.168.2.21)' can't be established.
ECDSA key fingerprint is SHA256:+Af/QmT0RYpmaMg+sirtEbN4+p435Jt3PoCGRyvS2AE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.2.21' (ECDSA) to the list of known hosts.
[email protected]'s password: 
 _   _ ____  _   _____ _          _____ 
| \ | |  _ \(_) |  ___(_)_ __ ___|___ / 
|  \| | |_) | | | |_  | | '__/ _ \ |_ \ 
| |\  |  __/| | |  _| | | | |  __/___) |
|_| \_|_|   |_| |_|   |_|_|  \___|____/ 
                                        
Welcome to Armbian buster with Linux 4.14.171-s5p6818

System load:   0.24 0.13 0.05   Up time:       1 min    
Memory usage:  7 % of 991MB   IP:            192.168.2.21
CPU temp:      55°C             
Usage of /:    1% of 58G      

[ Menu-driven system configuration (beta): sudo apt update && sudo apt install armbian-config ]

Last login: Sat Mar  7 19:27:17 2020 from 192.168.4.3

[email protected]:~# 

Set the hostname

[email protected]:~# hostnamectl
   Static hostname: nanopifire3
         Icon name: computer
        Machine ID: 25bb06f94489458aabb5147c4aef8708
           Boot ID: 1a00a0cd659d4033991b6314c39bb4f8
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.14.171-s5p6818
      Architecture: arm64
[email protected]:~# hostnamectl set-hostname nano1.home.lan
[email protected]:~# logout
Connection to 192.168.2.21 closed.
Paul-Errington's-MacBook-Pro:~ paulerrington$ ssh [email protected]
 _   _ ____  _   _____ _          _____ 
| \ | |  _ \(_) |  ___(_)_ __ ___|___ / 
|  \| | |_) | | | |_  | | '__/ _ \ |_ \ 
| |\  |  __/| | |  _| | | | |  __/___) |
|_| \_|_|   |_| |_|   |_|_|  \___|____/ 
                                        
Welcome to Armbian buster with Linux 4.14.171-s5p6818

System load:   0.00 0.25 2.56   Up time:       14:40 hours    
Memory usage:  19 % of 991MB    IP:            192.168.2.21
CPU temp:      63°C             
Usage of /:    5% of 58G      

[ Menu-driven system configuration (beta): sudo apt update && sudo apt install armbian-config ]

Last login: Wed Mar 11 11:25:25 2020 from 192.168.4.3

[email protected]:~# hostnamectl
   Static hostname: nano1.home.lan
         Icon name: computer
        Machine ID: 25bb06f94489458aabb5147c4aef8708
           Boot ID: 1a00a0cd659d4033991b6314c39bb4f8
  Operating System: Debian GNU/Linux 10 (buster)
            Kernel: Linux 4.14.171-s5p6818
      Architecture: arm64
[email protected]:~# hostname
nano1.home.lan

Checkpoint

Repeat the previous steps until all 3 nano pis have passwords set, static ip addresses and hostnames assigned.

Installing requirements.

Python is required as i am using ansible playbooks to provision software requirements for kubernetes.

apt-get install python python-dev python-pip libffi-dev -y

provision

before installing k8s, docker is required. below playbook has tasks to configure base requirements. For in depth build see the google documentation

ansible playbook for armbian to install docker and k8s

$ ansible-playbook docker.yml -i inventory –diff -u root -k

---

- hosts: all
  become: yes
  vars:

  tasks:

  - name: Set the max open file descriptors 
    sysctl: name=fs.file-max value=500000 state=present

  - name: Add or modify hard nofile limits for wildcard domain
    pam_limits:
      domain: '*'
      limit_type: hard
      limit_item: nofile
      value: 500000

  - name: Add or modify soft nofile limits for wildcard domain
    pam_limits:
      domain: '*'
      limit_type: soft
      limit_item: nofile
      value: 500000

  # - selinux:
  #     state: disabled

  - sysctl:
      name: vm.swappiness
      value: 0
      state: present

  - name: Remove swapfile from /etc/fstab
    mount:
      name: swap
      fstype: swap
      state: absent

  - name: Disable swap
    command: swapoff -a

  - name: install requirements
    apt:
      name: "{{ packages }}"
      state: present
    vars:
      packages:
      - vim
      - aptitude
      - apt-transport-https
      - ca-certificates
      - curl
      - gnupg2
      - software-properties-common
      - python
      - python-dev
      - python-pip
      - libffi-dev
      - libssl-dev
      - build-essential
      - python-setuptools
      - iptables 
      - arptables
      - ebtables

  - name: switch to legacy versions (iptables)
    alternatives:
      name: iptables
      path: /usr/sbin/iptables-legacy
  - name: switch to legacy versions (ip6tables)
    alternatives:
      name: ip6tables
      path: /usr/sbin/ip6tables-legacy
  - name: switch to legacy versions (arptables)
    alternatives:
      name: arptables
      path: /usr/sbin/arptables-legacy
  - name: switch to legacy versions (ebtables)
    alternatives:
      name: ebtables
      path: /usr/sbin/ebtables-legacy

  - name: Add apt signing key for docker debian
    apt_key:
      url: https://download.docker.com/linux/debian/gpg 
      state: present

  - name: Add docker repository
    apt_repository:
      repo: "deb [arch=arm64] https://download.docker.com/linux/debian buster stable"
      state: present
    tags: docker

  - name: update cache
    apt:
      update_cache: yes

  - name: Upgrade all packages to the latest version
    apt:
      name: "*"
      state: latest

  - name: install docker
    apt:
      name: "{{ packages }}"
    vars:
      packages:
      - docker-ce=5:19.03.4~3-0~debian-buster
      - docker-ce-cli=5:19.03.4~3-0~debian-buster
      - containerd.io=1.2.10-3

  - copy:
      dest: /etc/docker/daemon.json
      content: |
        {
          "exec-opts": ["native.cgroupdriver=systemd"],
          "log-driver": "json-file",
          "log-opts": {
            "max-size": "100m"
          },
          "storage-driver": "overlay2"
        }

  - name: reload systemd and restart docker
    systemd:
      state: restarted
      daemon_reload: yes
      name: docker

  - copy:
      dest: /etc/apt/preferences.d/docker-ce
      content: |
        Package: docker-ce
        Pin: version 5:19.03.4*
        Pin-Priority: 1000

  - copy:
      dest: /etc/apt/preferences.d/docker-ce-cli
      content: |
        Package: docker-ce-cli
        Pin: version 5:19.03.4*
        Pin-Priority: 1000

  - copy:
      dest: /etc/apt/preferences.d/containerd
      content: |
        Package: containerd.io
        Pin: version 1.2.10-3*
        Pin-Priority: 1000

  - name: install docker-compose with pip
    pip:
      name:
      - setuptools
      - wheel
      - virtualenv
      - docker-compose

  - name: Add google gpg key
    apt_key:
      url: https://packages.cloud.google.com/apt/doc/apt-key.gpg 
      state: present

  - name: Add kube repository
    apt_repository:
      repo: "deb [arch=arm64] https://apt.kubernetes.io/ kubernetes-xenial main"
      state: present

  - name: update cache
    apt:
      update_cache: yes

#/etc/default/kubelet
#KUBELET_EXTRA_ARGS=--cgroup-driver=systemd

  - name: install kubernets (1.17.3)
    apt:
      name: "{{ packages }}"
    vars:
      packages:
      - kubelet=1.17.3-00
      - kubeadm=1.17.3-00
      - kubectl=1.17.3-00

With the base OS installed, network configured and software requirments install. Next configuring a cluster.