Cloud-init

Figure 994. Introduction and reference Slide presentation

Figure 995. In a nutshell Slide presentation
  • Distribution image containing pre-installed Cloud Init

  • Script configurable installation options


Figure 996. Configuration options Slide presentation
  • Individual CRUD file operations

  • Supplying ssh user and host keys.

  • Adding users

  • ...

  • Installing packages

  • System Upgrade + reboot

  • Arbitrary command execution


Figure 997. Terraform interface to Cloud Init Slide presentation
resource "hcloud_server" "web" {
  name         = var.server_name
       ...
  user_data = file("userData.yml")
}

Figure 998. »hello, world ...« userData.yml file Slide presentation
#cloud-config
packages:
  - nginx
runcmd:
  - systemctl enable nginx
  - rm /var/www/html/*
  - >
    echo "I'm Nginx @ $(dig -4 TXT +short o-o.myaddr.l.google.com @ns1.google.com) 
    created $(date -u)" >> /var/www/html/index.html

Figure 999. Watch out for your enemies! Slide presentation
root@hello:~# journalctl -f
May 06 04:41:20 hello cloud-init[898]: Cloud-init v. 22.4.2 finished at Mon, 06 May 2024 04:41:20 +0000. Datasource DataSourceHetzner.  Up 11.78 seconds
   ...
May 06 04:46:16 hello sshd[927]: Invalid user abc from 43.163.218.130 port 33408
May 06 04:46:17 hello sshd[927]: Received disconnect from 43.163.218.130 port 33408:11: Bye Bye [preauth]
May 06 04:46:17 hello sshd[927]: Disconnected from invalid user abc 43.163.218.130 port 33408 [preauth]
   ...
May 06 04:50:54 hello sshd[930]: fatal: Timeout before authentication for 27.128.243.225 port 59866
   ...
May 06 04:52:45 hello sshd[933]: Invalid user cos from 43.163.218.130 port 59776
   ...
May 06 04:53:04 hello sshd[935]: Invalid user admin from 194.169.175.35 port 51128
May 06 04:53:49 hello sshd[937]: User root from 43.163.218.130 not allowed because not listed in AllowUsers
May 06 04:53:49 hello sshd[937]: Disconnected from invalid user root 43.163.218.130 port 50592 [preauth]

exercise No. 4

Working on Cloud-init

Q:

We continue our exercise series Incrementally creating a base system by adding a Cloud-init configuration:

  1. Follow Figure 997, “Terraform interface to Cloud Init and Figure 998, “»hello, world ...« userData.yml file ” to create simple web server.

    Tip

    You will have to extend your current firewall configuration allowing inbound traffic to port 80.

    On success pointing your web browser of choice to http://<your server's IP> should result in something similar to:

    I'm Nginx @ "95.217.154.104" created Sun May 5 06:58:37 PM UTC 2024
  2. With respect to Figure 999, “Watch out for your enemies! ” inspect the output of journalctl -f on your own server for a while. Then modify your current sshd configuration:

    On success the following sequence should be possible:

    $ ssh -v devops@95.217.154.104
    ...
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey 
    debug1: Next authentication method: publickey
    ...
    Last login: Sun May  5 19:21:12 2024 from 217.245.243.187
    devops@hello:~$ sudo su - 
    root@hello:~# hostname
    hello

    password is not among the list of allowed authentication methods.

    User devops may execute sudo commands by virtue of his membership in group sudo.

    On contrary ssh root login must be prohibited:

    $ ssh root@95.217.154.104
    root@95.217.154.104: Permission denied (publickey).
  3. Read the cloud-init documentation and/or related tutorials to:

    • Currently your (most likely outdated) cloud provider supplied distribution does not get upgraded on installation time:

      $ ./bin/ssh 
      ...
      devops@hello:~$ sudo su -
      root@hello:~# apt update
      Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
      Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
      ...
      Reading package lists... Done
      Building dependency tree... Done
      Reading state information... Done
      6 packages can be upgraded. Run 'apt list --upgradable' to see them.
      
      # apt list --upgradable
      Listing... Done
      less/stable-security,stable-security 590-2.1~deb12u2 amd64 [upgradable from: 590-2]
      libc-bin/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      libc-l10n/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]
      libc6/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales-all/stable-security,stable-security 2.36-9+deb12u7 amd64 [upgradable from: 2.36-9+deb12u6]
      locales/stable-security,stable-security 2.36-9+deb12u7 all [upgradable from: 2.36-9+deb12u6]

      Modify your Cloud-init configuration to upgrade your distribution at server creation time. If so required your system should also reboot.

    • Install and configure fail2ban limiting ssh failed connection attempts.

      Tip

    • Install the plocate file indexer package and initialize it.

    On success all packages should be up to date:

    $ ./bin/ssh 
    ...
    devops@hello:~$ sudo su -
    root@hello:~# apt update
    Hit:1 http://security.debian.org/debian-security bookworm-security InRelease
    Hit:2 http://deb.debian.org/debian bookworm InRelease                                              
    ...
    Reading package lists... Done
    Building dependency tree... Done
    Reading state information... Done
    All packages are up to date.

    Failed login attempts should be banned: Keep a second login open in advance when trying to simulate login failures! You should then see a report similar to:

    root@hello:~# fail2ban-client status sshd
    Status for the jail: sshd
    |- Filter
    |  |- Currently failed:	2
    |  |- Total failed:	14
    |  `- Journal matches:	_SYSTEMD_UNIT=sshd.service + _COMM=sshd
    `- Actions
       |- Currently banned:	2
       |- Total banned:	2
       `- Banned IP list:	170.64.133.30 213.136.94.219

    Searching for file name components should work like e.g.:

    root@hello:~# locate ssh_host
    /etc/ssh/ssh_host_ed25519_key
    /etc/ssh/ssh_host_ed25519_key.pub
Figure 1000. Problem: Duplicate known_hosts entry on re-creating server Slide presentation

Problem of repeated terraform apply:

$ ssh root@128.140.108.60
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!

Figure 1001. Solution: Generating known_hosts ... Slide presentation
resource "local_file" "known_hosts" {
  content         = "${hcloud_server.helloServer.ipv4_address} ...
                      ... ${tls_private_key.host.public_key_openssh}"
  filename        = "gen/known_hosts"
  file_permission = "644"
}

Figure 1002. ... and ssh wrapper Slide presentation
main.tf tpl/ssh.sh
resource "local_file" "ssh_script" {
  content = templatefile("tpl/ssh.sh", {
    ip=hcloud_server.hello.ipv4_address
  })
  filename        = "bin/ssh"
  file_permission = "700"
  depends_on      = [local_file.known_hosts]
}
#!/usr/bin/env bash

GEN_DIR=$(dirname "$0")/../gen

ssh -o UserKnownHostsFile= \
  "$GEN_DIR/known_hosts" devops@${ip} "$@"

exercise No. 5

Solving ~/.ssh/known_hosts quirk

Q:

Extend Working on Cloud-init generating both a bin/ssh wrapper and a corresponding gen/known_hosts file.