<?xml version="1.0" encoding="UTF-8"?> <chapter annotations="slide" version="5.1" xml:id="sdi_cloudProvider" xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xila="http://www.w3.org/2001/XInclude/local-attributes" xmlns:xi="http://www.w3.org/2001/XInclude" xmlns:trans="http://docbook.org/ns/transclusion" xmlns:svg="http://www.w3.org/2000/svg" xmlns:m="http://www.w3.org/1998/Math/MathML" xmlns:html="http://www.w3.org/1999/xhtml" xmlns:db="http://docbook.org/ns/docbook"> <title>Cloud provider</title> <section xml:id="sdi_cloudProvider_webAdminGui"> <title><orgname xlink:href="https://hetzner.com">Hetzner</orgname> cloud administration GUI</title> <figure xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair"> <title>Preliminary: Create an <command>ssh</command> key pair</title> <screen>sdiuser@martin-pc-dachboden:~$ <command xlink:href="https://linux.die.net/man/1/ssh-keygen">ssh-keygen</command> -t ed25519 <co linkends="sdi_cloudProvider_webAdminGui_createSshKeyPair-1" xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair-1-co"/> Generating public/private ed25519 key pair. Enter file in which to save the key (/home/sdiuser/.ssh/id_ed25519): Created directory '/home/sdiuser/.ssh'. Enter passphrase (empty for no passphrase): <co linkends="sdi_cloudProvider_webAdminGui_createSshKeyPair-2" xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair-2-co"/> Enter same passphrase again: Your identification has been saved in /home/sdiuser/.ssh/id_ed25519 <co linkends="sdi_cloudProvider_webAdminGui_createSshKeyPair-3" xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair-3-co"/> Your public key has been saved in /home/sdiuser/.ssh/id_ed25519.pub <co linkends="sdi_cloudProvider_webAdminGui_createSshKeyPair-4" xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair-4-co"/></screen> <calloutlist role="slideExclude"> <callout arearefs="sdi_cloudProvider_webAdminGui_createSshKeyPair-1-co" xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair-1"> <para>Create an elliptic rather than default <abbrev>RSA</abbrev> type key.</para> </callout> <callout arearefs="sdi_cloudProvider_webAdminGui_createSshKeyPair-2-co" xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair-2"> <para>Security aware folks will choose a decent passphrase protecting the private key being generated.</para> </callout> <callout arearefs="sdi_cloudProvider_webAdminGui_createSshKeyPair-3-co" xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair-3"> <para>The generated private key.</para> </callout> <callout arearefs="sdi_cloudProvider_webAdminGui_createSshKeyPair-4-co" xml:id="sdi_cloudProvider_webAdminGui_createSshKeyPair-4"> <para>The generated public key.</para> <note> <para>Different implementations like e.g. <command xlink:href="https://www.putty.org">putty</command> may use different key storage formats.</para> </note> </callout> </calloutlist> </figure> <figure xml:id="sdi_cloudProvider_webAdminGui_hetznerSignUp"> <title>Create a <orgname>Hetzner</orgname> account</title> <itemizedlist> <listitem> <para>Sign up at <link xlink:href="https://accounts.hetzner.com/signUp">https://accounts.hetzner.com/signUp</link> using an account name of your choice.</para> </listitem> <listitem> <para>Optionally: Activate 2-factor authentication.</para> </listitem> <listitem> <para>You may validate your account by ID card or similar. No payment required!</para> </listitem> <listitem> <para>Publish your <orgname>Hetzner</orgname> account's username to your SDI course's group at <link xlink:href="https://learn.mi.hdm-stuttgart.de">https://learn.mi.hdm-stuttgart.de</link>.</para> </listitem> </itemizedlist> </figure> <figure xml:id="sdi_cloudProvider_webAdminGui_accessProject"> <title>Access your project space</title> <para>Upon confirmation your <orgname>Hetzner</orgname> project space sdi_gxy (e.g. sdi_g01 corresponding to group 1) should be accessible.</para> </figure> <figure xml:id="sdi_cloudProvider_webAdminGui_"> <title>Create a server</title> <informaltable border="0"> <tr> <td valign="top"><orderedlist> <listitem> <para>Create a default firewall allowing <command xlink:href="https://linux.die.net/man/8/ping">ping</command> and <command xlink:href="https://linux.die.net/man/1/ssh">ssh</command></para> </listitem> <listitem> <para><productname>Ubuntu</productname> latest</para> </listitem> <listitem> <para>Shared vCPU / x86 / CX11 (<link xlink:href="https://www.hetzner.com/cloud/#pricing">the cheapest</link>)</para> </listitem> <listitem> <para>Add your personal <command>ssh</command> public key from <xref linkend="sdi_cloudProvider_webAdminGui_createSshKeyPair"/></para> </listitem> </orderedlist></td> <td valign="top"><orderedlist continuation="continues"> <listitem> <para>Omit volume, labels and cloud config</para> </listitem> <listitem> <para>Note the <guimenuitem>Networking</guimenuitem> / <guisubmenu>Public IPv4</guisubmenu> address for later reference</para> </listitem> <listitem> <para>Click »Create & Buy now«</para> </listitem> </orderedlist></td> </tr> </informaltable> </figure> <figure xml:id="sdi_cloudProvider_webAdminGui_accessServer"> <title>Access your server</title> <itemizedlist> <listitem> <para>Ping your server:</para> <note> <para>The IP 91.107.232.156 serves just as a sample value irrespective of your individual actual server IP.</para> </note> <screen>sdiuser:~$ ping 91.107.232.156 PING 91.107.232.156 (91.107.232.156) 56(84) bytes of data. 64 bytes from 91.107.232.156: icmp_seq=1 ttl=49 time=18.3 ms 64 bytes from 91.107.232.156 ...</screen> </listitem> <listitem> <para>Login via <command>ssh</command>:</para> <screen>ssh root@91.107.232.156</screen> </listitem> </itemizedlist> </figure> <figure xml:id="sdi_cloudProvider_webAdminGui_updateServer"> <title>Update and reboot</title> <orderedlist> <listitem> <para>apt update</para> </listitem> <listitem> <para>apt upgrade</para> </listitem> <listitem> <para>reboot</para> </listitem> </orderedlist> </figure> <figure xml:id="sdi_cloudProvider_webAdminGui_installNginx"> <title>Install a web server</title> <screen>root@topsy:~# apt install nginx</screen> </figure> <figure xml:id="sdi_cloudProvider_webAdminGui_localHttpAccess"> <title>Check local <acronym>http</acronym> web access</title> <screen>root@topsy:~# wget -O - 91.107.232.156 --2024-04-07 18:59:13-- http://91.107.232.156/ Connecting to 91.107.232.156:80... connected. <html> <head> <title>Welcome to nginx!</title> ...</screen> </figure> <figure xml:id="sdi_cloudProvider_cloudAdminGui_externHttp"> <title>External <acronym>http</acronym> web access</title> <para>Point your browser to http://91.107.232.156.</para> <screen>sdiuser:~$ telnet 91.107.232.156 80 Trying 91.107.232.156...</screen> <para>Why is there no answer?</para> </figure> <figure xml:id="sdi_cloudProvider_cloudAdminGui_allowHttp"> <title>Add port 80 / <acronym>http</acronym> firewall rule</title> <screen>sdiuser:~$ telnet 91.107.232.156 80 Trying 91.107.232.156... Connected to 91.107.232.156. Escape character is '^]'</screen> <para>Congrats: External Browser access is working now!</para> </figure> <figure xml:id="sdi_cloudProvider_cloudAdminGui_cleanUp"> <title>Cleaning up!</title> <caution> <para>This is about <emphasis role="red">$$$ MONEY $$$</emphasis></para> </caution> <itemizedlist> <listitem> <para>Delete your server including the IPv4 address.</para> </listitem> <listitem> <para>You may delete your firewall</para> </listitem> </itemizedlist> </figure> </section> <section xml:id="sdi_cloudProvider_terra"> <title>Working with <productname xlink:href="https://www.terraform.io">Terraform</productname></title> <figure xml:id="sdi_cloudProvider_terra_pledge"> <title>What's it all about?</title> <para><link xlink:href="https://developer.hashicorp.com/terraform/intro">Quote:</link></para> <para><quote><productname>Terraform</productname> is an infrastructure as code tool that lets you build, change, and version cloud and <abbrev>on-prem</abbrev> resources safely and efficiently.</quote></para> </figure> <figure xml:id="sdi_cloudProvider_terra_installSoftware"> <title><productname>Terraform</productname> resources</title> <itemizedlist> <listitem> <para><link xlink:href="https://developer.hashicorp.com/terraform/intro#why-terraform">Why <productname>Terraform</productname>?</link></para> </listitem> <listitem> <para><link xlink:href="https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli">Install <productname>Terraform</productname></link>.</para> </listitem> </itemizedlist> </figure> <figure xml:id="sdi_cloudProvider_terra_HetznerApiToken"> <title><orgname>Hetzner</orgname> API token</title> <itemizedlist> <listitem> <para>Access you cloud project using the <orgname>Hetzner</orgname> <xref linkend="glo_GUI"/> interface.</para> </listitem> <listitem> <para>Go to <option>Security</option> --> <option>API Tokens</option> --> <option>Generate API token</option></para> </listitem> <listitem> <para>Provide a name and hit Generate API token.</para> </listitem> <listitem> <para>Copy the generated token's value and store it in a secure location e.g. a password manager.</para> <caution> <para>The <orgname>Hetzner</orgname> <xref linkend="glo_GUI"/> blocks future access to the token.</para> </caution> </listitem> </itemizedlist> </figure> <figure xml:id="sdi_cloudProvider_terra_minimalConfig"> <title>Minimal <productname>Terraform</productname> configuration</title> <programlisting language="terraform"># Define Hetzner cloud provider terraform { required_providers { hcloud = { source = "hetznercloud/hcloud" } } required_version = ">= 0.13" } # Configure the Hetzner Cloud API token provider "hcloud" { token = "your_api_token_goes_here" } # Create a server resource "hcloud_server" "helloServer" { name = "hello" image = "debian-12" server_type = "cx11" location = "nbg1" }</programlisting> </figure> <figure xml:id="sdi_cloudProvider_terra_init"> <title><productname>Terraform</productname> <command xlink:href="https://developer.hashicorp.com/terraform/cli/commands/init">init</command></title> <programlisting language="terraform">$ terraform <command xlink:href="https://developer.hashicorp.com/terraform/cli/commands/init">init</command> Initializing the backend... Initializing provider plugins... - Finding latest version of hetznercloud/hcloud... - Installing hetznercloud/hcloud v1.46.1... - Installed hetznercloud/hcloud v1.46.1 (signed by a HashiCorp partner, key ID 5219EACB3A77198B) ... Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. ...</programlisting> </figure> <figure xml:id="sdi_cloudProvider_terra_plan"> <title><productname>Terraform</productname> <command xlink:href="https://developer.hashicorp.com/terraform/cli/commands/plan">plan</command></title> <programlisting language="terraform">$ terraform <command xlink:href="https://developer.hashicorp.com/terraform/cli/commands/plan">plan</command> Terraform used the selected providers to generate the following execution plan. Resource actions ... + create Terraform will perform the following actions: # hcloud_server.helloServer will be created + resource "hcloud_server" "helloServer" { + allow_deprecated_images = false + backup_window = (known after apply) ... } Plan: 1 to add, 0 to change, 0 to destroy.</programlisting> </figure> <figure xml:id="sdi_cloudProvider_terra_apply"> <title><productname>Terraform</productname> <command xlink:href="https://developer.hashicorp.com/terraform/cli/commands/apply">apply</command></title> <programlisting language="terraform">$ terraform <command xlink:href="https://developer.hashicorp.com/terraform/cli/commands/apply">apply</command> ... Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: <emphasis role="red">yes</emphasis> hcloud_server.helloServer: Creating... hcloud_server.helloServer: Still creating... [10s elapsed] hcloud_server.helloServer: Creation complete after 14s [id=45822789] Apply complete! Resources: 1 added, 0 changed, 0 destroyed.</programlisting> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_email"> <title><productname>Credentials by E-Mail</productname></title> <screen>Your server "hello" was created! You can access your server with the following credentials: IPv4 <emphasis role="red">128.140.108.60</emphasis> IPv6 2a01:4f8:1c1c:8e3a::/64 User root Password rJ3pNvJXbqMp3XNTvFdq You will be prompted to change your password on your first login. To improve security, we recommend that you add an SSH key when creating a server.</screen> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_problems"> <title><productname>Problems</productname>: 😟</title> <itemizedlist> <listitem> <para>Firewall blocks <command xlink:href="https://linux.die.net/man/1/ssh">ssh</command> server access:</para> <screen>$ ssh root@<emphasis role="red">128.140.108.60</emphasis> ssh: connect to host 128.140.108.60 port 22: Connection refused</screen> <para>Access by <xref linkend="glo_Vnc"/> <link xlink:href="https://docs.hetzner.com/cloud/servers/getting-started/vnc-console">console login</link> only</para> </listitem> <listitem> <para>IP and (initial) credentials by email 😱</para> </listitem> </itemizedlist> <para>Solution:</para> <orderedlist> <listitem> <para>Add firewall inbound <command xlink:href="https://linux.die.net/man/1/ssh">ssh</command> access rule.</para> </listitem> <listitem> <para>Configure <command xlink:href="https://linux.die.net/man/1/ssh">ssh</command> public key login.</para> </listitem> </orderedlist> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_sshAccessFw"> <title><command xlink:href="https://linux.die.net/man/1/ssh">ssh</command> access, firewall</title> <programlisting language="terraform">resource "hcloud_firewall" "<emphasis role="red">sshFw</emphasis>" { name = "ssh-firewall" rule { direction = "in" protocol = "tcp" port = "22" source_ips = ["0.0.0.0/0", "::/0"] } } ... resource "hcloud_server" "helloServer" { ... firewall_ids = [hcloud_firewall.<emphasis role="red">sshFw</emphasis>.id] }</programlisting> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_sshAccessKey"> <title><command xlink:href="https://linux.die.net/man/1/ssh">ssh</command> access, public key</title> <programlisting language="terraform">resource "hcloud_ssh_key" "<emphasis role="red">goik</emphasis>" { name = "goik@hdm-stuttgart.de" public_key = file("~/.ssh/id_ed25519.pub") } ... resource "hcloud_server" "helloServer" { ... ssh_keys = [hcloud_ssh_key.<emphasis role="red">goik</emphasis>.id] }</programlisting> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_sshApply"> <title>Apply <command xlink:href="https://linux.die.net/man/1/ssh">ssh</command> key access</title> <screen>$ terraform apply # hcloud_firewall.sshFw will be created + resource "hcloud_firewall" "sshFw" { ... # hcloud_server.helloServer will be created + resource "hcloud_server" "helloServer" { ... # hcloud_ssh_key.goik will be created + resource "hcloud_ssh_key" "goik" { ... Plan: 3 to add, 0 to change, 0 to destroy. ... Apply complete! Resources: 3 added, 0 changed, 0 destroyed.</screen> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_sshApplyOutputValuesDetails1"> <title>Output data details #1/2</title> <para>See <link xlink:href="https://developer.hashicorp.com/terraform/language/values/outputs"> terraform output documentation</link>:</para> <informaltable border="1"> <tr> <th>File <filename>outputs.tf</filename></th> <th>Result</th> </tr> <tr> <td rowspan="2" valign="top"><programlisting language="terraform">output "hello_ip_addr" { value = hcloud_server.helloServer.ipv4_address description = "The server's IPv4 address" } output "hello_datacenter" { value = hcloud_server.helloServer.datacenter description = "The server's datacenter" }</programlisting></td> <td valign="top"><screen language="properties">$ <command>terraform</command> output hello_datacenter = "nbg1-dc3" hello_ip_addr = "159.69.152.37"</screen></td> </tr> <tr> <td valign="top"><screen language="properties">$ <command>terraform</command> output hello_ip_addr "159.69.152.37"</screen></td> </tr> </informaltable> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_sshApplyOutputValuesDetails"> <title>Output data details #2/2</title> <informaltable border="1"> <tr> <th>File <filename>outputs.tf</filename></th> <th><command>terraform</command> output <option>-json</option></th> </tr> <tr> <td valign="top"><programlisting language="terraform">output "hello_ip_addr" { value = hcloud_server.helloServer.ipv4_address description = "The server's IPv4 address" } output "hello_datacenter" { value = hcloud_server.helloServer.datacenter description = "The server's datacenter" }</programlisting></td> <td valign="top"><programlisting language="json">{ "hello_datacenter": { "sensitive": false, "type": "string", "value": "nbg1-dc3" }, "hello_ip_addr": { "sensitive": false, "type": "string", "value": "159.69.152.37" } }</programlisting></td> </tr> </informaltable> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_sshProblemApiToken"> <title><productname>Problem 2</productname>: <xref linkend="glo_VCS"/> and visible provider API token 😱</title> <para><emphasis role="red">Versioned</emphasis> file <filename>main.tf</filename>:</para> <programlisting language="terraform">... provider "hcloud" { token = "<emphasis role="red">xdaGfz9LmwO8SWkg ... </emphasis>"} ...</programlisting> <para>Solution:</para> <itemizedlist> <listitem> <para>Declare a <link xlink:href="https://developer.hashicorp.com/terraform/language/values/variables">variable</link> <varname>hcloud_token</varname> in a <filename>variables.tf</filename> file</para> </listitem> <listitem> <para>Add a non-versioned file <filename>secrets.auto.tfvars</filename>.</para> </listitem> <listitem> <para>Optional: Provide a versioned <filename>secrets.auto.tfvars.template</filename> documenting file</para> </listitem> </itemizedlist> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_sshProblemApiTokenSolve"> <title>Solution</title> <informaltable border="0"> <tr> <td valign="top"><para>Declaring <emphasis role="red"><varname>hcloud_token</varname></emphasis> in <filename>variables.tf</filename>:</para><programlisting language="terraform">variable "<emphasis role="red">hcloud_token</emphasis>" { # See secret.auto.tfvars nullable = false sensitive = true }</programlisting></td> <td valign="top"><para>Defining <emphasis role="red"><varname>hcloud_token</varname></emphasis>'s value in <filename>secrets.auto.tfvars</filename>:</para><programlisting language="terraform"><emphasis role="red">hcloud_token</emphasis>="xdaGfz9LmwO8SWkg ... "</programlisting></td> </tr> <tr> <td valign="top"><para>Using <emphasis role="red"><varname>hcloud_token</varname></emphasis> in <filename>main.tf</filename>:</para><programlisting language="terraform">provider "hcloud" { token = var.<emphasis role="red">hcloud_token</emphasis> }</programlisting></td> <td valign="top"><para>Example in <filename>secrets.auto.tfvars.template:</filename></para><programlisting language="terraform"><emphasis role="red">hcloud_token</emphasis>="your_api_token_goes_here"</programlisting></td> </tr> </informaltable> </figure> <figure xml:id="sdi_cloudProvider_terra_hello_kownHostsDuplicateProblem"> <title>Duplicate known_hosts entry on re-creating server</title> <screen>$ ssh root@128.140.108.60 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (<emphasis role="red">man-in-the-middle attack</emphasis>)!</screen> </figure> </section> <section xml:id="sdi_cloudProvider_cloudInit"> <title>Cloud Init</title> <figure xml:id="sdi_cloudProvider_cloudInit_cloudStackTalk"> <title>Introduction and reference</title> <itemizedlist> <listitem> <para>Cloud Stack <link xlink:href="https://mirror.mi.hdm-stuttgart.de/Videos/Cloud/cloud-init_CloudStackCollaborationConference2022.mp4">Conference talk</link>.</para> </listitem> <listitem> <para><link xlink:href="https://cloudinit.readthedocs.io/en/latest">Cloud-init documentation</link></para> </listitem> </itemizedlist> </figure> <figure xml:id="sdi_cloudProvider_cloudInit_nutshell"> <title>In a nutshell</title> <itemizedlist> <listitem> <para>Distribution image containing pre-installed <productname>Cloud Init</productname></para> </listitem> <listitem> <para>Script configurable installation options</para> </listitem> </itemizedlist> </figure> <figure xml:id="sdi_cloudProvider_cloudInit_configOverview"> <title>Configuration options</title> <informaltable border="0"> <tr> <td valign="top"><itemizedlist> <listitem> <para>Individual CRUD file operations</para> </listitem> <listitem> <para>Supplying <productname>ssh</productname> user and host keys.</para> </listitem> <listitem> <para>Adding users</para> </listitem> <listitem> <para>...</para> </listitem> </itemizedlist></td> <td valign="top"><itemizedlist> <listitem> <para>Installing packages</para> </listitem> <listitem> <para>System Upgrade + reboot</para> </listitem> <listitem> <para>Arbitrary command execution</para> </listitem> </itemizedlist></td> </tr> </informaltable> </figure> <figure xml:id="sdi_cloudProvider_cloudInit_terraformInterfaceCloudinit"> <title><productname>Terraform</productname> interface to <productname>Cloud Init</productname></title> <programlisting language="terraform">resource "hcloud_server" "web" { name = var.server_name ... user_data = file("Server/web/web.yml") }</programlisting> </figure> </section> </chapter>