source: main/waeup-ansible/README.rst @ 17133

=============================================
 Playbooks for administrating WAeUP servers.
=============================================

These are materials to use with our servers.

For starters: the tutorial given on

  https://github.com/leucos/ansible-tuto

is a really nice hands-on intro to `ansible`. Please read it!

If you want to devel/test scripts in here, try to work with virtual machines
first. The ``Vagrant`` section below explains the details.

Server Lifecircle
=================

When we get a server freshly installed from Hetzner, we want to make sure, at
least some common security holes are closed.


Right after first install: `bootstrap.yml`
------------------------------------------

For starters we "bootstrap" a server install with the ``bootstrap.yml``
playbook. This playbook does three things:

- It secures the ``SSHD`` config according to infos from
  https://bettercrypto.org
- It adds accounts for admin users (including sudo rights)
- It disables root login via SSH.

Before the playbook can be run, you have to fix some things.

1) Make sure you can ssh into the systems as ``root``.

2) Make sure, Python2.x is installed on the target systems. This is not the
   case anymore for instance for minimal Ubuntu images starting with 16.04 LTS.

   If Python2.x is not installed, do::

     # apt-get update
     # apt-get install python python-simplejson

   as `root` on each targeted system.

3) For each server to handle, make an entry in the ``[yet_untouched]`` section
   of the ``hosts`` file like this::

     # hosts
     [yet_untouched]
     h23.waeup.org ansible_user=root ansible_ssh_pass=so-secret ansible_sudo_pass="{{ ansible_ssh_pass }}"
     h24.waeup.org ansible_user=root ansible_ssh_pass=123456789 ansible_sudo_pass="{{ ansible_ssh_pass }}"

   The ``ansible_sudo_pass`` is not neccessary for now, but will be needed if
   you want to run everything as a normal user. And it is just a blank copy of
   ``ansible_ssh_pass``.

   Yes, this is a very dangerous part and you should not check this
   modifications in. Instead you should remove the entries after you are done.

4) Update the ``vars`` in ``bootstrap.yml``. Tell, whether SSH root access
   should stay enabled and say ``no`` or ``false``.

   Then, you have to create a dict of admin users. For each user we need a name
   (key) and a hashed password. This can be done like this::

     $ diceware -d '-' -n 6 --no-caps | tee mypw | mkpasswd -s --method=sha-512 >> mypw

   which will create a random password and its SHA512-hashed variant in a file
   called ``mypw``. If you do not have `diceware` installed, you can use
   `pwgen` (or any other password maker)::

     $ pwgen -s 33 | tee mypw | mkpasswd -s --method=sha-512 >> mypw

   The hashed variant then has to be entered as ``hashed_pw`` in the `vars` of
   ``bootstrap.yml``.

   In the end, there should be something like::

     # bootstrap.yml
     # ...
     vars:
       permit_ssh_root: false
       admin_users:
         user1:
           hashed_pw: "$6$Wsdfhwelkl32lslk32lkdslk43...."
         user2:
           hashed_pw: "$6$FDwlkjewlkWs2434SVRDE65DFF...."
     ...

   Please note, that all users listed in this dict will have the same passwords
   on all servers handled when running the script.

5) Finally, run the play::

     $ ansible-playbook -i hosts -C bootstrap.yml

   to see, whether setup is fine (dry run) and::

     $ ansible-playbook -i hosts bootstrap.yml

   to actually perform the changes.

6) In `hosts` move the host we handle from ``[yet_untouched]`` over to
   ``[bootstapped]``.


Setup
=====

After bootstrapping, there should be a user account we can use.

1) Create a local SSH key to connect to the new server and copy it over::

     $ ssh-keygen -t ed25519 -C "uli@foo to myremote" -f ~/.ssh/id_myremote

   Where ``myremote`` is normally one of h1, h2, ...., hN. Then::

     $ ssh-copy-id -i ~/.ssh/id_myremote user@myremote.waeup.org

   and eventually edit ``~/.ssh/config`` to register your new key.
   If you are out for adventure, do not create a new key but use the one you
   use on all other machines as well. This is, of course, not recommended.

2) Update the entry of the handled host in the local `hosts` inventory:
   - Remove ``ansible_user=root``
   - Remove ``ansible_ssh_pass``.
   - Set ``ansible_sudo_pass`` to the password of the user you connect as.

3) Update the server::

     $ ansible -i hosts hmyremote.waeup.org -b -m apt -a "upgrade=safe update_cache=yes"

   This way we can ensure that your SSH setup works correctly.

4) Run setup.py::

     $ ansible-playbook -i hosts -l hmyremote.waeup.org -C setup.yml

   (for a dry run) and::

     $ ansible-playbook -i hosts -l hmyremote.waeup.org setup.yml

   for the real run.


Vagrant
=======

In `Vagrantfile` we set up a vagrant environment which provides three
hosts as virtualbox:

  ``vh5.sample.org``, ``vh6.sample.org``, ``vh7.sample.org``

running Ubuntu 14.04. ``vh5`` represents "virtual host 5" and should
reflect h5.waeup.org. The same holds for ``vh6`` and ``vh7``
accordingly.

The three virtual hosts are for testing any upcoming ansible
playbooks. They should be used before running playbooks on the real
hosts!


Initialize Vagrant Env
----------------------

You must have `vagrant` installed, if possible in a fairly recent
version. I (uli) use `vagrant 1.8.1` (latest as time of writing).  As
Ubuntu 14.04 is pretty outdated in that respect, I had to grab a .deb
package from

  https://www.vagrantup.com/downloads.html

that could be installed with::

 $ sudo dpkg -i vagrant_1.8.1_x86_64.deb


When everything is in place, change into this directory and run::

  $ vagrant up
  Bringing machine 'vh5' up with 'virtualbox' provider...
  Bringing machine 'vh6' up with 'virtualbox' provider...
  Bringing machine 'vh7' up with 'virtualbox' provider...
  ==> vh5: Importing base box 'ubuntu/trusty32'...
  ...

This will fetch Vagrant virtualbox images for trusty32, i.e. Ubuntu
14.04 images, 32bit version (plays nice also on 64bit hosts).

When hosts are being supplied by Hetzner or another hosting provider,
then we normally get access as `root` user only. Therefore, After base
init the root accounts of all hosts are enabled with password
``vagrant``. This is done by the ansible playbook in
``vagrant-provision.yml``.

All three hosts provide ssh access via::

  $ vagrant ssh vh0

or equivalent commands. They have a user 'vagrant' installed, which
can sudo without password.

After install all three hosts can also be accessed as `root` using
password `vagrant` (for example vh5):

  $ ssh -l root 192.168.36.10

See ``Vagrantfile`` for the IP addresses set.

You can halt (all) the virtual hosts with::

  $ vagrant halt



Ansible Environment
===================

The ansible environment should provide ansible roles and playbooks for
WAeUP related server administration.

The general file-layout and naming should follow

  https://docs.ansible.com/ansible/playbooks_best_practices.html#directory-layout


Bootstrapping - Freshmechs
--------------------------

We call those machines "freshmech" that are freshly delivered from the
hosting provider or that were freshly provisioned by `vagrant` (see
above).

These machines are expected to have only a single root account and
normally a (security-wise) poor SSH configuration.

Bootstrapping these machines means we secure SSH, restart the SSH
daemon and then add important accounts: "uli", "henrik", "ansible".

To make sure, the connection to a "freshmech" works, you should at
least one time login via SSH before proceeding with ansible and all
bells and whistles::

  ssh -l root 192.168.36.10

(with the real IP of the machine you want to reach, of course).

Any host you want to "bootstrap" must be entered in a local hosts
file, normally ``hosts-virtual``, with a line like this:

  [yet_untouched]
  vh5.sample.org ansible_host=192.168.36.10 ansible_user=root

in the "yet_untouched" section.

Afterwards try:

  $ ansible-playbook -i hosts-virtual --ask-pass bootstrap.yml

The ``ask-pass`` parameter is needed to enter the password given by
the provider on the commandline. For the local `vagrant` machines this
will be `vagrant`.

If run on local virtual machines, you might want to make sure that
your local `known_hosts` file does not contain an old ssh host
fingerprint. Otherwise you have to remove entries for::

  192.168.36.10
  192.168.36.11
  192.168.36.12

respectively before running `bootstrap.yml`.

Alternatively you can run everything with the
`ANSIBLE_HOST_KEY_CHECKING` environment variable set to ``False``::

  $ ANSIBLE_HOST_KEY_CHECKING=False  ansible-playbook -i hosts-virtual --ask-pass bootstrap.yml

This will suppress host fingerprint checking.