Kubernetes in TripleO with multiple masters and multiple nodes

TripleO already deploys OpenStack into containers. Going forward we would like to integrate a container orchestration engine. We have been experimenting with Kubernetes and OpenShift. Let’s look at deploying vanilla Kubernetes in this blog post. We’ll deploy a Kubernetes cluster of 3 masters + 3 nodes, utilizing the TripleO composable services framework and the Kubespray installer, and drive the deployment using TripleO Quickstart.

Architecture

First let’s skim through the architecture enhancement made to support external installers within the new Ansible deployment mechanism.

We now allow the composable service templates to emit external_deploy_tasks. These are Ansible tasks executed on the undercloud. They can read the full Ansible inventory of the deployment playbook when needed, which makes them fit for running complex out-of-tree (meaning out-of-tripleo-heat-templates) installers like Kubespray or Ceph-Ansible.

We had two options how to approach these external installers – either pull them in via a playbook include directly to the ansible-playbook process for overcloud deployment, or run a nested ansible-playbook subprocess (e.g. for Kubespray) from the outer ansible-playbook process. We went for the latter as it provides more control and can execute also non-Ansible installers. The summary of important differences between the approaches is in the following table:

The external_deploy_tasks are step-based, and are interleaved between the “normal” deployment steps which currently run Puppet and manage containers. This way we can decide in which particular step we want an external installer to be run. The situation is depicted (in a simplified way) in the following diagram:

Implementation for Kubespray

In Kubespray’s case specifically, we use the external_deploy_tasks to generate files needed by the Kubespray installer and then execute the installer. The files used by Kubespray that we generate are:

• an inventory,

• a simple playbook (including Kubespray’s cluster.yml),

• a file with Ansible variables that configures Kubespray.

If you want to explore the code, see the service template kubernetes-master.yaml as of 20th November 2017. There’s also kubernetes-worker.yaml service template, which is just used for tagging nodes to be recognized by the inventory generator as workers, and for setting up worker node firewall rules.

Deployment

Prepare the environment

The assumed starting point for deploying Kubernetes will be having a deployed undercloud with 6 virtual baremetal nodes defined and ready for use.

There are several paths to do this with TripleO Quickstart, the easiest one is probably to deploy full 3 controller + 3 compute environment using --nodes config/nodes/3ctlr_3comp.yml, and then delete the overcloud stack. If your virt host doesn’t have enough capacity for that many VMs, you can use a smaller configuration, e.g. 1ctlr_1comp.yml or just 1ctlr.yml.

For detailed information how to deploy with Quickstart, please refer to TripleO Quickstart docs.

Deploy the overcloud

Let’s prepare extra-oooq-vars.yml file. It’s a file with Quickstart variables, so it will have to be on the host where you run Quickstart. The contents will be as follows:

# use t-h-t with our cherry-picks
overcloud_templates_path: /home/stack/tripleo-heat-templates

# specify NTP server this way. Improved in https://review.openstack.org/516958
extra_tht_config_args: --ntp-server pool.ntp.org

# make validation errors non-fatal
validation_args: ''

# network config in the featureset is for CI, override it back to defaults
network_args: -e /home/stack/net-config-defaults.yaml

# deploy with config-download mechanism, we'll execute the actual
# software deployment via ansible subsequently
config_download_args: >-
  -e /home/stack/tripleo-heat-templates/environments/config-download-environment.yaml
  --disable-validations
  --verbose

# do not run the workflow
deploy_steps_ansible_workflow: false

And /home/stack/net-config-defaults.yaml will have to be on the undercloud, and it has these contents:

resource_registry:
  OS::TripleO::Controller::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-bridge.yaml
  OS::TripleO::Compute::Net::SoftwareConfig: /usr/share/openstack-tripleo-heat-templates/net-config-noop.yaml

Now let’s reuse the undercloud deployed previously by Quickstart, and deploy the overcloud Heat stack. This could be done with quickstart.sh too, but personally i prefer running ansible-playbook for more direct control:

# run this where you run Quickstart (likely not the undercloud)

# VIRTHOST must point to the machine that hosts your Quickstart VMs,
# edit this if necessary
export VIRTHOST=$(hostname -f)
# WORKSPACE must point to your Quickstart workspace directory,
# edit this if necessary
export WORKSPACE=$HOME/.quickstart
source $WORKSPACE/bin/activate
export ANSIBLE_ROLES_PATH=$WORKSPACE/usr/local/share/ansible/roles:$WORKSPACE/usr/local/share/tripleo-quickstart/roles
export ANSIBLE_LIBRARY=$WORKSPACE/usr/local/share/ansible:$WORKSPACE/usr/local/share/tripleo-quickstart/library
export SSH_CONFIG=$WORKSPACE/ssh.config.ansible
export ANSIBLE_SSH_ARGS="-F ${SSH_CONFIG}"

ansible-playbook -v \
    -i $WORKSPACE/hosts \
    -e local_working_dir=$WORKSPACE \
    -e virthost=$VIRTHOST \
    -e @$WORKSPACE/config/release/tripleo-ci/master.yml \
    -e @$WORKSPACE/config/nodes/3ctlr_3comp.yml \
    -e @$WORKSPACE/config/general_config/featureset026.yml \
    -e @extra-oooq-vars.yml \
    $WORKSPACE/playbooks/quickstart-extras-overcloud.yml

With the overcloud Heat stack created, we can now fetch the overcloud software config definition and deploy it with Ansible. In the future this will be neatly hidden behind CLI and WUI interface, but for now it’s another WIP feature, so we’ll take a bit more explicit approach:

# clean any previous config downloads
rm -rf ~/config-download/tripleo*

# produce Ansible playbooks from Heat stack outputs
tripleo-config-download -s overcloud -o ~/config-download

# skip this in case you want to manually check fingerprints
export ANSIBLE_HOST_KEY_CHECKING=no

# deploy the software configuration of overcloud
ansible-playbook \
    -v \
    -i /usr/bin/tripleo-ansible-inventory \
    ~/config-download/tripleo-*/deploy_steps_playbook.yaml

This applied the software configuration, including installation of Kubernetes via Kubespray.

Hello Kubernetes in TripleO

For now it’s best to ssh to an overcloud controller node to manage Kubernetes with kubectl. External HA access to the Kubernetes API is being worked on.

After smoke testing with e.g. kubectl get nodes, you can try deploying something on the Kubernetes cluster, e.g. Use a Service to Access an Application in a Cluster tutorial from the Kubernetes docs is a nice one.