Tag Archives: Kibana

Using CALM Blueprints – Automation is the new punk!


I can imagine there are a lot of people like me who are continually setting up and tearing down environments in order to run application benchmarks, test out APIs or run various new features etc. The consequence of this is that I have umpteen sources of best practice notes for each and every technology stack I get involved with. What’s worse is that some of the configurations and their changes are identical across multiple applications. So often, I am digging around in directories entitled somewhat unhelpfully like “Notes” and “Best_Practices” or …. wait for it …..”Tunings”.

As part of an ongoing move towards Infrastructure as Code, I am on a mission now to get all of the crufty bits of info I keep here, there and everywhere, into a source code repository format. To that end I have been looking at using the Multi-VM blueprint functionality of Nutanix Calm (Automated Lifecycle Management). Calm allows me to create a blueprint and reuse all my original code snippets and config edits. They can be in Bash, Python or Powershell and so on. Once created, the blueprint can be stored in a repository on Github, for example. Then everytime I use that blueprint I get a repeatable deployment that is the same, each & every time I run it.

Here’s one I made earlier

Let’s take a look at building out a stack to benchmark Elasticsearch using esrally. I covered some of this in a my last post. I want to start off by discussing a few prerequisites that will be needed. First and foremost – the image used to create the virtual machines (VMs). I used CentOS 7 cloud images which will require ssh key based access for the default user (centos). This means I need to store both public and private keys in the various parts of the configuration. See below for the Configuration > DOWNLOADABLE IMAGE CONFIGURATION and Credentials sections in the blueprint

The blueprint automatically creates three virtual machines (VMs), one to host a single Elasticsearch instance, one for the Kibana instance and another that will run the esrally workload generator. See below for the basic layout of the blueprint. As the Kibana instance needs to know the address of the Elasticsearch instance, I need to create a dependency between the Elasticsearch and Kibana services. I do this by creating “an edge” between the services. This is delineated by the white line. That way the Kibana configuration/install only proceeds when the Elasticsearch configuration/install has completed. However, all underlying VMs are created simultaneously.

Services and dependencies

Each service requires a virtual machine in order to provide that service. So configure each VM with storage (vDISKS), network (NIC), ssh access (Credentials), along with any guest customisation and so on.  For the Search_Index (Elasticsearch) service, I built the Elasticsearch VMs to host six 200GB vdisks, and used the cloud-config already installed in the image to set access keys and permissions. See below…

Application Profiles and variables

The use of application profiles not only allows you to specify the platform (or substrate in Calm speak). You can also encapsulate variables which are then passed to that application. I am deploying to a Nutanix platform in this case. This works just as well however, with AWS, GCP and Azure. You can see from the application profile below the variables I have created. I could very quickly deploy several application stacks using this in a blueprint and each one could have a different java heap size. I could then make performance comparisons between the two. Each application stack would be exactly the same apart from the one changed variable. By extension I could add other variables I am interested in, like LVM stripe width or filesystem block size and so on.

Application Installation and Configuration

How variables in the application profiles get used, can be shown below in the package install task. The bulk of any configuration is done here. Tasks can be assigned to any action that are related to a service or the application profile. So a start, restart, stop or delete can have an associated task. For each service there’s a package install task and that’s where we use the application profile variables. Each of the services I configured have a package install task, below is the task for the Elasticsearch/Search_Index service 

The canvas (above) shows a number of ways to update or edit files based on various patterns. Note that all config file edits/updates are done in place. You should avoid using a CLI that relies on creating temporary files. Your package install script could end up trying to write/access files outside of the deployment environment. This is a potential security hole which Calm will not allow. Notice how the variable macros in the above package tasks are invoked below :

sudo sed -i 's/-Xms1g/-Xms@@{java_heap_size}@@g/' /etc/elasticsearch/jvm.options
sudo sed -i 's%path.data: /var/lib/elasticsearch%path.data: @@{elastic_data_path}@@%' /etc/elasticsearch/elasticsearch.yml

Calm internal macros are also available. For example: passing the address of one service into another – this is from the package task for the Data Visualisation service (kibana instance):

sudo sed -i 's%^#elasticsearch.hosts: \["http://localhost:9200"\]%elasticsearch.hosts: \["http://@@{Search_Index.address}@@:@@{elastic_http_port}@@"\]%' /etc/kibana/kibana.yml

or for cardinal numbers for unique VM names (see the VM configuration section of any service):


Provisioning and Auditing

That’s the the blueprint complete. It should be saved without errors or warnings. Now it’s time to launch the blueprint to build the application stack. At this point you can name what will be your running application instance and change/set any runtime variables. Once launched the blueprint is queued, verified and then cloned ready to run. While its running you can audit the steps of the workflow in the blueprint:


Once the application is marked RUNNING, you can then either connect to individual VMs, or access an application via a browser. It’s common for all means of VM or application access to be placed in the blueprint description (Note: it also expands macro variables – see below):

The following is an example of the /etc/motd when logging into the VM installed with esrally

# ssh -i ./keys.pem -l centos
Last login: Wed Jul 17 15:46:57 2019 from

Configuration successfully written to /home/centos/.rally/rally.ini. Happy benchmarking!

More info about Rally:

* Type esrally --help
* Read the documentation at https://esrally.readthedocs.io/en/1.2.1/
* Ask a question on the forum at https://discuss.elastic.co/c/elasticsearch/rally

To get started:
esrally list tracks


esrally --pipeline=benchmark-only --target-hosts= \
--track=eventdata --track-repository=eventdata --challenge=bulk-size-evaluation


The final version (for now) of the blueprint is available to clone or download at:


Upload the blueprint to the Calm service on Prism Central. Then work through it as you read this post. Make your own changes if required. At the end (~10 minutes) you will have a running environment with which to test various Elasticsearch workloads. I intend to work through more blueprints related to other cloud native applications, with a view to developing larger scale deployments. Stay tuned,


ELK on Nutanix : Kibana

It might seem like I am doing things out of sequence by looking at the visualisation layer of the ELK stack next. However, recall in my original post , that I wanted to build sets  of unreplicated indexes and then use Logstash to fire test workloads at them. Hence, I am covering Elasticsearch and Kibana initially. This brings me to another technical point that I need to cover. In order for a single set of indexes to be actually recoverable, when running on a single node, we need to invoke the following parameters in our Elasticsearch playbook :

So in file: roles/elastic/vars/main.yml
elasticsearch_gateway.recover_after_nodes: 1
elasticsearch_gateway.recover_after_time: 5m
elasticsearch_gateway.expected_nodes: 1

These are then set in the elasticsearch.yml.j2 file as follows:

# file: roles/elastic/templates/elasticsearch.yml.j2
#{{ ansible_managed }}


# Allow recovery process after N nodes in a cluster are up:
#gateway.recover_after_nodes: 2
{% if elasticsearch_gateway_recover_after_nodes is defined %}
gateway.recover_after_nodes : {{ elasticsearch_gateway_recover_after_nodes}}
{% endif %}

and so on ....

This allows the indexes to be recovered when there is only a single node in the cluster. See below for the state of my indexes after a reboot:

[root@elkhost01 elasticsearch]# curl -XGET http://localhost:9200/_cluster/health?pretty
 "cluster_name" : "nx-elastic",
 "status" : "yellow",
 "timed_out" : false,
 "number_of_nodes" : 1,
 "number_of_data_nodes" : 1,
 "active_primary_shards" : 4,
 "active_shards" : 4,
 "relocating_shards" : 0,
 "initializing_shards" : 0,
 "unassigned_shards" : 4,
 "delayed_unassigned_shards" : 0,
 "number_of_pending_tasks" : 0,
 "number_of_in_flight_fetch" : 0

Lets now look at the Kibana playbook I am attempting. Unfortunately, Kibana is distributed as a compressed tar archive. This means that the yum or dnf modules are no help here. There is however a very useful unarchive module, but first we need to download the tar bundle using get_url as follows :

- name: download kibana tar file
 get_url: url=https://download.elasticsearch.org/kibana/kibana/kibana-{{ kibana_version }}-linux-x64.tar.gz
 dest=/tmp/kibana-{{ kibana_version }}-linux-x64.tar.gz mode=755
 tags: kibana

I initially tried unarchiving the Kibana bundle into /tmp. I then intended to copy everything below the version specific directory (/tmp/kibana-4.0.1-linux-x64) into the Ansible created /opt/kibana directory. This proved problematic as neither the synchronize nor the copy modules seemed setup to do mass copy/transfer between one directory structure to another. Maybe I am just not getting it – I even tried using with_item loops but no joy as fileglobs are not recursive. Answers on a postcard are always appreciated? In the end I just did this :

- name: create kibana directory
 become: true
 file: owner=kibana group=kibana path=/opt/kibana state=directory
 tags: kibana

- name: extract kibana tar file
 become: true
 unarchive: src=/tmp/kibana-{{ kibana_version }}-linux-x64.tar.gz dest=/opt/kibana copy=no
 tags: kibana

The next thing to do was to create a systemd service unit. There isn’t one for Kibana as there is no rpm package available. Usual templating applies here :

- name: install kibana as systemd service
 become: true
 template: src=kibana4.service.j2 dest=/etc/systemd/system/kibana4.service owner=root \
           group=root mode=0644
 - restart kibana
 tags: kibana

And the service unit file looked like:

[ansible@ansible-host01 templates]$ cat kibana4.service.j2
{{ ansible_managed }}

ExecStart=/opt/kibana/kibana-{{ kibana_version }}-linux-x64/bin/kibana


This all seemed to work as I could now access Kibana via my browser. No indexes yet of course :


There are one or two plays I would like still like to document. Firstly, the ‘notify’ actions in some of the plays. These are used to call – in my case – the restart handlers. Which in turn causes the service in question to be restarted – see the next section :

# file: roles/kibana/handlers

- name: restart kibana
 become: true
 service: name=kibana state=restarted

I wanted to document this next feature simply because it’s so useful – tags. I have assigned a tag to every play/task in the playbook so far you will have noticed. For testing purposes they allow you to run specific plays. You can then troubleshoot just that particular play and see what’s going on.

 ansible-playbook -i ./production site.yml --tags "kibana" --ask-sudo-pass

Now that I have the basic plays to get my Elasticsearch and Kibana services up and running via Ansible, it’s time to start looking at Logstash. Next time I post on ELK type stuff, I will try to look at logging and search use cases. Once I crack how they work of course.

ELK on Nutanix : Elasticsearch

In this second post on using Ansible to deploy the ELK stack on Nutanix, I will cover my initial draft at a playbook for Elasticsearch (ES).  Recall from my previous post, the playbook layout looks like:

[ansible@ansible-host01 roles]$ tree elastic
├── files
│   └── elasticsearch.repo
├── handlers
│   └── main.yml
├── tasks
│   └── main.yml
├── templates
│   ├── elasticsearch.default.j2
│   ├── elasticsearch.in.sh.j2
│   └── elasticsearch.yml.j2
└── vars
 └── main.yml

There’s also an additional role at play here, config – which is the basic config for the underlying VM guest OS, which we also need to look at :

[ansible@ansible-host01 roles]$ tree config
├── files
├── handlers
├── tasks
│   └── main.yml
├── templates
└── vars
 └── main.yml

the common role is where I set things via the Ansible sysctl module, or add entries to files (using lineinfile) in order to set max memory and ulimits etc. It’s generic system configuration, so for example:

#installing java runtime pkgs (pre-req for ELK)
- name: install java 8 runtime
 become: true
 yum: name=java state=installed
 tags: config

#set system max/min numbers...
- name: set maximum map count in sysctl/systemd
 become: true
 sysctl: name=vm.max_map_count value={{ os_max_map_count }} state=present
 tags: config


- name: set soft limits for open files
 become: true
 lineinfile: dest=/etc/security/limits.conf line="{{ elasticsearch_user }} soft nofile {{ elasticsearch_max_open_files }}" insertafter=EOF backup=yes
 tags: config

- name: set max locked memory
 become: true
 lineinfile: dest=/etc/security/limits.conf line="{{ elasticsearch_user }} - memlock {{ elasticsearch_max_locked_memory }}" insertafter=EOF backup=yes
 tags: config


Here might be a good time to touch upon how Ansible allows you to set variables. Within the directory of each role there’s a subdir called vars and all the variables needed for that role are contained in the YAML file (main.yml). Here’s a snippet:

# can use vars to set versioning and user 
elasticsearch_version: 1.7.0
elasticsearch_user: elasticsearch

# here's how we can specify the data volumes that ES will use 
elasticsearch_data_dir: /esdata/data01,/esdata/data02,/esdata/data03,/esdata/data04,/esdata/data05,/esdata/data06


# Virtual memory settings - ES heap is set to half my current VM RAM
# but no greater than 32GB for performance reasons
elasticsearch_heap_size: 16g
elasticsearch_max_locked_memory: unlimited
elasticsearch_memory_bootstrap_mlockall: "true"


# Good idea not to go with the ES default names of Franz Kafka etc
elasticsearch_cluster_name: nx-elastic
elasticsearch_node_name: nx-esnode01

# My initial nodes will be both cluster quorum members and data "workhorse" nodes.
# I will # separate duties as I scale. Also I set the min master nodes to 1 so that 
# my ES cluster comes up while initially testing a single index 
elasticsearch_node_master: "true"
elasticsearch_node_data: "true"
elasticsearch_discovery_zen_minimum_master_nodes: 1

We’ll see how we use these variables as we cover more features. Next up I used some nice features like shell and also register variables to be able to provide conditional behaviour for package install :

- name: check for previous elasticsearch installation
 shell: if [ -e /usr/share/elasticsearch/lib/elasticsearch-{{ elasticsearch_version }}.jar ]; then echo yes; else echo no; fi;
 register: version_exists
 always_run: True
 tags: elastic

- name: uninstalling previous version if applicable
 become: true
 command: yum erase -y elasticsearch
 when: version_exists.stdout == 'no'
 ignore_errors: true
 tags: elastic

and similarly for the marvel plugin :

- name: check marvel plugin installed
 become: true
 stat: path={{ elasticsearch_home_dir }}/plugins/marvel
 register: marvel_installed
 tags: elastic

- name: install marvel plugin
 become: true
 command: "{{ elasticsearch_home_dir }}/bin/plugin -i elasticsearch/marvel/latest"
 - restart elasticsearch
 when: not marvel_installed.stat.exists
 tags: elastic

The Marvel plugin stanza above also makes use of the stat module – this is a really great module. It returns all kinds of goodness you would normally expect from a stat() system call and yet you are doing it in your Ansible playbook.  There are a couple more things I will cover and then leave the rest for when I talk about Kibana and Logstash in a follow up post. First up then are templates. Ansible uses Jinja2 templating in order to transform a file and install it on your host, you can create a file with appropriate templating as below. The variables in {{ .. }} are from the roles ../var directory containing the yaml file already described earlier.

Note : I stripped all comment lines for sake of brevity:

[ansible@ansible-host01 templates]$ pwd
[ansible@ansible-host01 templates]$ grep -v ^# elasticsearch.yml.j2
{% if elasticsearch_cluster_name is defined %}
cluster.name: {{ elasticsearch_cluster_name }}
{% endif %}


{% if elasticsearch_node_name is defined %}
node.name: {{ elasticsearch_node_name }}
{% endif %}


{% if elasticsearch_node_master is defined %}
node.master: {{ elasticsearch_node_master }}
{% endif %}
{% if elasticsearch_node_data is defined %}
node.data: {{ elasticsearch_node_data }}
{% endif %}


{% if elasticsearch_memory_bootstrap_mlockall is defined %}
bootstrap.mlockall: {{ elasticsearch_memory_bootstrap_mlockall }}
{% endif %}

The template  file when run in the play is then transformed using the provided variables and copied into place on my  ELK host target VM…

- name: copy elasticsearch defaults file
 become: true
 template: src=elasticsearch.default.j2 dest=/etc/sysconfig/elasticsearch owner={{ elasticsearch_user }} group={{ elasticsearch_group }} mode=0644
 - restart elasticsearch
 tags: elastic

So let’s see how our playbook runs and what the output looks like

[ansible@ansible-host01 elk]$ ansible-playbook -i ./production site.yml \
--tags "config,elastic" --ask-sudo-pass
SUDO password:

PLAY [elastic-hosts] **********************************************************

GATHERING FACTS ***************************************************************
ok: []

TASK: [config | install java 8 runtime] ***************************************
ok: []

TASK: [config | set swappiness in sysctl/systemd] *****************************
ok: []

TASK: [config | set maximum map count in sysctl/systemd] **********************
ok: []

TASK: [config | set hard limits for open files] *******************************
ok: []

TASK: [config | set soft limits for open files] *******************************
ok: []

TASK: [config | set max locked memory] ****************************************
ok: []

TASK: [config | Install wget package (Fedora based)] **************************
ok: []

TASK: [elastic | install elasticsearch signing key] ***************************
changed: []

TASK: [elastic | copy elasticsearch repo] *************************************
ok: []

TASK: [elastic | check for previous elasticsearch installation] ***************
changed: []

TASK: [elastic | uninstalling previous version if applicable] *****************
skipping: []

TASK: [elastic | install elasticsearch pkgs] **********************************
skipping: []

TASK: [elastic | copy elasticsearch configuration file] ***********************
ok: []

TASK: [elastic | copy elasticsearch defaults file] ****************************
ok: []

TASK: [elastic | set max memory limit in systemd file (RHEL/CentOS 7+)] *******
changed: []

TASK: [elastic | set log directory permissions] *******************************
ok: []

TASK: [elastic | set data directory permissions] ******************************
ok: []

TASK: [elastic | ensure elasticsearch running and enabled] ********************
ok: []

TASK: [elastic | check marvel plugin installed] *******************************
ok: []

TASK: [elastic | install marvel plugin] ***************************************
skipping: []

NOTIFIED: [elastic | restart elasticsearch] ***********************************
changed: []

PLAY RECAP ******************************************************************** : ok=19 changed=4 unreachable=0 failed=0

[ansible@ansible-host01 elk]$

I can verify that my ES cluster is working by querying the Cluster API – note that the red status is down to the fact I have no other cluster nodes yet on which to replicate the index shards:

# curl -XGET http://localhost:9200/_cluster/health?pretty
 "cluster_name" : "nx-elastic",
 "status" : "red",
 "timed_out" : false,
 "number_of_nodes" : 1,
 "number_of_data_nodes" : 1,
 "active_primary_shards" : 0,
 "active_shards" : 0,
 "relocating_shards" : 0,
 "initializing_shards" : 0,
 "unassigned_shards" : 0,
 "delayed_unassigned_shards" : 0,
 "number_of_pending_tasks" : 0,
 "number_of_in_flight_fetch" : 0

You can use further API queries to verify that the desired configuration is in place and at that point you have a solid, repeatable deployment with a known outcome ie: you are doing DevOps.

Using Ansible to deploy ELK stack on Nutanix

Just recently my colleague Andrew Nelson (@vmwnelson) posted an article on setting up Ansible on the Nutanix platform. I am also using Ansible to develop playbooks and the like to deploy the ELK stack components (Elasticsearch-Logstash-Kibana) on a block here at Nutanix. My initial aim is to setup a single index in an Elasticsearch (single node for now) cluster and use Logstash to pipe in data to be indexed. On top of that I intend to use Kibana and the Marvel plugin to measure at which point my index begins to struggle (based on stuff like OS level resource consumption, etc) as viewed from Marvel.

From a virtual machine perspective I have a Fedora 22 based gold image. From this base image I clone one VM to be the Ansible master that I will run playbooks (orchestration) from, and another VM which I will deploy my ELK stack to. This second “target” VM has had 7 vDisks added to it. The idea here being that Elasticsearch (ES) can use a comma separated list of vDisks (in my case I created them as six Linear LVM volumes). These are written to in a round robin fashion by ES and so the data gets “striped”. Nutanix vDisks are already redundant so we are getting a kind of RAID 10 for free! Here’s how my disk layout looks once configured and mounted (I am using XFS as my filesystem) on the target VM:

[root@elkhost01 ~]# df -h
/dev/mapper/esdata05-esdata05 200G 271M 200G 1% /esdata/data05
/dev/mapper/esdata03-esdata03 200G 291M 200G 1% /esdata/data03
/dev/mapper/esdata04-esdata04 200G 273M 200G 1% /esdata/data04
/dev/mapper/esdata02-esdata02 200G 271M 200G 1% /esdata/data02
/dev/mapper/esdata06-esdata06 200G 291M 200G 1% /esdata/data06
/dev/mapper/eslog-eslog 100G 150M 100G 1% /var/log/elasticsearch
/dev/mapper/esdata01-esdata01 200G 279M 200G 1% /esdata/data01


[root@elkhost01 ~]# lvs
 LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
 esdata01 esdata01 -wi-ao---- 200.00g
 esdata02 esdata02 -wi-ao---- 200.00g
 esdata03 esdata03 -wi-ao---- 200.00g
 esdata04 esdata04 -wi-ao---- 200.00g
 esdata05 esdata05 -wi-ao---- 200.00g
 esdata06 esdata06 -wi-ao---- 200.00g
 eslog eslog -wi-ao---- 100.00g

The next step is to install and configure Ansible. First off, configure an ansible user on both the orchestration host and target host and sync ssh keys between the two – (there’s a module that does ssh key exchange in Ansible and I will cover that at some stage)  – like so:

on both VMs :

useradd ansible
passwd ansible

# generate pub and priv keys ....
ssh-keygen -t rsa

If using strictmodes (default) in sshd_config file 
then ensure correct perms on .ssh directory and files 

chmod 700 ~/.ssh 
chmod 600 ~/.ssh/authorized_keys
[ansible@elkhost01 ~]$ ls -l ~/.ssh
total 12
-rw-------. 1 ansible ansible 404 Oct 1 13:38 authorized_keys
-rw-------. 1 ansible ansible 1675 Oct 1 13:31 id_rsa
-rw-------. 1 ansible ansible 402 Oct 1 13:31 id_rsa.pub

Exchange public keys (copy into remote hosts authorized_keys file) 
for passwordless access

[ansible@ansible-host01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
ansible@'s password:

Number of key(s) added: 1

Now try logging into the machine, with: "ssh ''"
and check to make sure that only the key(s) you wanted were added.

[ansible@ansible-host01 ~]$ 

[ansible@ansible-host01 ~]$ ssh
Last login: Thu Oct 1 13:38:35 2015 from
[ansible@elkhost01 ~]$

Once you have passwordless ssh configured between your hosts – go ahead and install Ansible on the orchestration host:

# yum install ansible -y

Once installed, there are a few post install steps and tests to make sure that Ansible is working. First off set up a Ansible hosts inventory file that will eventually contain all the hostnames broken out by deployment type. The default location for this file is  /etc/ansible/hosts. In this instance I have chosen to specify a non standard name/location in order keep my hosts file within my proposed playbook.

[ansible@ansible-host01 elk]$ pwd
[ansible@ansible-host01 elk]$ cat production
# file: production




And if the passwordless ssh setup is correct – we can test as follows :

[ansible@ansible-host01 elk]$ ansible all -m ping | success >> {
 "changed": false,
 "ping": "pong"

Ansible machine configuration is done via playbooks, which are based on YAML syntax. There’s a great best practice guide here. I have followed that same best practice guide on playbook directory layout below …

├── elastic.yml
├── group_vars
├── host_vars
├── kibana.yml
├── production 
├── roles
│   ├── common
│   │   ├── files
│   │   ├── handlers
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   └── vars
│   │   └── main.yml
│   ├── elastic
│   │   ├── files
│   │   │   └── elasticsearch.repo
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   │   ├── elasticsearch.default.j2
│   │   │   ├── elasticsearch.in.sh.j2
│   │   │   └── elasticsearch.yml.j2
│   │   └── vars
│   │   └── main.yml
│  └-- kibana
│      ├── files
│      ├── handlers
│      │   └── main.yml
│      ├── tasks
│      │   └── main.yml
│      ├── templates
│      │   └── kibana4.service.j2
│      └── vars
│      └── main.yml
├── site.yml

I am going to cover the individual roles for elasticsearch, logstash and kibana in subsequent posts. For now there’s a main site wide playbook :

[ansible@ansible-host01 elk]$ cat site.yml
# file: site.yml
- include: elastic.yml
- include: kibana.yml
- include: logstash.yml
#- include: log-forwarder.yml
#- include: redis.yml
#- include: nginx.yml

Which is then broken up into individual service specific playbooks :

[ansible@ansible-host01 elk]$ cat elastic.yml
#file: elastic.yml
- hosts: elastic-hosts
 - common
 - elastic
[ansible@ansible-host01 elk]$ cat kibana.yml
#file: kibana.yml
- hosts: kibana-hosts
 - kibana

I will discuss the individual roles and their associated tasks etc next time. For now this should be enough to get basic Ansible functionality going.