I was asked recently to take a look at an API issue around uploading an image file from a desktop directory location into the Acropolis Image Service. The REST calls for this workflow are part of version 0.8 of the Nutanix Management API. As such they only work on an endpoint supplied by Prism Element.
REST Explorer
POST
We covered image upload briefly in a previous post were we touched upon uploading from an external URL. The mechanism for image upload discussed here is in two parts. The first requires a POST call that will populate the image entry in the repository with the various metadata. For example, name, image type and so on. The second phase is the actual “image binary data” upload and this is done via a PUT call. Once both parts are successfully returned, we have the fully completed image. Let’s start with the initial POST call to upload a small CirrOS cloud image, in which we specify a name for the image and its image type (imageType).
The resulting task UUID could be monitored for progress and status if required. However, the operation often completes before we get the chance. Next we review the returned JSON response from a REST call listing all current images. The imageState at this point is currently INACTIVE. It needs to be in this status for the subsequent update or PUT step to proceed.
In order to load the image data to its previously created metadata, we need both the current image UUID and the UUID of the storage container we will upload the image to. The image UUID we already have from the /images call above. We can do similar to GET the storage container UUID
The Nutanix API dev team are continually working on making improvements to our REST automation capabilities. If this is something that interests you then feel free to reach out with any suggestions. Maybe you want be able to take a first look at our latest API innovations or help drive those changes with feature requests? In either case, please reach out to us to get the ball rolling.
Note:
In this post I used api/nutanix/v0.8 and api/nutanix/v2.0 in my base/endpoint URLs. While the defined endpoints in older documentation are PrismGateway/services/rest/v[0.8,2.0]/, there has been a move (within REST Explorer) to standardise on a single endpoint syntax while still calling out the version of API albeit v0.8 or v2.0. Both base url formats are interchangeable, however I recommend you always stick to the latest format for sake of consistency.
For the non European football fans out there, Kenny Dalglish,or “King Kenny” as he was known to both the Liverpool and Celtic faithful, was once described in a match commentary as “the creator supreme”. In this short series of posts covering the REST capabilities for managing the Nutanix Enterprise Cloud, I hope to show its possible that we can all be a “creator supreme” just like King Kenny!
Swagger
The first place to look when working with the API is the REST API Explorer itself. You invoke the Explorer page by right-clicking the admin pull-down menu (top right of Prism home page) and selecting the REST API Explorer option.
It’s good practice to create a separate user login for the REST Explorer. The REST API Explorer is essentially the documentation to the API. It’s produced via swagger, which is an open standard that takes your API spec and generates interactive documentation. The documentation can be viewed and you are able to interactively test API calls via a browser.
Images
Let’s start by taking a look at a simple POST method that will list all currently available images:
POST /images/list
Select the above method in the images section of the REST API Explorer to expand the method details:
In order to try out the REST method, double click on the Model Schema box (right hand side above) – this will then be populated into the get_entities_request box on the left hand side. You can edit the entities according to how you want to retrieve the available information. For example here’s the bare minimum you need as a JSON payload to request information from the Images catalogue.
{
"kind": "image",
"offset": 0,
"length": 10
}
Note that with our pagination we are starting at offset zero – so the first image – until the tenth image, defined by the length parameter. With the JSON payload entered as above we can press the Try it out! button and see the method in action.
The results of the method call are displayed below. The Curl syntax for invoking the method and json payload are shown, along with the individual Request URL and the Response Body. We can use the Curl syntax to programmatically call the method outside of the Explorer, either in Bash or Python, for example.
Once we begin to use the methods independently of the Explorer, then in addition to curl you should consider installing a JSON command line processor like jq, and use a JSON Linter to validate your JSON syntax for data payloads. How the tools might be used will be shown throughout this post
CURL
Lets recap the previous POST method but this time run it from the command line. In this instance we load the JSON payload (see above) from the file list_images_v3.json using the -d option. The -k option , or — insecure allows the command to proceed even though I am using self-signed SSL/TLS certs. The -s option simply disables all progress indicators.
Piping the output from the curl command into jq, provides a formatted and syntax highlighted output that’s easier to read. To make this more obvious, let’s use some additional options to the jq command line and pull out just one image reference:
One of the prime uses for this kind of command is to retrieve only the info required when populating a schema for another REST method (see below shortly). For example, you may only want a subset of entries and perhaps they need to be conveniently labelled:
Notice the PENDING status in the output above. We can follow the progress of the image upload by using the task_uuid entry in a tasks method call. The call can be run again and again until a task, in this case the upload, is complete
Finally, lets delete the image. This is done by specifying the image UUID in the delete method call. We covered how to get a UUID for an image (or any entity really) above. So let’s just show the call
Hopefully this will help get people started on their API path, we haven’t really scratched the surface of what can be done. Hopefully, this post has at least demystified where and how to make a start. In subsequent posts I hope to show more ways to glean info from the API Explorer itself and how to use it to build more complex REST methods. Until then, check out Nutanix Developer Community site. Good luck creators!
The answer to the question : “how big should I size my Elasticsearch VMs and how what kind of performance will I get?”, always comes down to the somewhat disappointing answer of “It depends!?” It depends on the workload – be it index or search heavy, on the type of data being transformed and so on.
The way to size your Elasticsearch environment is by finding your “unit of scale”, this is the performance characteristics you will get for your workload via a single shard index running in a single Virtual Machine (VM). Once you have a set of numbers for a particular VM config then you can scale throughput etc, via increasing the number of VMs and/or indexes to handle additional workload.
Virtual Machine Settings
The accepted sweet spot for VM sizing an indexing workload is something like 64GB RAM/ 8+ vCPUs. You can of course right size this further where necessary, thanks to virtualisation. I assign just below half the RAM (31GB) to the heap for the Elasticsearch instance. This is to ensure that the JVM uses compressed Ordinary Object Pointers (OOPs) on a 64 bit system. This heap memory also needs to be locked into RAM
# grep -v ^# /etc/elasticsearch/elasticsearch.yml
cluster.name: esrally
node.name: esbench
path.data: /elastic/data01 # <<< single striped data volume
bootstrap.memory_lock: true # <<< lock heap in RAM
network.host: 10.68.68.202
http.port: 9200
discovery.zen.minimum_master_nodes: 1 # <<< single node test cluster
xpack.security.enabled: false
# grep -v ^# /etc/elasticsearch/jvm.options
…
-Xms31g
-Xmx31g
…
From the section above , notice the single mount point for the path.data entry. I am using a 6 vdisk LVM stripe. While you can specify per-vdisk mount points in a comma separated list, unless you have enough indices to make sure all the spindles turn (all the time) then you are better off with logical volume management. You can ensure you are using compressed OOPs by checking for the following log entry at startup
Disable swapping, either via the cli or remove swap entries from /etc/fstab
# sudo swapoff –a
Elasticsearch Bulk Index Tuning
In order to improve indexing rate and increase shard segment size, you can disable refresh interval on an initial load. Afterwards, setting this to 30s (default=1s) in production means larger segments sizes and potentially less merge pressure at a later date.
Recall that we only want a single shard index and no replication for our testing. We can achieve this by either disabling replication on the fly or creating a template that configures the desired settings at index creation
Disable replication globally ...
curl -X PUT "10.68.68.202:9200/_settings" -H 'Content-Type: application/json' -d '{"index" : {"number_of_replicas" : 0}}’
or create a template - in this case, for a series of index name regex patterns...
# cat template.json
{
“index_patterns": [ “ray*”, "elasticlogs”],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
curl -s -X PUT "10.68.68.202:9200/_template/test_template" -H 'Content-Type: application/json' -d @template.json
Elasticsearch Benchmarking tools
esrally is a macrobenchmarking tool for elasticsearch. To install and configure – use the following quickstart guide. Full information is available here :
rally-eventdata-track – is repository containing a Rally track for simulating event-based data use-cases. The track supports bulk indexing of auto-generated events as well as simulated Kibana queries.
eventdata bulk index – 5000 events/request highlighted @indexing rate of ~50k docs/sec
httpd logs index test – highlighted @indexing rate ~80k docs/s
Elasticsearch is just one of a great many cloud native applications that can run successfully on Nutanix Enterprise Cloud. I am seeing more and more opportunities to assist our account teams in the sizing and deployment of Elasticsearch. However, unlike other Search and Analytics platforms Elasticsearch has no ready made formula for sizing. This post will hopefully allow people to make a start on their Elasticsearch sizing on Nutanix and, in addition, help identify future steps to improve their performance numbers.
It might seem like I am doing things out of sequence by looking at the visualisation layer of the ELK stack next. However, recall in my original post , that I wanted to build sets of unreplicated indexes and then use Logstash to fire test workloads at them. Hence, I am covering Elasticsearch and Kibana initially. This brings me to another technical point that I need to cover. In order for a single set of indexes to be actually recoverable, when running on a single node, we need to invoke the following parameters in our Elasticsearch playbook :
So in file: roles/elastic/vars/main.yml
...
elasticsearch_gateway.recover_after_nodes: 1
elasticsearch_gateway.recover_after_time: 5m
elasticsearch_gateway.expected_nodes: 1
...
These are then set in the elasticsearch.yml.j2 file as follows:
# file: roles/elastic/templates/elasticsearch.yml.j2
#{{ ansible_managed }}
...
# Allow recovery process after N nodes in a cluster are up:
#
#gateway.recover_after_nodes: 2
{% if elasticsearch_gateway_recover_after_nodes is defined %}
gateway.recover_after_nodes : {{ elasticsearch_gateway_recover_after_nodes}}
{% endif %}
and so on ....
This allows the indexes to be recovered when there is only a single node in the cluster. See below for the state of my indexes after a reboot:
Lets now look at the Kibana playbook I am attempting. Unfortunately, Kibana is distributed as a compressed tar archive. This means that the yumor dnfmodules are no help here. There is however a very useful unarchivemodule, but first we need to download the tar bundle using get_url as follows :
I initially tried unarchiving the Kibana bundle into /tmp. I then intended to copy everything below the version specific directory (/tmp/kibana-4.0.1-linux-x64) into the Ansible created /opt/kibana directory. This proved problematic as neither the synchronizenor the copymodules seemed setup to do mass copy/transfer between one directory structure to another. Maybe I am just not getting it – I even tried using with_item loops but no joy as fileglobsare not recursive. Answers on a postcard are always appreciated? In the end I just did this :
The next thing to do was to create a systemd service unit. There isn’t one for Kibana as there is no rpm package available. Usual templating applies here :
This all seemed to work as I could now access Kibana via my browser. No indexes yet of course :
There are one or two plays I would like still like to document. Firstly, the ‘notify’ actions in some of the plays. These are used to call – in my case – the restart handlers. Which in turn causes the service in question to be restarted – see the next section :
I wanted to document this next feature simply because it’s so useful – tags. I have assigned a tag to every play/task in the playbook so far you will have noticed. For testing purposes they allow you to run specific plays. You can then troubleshoot just that particular play and see what’s going on.
Now that I have the basic plays to get my Elasticsearch and Kibana services up and running via Ansible, it’s time to start looking at Logstash. Next time I post on ELK type stuff, I will try to look at logging and search use cases. Once I crack how they work of course.
