One node, one index, one shard
The answer to the question : “how big should I size my Elasticsearch VMs and how what kind of performance will I get?”, always comes down to the somewhat disappointing answer of “It depends!?” It depends on the workload – be it index or search heavy, on the type of data being transformed and so on.
The way to size your Elasticsearch environment is by finding your “unit of scale”, this is the performance characteristics you will get for your workload via a single shard index running in a single Virtual Machine (VM). Once you have a set of numbers for a particular VM config then you can scale throughput etc, via increasing the number of VMs and/or indexes to handle additional workload.
Virtual Machine Settings
The accepted sweet spot for VM sizing an indexing workload is something like 64GB RAM/ 8+ vCPUs. You can of course right size this further where necessary, thanks to virtualisation. I assign just below half the RAM (31GB) to the heap for the Elasticsearch instance. This is to ensure that the JVM uses compressed Ordinary Object Pointers (OOPs) on a 64 bit system. This heap memory also needs to be locked into RAM
# grep -v ^# /etc/elasticsearch/elasticsearch.yml
cluster.name: esrally
node.name: esbench
path.data: /elastic/data01 # <<< single striped data volume
bootstrap.memory_lock: true # <<< lock heap in RAM
network.host: 10.68.68.202
http.port: 9200
discovery.zen.minimum_master_nodes: 1 # <<< single node test cluster
xpack.security.enabled: false
# grep -v ^# /etc/elasticsearch/jvm.options
…
-Xms31g
-Xmx31g
…From the section above , notice the single mount point for the path.data entry. I am using a 6 vdisk LVM stripe. While you can specify per-vdisk mount points in a comma separated list, unless you have enough indices to make sure all the spindles turn (all the time) then you are better off with logical volume management. You can ensure you are using compressed OOPs by checking for the following log entry at startup
[2017-08-07T11:06:16,849][INFO ][o.e.e.NodeEnvironment ] [esrally02] heap size [30.9gb], compressed ordinary object pointers [true]
Operation System Settings
Set the required kernel settings
# sysctl -p
…
vm.swappiness = 0
vm.overcommit_memory = 0
vm.max_map_count = 262144
…
Ensure file descriptors limits are increased
# ulimit –n 65536
verify...
curl –XGET http://10.68.68.202:9200/_nodes/stats/process?filter_path=**.max_file_descriptors
…
{"process":{"max_file_descriptors":65536}}}}
…
Disable swapping, either via the cli or remove swap entries from /etc/fstab
# sudo swapoff –a Elasticsearch Bulk Index Tuning
In order to improve indexing rate and increase shard segment size, you can disable refresh interval on an initial load. Afterwards, setting this to 30s (default=1s) in production means larger segments sizes and potentially less merge pressure at a later date.
curl -X PUT "10.68.68.202:9200/elasticlogs/_settings" -H 'Content-Type: application/json' -d'
{
"index" : {
"refresh_interval" : "-1"
}
}’
Recall that we only want a single shard index and no replication for our testing. We can achieve this by either disabling replication on the fly or creating a template that configures the desired settings at index creation
Disable replication globally ...
curl -X PUT "10.68.68.202:9200/_settings" -H 'Content-Type: application/json' -d '{"index" : {"number_of_replicas" : 0}}’
or create a template - in this case, for a series of index name regex patterns...
# cat template.json
{
“index_patterns": [ “ray*”, "elasticlogs”],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0
}
}
curl -s -X PUT "10.68.68.202:9200/_template/test_template" -H 'Content-Type: application/json' -d @template.jsonElasticsearch Benchmarking tools
esrally is a macrobenchmarking tool for elasticsearch. To install and configure – use the following quickstart guide. Full information is available here :
https://github.com/elastic/rally
rally-eventdata-track – is repository containing a Rally track for simulating event-based data use-cases. The track supports bulk indexing of auto-generated events as well as simulated Kibana queries.
https://github.com/elastic/rally-eventdata-track
esrally --pipeline=benchmark-only --target-hosts=10.68.68.202:9200
--track=eventdata --track-repository=eventdata --challenge=bulk-size-evaluation
Elasticsearch is just one of a great many cloud native applications that can run successfully on Nutanix Enterprise Cloud. I am seeing more and more opportunities to assist our account teams in the sizing and deployment of Elasticsearch. However, unlike other Search and Analytics platforms Elasticsearch has no ready made formula for sizing. This post will hopefully allow people to make a start on their Elasticsearch sizing on Nutanix and, in addition, help identify future steps to improve their performance numbers.
