Category Archives: REST API

Automating S3 compliant Object stores via Nutanix Objects API

As part of an API first strategy within the company, the Objects team at Nutanix has developed a REST API to enable the automated creation, deletion, management and monitoring of S3 compliant Object stores. I was fortunate to be given early access to the developing API. As part of this preview work, I have been looking at how to use CALM’s built-in support for “chaining” REST calls together, in order to build a JSON payload that creates an object store via its API. 

POST /objectstores

Let’s take a brief look at a subset of the Objects API. In order to create our objectstore, we need to make several intermediate calls to the standard v3 API. These calls are used to obtain (for example) reference UUIDs from entities like the underlying Nutanix cluster or required networks. The image below shows how the desired objectstore payload is pre-populated using macro variables that are either entered as part of the initial CALM blueprint configuration – @@{objectstore_name}@@ or generated from CALM tasks that pass in a variable at runtime – @@{CLUSTER}@@. We’ll discuss the latter shortly.

The Objects API (OSS) is accessed via a Prism Central (PC) endpoint. Notice the Objects API endpoint URL, where @@{address}@@ defines the PC IP address.

https://@@{address}@@:9440/oss/api/nutanix/v3/objectstores

The REST call to create the objectstore is then handled by the CALM provided URL request function, urlreq(). The underlying call is still made via the Python requests module however. See below for how it was used in this scenario. More details on the various supported CALM functions can be found on the Nutanix documentation portal

Task type: Set Variable

Let’s look at how we generate the various saved UUIDs and other required entities, in order to pass them around our code. Recall that such entities are used to build the final JSON payload for the objectstore creation step we have already covered above. CALM provides a task framework that performs various functions. For example, to run a script or some Python code. There’s also a task option that results in the setting of a required variable. Once such a variable is created or set, it is then available to all other tasks. The next image below shows how we configure a task to set a variable.

Application profile : Objects

On the left hand pane in the above image, you will see an Application profile, entitled Objects. This profile gives me a set of default actions for my object store, such as Create, Start, Restart, etc. It also allows the creation of custom actions. We will look at REST_Create as an example of a custom action. From the list of tasks associated with REST_Create in the central canvas, we have an a task entitled, GetClusterUUID. The right hand pane shows how this task is configured. Note the task type is “Set Variable”. We also run a Python request, in the Script canvas. This populates an Output variable entitled CLUSTER. CLUSTER contains the Nutanix cluster UUID. We can see how this works in a little more detail below.

Script

First, we set the credentials for Prism Central access. How credentials get set up in this kind of configuration, will be discussed later in the post. Next step is to populate the REST headers, URL and the JSON payload. Payload here is empty, but you can choose to either limit the number of clusters returned or use pagination if preferred. Pagination will require additional coding however.

We cycle through the response content of cluster entities looking for a match against our supplied cluster name – @@{cluster_name}@@. If found, we have guardrail code that ensures we only proceed if both hypervisor and version of AOS are supported. We do this in the GetClusterUUID task as its the first call we make. In doing so we exit as early as possible if we find a problem.

The matching cluster UUID from the response is saved into the CLUSTER variable. This UUID is then available to other tasks in the blueprint. Similar patterns are repeated in the tasks GetInfraNetUUID and GetClientNetUUID. Both tasks populate a variable with their respective network references (UUIDs). These variables are both used in the CreateObjectstore task, covered above. Without going into too much architecture detail, the Objects feature set is built on a microservice architecture. The networks mentioned are required for the internal Kubernetes inter-node/pod communication.

CALM Service

I will quickly go over the creation of the required Objects_Store service in CALM. This will cover the previously only mentioned credentials setup and so on. I think the image below is fairly self explanatory. It shows how to configure a blueprint to run against the incumbent Prism Central instance, and deploy the application (in our case an Object store) on an existing cluster infrastructure.

The CALM blueprint discussed here for automated Object store creation is available here (in its current form):

https://github.com/rayhassan/calm-bp-objects

As the API develops towards General Availability, I hope to add more functionality to the blueprint (DELETE, Replace Certs, and so on). For now, here’s a quick run through of how the blueprint deploys the Objectstore via API. The image below shows the running application after the blueprint is launched.

The objectstore is then “managed” via the now provisioned application. To then create an objectstore according to the options set at the blueprint launch, we run the custom actions we previously created. Select first the Manage tab and then the REST_Create task

While the objectstore is being created, we can run other tasks that perform API calls that monitor objectstore progress and status. The output from the Audit tab is how ever we decided to format the JSON response in our REST_Status task. For example….

This ties in with exactly what we see in the Prism GUI at that time.

Big Data use case

In addition to the use cases outlined below, I am interested in investigating how Nutanix Objects¬† will play in the Big Data space. In particular, how Objects can be used to create standby environments for an Hadoop ecosystem. Ideally in another location. This is something that usually requires a large amount of work. Using Objects there’s the potential to de-risk the data lake replication part to a large extent. I hope to make this investigation a part of our upcoming Hadoop certification work.

Current Use Cases

  • Backup: Consolidate Nutanix and non-Nutanix primary infrastructure.
  • Long Term Retention (e.g.Splunk cold tier, Doc archives,Images/Videos): Cheap & deep,
    with regulatory content retention.
  • DevOps: Enable IT to provide an AWS S3 like service, on-premises, for cloud-native

Let me know if you find the Objects blueprint useful or feel free to share your experience of Nutanix Objects and how we can make things work better.

The Creator Supreme! …..or How to be the Kenny Dalglish of Nutanix API automation

For the non European football fans out there, Kenny Dalglish,or “King Kenny” as he was known to both the Liverpool and Celtic faithful, was once described in a match commentary as “the creator supreme”.  In this short series of posts covering the REST capabilities for managing the Nutanix Enterprise Cloud, I hope to show its possible that we can all be a “creator supreme” just like King Kenny!

Swagger

The first place to look when working with the API is the REST API Explorer itself. You invoke the Explorer page by right-clicking the admin pull-down menu (top right of Prism home page) and selecting the REST API Explorer option.

It’s good practice to create a separate user login for the REST Explorer. The REST API Explorer is essentially the documentation to the API. It’s produced via swagger, which is an open standard that takes your API spec and generates interactive documentation. The documentation can be viewed and you are able to interactively test API calls via a browser.

Images

Let’s start by taking a look at a simple POST method that will list all currently available images:

POST /images/list

Select the above method in the images section of the REST API Explorer to expand the method details:

In order to try out the REST method, double click on the Model Schema box (right hand side above) – this will then be populated into the get_entities_request box on the left hand side. You can edit the entities according to how you want to retrieve the available information. For example here’s the bare minimum you need as a JSON payload to request information from the Images catalogue.

 {
   "kind": "image",
   "offset": 0,
   "length": 10
 }

Note that with our pagination we are starting at offset zero – so the first image – until the tenth image, defined by the length parameter. With the JSON payload entered as above we can press the Try it out! button and see the method in action.

The results of the method call are displayed below. The Curl syntax for invoking the method and json payload are shown, along with the individual Request URL and the Response Body. We can use the Curl syntax to programmatically call the method outside of the Explorer, either in Bash or Python, for example.

Once we begin to use the methods independently of the Explorer, then in addition to curl you should consider installing a JSON command line processor like jq, and use a JSON Linter to validate your JSON syntax for data payloads. How the tools might be used will be shown throughout this post

CURL

Lets recap the previous POST method but this time run it from the command line. In this instance we load the JSON payload (see above) from the file list_images_v3.json using the -d option. The -k option , or — insecure allows the command to proceed even though I am using self-signed SSL/TLS certs. The -s option simply disables all progress indicators.

curl -s --user api:<password> -k -X POST --header "Content-Type: application/json" --header "Accept: application/json" -d @list_images_v3.json "https://10.XX.XX.60:9440/api/nutanix/v3/images/list" | jq 

Piping the output from the curl command into jq, provides a formatted and syntax highlighted output that’s easier to read. To make this more obvious, let’s use some additional options to the jq command line and pull out just one image reference:

curl -s --user apiuser:<password> -k -X POST --header "Content-Type: application/json" --header "Accept: application/json" -d @list_images_v3.json "https://10.68.64.60:9440/api/nutanix/v3/images/list" | jq '.entities[] | select (.spec.name=="CentOS7-x86_64-Generic Cloud")'

 {
   "status": {
     "state": "COMPLETE",
     "name": "CentOS7-x86_64-Generic Cloud",
     "resources": {
       "retrieval_uri_list": [
         "https://127.0.0.1:9440/api/nutanix/v3/images//file"
       ],
       "image_type": "DISK_IMAGE",
       "architecture": "X86_64",
       "size_bytes": 8589934592
     },
     "description": "Generic Cloud"
   },
   "spec": {
     "name": "CentOS7-x86_64-Generic Cloud",
     "resources": {
       "image_type": "DISK_IMAGE",
       "architecture": "X86_64"
     },
     "description": "Generic Cloud"
   },
   "metadata": {
     "last_update_time": "2019-03-27T10:47:15Z",
     "kind": "image",
     "uuid": "04a18eb0-a3ed-4ff7-aa43-bdbb055a96ef",
     "spec_version": 0,
     "creation_time": "2019-03-27T10:47:15Z",
     "categories": {}
   }
 }

All well and good if you know the exact name of your image. What about if you don’t ? See below

curl -s --user apiuser:<password> -k -X POST --header "Content-Type: application/json" --header "Accept: application/json" -d @list_images_v3.json "https://10.68.64.60:9440/api/nutanix/v3/images/list" | jq '.entities[] | select (.spec.name | . and contains("CentOS"))'

{
  "status": {
    "state": "COMPLETE",
    "name": "CentOS7-x86_64-Minimal",
    "resources": {
      "retrieval_uri_list": [
        "https://127.0.0.1:9440/api/nutanix/v3/images//file"
      ],
      "image_type": "ISO_IMAGE",
      "architecture": "X86_64",
      "size_bytes": 713031680
    },
    "description": "Minimal"
  },
  "spec": {
    "name": "CentOS7-x86_64-Minimal",
    "resources": {
      "image_type": "ISO_IMAGE",
      "architecture": "X86_64"
    },
    "description": "Minimal"
  },
  "metadata": {
    "last_update_time": "2019-03-27T10:47:15Z",
    "kind": "image",
    "uuid": "dd482003-99f4-45df-9406-1dc9859418c4",
    "spec_version": 0,
    "creation_time": "2019-03-27T10:47:15Z",
    "categories": {}
  }
}
{
  "status": {
    "state": "COMPLETE",
    "name": "CentOS7-x86_64-Generic Cloud",
    "resources": {
      "retrieval_uri_list": [
        "https://127.0.0.1:9440/api/nutanix/v3/images//file"
      ],
      "image_type": "DISK_IMAGE",
      "architecture": "X86_64",
      "size_bytes": 8589934592
    },
    "description": "Generic Cloud"
  },
  "spec": {
    "name": "CentOS7-x86_64-Generic Cloud",
    "resources": {
      "image_type": "DISK_IMAGE",
      "architecture": "X86_64"
    },
    "description": "Generic Cloud"
  },
  "metadata": {
    "last_update_time": "2019-03-27T10:47:15Z",
    "kind": "image",
    "uuid": "04a18eb0-a3ed-4ff7-aa43-bdbb055a96ef",
    "spec_version": 0,
    "creation_time": "2019-03-27T10:47:15Z",
    "categories": {}
  }
}

One of the prime uses for this kind of command is to retrieve only the info required when populating a schema for another REST method (see below shortly). For example, you may only want a subset of entries and perhaps they need to be conveniently labelled:

curl -s --user apiuser:<password> -k -X POST --header "Content-Type: application/json" --header "Accept: application/json" -d @list_images_v3.json "https://10.68.64.60:9440/api/nutanix/v3/images/list" | jq '.entities[] | {name: .spec.name, type: .spec.resources.image_type, uuid: .metadata.uuid} | select (.name | . and contains("CentOS"))'
{
"name": "CentOS7-x86_64-Minimal",
"type": "ISO_IMAGE",
"uuid": "dd482003-99f4-45df-9406-1dc9859418c4"
}
{
"name": "CentOS7-x86_64-Generic Cloud",
"type": "DISK_IMAGE",
"uuid": "04a18eb0-a3ed-4ff7-aa43-bdbb055a96ef"
}

Upload

Let’s have a look at uploading an image to the image repository on your Prism Central instance. The following is the required schema :

cat upload_image_v3.json
{
     "spec": {
         "name": "test",
         "resources": {
             "version": {
                 "product_version": "test",
                 "product_name": "test"
             },
             "architecture": "X86_64",
             "image_type": "DISK_IMAGE",
             "source_uri": "http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img"
         }
     },
     "api_version": "3.1.0",
     "metadata": {
         "kind": "image"
     }
 }

which we can use as follows :

curl -s --user apiuser:<password> -k -X POST --header "Content-Type: application/json" --header "Accept: application/json" -d @upload_image_v3.json "https://10.68.64.60:9440/api/nutanix/v3/images" | jq .
 {
   "status": {
     "state": "PENDING",
     "execution_context": {
       "task_uuid": "f1456be3-21a8-45ab-9dc3-c323973e6f3f"
     }
   },
   "spec": {
     "name": "test",
     "resources": {
       "image_type": "DISK_IMAGE",
       "source_uri": "http://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-disk1.img",
       "version": {
         "product_version": "test",
         "product_name": "test"
       },
       "architecture": "X86_64"
     }
   },
   "api_version": "3.1",
   "metadata": {
     "owner_reference": {
       "kind": "user",
       "uuid": "00000000-0000-0000-0000-000000000000",
       "name": "admin"
     },
     "kind": "image",
     "spec_version": 0,
     "uuid": "b87c7183-8716-4051-8a35-da69fdbf1e60"
   }
 }

Tasks

Notice the PENDING status in the output above. We can follow the progress of the image upload by using the task_uuid entry in a tasks method call. The call can be run again and again until a task, in this case the upload, is complete

curl -s --user apiuser:<password> -k -X GET --header "Content-Type: application/json" --header "Accept: application/json" "https://10.68.64.60:9440/api/nutanix/v3/tasks/f1456be3-21a8-45ab-9dc3-c323973e6f3f" | jq .
 {
   "status": "RUNNING",
   "last_update_time": "2019-05-30T13:19:00Z",
   "logical_timestamp": 1,
   "entity_reference_list": [
     {
       "kind": "image",
       "uuid": "b87c7183-8716-4051-8a35-da69fdbf1e60"
     }
   ],
   "start_time": "2019-05-30T13:19:00Z",
   "creation_time": "2019-05-30T13:18:59Z",
   "start_time_usecs": 1559222340033005,
   "cluster_reference": {
     "kind": "cluster",
     "uuid": "e0cca748-66c4-45fb-95e2-10836439ea15"
   },
   "subtask_reference_list": [],
   "progress_message": "create_image_intentful",
   "creation_time_usecs": 1559222339906023,
   "operation_type": "create_image_intentful",
   "percentage_complete": 0,
   "api_version": "3.1",
   "uuid": "f1456be3-21a8-45ab-9dc3-c323973e6f3f"

Delete

Finally, lets delete the image. This is done by specifying the image UUID in the delete method call. We covered how to get a UUID for an image (or any entity really) above. So let’s just show the call

curl -s --user apiuser:<password> -k -X DELETE --header "Content-Type: application/json" --header "Accept: application/json" "https://10.68.64.60:9440/api/nutanix/v3/images/081e562f-6c26-4897-bc36-a74e4843bb57" | jq .
 {
     "status": 
     {
         "state": "DELETE_PENDING", 
         "execution_context": 
             {
                 "task_uuid": "b9ae5dce-c79d-4bca-b77d-b322949f71e5"
             }
     }, 
     "spec": "", 
     "api_version": "3.1", 
     "metadata": 
         {
             "kind": "image"
         }
 }

You can of course track the deletion progress via the tasks method using the task_uuid produced above.

Conclusion

Useful Resources recap:
HTTP Response status codes 
jq
curl
JSON Lint - The JSON Validator

Hopefully this will help get people started on their API path, we haven’t really scratched the surface of what can be done. Hopefully, this post has at least demystified where and how to make a start. In subsequent posts I hope to show more ways to glean info from the API Explorer itself and how to use it to build more complex REST methods. Until then, check out Nutanix Developer Community site. Good luck creators!