Merge pull request #256 from mattkirby/config_changes_squash

(POOLER-107) Add configuration API endpoint

docs/API.md

@@ -1,8 +1,17 @@
-### API
+# Table of contents
+1. [API](#API)
+2. [Token operations](#token)
+3. [VM operations](#vmops)
+4. [Add disks](#adddisks)
+5. [VM snapshots](#vmsnapshots)
+6. [Status and metrics](#statusmetrics)
+7. [Pool configuration](#poolconfig)
+
+### API <a name="API"></a>
 
 vmpooler provides a REST API for VM management.  The following examples use `curl` for communication.
 
-#### Token operations
+#### Token operations <a name="token"></a>
 
 Token-based authentication can be used when requesting or modifying VMs.  The `/token` route can be used to create, query, or delete tokens.  See the provided YAML configuration example, [vmpooler.yaml.example](vmpooler.yaml.example), for information on configuring an authentication store to use when performing token operations.
 
@@ -76,7 +85,7 @@ Enter host password for user 'jdoe':
 }
 ```
 
-#### VM operations
+#### VM operations <a name="vmops"></a>
 
 ##### GET /vm
 
@@ -230,7 +239,7 @@ $ curl -X DELETE --url vmpooler.company.com/api/v1/vm/fq6qlpjlsskycq6
 }
 ```
 
-#### Adding additional disk(s)
+#### Adding additional disk(s) <a name="adddisks"></a>
 
 ##### POST /vm/&lt;hostname&gt;/disk/&lt;size&gt;
 
@@ -270,7 +279,7 @@ $ curl --url vmpooler.company.com/api/v1/vm/fq6qlpjlsskycq6
 
 ````
 
-#### VM snapshots
+#### VM snapshots <a name="vmsnapshots"></a>
 
 ##### POST /vm/&lt;hostname&gt;/snapshot
 
@@ -322,7 +331,7 @@ $ curl X POST -H X-AUTH-TOKEN:a9znth9dn01t416hrguu56ze37t790bl --url vmpooler.co
 }
 ````
 
-#### Status and metrics
+#### Status and metrics <a name="statusmetrics"></a>
 
 ##### GET /status
 
@@ -540,3 +549,97 @@ $ curl -G -d 'from=2015-03-10' -d 'to=2015-03-11' --url vmpooler.company.com/api
   ]
 }
 ```
+
+#### Managing pool configuration via API <a name="poolconfig"></a>
+
+##### GET /config
+
+Returns the running pool configuration
+
+Responses:
+* 200 - OK
+* 404 - No configuration found
+```
+$ curl https://vmpooler.company.com/api/v1/config
+```
+```json
+{
+  "pool_configuration": [
+    {
+      "name": "redhat-7-x86_64",
+      "template": "templates/redhat-7.2-x86_64-0.0.3",
+      "folder": "vmpooler/redhat-7-x86_64",
+      "datastore": "stor1",
+      "size": 1,
+      "datacenter": "dc1",
+      "provider": "vsphere",
+      "capacity": 1,
+      "major": "redhat",
+      "template_ready": true
+    }
+  ],
+  "status": {
+    "ok": true
+  }
+}
+```
+
+Note: to enable poolsize and pooltemplate config endpoints it is necessary to set 'experimental_features: true' in your vmpooler configuration. A 405 is returned when you attempt to interact with these endpoints when this configuration option is not set.
+
+##### POST /config/poolsize
+
+Change pool size without having to restart the service.
+
+All pool template changes requested must be for pools that exist in the vmpooler configuration running, or a 404 code will be returned
+
+When a pool size is changed due to the configuration posted a 201 status will be returned. When the pool configuration is valid, but will not result in any changes, 200 is returned.
+
+Pool size configuration changes persist through application restarts, and take precedence over a pool size value configured in the pool configuration provided when the application starts. This persistence is dependent on redis. So, if the redis data is lost then the configuration updates revert to those provided at startup at the next application start.
+
+An authentication token is required in order to change pool configuration when authentication is configured.
+Responses:
+* 200 - No changes required
+* 201 - Changes made on at least one pool with changes requested
+* 400 - An invalid configuration was provided causing requested changes to fail
+* 404 - An unknown error occurred
+* 405 - The endpoint is disabled because experimental features are disabled
+```
+$ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"2","debian-7-x86_64":"1"}' --url https://vmpooler.company.com/api/v1/config/poolsize
+```
+```json
+{
+  "ok": true
+}
+```
+
+##### POST /config/pooltemplate
+
+Change the template configured for a pool, and replenish the pool with instances built from the new template.
+
+All pool template changes requested must be for pools that exist in the vmpooler configuration running, or a 404 code will be returned
+
+When a pool template is changed due to the configuration posted a 201 status will be returned. When the pool configuration is valid, but will not result in any changes, 200 is returned.
+
+A pool template being updated will cause the following actions, which are logged in vmpooler.log:
+* Destroy all instances for the pool template being updated that are in the ready and pending state
+* Halt repopulating the pool while creating template deltas for the newly configured template
+* Unblock pool population and let the pool replenish with instances based on the newly configured template
+
+Pool template changes persist through application restarts, and take precedence over a pool template configured in the pool configuration provided when the application starts. This persistence is dependent on redis. As a result, if the redis data is lost then the configuration values revert to those provided at startup at the next application start.
+
+An authentication token is required in order to change pool configuration when authentication is configured.
+
+Responses:
+* 200 - No changes required
+* 201 - Changes made on at least one pool with changes requested
+* 400 - An invalid configuration was provided causing requested changes to fail
+* 404 - An unknown error occurred
+* 405 - The endpoint is disabled because experimental features are disabled
+```
+$ curl -X POST -H "Content-Type: application/json" -d '{"debian-7-i386":"templates/debian-7-i386"}' --url https://vmpooler.company.com/api/v1/config/pooltemplate
+```
+```json
+{
+  "ok": true
+}
+```

lib/vmpooler/api/helpers.rb

@@ -378,6 +378,29 @@ def get_task_summary(backend, task_str, from_date, to_date, opts = {})
         result
       end
 
+      def pool_index(pools)
+        pools_hash = {}
+        index = 0
+        for pool in pools
+          pools_hash[pool['name']] = index
+          index += 1
+        end
+        pools_hash
+      end
+
+      def template_ready?(pool, backend)
+        prepared_template = backend.hget('vmpooler__template__prepared', pool['name'])
+        return false if prepared_template.nil?
+        return true if pool['template'] == prepared_template
+        return false
+      end
+
+      def is_integer?(x)
+        Integer(x)
+        true
+      rescue
+        false
+      end
     end
   end
 end

lib/vmpooler/api/v1.rb

@@ -120,6 +120,74 @@ def atomically_allocate_vms(payload)
       result
     end
 
+    def update_pool_size(payload)
+      result = { 'ok' => false }
+
+      pool_index = pool_index(pools)
+      pools_updated = 0
+      sync_pool_sizes
+
+      payload.each do |poolname, size|
+        unless pools[pool_index[poolname]]['size'] == size.to_i
+          pools[pool_index[poolname]]['size'] = size.to_i
+          backend.hset('vmpooler__config__poolsize', poolname, size)
+          pools_updated += 1
+          status 201
+        end
+      end
+      status 200 unless pools_updated > 0
+      result['ok'] = true
+      result
+    end
+
+    def update_pool_template(payload)
+      result = { 'ok' => false }
+
+      pool_index = pool_index(pools)
+      pools_updated = 0
+      sync_pool_templates
+
+      payload.each do |poolname, template|
+        unless pools[pool_index[poolname]]['template'] == template
+          pools[pool_index[poolname]]['template'] = template
+          backend.hset('vmpooler__config__template', poolname, template)
+          pools_updated += 1
+          status 201
+        end
+      end
+      status 200 unless pools_updated > 0
+      result['ok'] = true
+      result
+    end
+
+    def sync_pool_templates
+      pool_index = pool_index(pools)
+      template_configs = backend.hgetall('vmpooler__config__template')
+      unless template_configs.nil?
+        template_configs.each do |poolname, template|
+          if pool_index.include? poolname
+            unless pools[pool_index[poolname]]['template'] == template
+              pools[pool_index[poolname]]['template'] = template
+            end
+          end
+        end
+      end
+    end
+
+    def sync_pool_sizes
+      pool_index = pool_index(pools)
+      poolsize_configs = backend.hgetall('vmpooler__config__poolsize')
+      unless poolsize_configs.nil?
+        poolsize_configs.each do |poolname, size|
+          if pool_index.include? poolname
+            unless pools[pool_index[poolname]]['size'] == size.to_i
+              pools[pool_index[poolname]]['size'] == size.to_i
+            end
+          end
+        end
+      end
+    end
+
     # Provide run-time statistics
     #
     # Example:
@@ -196,6 +264,8 @@ def atomically_allocate_vms(payload)
         }
       }
 
+      sync_pool_sizes
+
       result[:capacity] = get_capacity_metrics(pools, backend) unless views and not views.include?("capacity")
       result[:queue] = get_queue_metrics(pools, backend) unless views and not views.include?("queue")
       result[:clone] = get_task_metrics(backend, 'clone', Date.today.to_s) unless views and not views.include?("clone")
@@ -502,6 +572,30 @@ def invalid_templates(payload)
       invalid
     end
 
+    def invalid_template_or_size(payload)
+      invalid = []
+      payload.each do |pool, size|
+        invalid << pool unless pool_exists?(pool)
+        unless is_integer?(size)
+          invalid << pool
+          next
+        end
+        invalid << pool unless Integer(size) >= 0
+      end
+      invalid
+    end
+
+    def invalid_template_or_path(payload)
+      invalid = []
+      payload.each do |pool, template|
+        invalid << pool unless pool_exists?(pool)
+        invalid << pool unless template.include? '/'
+        invalid << pool if template[0] == '/'
+        invalid << pool if template[-1] == '/'
+      end
+      invalid
+    end
+
     post "#{api_prefix}/vm/:template/?" do
       content_type :json
       result = { 'ok' => false }
@@ -747,6 +841,95 @@ def invalid_templates(payload)
 
       JSON.pretty_generate(result)
     end
+
+    post "#{api_prefix}/config/poolsize/?" do
+      content_type :json
+      result = { 'ok' => false }
+
+      if config['experimental_features']
+        need_token! if Vmpooler::API.settings.config[:auth]
+
+        payload = JSON.parse(request.body.read)
+
+        if payload
+          invalid = invalid_template_or_size(payload)
+          if invalid.empty?
+            result = update_pool_size(payload)
+          else
+            invalid.each do |bad_template|
+              metrics.increment("config.invalid.#{bad_template}")
+            end
+            result[:bad_templates] = invalid
+            status 400
+          end
+        else
+          metrics.increment('config.invalid.unknown')
+          status 404
+        end
+      else
+        status 405
+      end
+
+      JSON.pretty_generate(result)
+    end
+
+    post "#{api_prefix}/config/pooltemplate/?" do
+      content_type :json
+      result = { 'ok' => false }
+
+      if config['experimental_features']
+        need_token! if Vmpooler::API.settings.config[:auth]
+
+        payload = JSON.parse(request.body.read)
+
+        if payload
+          invalid = invalid_template_or_path(payload)
+          if invalid.empty?
+            result = update_pool_template(payload)
+          else
+            invalid.each do |bad_template|
+              metrics.increment("config.invalid.#{bad_template}")
+            end
+            result[:bad_templates] = invalid
+            status 400
+          end
+        else
+          metrics.increment('config.invalid.unknown')
+          status 404
+        end
+      else
+        status 405
+      end
+
+      JSON.pretty_generate(result)
+    end
+
+    get "#{api_prefix}/config/?" do
+      content_type :json
+      result = { 'ok' => false }
+      status 404
+
+      if pools
+        sync_pool_sizes
+        sync_pool_templates
+
+        pool_configuration = []
+        pools.each do |pool|
+          pool['template_ready'] = template_ready?(pool, backend)
+          pool_configuration << pool
+        end
+
+        result = {
+          pool_configuration: pool_configuration,
+          status: {
+            ok: true
+          }
+        }
+
+        status 200
+      end
+      JSON.pretty_generate(result)
+    end
   end
   end
 end