Add metrics server support for provision controller

[cofyc]

What this PR does / why we need it:

This pr add metrics server support for provision controller.

Like kubelet's storage_operation_xxx metrics, it's better we can have some metrics to monitor latency of provision/delete operations and failures in provisioner.

Current metrics:

• controller_persistentvolumeclaim_provision_total
• controller_persistentvolumeclaim_provision_failed_total
• controller_persistentvolumeclaim_provision_duration_seconds
• controller_persistentvolume_delete_total
• controller_persistentvolume_delete_failed_total
• controller_persistentvolume_delete_duration_seconds

Metrics server is disabled by default. Each provisioner can enable based on needs.
rbd/cephfs example: https://github.com/kubernetes-incubator/external-storage/pull/797/files

Examples:

...
controller_persistentvolume_delete_duration_seconds_bucket{class="rbd",le="10"} 4
controller_persistentvolume_delete_duration_seconds_bucket{class="rbd",le="+Inf"} 4
controller_persistentvolume_delete_duration_seconds_sum{class="rbd"} 1.0242772740000001
controller_persistentvolume_delete_duration_seconds_count{class="rbd"} 4
controller_persistentvolume_delete_total{class="rbd"} 4
...
controller_persistentvolumeclaim_discovery_duration_seconds_bucket{class="rbd",le="10"} 4
controller_persistentvolumeclaim_discovery_duration_seconds_bucket{class="rbd",le="+Inf"} 4
controller_persistentvolumeclaim_discovery_duration_seconds_sum{class="rbd"} 1.467650504
controller_persistentvolumeclaim_discovery_duration_seconds_count{class="rbd"} 4
controller_persistentvolumeclaim_provision_failed_total{class="rbd"} 9
controller_persistentvolumeclaim_provision_total{class="rbd"} 4

[cofyc]

/area lib
/assign @rootfs

[cofyc]

cc @childsb What do you think?

[cofyc]

cc @wongma7

[rootfs]

/assign @jsafrane

[wongma7]

lgtm, there is just one question I have about the port

[wongma7]

Should there be a port other than random we should default to? Like some convention for all provisioners. Will the port need to be exposed via the pod yaml? Because then a convention would make it easy to distribute yamls with the port already filled in

[cofyc]

Though port 0 means picking a random port in network programming, I used it to disabling metrics server here, like kubelet --cadvisor-port flag. Sorry, comments here is a bit confusing.

Will the port need to be exposed via the pod yaml? Because then a convention would make it easy to distribute yamls with the port already filled in

It depends on provisioner implementation. Provisioner can have a non-zero metrics port and pass it to NewProvisionController() to enable metrics server by default.

My intent is to make this feature optional for backward compatibility, each provisioner can customize based on needs, e.g. disable/enable by default, choose default port. We can have some convention for default port (e.g. 8000), but we may need to add another option to toggle feature.

What do you prefer? Do you want to enable metrics server by default?

[wongma7]

hmm okay I agree with you actually let's disable it by default

[wongma7]

/lgtm

[sandaymin123]

@cofyc and @wongma7
I want to clarify if these metrics for out-of-tree flex drivers are also supported:

kubelet_volume_stats_available_bytes
kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_inodes
kubelet_volume_stats_inodes_free
kubelet_volume_stats_inodes_used
kubelet_volume_stats_used_bytes``` with this PR?

[wongma7]

@sandaymin123 no, those stats are calculated by kubernetes for in-tree volumes that support the GetMetrics call. We will have to figure out a different solution for out-of-tree.