A VELOCLOUD KUBERNETES OPERATOR !
As Kubernetes is becoming the de-facto standard to run the new modern applications, there is a need to include the SDWAN configuration as part of the application development.
Let’s imagine a dev coding a new app. As part of the application testing, it can be useful to adapt the SDWAN policy, such as bandwidth, the QoS settings, the link steering and so on. The dev could use the same YAML files and kubectl commands that he loves and just includes the SDWAN parameters.
Obviously, the settings that the dev could use should be limited to a range that has been defined by infra admins. Some namespaces could consume more bandwidth than others, for example. The link steering policy could be limited to “Internet Only” for pre-prod namespaces, as another example.
In this post, I’m going to cover the creation of such integration. We’re going to create a Velocloud CRD (Custom Resource Definition), that the infra admins will be able to use to define the set of settings allowed for a dev. Then, we’ll create a Kubernetes Operator, which is going to talk with the Velocloud Orchestrator to implement the SDWAN logic. Finally, we’ll define some YAML samples to demonstrate how a dev can use such a Velocloud CRD.
So we need 2 things to start coding: a Velocloud Orchestrator and a Kubernetes 😉
For Kubernetes, I will use Kind as a very simple Kubernetes running on my laptop. The setup is really easy to do, and you can have a running Kube on your laptop after only a few minutes: https://kind.sigs.k8s.io/docs/user/quick-start/
I’m going the create 2 namespaces:
1adeleporte@adeleporte-a01 ns1 % kubectl create ns ns1
2namespace/ns1 created
3adeleporte@adeleporte-a01 ns1 % kubectl create ns ns2
4namespace/ns2 created
5adeleporte@adeleporte-a01 ns1 % kubectl get ns
6NAME STATUS AGE
7default Active 8m27s
8kube-node-lease Active 8m29s
9kube-public Active 8m29s
10kube-system Active 8m29s
11local-path-storage Active 8m23s
12ns1 Active 6s
13ns2 Active 5s
For the Velocloud Orchestrator, I will use a new feature, the API tokens. Once logged as a Administrator, you can now create/revoke some API tokens, which seems to be a good idea in my own opinion.
Velocloud CRD definition
Now, let’s start coding a very basic Velocloud CRD in a crd.yaml file:
1---
2apiVersion: apiextensions.k8s.io/v1beta1
3kind: CustomResourceDefinition
4metadata:
5 name: velocloudbps.vcn.cloud
6spec:
7 scope: Namespaced
8 group: vcn.cloud
9 version: v1
10 names:
11 kind: VelocloudBP
12 plural: velocloudbps
13 singular: velocloudbp
14 shortNames:
15 - velo
16---
17apiVersion: rbac.authorization.k8s.io/v1
18kind: Role
19metadata:
20 name: velocloud-group-admin
21rules:
22- apiGroups:
23 - vcn.cloud
24 resources:
25 - velocloudbps
26 - velocloudbps/finalizers
27 verbs: [ get, list, create, update, delete, deletecollection, watch ]
28
29---
30apiVersion: rbac.authorization.k8s.io/v1beta1
31kind: RoleBinding
32metadata:
33 name: velocloud-group-rbac
34subjects:
35- kind: ServiceAccount
36 name: default
37roleRef:
38 kind: Role
39 name: velocloud-group-admin
40 apiGroup: rbac.authorization.k8s.io
1adeleporte@adeleporte-a01 ns1 % kubectl apply -f crd.yaml -n ns1
2customresourcedefinition.apiextensions.k8s.io/velocloudbps.vcn.cloud created
Here is what we’ve done:
We have a new kind of CRD, called velocloudbps + a new role, called velocloud-group-admin, which can use get, list, create and watch commands, and this role is binded to the default service account.
By doing so, we now have some new kubectl commands:-)
1adeleporte@adeleporte-a01 ns1 % kubectl get velocloudbps
2No resources found.
3
4adeleporte@adeleporte-a01 ns1 % kubectl get velocloudbps
5No resources found.
6adeleporte@adeleporte-a01 ns1 % kubectl get velo
7No resources found.
Ok, not very useful right now, but we have our new Velocloud CRD! Let’s modify the initial crd.yaml file to include some validation. This will enable us (and the infra admins) to define which parameter can be set on this CRD. Let’s say that we want to define the Velocloud Configuration Profile that is going to be used, the FQDN of the application currently being developed by the dev, the TCP port, and the Business Policy settings (Link steering policy, QoS priority, Bandwidth Limit). We also define the column to be displayed when using commands such as “kubectl get velocloudbps -o wide”
1---
2apiVersion: apiextensions.k8s.io/v1beta1
3kind: CustomResourceDefinition
4metadata:
5 name: velocloudbps.vcn.cloud
6spec:
7 scope: Namespaced
8 group: vcn.cloud
9 version: v1
10 names:
11 kind: VelocloudBP
12 plural: velocloudbps
13 singular: velocloudbp
14 shortNames:
15 - velo
16 validation:
17 openAPIV3Schema:
18 properties:
19 spec:
20 type: object
21 properties:
22 profile:
23 type: string
24 description: profile
25 enum:
26 - Quick Start Profile
27 name:
28 type: string
29 description: name
30 fqdn:
31 type: string
32 description: fqdn
33 pattern: '^(.*).vcn.cloud$'
34 dport:
35 type: integer
36 description: dport
37 service-class:
38 type: string
39 description: sc
40 enum:
41 - realtime
42 - transactional
43 - bulk
44 link-policy:
45 type: string
46 description: lp
47 enum:
48 - auto
49 - fixed
50 service-group:
51 type: string
52 description: sg
53 enum:
54 - PRIVATE_WIRED
55 - PUBLIC_WIRED
56 - ALL
57 priority:
58 type: string
59 description: priority
60 enum:
61 - high
62 - normal
63 - low
64 bandwidth-limit:
65 type: integer
66 description: bpl
67 minimum: -1
68 maximum: 50
69additionalPrinterColumns:
70 - name: Profile
71 type: string
72 description: The profile
73 JSONPath: .spec.profile
74 - name: FQDN
75 type: string
76 description: The fqdn
77 JSONPath: .spec.fqdn
78 - name: Service Class
79 type: string
80 priority: 1
81 description: The service class
82 JSONPath: .spec.service-class
83 - name: Priority
84 type: string
85 priority: 1
86 description: The priority
87 JSONPath: .spec.priority
88 - name: Link Policy
89 type: string
90 priority: 1
91 description: The link Policy
92 JSONPath: .spec.link-policy
93 - name: Bandwidth Limit
94 type: integer
95 priority: 1
96 description: The bandwidth Limit
97 JSONPath: .spec.bandwidth-limit
Velocloud Kubernetes Operator
Finally, we need an operator. Basically, we want Kubernetes to run our Velocloud operator when a CRD is created/updated/deleted. For that, we’ll create a deployment of a pod with 2 containers: the operator itself (which is going to talk to the Velocloud Orchestrator) and a kubectl proxy (to handle authentication).
Let’s change again our crd.yaml file to include this deployment as part of the CRD definition:
1---
2apiVersion: apps/v1
3kind: Deployment
4metadata:
5 name: operator
6 labels:
7 app: operator
8spec:
9 selector:
10 matchLabels:
11 app: operator
12 template:
13 metadata:
14 labels:
15 app: operator
16 spec:
17 containers:
18 - name: proxycontainer
19 image: lachlanevenson/k8s-kubectl
20 command: ["kubectl","proxy","--port=8001"]
21 - name: app
22 image: adeleporte/velokube
23 env:
24 - name: res_namespace
25 valueFrom:
26 fieldRef:
27 fieldPath: metadata.namespace
28 - name: VCO_URL
29 value: https://vco22-fra1.velocloud.net/portal/rest
30 - name: VCO_TOKEN
31 value: changeme
32 - name: verbose
33 value: log
1adeleporte@adeleporte-a01 ns1 % kubectl apply -f crd.yaml -n ns1
2customresourcedefinition.apiextensions.k8s.io/velocloudbps.vcn.cloud created
At this stage, the Velocloud Kubernetes operator logic is defined. The adeleporte/velokube container (we’ll discuss about this container right after) is responsible for watching at velocloudbps CRD changes. This container will then implement the necessary changes over the Velocloud Orchestrator. Let’s now go deeper into the adeleporte/velokube container:
1import requests
2import warnings
3import json
4import os
5from copy import deepcopy
6import logging
7import sys
8
9# Env variables
10vco_url = os.getenv("VCO_URL")
11token = os.getenv("VCO_TOKEN")
12verbose = os.getenv("verbose")
13
14base_url = "http://127.0.0.1:8001"
15namespace = os.getenv("res_namespace", "default")
16
17# Logging
18log = logging.getLogger(__name__)
19out_hdlr = logging.StreamHandler(sys.stdout)
20out_hdlr.setFormatter(logging.Formatter('%(asctime)s %(message)s'))
21out_hdlr.setLevel(logging.INFO)
22log.addHandler(out_hdlr)
23log.setLevel(logging.INFO)
24
25
26def client(method, body):
27 warnings.filterwarnings('ignore', message='Unverified HTTPS request')
28 headers = {'Content-Type': 'application/json', 'Authorization': 'Token {}'.format(token)}
29 try:
30 resp = requests.post('{}/{}'.format(vco_url, method), data=json.dumps(body), headers=headers, verify=False)
31 if (resp.status_code == 200):
32 resp_parsed = json.loads(resp.text)
33 return resp_parsed
34 else:
35 log.info(resp.status_code)
36 return False
37 except:
38 log.info('Request failed')
39 return False
40
41
42def GetConfiguration(profilename):
43
44 profiles = client('enterprise/getEnterpriseConfigurationsPolicies', {})
45 for profile in profiles:
46 if profile['name'] == profilename:
47 config = client('configuration/getConfiguration', {"id": profile['id'], "with": ["modules"]})
48 return config
49
50 log.info('Failed to find Profile')
51 return False
52
53def GetQosModule(config):
54 for module in config['modules']:
55 if module['name'] == "QOS":
56 qos_module = module
57 return qos_module
58 break
59 log.info('Cant find Qos Module')
60 return None
61
62def DeleteQosRule(qos_module, spec):
63 # Look for rule to delete
64 index = 0
65 for rule in qos_module['data']['segments'][0]['rules']:
66 if rule['name'] == spec['name']:
67 qos_module['data']['segments'][0]['rules'].pop(index)
68 index = index+1
69
70 # Update the module
71 update = client('configuration/updateConfigurationModule', {"id": qos_module['id'], "_update": {"name": "QOS", "data": qos_module['data']}})
72
73 log.info('rule deleted')
74 return update
75
76def SetQosRule(qos_module, spec):
77 template_rule = None
78 for rule in qos_module['data']['segments'][0]['defaults']:
79 if rule['name'] == "Default-Any-Other":
80 template_rule = rule
81 break
82
83 if (template_rule == None):
84 log.info('Default rule Default-Any-Other not found')
85 return False
86
87 new_rule = deepcopy(template_rule)
88
89 # Name
90 new_rule['name'] = spec['name']
91
92 # Match
93 new_rule['match']['hostname'] = spec['fqdn']
94 new_rule['match']['dport_high'] = spec['dport']
95 new_rule['match']['dport_low'] = spec['dport']
96 new_rule['match']['proto'] = 6
97
98 # Service Class
99 new_rule['action']['QoS']['type'] = spec['service-class']
100
101 # Priority
102 new_rule['action']['QoS']['rxScheduler']['priority'] = spec['priority']
103 new_rule['action']['QoS']['txScheduler']['priority'] = spec['priority']
104
105 # Bandwidth Limit
106 new_rule['action']['QoS']['rxScheduler']['bandwidthCapPct'] = spec['bandwidth-limit']
107 new_rule['action']['QoS']['txScheduler']['bandwidthCapPct'] = spec['bandwidth-limit']
108
109 # Link Steering
110 new_rule['action']['edge2CloudRouteAction']['linkPolicy'] = spec['link-policy']
111 new_rule['action']['edge2CloudRouteAction']['serviceGroup'] = spec['service-group']
112 new_rule['action']['edge2DataCenterRouteAction']['linkPolicy'] = spec['link-policy']
113 new_rule['action']['edge2DataCenterRouteAction']['serviceGroup'] = spec['service-group']
114 new_rule['action']['edge2EdgeRouteAction']['linkPolicy'] = spec['link-policy']
115 new_rule['action']['edge2EdgeRouteAction']['serviceGroup'] = spec['service-group']
116
117 # Look if rule already exists
118 index = 0
119 for rule in qos_module['data']['segments'][0]['rules']:
120 if rule['name'] == spec['name']:
121 qos_module['data']['segments'][0]['rules'][index] = new_rule
122 update = client('configuration/updateConfigurationModule', {"id": qos_module['id'], "_update": {"name": "QOS", "data": qos_module['data']}})
123 log.info('rule updated')
124 return update
125 index = index+1
126
127 # Rule not found, insert it at the top
128 qos_module['data']['segments'][0]['rules'].insert(0, new_rule)
129 update = client('configuration/updateConfigurationModule', {"id": qos_module['id'], "_update": {"name": "QOS", "data": qos_module['data']}})
130 log.info('rule added')
131 return update
132
133
134def main():
135 config = GetConfiguration()
136 qos_module = GetQosModule(config)
137 update = SetQosRule(qos_module, fqdn)
138
139
140def event_loop():
141 log.info("Starting the service")
142 url = '{}/apis/vcn.cloud/v1/namespaces/{}/velocloudbps?watch=true"'.format(
143 base_url, namespace)
144 r = requests.get(url, stream=True)
145 # We issue the request to the API endpoint and keep the conenction open
146 for line in r.iter_lines():
147 obj = json.loads(line)
148 # We examine the type part of the object to see if it is MODIFIED
149 if (verbose == 'log'):
150 log.info(obj)
151
152 config = GetConfiguration(obj['object']['spec']['profile'])
153 qos_module = GetQosModule(config)
154
155 if (obj['type'] == 'ADDED'):
156 SetQosRule(qos_module, obj['object']['spec'])
157 if (obj['type'] == 'MODIFIED'):
158 SetQosRule(qos_module, obj['object']['spec'])
159 if (obj['type'] == 'DELETED'):
160 DeleteQosRule(qos_module, obj['object']['spec'])
161
162
163event_loop()
The code is written is Python. The interesting parts are:
- we use a Stream Request to watch at velocloudbps events (/apis/vcn.cloud/v1/namespaces/{}/velocloudbps?watch=true)
- this socket acts as a loop and generates a line for each event
- when a new event comes, we decode the details (obj = json.loads(line))
- depending of the type of event (creation/modification/deletion) (obj[‘type’]), we use a different Python function
- the Velocloud Business Profile is stored inside a Configuration module (the profile) and a Qos module (the business policies), so we have a implement some code to fetch these values
- in case of creation, we copy the default QoS rule, change some parameters according to the CRD, and update the QoS module
This code is copied in a Python container and uploaded to docker hub. Here is the docker file to generate the adeleporte/velokube container:
1FROM python:latest
2RUN pip install requests
3COPY operator/main.py /main.py
4ENTRYPOINT [ "python", "/main.py" ]
If everything works as expected, you should see your deployment as ok in Kube:
1adeleporte@adeleporte-a01 ns1 % kubectl get deployments -n ns1
2NAME READY UP-TO-DATE AVAILABLE AGE
3operator 1/1 1 1 96m
4adeleporte@adeleporte-a01 ns1 % kubectl describe deployment operator -n ns1
5Name: operator
6Namespace: ns1
7CreationTimestamp: Tue, 21 Jul 2020 14:02:31 +0200
8Labels: app=operator
9Annotations: deployment.kubernetes.io/revision: 1
10 kubectl.kubernetes.io/last-applied-configuration:
11 {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"operator"},"name":"operator","namespace":"ns1"},...
12Selector: app=operator
13Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
14StrategyType: RollingUpdate
15MinReadySeconds: 0
16RollingUpdateStrategy: 25% max unavailable, 25% max surge
17Pod Template:
18 Labels: app=operator
19 Containers:
20 proxycontainer:
21 Image: lachlanevenson/k8s-kubectl
22 Port: <none>
23 Host Port: <none>
24 Command:
25 kubectl
26 proxy
27 --port=8001
28 Environment: <none>
29 Mounts: <none>
30 app:
31 Image: adeleporte/velokube
32 Port: <none>
33 Host Port: <none>
34 Environment:
35 res_namespace: (v1:metadata.namespace)
36 VCO_URL: https://vco22-fra1.velocloud.net/portal/rest
37 VCO_TOKEN: changeme
38 verbose: log
39 Mounts: <none>
40 Volumes: <none>
41Conditions:
42 Type Status Reason
43 ---- ------ ------
44 Progressing True NewReplicaSetAvailable
45 Available True MinimumReplicasAvailable
46OldReplicaSets: <none>
47NewReplicaSet: operator-5bf4b944cd (1/1 replicas created)
48Events: <none>
49adeleporte@adeleporte-a01 ns1 %
50adeleporte@adeleporte-a01 ns1 % kubectl get pods -n ns1
51NAME READY STATUS RESTARTS AGE
52operator-5bf4b944cd-tnfh9 2/2 Running 2 98m
53adeleporte@adeleporte-a01 ns1 %
54adeleporte@adeleporte-a01 ns1 % kubectl logs operator-5bf4b944cd-tnfh9 app -n ns1
552020-07-21 13:14:51,674 Starting the service
56adeleporte@adeleporte-a01 ns1 %
Using the Velocloud Kubernetes Operator
Everything is done! Let’s test our new Velocloud Kubernetes Operator 😉 So now, I’ll acting as a dev, let’s create a velocloud.yaml as part of my application manifest:
1---
2apiVersion: vcn.cloud/v1
3kind: VelocloudBP
4metadata:
5 name: velokube1
6spec:
7 profile: Quick Start Profile
8 name: velokube1
9 fqdn: velokube1.vcn.cloud
10 dport: 8080
11 service-class: realtime
12 link-policy: auto
13 service-group: ALL
14 priority: high
15 bandwidth-limit: -1
16---
17apiVersion: vcn.cloud/v1
18kind: VelocloudBP
19metadata:
20 name: velokube2
21spec:
22 profile: Quick Start Profile
23 name: velokube2
24 fqdn: velokube2.vcn.cloud
25 dport: 443
26 service-class: realtime
27 link-policy: fixed
28 service-group: PUBLIC_WIRED
29 priority: high
30 bandwidth-limit: 40
1adeleporte@adeleporte-a01 ns1 % kubectl create -f velocloud.yaml --namespace ns1
2velocloudbp.vcn.cloud/velokube1 created
3velocloudbp.vcn.cloud/velokube2 created
4adeleporte@adeleporte-a01 ns1 %
5adeleporte@adeleporte-a01 ns1 % kubectl get velocloudbps -n ns1
6NAME PROFILE FQDN
7velokube1 Quick Start Profile velokube1.vcn.cloud
8velokube2 Quick Start Profile velokube2.vcn.cloud
9adeleporte@adeleporte-a01 ns1 % kubectl get velocloudbps -n ns1 -o wide
10NAME PROFILE FQDN SERVICE CLASS PRIORITY LINK POLICY BANDWIDTH LIMIT
11velokube1 Quick Start Profile velokube1.vcn.cloud realtime high auto -1
12velokube2 Quick Start Profile velokube2.vcn.cloud realtime high fixed 40
13adeleporte@adeleporte-a01 ns1 %
For my application velokube1.vcn.cloud, I want to have a RealTime Service Class, use all the available links, have a high QoS priority and no bandwidth limit.
For my application velokube2.vcn.cloud, I want to have a RealTime Service Class, use only Internet Links, have a High QoS priority and a bandwidth limit set to 40%.
Let’s check the Velocloud configuration:
And the operator logs
1deleporte@adeleporte-a01 ns1 % kubectl logs operator-5bf4b944cd-tnfh9 app -n ns1
22020-07-21 13:14:51,674 Starting the service
32020-07-21 13:47:05,250 {'type': 'ADDED', 'object': {'apiVersion': 'vcn.cloud/v1', 'kind': 'VelocloudBP', 'metadata': {'creationTimestamp': '2020-07-21T13:47:05Z', 'generation': 1, 'name': 'velokube1', 'namespace': 'ns1', 'resourceVersion': '18703', 'selfLink': '/apis/vcn.cloud/v1/namespaces/ns1/velocloudbps/velokube1', 'uid': 'b2767e70-642e-4e4b-aad8-cd78fd089b7b'}, 'spec': {'bandwidth-limit': -1, 'dport': 8080, 'fqdn': 'velokube1.vcn.cloud', 'link-policy': 'auto', 'name': 'velokube1', 'priority': 'high', 'profile': 'Quick Start Profile', 'service-class': 'realtime', 'service-group': 'ALL'}}}
42020-07-21 13:47:05,629 rule added
52020-07-21 13:47:05,629 {'type': 'ADDED', 'object': {'apiVersion': 'vcn.cloud/v1', 'kind': 'VelocloudBP', 'metadata': {'creationTimestamp': '2020-07-21T13:47:05Z', 'generation': 1, 'name': 'velokube2', 'namespace': 'ns1', 'resourceVersion': '18704', 'selfLink': '/apis/vcn.cloud/v1/namespaces/ns1/velocloudbps/velokube2', 'uid': '646346aa-d8b5-4ef3-bbb0-c8fb0ff7b062'}, 'spec': {'bandwidth-limit': 40, 'dport': 443, 'fqdn': 'velokube2.vcn.cloud', 'link-policy': 'fixed', 'name': 'velokube2', 'priority': 'high', 'profile': 'Quick Start Profile', 'service-class': 'realtime', 'service-group': 'PUBLIC_WIRED'}}}
62020-07-21 13:47:05,997 rule added
7adeleporte@adeleporte-a01 ns1 %
Now let’s imagine that I want to change my mind for application2. I change some settings in the YAML file (Link Policy Public to Private, Priority High to Low, Bandwidth Limit from 40 to 10)
1---
2apiVersion: vcn.cloud/v1
3kind: VelocloudBP
4metadata:
5 name: velokube2
6spec:
7 profile: Quick Start Profile
8 name: velokube2
9 fqdn: velokube2.vcn.cloud
10 dport: 443
11 service-class: realtime
12 link-policy: fixed
13 service-group: PRIVATE_WIRED
14 priority: low
15 bandwidth-limit: 10
1
2adeleporte@adeleporte-a01 ns1 % kubectl apply -f velocloud.yaml --namespace ns1
3velocloudbp.vcn.cloud/velokube1 unchanged
4velocloudbp.vcn.cloud/velokube2 configured
5adeleporte@adeleporte-a01 ns1 %
Set some limits to devs
Now, let’s imagine that I want to change the bandwidth limit to 75%
1---
2apiVersion: vcn.cloud/v1
3kind: VelocloudBP
4metadata:
5 name: velokube2
6spec:
7 profile: Quick Start Profile
8 name: velokube2
9 fqdn: velokube2.vcn.cloud
10 dport: 443
11 service-class: realtime
12 link-policy: fixed
13 service-group: PRIVATE_WIRED
14 priority: low
15 bandwidth-limit: 75
1velocloudbp.vcn.cloud/velokube1 unchanged
2The VelocloudBP "velokube2" is invalid: spec.bandwidth-limit: Invalid value: 50: spec.bandwidth-limit in body should be less than or equal to 50
3adeleporte@adeleporte-a01 ns1 %
As a developer, I’m not allowed to set more than 50% of bandwidth, because this limit has been set by the infra team when defining the CRD.
As a result, the infra team can define very specific limits for each namespace and developers are free to set the Velocloud configutation only within these limit.
Conclusion
I hope that this post will give you some insights about what can be achieved when integrating such great solutions together, like Velocloud and Kubernetes. The infra/network/wan teams can now work together with the dev team to align infrastructure needs and business/dev agility.