We always want to automate things and while Kubernetes already has a lot of unique features, the autoscaling part is missing. Perhaps I haven’t looked at the latest releases and I’m already missing something.
Here I’d like to outline my findings and the way I did the autoscaling on Kubernetes.
Requirements for the setup:
- Patience
- Working Kubernetes cluster
The setup involves time-series database InfluxDB and Heapster for monitoring and some custom scripting for evaluation and scaling.
For the application, I used simple WordPress container that had one container running by default. The replication controller config looked like this.
apiVersion: v1beta3
kind: ReplicationController
metadata:
labels:
name: wordpress
name: wordpress
spec:
replicas: 1
selector:
name: wordpress
template:
metadata:
labels:
name: wordpress
spec:
containers:
- image: wordpress
name: wordpress
env:
- name: WORDPRESS_DB_PASSWORD
value: yourpassword
ports:
- containerPort: 80
name: wordpress
For the WordPress service, we add configuration into Heapster. I did the following:
"targets": [
{
"target": "randomWalk('random walk')",
"function": "derivative",
"column": "value",
"series": "cpu/usage_ns_cumulative",
"query": "select container_name, derivative(value) from
\"cpu/usage_ns_cumulative\" where $timeFilter and labels =~ /name:wordpress/ group by
time($interval), container_name fill(0) order asc",
"condition_filter": true,
"interval": "5s",
"groupby_field_add": true,
"groupby_field": "container_name",
"alias": "",
"condition": "labels =~ /name:wordpress/",
"rawQuery": false,
"fill": "0"
}
],
With this configuration I was able to monitor all containers that belonged to the WordPress service in Kubernetes which dynamically allocated the total capacity and the used capacity of all containers of WordPress application.
After that I used httperf to see if it actually works. I could see the following graph in Grafana.
Now the interesting part comes with getting the state of the container group.
I managed to get the data of last 5 minutes with curl.
curl -G 'http://ip_address:8086/db/k8s/series?u=root&p=root&pretty=true' --data-urlencode
"q=select container_name, derivative(value)from \"cpu/usage_ns_cumulative\" where labels =~
/name:wordpress/ and time > now() - 5m group by time(5s)"
curl -G 'http://ip_address:8086/db/k8s/series?u=root&p=root&pretty=true' --data-urlencode
"q=select container_name, mean(value) from \"memory/usage_bytes_gauge\" where labels =~
/name:wordpress/ and time > now() - 5m group by time(5s)"
To use the data, in simple PHP script, I did the following:
$options = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_SSL_VERIFYHOST => 2,
CURLOPT_HTTPGET => true,
);
curl_setopt_array($curl, $options);
$result = curl_exec($curl);
$result = json_decode($result, true);
echo $result[0]['points'][0][1]
Now the last point is to trigger the autoscaling. For that I used cron to schedule the check every 5 minutes with a simple shell script that would have the thresholds defined and operations with kubectl utility.
Basically, it takes the data from Heapster and compares to the defined threshold.
mb=1048576
57
mem=$((mem / mb))
getcurrent=`kubectl describe rc wordpress > /tmp/wprc`
replicas=`grep "Replicas" /tmp/wprc | awk '{print $2}'`
triggerpoint=500
currentstatus=$((mem * replicas))
triggerneeded=$((triggerpoint * replicas))
new=$((replicas+1))
if [ $currentstatus -gt $triggerneeded ]; then
echo "Memory usage above threshold, adding replica" \
`/opt/bin/kubectl resize rc wordpress --replicas=$new`
else
echo "Nothing to do this time"
fi
I used httperf tool to simulate user requests in order to verify our autoscaler works. During the test we succeeded with the autoscaling script and new containers were scheduled.
Name: wordpress
Image(s): wordpress
Selector: name=wordpress
Labels: name=wordpress
Replicas: 3 current / 3 desired
Pods Status: 2 Running / 1 Waiting / 0 Succeeded / 0 Failed
That’s it, the dirty autoscaling works on Kubernetes. I not a developer therefore I could not really go deep with coding, nevertheless, perhaps someone will find it interesting to pick up as a starting point to play with Kubernetes autoscaling.
Cheers
itilbox 