Introduction to kubernetes

Kubernetes becomes defacto standard runtime environment for modern cloud-native applications. As a developer, you should understand the basic concepts behind and operate the application reasonably in DevOps mode. This post aims to explain basic concepts and provide a solid grounding you can further improve your knowledge and prepare you for day-to-day work and potential interview questions. This post is organized as a series of closed topics where understanding those areas should allow you to answer a range of questions. Kubernetes is a complex system, so we simplify things. If you want to prepare for Kubernetes Certified Developer (KCD), this post can be a stepping stone.

What is Kubernetes

Kubernetes is an open-source container orchestration tool for managing multi-container (e.g. by docker) applications. It is in some sense similar to docker swarm or loosely related to docker-compose. Kubernetes was initially developed by Google and released as open-source in 2014.

Simplified kubernetes architecture diagram

After reading this section, you should be able to answer the following questions:

• What are the components of the kubernetes cluster?
• What are the row responsibilities of those components?

We are not going into great detail to provide a high-level context as this post is not aimed at Kubernetes administrators but rather developers.

Master node

The master node is responsible for the management of the cluster and is running control plane components. Kube-apiserver expose APIs for management. Controller managers consist of two distinct types: cloud and Kube related. Cloud controller embed a cloud-specific control logic while Kube related take care of kubernetes cluster related things, e.g. node statuses, the job runs, access token creations etc. All data are stored in etcd key-value store. Kube-scheduler is responsible for newly created pods and select nodes to run them. It takes into consideration various limitations and specifications for a given pod. 

Worker Nodes

Worker nodes are running workloads and consist of Kublet and kube-proxy. Kube-proxy is a network proxy running on each node in the cluster and is responsible for network communication according to configuration. Kublet runs on each node as well, and his responsibility is to make sure that pod containers are running and healthy.

Kubernetes basic concepts

After reading this section, you should be able to answer the following questions:

• What is kubernetes manifest?
• What is a kubernetes pod?
• What is a control manager?
• How kubernetes control loop operate?

Kubernetes control loop – the heart of the kubernetes

Kubernetes is based on the modern declarative DevOps principle that you describe the desired state of things rather than a set of steps that lead to that state. At the heart of Kubernetes, there is a control loop that constantly evaluates a current state versus a desired state and requests appropriate corrections that should lead towards it. The desired state in the Kubernetes is described as a Kubernetes manifests, a bunch of YAML files, that describes a state for various kubernetes objects called kinds, e.g. deployment. Controllers continuously execute the control loop and drive changes, usually through an apiserver. The controller is typically responsible for one or more kubernetes types/kinds. The basic kind which acts as a unit of scalability in kubernetes is called a pod. The pod can consist of one or more containers with specified parameters. 

Kubernetes object kinds

After reading this section, you should be able to answer the following questions:

• What is the difference between service and ingress?
• How are kubernetes objects organized?
• What is a daemonset good for?
• What is the difference between deployment and stateful set?
• How to store sensitive information in the cluster?
• What are lower-level kubernetes objects? 

Basic Kubernetes object kinds

Kubernetes deployment consists of various abstractions over infrastructure that are described in kubernetes as object kinds. They can be divided according to different criteria:

• High-level kinds vs lower level kinds
• Stateful vs stateless
• Ephemeral vs permanent kinds 

The image above describes the basic kinds used in kubernetes deployment. As mentioned earlier, the basic unit of scalability and deployment is a pod. Pod represents a set of containers with associated resources granted to them. Because a pod is a transient object that can be freely moved in the cluster, e.g. by re-scheduling, kubernetes provide a service object used as a permanent locator/address within a cluster (implements a service locator pattern). Service can be represented by, for example, a load balancer. The sole purpose of ingress is to reach the pods running in the cluster from the outside world. Ingress is typically represented by cloud global HTTPS load balancer and/or Nginx proxy server or similar. Trafic is further routed based on the host header to the appropriate service within the cluster. As the pod is transient, so the container does. To provide persistent storage to a container, the concept of persistent volume claim (pvc) addresses this requirement. ConfigMap object provides a set of “static” files, e.g. for container configuration purposes. In comparison, secrets offer a way how to access sensitive information from containers.

Those lower-level objects can be further abstracted by kubernetes higher-level kinds like deployment or stateful sets, which removes a need to specify replica sets. Deployment is primarily aimed for stateless applications, while stateful set provides ways to maintain pod identity through pod restarts or relocation, e.g. moving storage around or keeping communication to a given pod.

Different runtime requirements

To address the different needs for applications runtime, kubernetes introduces a few more concepts: Kubernetes jobs and deamon sets. A deamon set is a special kind of deployment pod that runs on every VM machine from a node pool. This kind of deployment is typically used for VM metrics collection etc. Kubernetes jobs provide a way how to run a single task in the kubernetes cluster. For example, spinnaker custom stages are managed this way.

Object versioning

Nearly every object in kubernetes has its version. So it is possible to restore any version running by activating those. Unfortunately, the only exceptions to those rules from the basic objects mentioned above are ConfigMaps and Secrets (if I am not mistaken). Restoring a deployment version without a corresponding version of configuration is one of the big shortcomings, and in my opinion, the decision was unlucky. There are techniques how to deal with that.

Interaction with kubernetes cluster, managing kubernetes deployment

After reading this section, you should be able to answer the following questions:

• What is a key tool for interaction with the cluster?
• What are the basic operational models/deployments?
• How do you manage resources?
• What is the difference between liveness and readiness probe?

Kube control

When interacting with the Kubernetes cluster directly, the key tool is Kube control (kubectl). Kubectl is a command-line utility that interacts with a cluster api server. It allows you to add or remove deployment, get the version of any manifest file. It also provides a set of administrative commands.

Using kubectl, you can drive a deployment as a rolling upgrade ( as a default scenario), a more advanced procedure via replica controllers or just simple scaling up or down pods (as shown in the example).

kubectl scale --replicas=3 deployment/mysql

Resource management

Every pod specification contains a section of manifest dedicated to resources. The resource requests section specifies resources necessary for starting the pod containers, which the scheduler uses to plan a deployment. The resource limit section defines resources that cannot be exceeded, and the pod will get killed or throttled. This is a fundamental mechanism for configuring resources from bursty to guaranteed load and one way that affects the quality of service.

    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Kubernetes probes

Every pod is evaluated in the kubernetes control loop to correct potential differences from the desired state. To make sure that the pod is ok kubernetes use liveness and readiness probes mechanism. The liveness probe checks whether your application is up and running. If not, kubernetes will kill and restart the pod. The readiness probe checks whether the pod can accept and process new requests. As a software developer, you should provide such endpoints that correctly implement the logic to provide such insight. Only with correctly implemented endpoints, you can achieve a zero-downtime deployment. 

Kubernetes advanced topics

After reading this section, you should be able to answer the following questions:

• How do you govern application uptime? ~ pod disruption budget
• How do you adopt an application to various load scenarios? ~ autoscaling
• How do you isolate applications from each other?
• How to achieve an advanced deployment strategy?

Advanced kubernetes kinds

A lower-level concept called Replica Set is responsible for scaling out a pod to handle more traffic. This concept is a key for basic and more advanced deployment strategies like rolling upgrade (basic) or blue-green deployment. By extension of this capability is to provide automatic scalability using a horizontal pod autoscaler, where the server changes the number of replicas for a given pod. Contrary to guaranteeing some application availability running in a pod, you can define a pod disruption budget that describes the rules of how pods can be changed.

Kubernetes deployment isolations

Every deployed object to kubernetes is deployed to a namespace which acts as a logical separation of deployments in a single cluster. If a namespace is not specified, then the default namespace is used. System namespaces are also created, e.g. kube-system or kube-public. Those are used for deployments supporting kubernetes cluster operations. 

To isolate deployments on the network level, you need to specify a network policy. Network policy allows you to restrict how a pod is allowed to communicate with various network entities. 

Conclusion

In this post, we briefly touched all core concepts in Kubernetes that developers should be aware of to build a secure, stable and resilient solution. The information loads this post, and we often oversimplified the concept to make it digestible for newcomers. Please, let me know if I missed some vital concept or oversimplified something that deserves a more detailed explanation or is not clear enough. You can leave a comment here or reach me on Twitter

Terraform introduction

HashiCorp Terraform is a popular tool for managing your cloud infrastructure as code (IaC) in a cloud-agnostic way (same tool for various cloud platforms). Instead of unifying all capabilities for different cloud platforms, the core concepts are exposed to the end-user via Terraform provider concept. Terraform offers providers for all major cloud vendors and other cloud services and technologies as well, e.g. Kubernetes.  

This blog post doesn’t aim to be an introduction to Terraform concepts (official documentation is quite ok) but instead sharing an experience with using a Terraform in a distributed team, tools that come in handy and all things that make life easier. Even though HashiCorp offers Terraform Enterprise this option is used quite rarely at least on “small/er” projects so we won’t be discussing this option here. I openly admit that I have zero experience with this service so I cannot objectively compare. I will solely focus on using the open-sourced part of the project and Terraform version 0.12.x and higher. 

Terraform state

Terraform maintains the state of the infrastructure that manages in the state file. Format of Terraform state file is version dependant without strict rules on version compatibility (at least I wasn’t able to find one that was reliably followed and guaranteed). Managing state files poses two main challenges:
1) manage/share state across the team
2) control/align the version used across the team

The first aspect, different teams solve differently. Some commit state file alongside the configuration to the version control system which is far from ideal as there might be multiple copies of such resources across the team and requires some team coordination. On top of that, state file contains sensitive information which is impossible to mask and such doesn’t belong to the source control system. A lot better approach is using Terraform remote backend, which allows a true concurrent approach. Capabilities depend on the concrete implementation used. The backend can be changed from local to remote easily as is migrated automatically. The only limitation is that merging and splitting state file is allowed only for the locally managed state. 

Terraform version lock

Managing Terraform version management is centred around providing frictionless version upgrades for different Terraform configurations that align across the team with assuring that state file won’t get upgraded accidentally. To make sure that your state file won’t get upgraded accidentally put version restriction to every configuration managed e.g.

terraform {
  required_version = "0.12.20"
}

TFENV introduction

To align the team on a uniform Terraform version for every single configuration managed use tool for Terraform version management, e.g. tfenv. Put the desired version of to .terraform-version file located in the folder together with configuration. Tfenv automatically switches to appropriate version as needed, when new version encountered you need to run tfenv install to download a new version. If you want to check version available:

$ tfenv list
* 0.12.20 (set by /Users/jakub/test/terraform/.terraform-version)
  0.12.19
  0.12.18

As the number of resources or organisation grows so does the state file. Which leads to increased time for configuration synchronisation and competing for a lock on a remote state file. To increase the throughput and allow team DevOps mode(clear ownership of the solution from end to end), you might want to divide the infrastructure and associated state files into smaller chunks with clear boundaries. To keep your configuration DRY hierarchical configuration tools like Terragrunt comes to the rescue and reduce repetition. 

Terrafrom code validation and formatting

A growing number of users poses challenges as well as benefits which are the same as on application code written in, e.g. java or any other programming language. What is the real motivation for Infrastructure as a Code (IaC). How to setup standards and best practices on the project? Terraform offers a bunch of tools embedded. To make sure that code is properly formatted according to standards use fmt utility, which is pluggable to your CI/CD pipeline or pre-commit hooks.

terraform fmt --recursive --check --diff

For your re-usable Terraform modules it is good to make sure they are valid though it doesn’t catch all the bugs as it doesn’t checks against cloud APIs, so it doesn’t replace integration tests.

terraform validate

Terraform operation

Getting an idea of what will change, diff of your current infrastructure against proposed changes can be easily achieved via the generated plan

terraform plan

Static security scanning – infrastructure security

Enforcing security and standards are a lot easier on IaC as you can use tools like tflint or checkov which allows writing custom policies. We conclude the tool section with awesome Terraform tools which provide a great source if you are looking for something specific.

Conclusion

In this blog post, we just scratched the surface of Terraform tooling and completely skipped design and testing, which are topics for separate posts. What are your favourite tools? What did you find really handy? Leave a comment, share your tips or you can ask me on twitter.

JIT watch introduction

Java HotSpot JITWatch configurations

-XX:+UnlockDiagnosticVMOptions

-XX:+LogCompilation

-XX:+TraceClassLoading

[info] 0x0000000103e5473d: test %r13,%r13
[info] 0x0000000103e54740: jne 0x0000000103e5472a
[info] 0x0000000103e54742: mov $0xfffffff6,%esi
[info] 0x0000000103e54747: mov %r14d,%ebp
[info] 0x0000000103e5474a: nop
[info] 0x0000000103e5474b: callq 0x0000000103d431a0 ; OopMap{off=112}
[info] ;*invokevirtual inc
[info] ; - com.jaksky.jvm.tests.jit.IncWhile::testJit@12 (line 19)
[info] ; {runtime_call}
[info] 0x0000000103e54750: callq 0x0000000102e85c18 ;*invokevirtual inc
[info] ; - com.jaksky.jvm.tests.jit.IncWhile::testJit@12 (line 19)
[info] ; {runtime_call}
[info] 0x0000000103e54755: xor %r13d,%r13d

JITWatch

jakub@MBook ~/Development/GIT_REPO (master) $ ./jarScan.sh --mode=maxMethodSize --limit=35 ./chess-challenge/target/scala-2.12/classes/
"cz.jaksky.chesschallenge","ChessChallange$","delayedEndpoint$cz$jaksky$chesschallenge$ChessChallange$1","",1281
"cz.jaksky.chesschallenge.solver","ChessBoardSolver$","placeFigures$1","scala.collection.immutable.List,scala.collection.immutable.Set",110
"cz.jaksky.chesschallenge.solver","ChessBoardSolver$","visualizeSolution","scala.collection.immutable.Set,int,int",102
"cz.jaksky.chesschallenge.domain","Knight","check","cz.jaksky.chesschallenge.domain.Position,cz.jaksky.chesschallenge.domain.Position",81
"cz.jaksky.chesschallenge.domain","Queen","equals","java.lang.Object",73
"cz.jaksky.chesschallenge.domain","Rook","equals","java.lang.Object",73
"cz.jaksky.chesschallenge.domain","Bishop","equals","java.lang.Object",73
"cz.jaksky.chesschallenge.domain","Knight","equals","java.lang.Object",73
"cz.jaksky.chesschallenge.domain","King","equals","java.lang.Object",73
"cz.jaksky.chesschallenge.domain","Position","Position","int,int",73
"cz.jaksky.chesschallenge.domain","Position","equals","java.lang.Object",72

Conclusion

Working on the next project using again awesome Apache Kafka and again fighting against a fundamental misunderstanding of the philosophy of this technology which probably usually comes from previous experience using traditional messaging systems. This blog post aims to make the mindset switch as easy as possible and to understand where this technology fits in. What pitfalls to be aware off and how to avoid them. On the other hand, this article doesn’t try to cover all or goes into much detail.

How Kafka compares to traditional messaging systems

Apache Kafka is system optimized for writes – essentially to keep up with whatever speed or amount producer sends. This technology can be configured to meet any required parameters. That is one of the motivations behind naming this technology after famous writer Franz Kafka. If you want to understand the philosophy of this technology you have to take a look with a fresh eye. Forget what you know from JMS, RabbitMQ, ZeroMQ, AMQP and others. Even though the usage patterns are similar internal workings are completely different – the opposite. Following table provides a quick comparison

Kafka queue

Core ideas in Apache Kafka come from RDBMS. I wouldn’t describe Kafka as a messaging system but rather as a distributed database commit log which in order to scale can be partitioned. Once the information is written to the commit log everybody interested can read it at its own pace and responsibility. It is consumers responsibility to read it not the responsibility of the system to deliver the information to the consumer. This is the fundamental twist. Information stays in the commit log for a limited time given by retention policy applied. During this period it can be consumed even multiple times by consumers. As the system has reduced set of responsibilities it is much easier to scale. It is also really fast – as sequence read from the disk is similar to random access memory read thanks to effective file system caching.

Kafka scalability

Topic partition is a basic unit of scalability when scaling out Kafka. Message in Kafka is simple key-value pair represented as byte arrays. When message producer is sending a message to Kafka topic a client partitioner decides to which topic partition message is persisted based on message key. It is a best practice that messages that belong to the same logical group are sent to the same partition.  As that guarantee clear ordering. On the client side, exact position of the client is maintained on per topic partition bases for the assigned consumer group. So point to point communication is achieved by using exactly the same consumer group id when clients are reading from the topic partition. While publish-subscribe is achieved by using distinct consumer group id for each client to topic partition. The offset is maintained for consumer group id and topic partition and can be reset if needed.

Topic partitions can be replicated zero or n times and distributed across the Kafka cluster. Each topic partition has one leader and zero or n followers depends on replication factor. The leader maintains so-called In Sync Replicas (ISR) defined by delay behind the partition leader is lower than replica.lag.max.ms. Apache Zookeeper is used for keeping metadata and offsets.

• acknowledge – broker acknowledge to producer message write
• commit – the message is written to all ISR and consumer can read

• 0 – producer doesn’t wait for confirmation
• 1 – wait for acknowledge from the leader
• ALL – wait for acknowledge from all ISR ~ message commit

Apache Kafka configuration options

Apache Kafka is quite flexible in configuration and as such, it can meet many different requirements in terms of throughput, consistency and scalability. Replication of topic partition brings read scalability on the consumer side but also poses some risk as it is some additional level of complexity to achieve this. If you are unaware of those corner cases it might lead to nasty surprises, especially for newcomers. So let’s take a closer look at following scenario.

Loosing messages scenario

We have topic partition with a replication factor 2. Producer requires highest consistency level, set to ack = all. Replica 1 is currently the leader. Message 10 is committed hence available to clients. Message 11 is not acknowledged nor committed due to the failure of replica 3. Replica 3 will be eliminated from ISR or put offline. That causes that message 11 becomes acknowledged and committed.

Is that a problem? Well, the correct answer is: It depends. There are scenarios where this behaviour is perfectly fine. Imagine collecting logs from all machines via sending them through Kafka. On the other hand, if we implement event sourcing and we just lost some events that we cannot recreate the application state correctly. Yes, we have a problem! Unfortunately, if that doesn’t change in latest releases, that is default configuration if you just install new fresh Kafka cluster. It is a set up which favour availability and throughput over other factors. But Kafka allows you to set it up in a way that it meets your requirements for consistency as well but will sacrifice some availability in order to achieve that (CAP theorem). To avoid the described scenario you should use the following configuration. The producer should require acknowledging level ALL. Do not allow kafka perform a new leader election for dirty replicas – use settings unclean.leader.election.enable = false. Use replication factor (default.replication.factor = 3) and require minimal number of replicas to be in sync state to higher than 1 (min.insync.replicas = 2).

Message delivery quarantees

We already quickly touched the topic of message delivery to the consumer. Kafka doesn’t guarantee that message was delivered to all consumers. It is the responsibility of the consumers to read messages. So there is no semantics of persistent message as known from traditional messaging brokers. All messages sent to Kafka are persistent meaning available for consumption by clients according to the retention policy. Retention policy essentially specifies how long the message will be available in Kafka. Currently, there are two basic concepts – limited by space used for keeping messages or time for which the message should be at least available. The one which gets violated first wins.

Data cleanup

When I need to clean the data from the Kafka (triggered by retention policy) there are two options. The simplest one just deletes the message. Or I can compact messages. Compaction is a process where for each message key is just one message, usually the latest one. That is actually the second semantics of key used in the message.

Kafka “missing” features

What features you cannot find in Apache Kafka compared to traditional messaging technologies? Probably the most significant is an absence of any selector in combination with listening (wake me on receive). For sure can be implemented via correlation id, but efficiency is on the completely different level. You have to read all messages, deserialize those and filter. Compared to a traditional selector which uses the custom field in message header where you don’t need even to deserialize message payload that is on the completely different level. Monitoring Kafka on production environment essentially concerns elementary question: Are the consumers fast enough? Hence monitoring consumers offsets with respect to the retention policy.

Kafka was created on LinkedIn to solve a specific problem of modern data-driven application to fill the gap in traditional ETL processes usually working with flat files and DB dumps. It is essentially enterprise service bus for data where software components need exchange data heavily. It unifies and decouples data exchange among components. Typical uses are in “BigData” pipeline together with Hadoop and Spark in lambda or kappa architecture.  It lays down foundations of modern data stream processing.

Conclusion

This post just scratches basic concepts in Apache Kafka. If you are interested in details I really suggest to read following sources which I found quite useful on my way when learning Kafka:

• This article  describes core ideas behind Kafka
• Official documentation  is pretty good as well
• If you are concerned about performance you can find your answers in this white paper
• Also, need to mention a great book from original creators here