Kubernetes- Part 2

Expose Deployment, Pods, Services, Replica set, Configmap, Centralized monitoring, monitoring GKE, Deployment rollouts with updates with no downtime, Autoscalling

11 min readJul 6, 2023

GitHub: https://github.com/nikitha2/Currency-Exchange-Conversion-Microservices-Backend.git
Linkedin: https://www.linkedin.com/in/nikitha-gullapalli/

In Part 1 we spoke about the architecture of kubernetes and how to connect to gcloud from terminal and run kubectl commands.

Now, lets looks at steps to run micro services on kubernetes.

Create cluster and connect to it from terminal/gcloud terminal.
Prepare micro services Kubernetes, generate docker images and push to docker hub.
Deploy Services. Deployment can be done in one of the three ways shown below

Deploy each service separately
Declarative deployment using yaml
Declarative deployment using yaml and configMap

4. Test the endpoints using the externally exposed IP address

1. Create cluster and connect to it from terminal/gcloud terminal.

Create cluster

Create a project in in gcloud console-> Goto project->Kubernetes Engine-> clusters -> click create. -> click SWITCH TO STANDARD CLUSTER-> Enter cluster name and create cluster with default values.

Connect to your cluster

Goto the clusters as shown below. Click on microservices101-cluster.

2. Click connect-> copy paste command-line access command to your gcloud terminal. This will connect you to your cluster.

2. Prepare micro services kubernetes, generate docker images and push to docker hub.

Prepare micro services kubernetes

Remove all spring cloud related services like Eureka, Zipkin, API-Gateway. Kubernetes will provide all these for free (All these changes are marked with a comment “CHANGE-KUBERNETES” in the github project.)

Enable liveliness probes in application.

management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true

2. Add feign client in feign proxy. In our case currency-exchange proxy. Kubernetes provides its own naming service.

@Component
@FeignClient(name = "currency-exchange-service",
    /** Will search for CURRENCY_EXCHANGE_SERVICE_HOST in env variables. 
       If not found will use http:localhost.. 
       -> Look at service discovery below to understand this better*/
    url = "${CURRENCY_EXCHANGE_SERVICE_HOST:http://localhost}:8000", 
    configuration = FeignClientConfiguration.class)
public interface CustomerExchangeProxy {

Service Discovery:

Kubctl expose command creates a service. E.g: Using below command will create a service with name: microservices101-deployment. Kubernetes will create a environment variable for every service.

kubectl expose deployment microservices101-deployment --type=LoadBalancer --port=8080

For example for a service of name: microservices101-deployment, environment name will be MICROSERVICE101_DEPLOYMENT__SERVICE_HOST.

<ServiceName in caps>_SERVICE_HOST

Say I created a deployment and exposed it. Which means service is created. How to see the environment variables? Below command will give the env variables of that pod.

kubectl exec -it <pod_name> -- env

/**
kubectl get pods -> gives list of pods and their names

Use the pod name to get environment variable. 

**/

Service names currency-exchange and currency-conversion

Now run services(currency-exchange and currency-conversion services) locally. Make sure they are running. Generate image.

Right click -> run as -> Maven build -> “nvm spring-boot:build-image -DskipTests — X” -> run

Detailed steps to generate image and push to docker hub here.

3. Deploy Services. Deployment can be done in one of the three ways shown below

a) Deploy each service separately

We have docker images from Step 2. Create depolyment and expose wach service as shown below.

#Create deployment. Pod, replica set, deployment will be created at this step.
kubectl create deployment <deploymentName> --image=<dockerId>/<dockerImageId>:<tag>
E.g: kubectl create deployment currency-conversion --image=<dockerID>/mmv3-currency-conversion-service:0.0.1-SNAPSHOT

#Expose deployment- service will be created at this step.
kubectl expose deployment currency-conversion --type=LoadBalancer --port=8100

It is too much work to deploy each service. In a real application we will have a lot of micro services. Deploying each service when a change is made is too much work. To avoid this we can do this using a yaml file.

b) Deployment with Yaml (Declarative-deployment)

See all services and deployments running.

// see all the services running -> 
kubectl get services

//see all the deployments running
kubectl get deployments

Lets create a deployment yaml file for currency-exchange.

Goto location you want to create the file. For me it was my project location.
To get yaml for a deployment use below command. It will open yaml file in terminal.

kubectl get deployment currency-exchange -o yaml

3. Save deployment yaml to local yaml file -> deployment.yaml. Now go to the location and refresh. You should see this file.

kubectl get deployment currency-exchange -o yaml >> deployment.yaml

4.Save service yaml to local yaml file -> service.yaml. Now go to the location and refresh. You should see this file.

kubectl get service currency-exchange -o yaml >> service.yaml

5. Save service yaml to local file -> service.yaml. Now go to the location and refresh. You should see this file.

5. I want both in same file. So copied service.yaml into deployment.yaml. Then deleted service.yaml file.

6. Delete all deployments and run fresh deployments with yaml

kubectl delete all -l app=currency-exchange
kubectl delete all -l app=currency-conversion

//command to make sure deployments and services deleted.
kubectl get all

7. I can simply deploy my currecy-exchange application by running the yaml file. I can further customize my deployment by editing the yaml file.

goto file location and run the below command.

#Deploy deployment and service
kubectl apply -f deployment.yaml

#compares deployments before vs after and tells us the differences
kubectl diff -f deployment.yaml

#To follow logs. -f is to follow the logs
kubectl logs -f <podId>

8. Optional: Clean up the deployment.yaml file and keep only the necessary info as shown here. Then deploy again.

9. Do the same for all your services and run deployment.yaml to deploy services in kubernetes.

10. Make sure to enable Logging and tracking APIs to trace the logs.

Goto Enable APIs and service in APIs and services.

Enable cloud loggin API

Enable stackdriver APIs

11. For currency-conversion feign url, we used CURRENCY_EXCHANGE_SERVICE_HOST in currency-exchange-proxy.java as mentioned in step 2. This will work fine as long as we name the the deployment currency-exchange.

To be independent of the deployment name, we can go one step ahead and name a custom environment variable in deployment.yaml of currency-converter file. This was we are not dependent on the deployed currenct-exchange service name.

c) Declarative deployment using yaml and configMap

Kubernetes allows us to do centralized configuration option. To do this we use configmap.

Step-by-step commands below to create configmap-> get yaml and create configmap.yaml file.

#Create configmap with env variables
kubectl create configmap <configmapName>  --from-literal=<envName>=<envValue>
E.g: kubectl create configmap currrency-converstion  --from-literal=CURRENCY_EXCHANGE_URI=http://currency-exchange

#Get all configmaps. Gives the names of the configmaps and related info.
kubectl get configmap  

#Get the yaml configuration of the configmap
kubectl get configmap <configmapName> -o yaml
E.g: kubectl get configmap currrency-converstion -o yaml

#Take out the yaml into a file. Goto the location you want to create the file at and enter below command
kubectl get configmap <configmapName> -o yaml >> <fileName>.yaml
E.g:kubectl get configmap currrency-converstion -o yaml >> configmap.yaml

Now copy-paste the yaml content from configmap.yaml to deployment.yaml -> remove custom env variable from step 2 -> add reference to configmap in deployment

Redeploy application

kubectl apply -f deployment.yaml 

// get pods to see new pod is created.
kubectl get pods

# If its confusing. Delete the deployment and create new one.
//Get all
kubectl get all
//delete the deployment/service you want to redeploy
kubectl delete all -l app=<deploymentName>
E.g: kubectl delete all -l app=currency-conversion
//Redeploy

4. Test the endpoints using the externally exposed IP address

Once deployed, run below command to get all services running. If deployment successful, you should see your service here

kubectl get services

You will get the EXTERNAL-IP from here. Run your endpoint with the id

E.g:Instead of using url:http://localhost:8000/currency-exchange/from/USD/to/INRI will be using url: http://34.66.121.88:8000/currency-exchange/from/USD/to/INR

Hope that was helpful. Once deployment is successfull. You can monitor your services and deployments with centralized logging

— — — — — — — - — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Centralized logging and monitoring GKE

Centralized logging

As discussed before, kubernetes provides centralized logging for free. Similar to zipkin. As discussed in part 1, we disabled zipkin because we wanted to use centralized logging provided by kubernetes. Lets look at steps to open these logs and what all they have to offer.

Open you cluster -> scroll down to features -> click on “View Logs” for logs.

click clear

click contaier

Select the pod name you want to monitor. In my case it's currency-convertion.

I ran watch curl http://34.69.81.179:8100/currency-conversion-feign/from/USD/to/INR/quantity/20 in terminal. This makes a call every 2seconds.

clicking on the log you want. If you have the below dependency in your application. You should see a “tracingId” generated. You can use this traceId to trace all matching entries.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-sleuth</artifactId>
 </dependency>

textPayload: "2023-07-06T22:08:47.464Z  INFO[<serviceName>, <tracingId>] 1 --- [nio-8100-exec-3] c.s.m.c.c.CurrencyConversionController   : findByFromAndToAndQuantityWithFeign called with from:USD, to: INR, and quantity: 20"

Add tracing Id to the query and run it. You should see all the calls made this this tracing Id. E.g: I made currency-convertion call which inturn made currency-exchange call. So I should see both call logs.

Monitoring GKE

Open you cluster -> scroll down to features -> click on “View GKE dashboard” to view dashboard. You should see everything related tot he cluster. Cpu used, containers, pods info. alerts, etc. This make monitoring of our deployments convenient.

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

Deployment rollouts with updates with no downtime

We have currency-exchange and currency-convertion services deployed and running. Awesome!

Now, I get new feature requirements and I finished development in my local. I need to deploy the changes now. How do I do that with zero downtime?

# Get all deployment names and info
kubectl get deployments

#Get rollout history of a deployment
kubectl rollout history deployment <deploymentName>
E.g: kubectl rollout history deployment currency-exchange

Simply update the image in deployment.yaml and apply the deployment

E.g: I changes image from nikithagullapalli/mmv3-currency-exchange-service-kubernetes:0.0.11-SNAPSHOT to nikithagullapalli/mmv3-currency-exchange-service-kubernetes:0.0.12-SNAPSHOT -> checked differences and applied deployment.

Application was down for a few minutes, but deployment was successful. I am able to v12 in my response.

#check difference
kubectl diff -f deployment.yaml      

#apply deployment
kubectl apply -f deployment.yaml

Like I mentioned, I saw some downtime. This is not good. How to resolve this?

We can resolve this issue by using liveness and readiness probes provided by kubernetes. Kubernetes uses probes to check the health of the micro services.

If readiness probe is not successful, no traffic is sent
If liveliness probe is not successful, pod restarts

Spring actuator(≥2.3) provides these probes. Make sure actuator dependency is added to pom.xml , application.properties and deployment.yaml

#pom.xml
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

#application.properties
management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true

#deployment.yaml
readinesProbe:
  httpGet:
    post: 8000
    path: /actuator/health/readiness
livenessProbe:
    httpGet:
    post: 8000
    path: /actuator/health/liveness

Now when you make a new deployment, these probes will make sure only when the container is ready to receive traffic, they will recieve traffic. Else traffic goes to older deployment until new deployment is UP.

Note: If the image in deployment.yaml is not valid, kubernetes run the previous deployment and keeps the application up and running.

E.g: In terminal cd to the currency-exchange location where my deployment.yaml file is located -> update image in deployment.yaml to “dummy” and save file-> In terminal check differences and then apply deployment.

#Gives changes made to the file between previous deployment and now. 
# In this case it will say image changed to "dummy"
kubectl diff -f deployment.yaml

#apply deployment
kubectl apply -f deployment.yaml

You will notice that Kubernetes will still run the previous valid image and shift of image:dummy is not made as it is an invalid image. If you run “kubectl get pods”, you will see it is still running the previous pod and new pod has a status “InvalidImageName”.

I can now undo my change in deployment.yaml and redeploy changes or use below command to undo rollout

kubectl rollout undo deployment currency-exchange to-revision=<revisionNo>
E.g: kubectl rollout undo deployment currency-exchange to-revision=1