In this blog, we would like to share the experience of attending Grafana’s recent Meetup, where Devarsh gave an insightful talk about how Grafana Tempo is being used at his firm. He spoke about why they chose Tempo over other observability tools and what made it stand out in their setup. Let's dive into what made this session so interesting!

What is Observability?

• O11y means the output which we will get from the API Calls, database, or they can can be received by Kubernetes or Docker containers.

• The output(or the three pillars of o11y) can be in the form of Metrics, Logs and Traces.

• Understanding each,

  • Metrics help us monitor the performance and resource usage—like CPU and memory—of the underlying infrastructure.

  • The Logs, the logs are, the real-time messages from the services, that what’s internal process are going internally. Like if we see the logs of the Node.js server, we know that we see the message → “Server is running on port 3000” or so on.

  • The Traces, this is something interesting, which we’re going to witness today. The traces take you through the path the request travels. Whereas, logs tell you this request was sent, and this request was received. The traces show you how many milliseconds (the latency) it took to reach that route.

Why O11y?

• If we go back a decade, we will see that companies used to use monolithic architecture. So, if something goes wrong, they will just check the logs and see which part is breaking and remediate it.

• But today, things are different. Microservices have an attraction, and companies are leveraging it, and here the need for o11y arises.

• The o11y follows this cycle → Detect → Investigate → and Remediate

• Detect → If we do not use O11y, the customer will call us or raise the issue that the particular microservice is misbehaving. But, using o11y, we will detect the issue beforehand can investigate and remediate the issue.

• What do I mean when I speak of investigating? → We will see what status code we’re seeing and which request is misbehaving, and ultimately, we can remediate the issue.

What is OpenTelemetry?

• Using OTel, we examine the internal state of o11y → otel o11y pipeline can be broken down into three main stages: the generation → the collection and → the export of the three pillars that we saw.

What is the need for OpenTelemetry?

• It generalises the way. There exist multiple tools which achieve this in their way. Tools, like Nagios, Jaeger, Zipkin, AWS X-ray, Grafana, Prometheus. But with multiple choices, there comes misery in selecting one. Hence, OTel.

• OTel will instrument our microservice to measure the changes. In our case, instrumenting Node.js.

Why did we choose Grafana Tempo?

• Simplified Documentation on What and How

• Integrate it with the Grafana ecosystem.

• Native support for OpenTelemetry

• Helped us to find out the root cause analysis and find Slow Traces

• Database support and accurate service graph

• Which eventually boosted our productivity and saved our costs, as it’s an open-source project

Why not Jaeger?

• For sure Jaeger will provide us Traces and Monitoring whereas Tempo has the Native support of Grafana alongwith the service graph and long term storage support.

Why not Zipkin?

• No doubt, Zipkin is reliable and lightweight. But, it doesn’t support Grafana and OpenTelemetry natively and have limited capabilities in terms of Service Graphs and Long-Term storage option.

Please refer to this GitHub repository. I have made changes to it and made it lightweight for my machine.
https://github.com/devarsh10/opentelemetry-demo/tree/lightweight

Finally, we will be able to see something like this,

Key Takeaways

• Metrics, logs, and traces are the 3 pillars of observability, all viewable in a unified Grafana dashboard experience.

• OpenTelemetry provides a vendor-neutral pipeline to generate, collect, and export telemetry data directly to Grafana's ecosystem.

• Grafana Tempo offers a distributed tracing backend that integrates with OpenTelemetry and Prometheus.

References

• Why O11y Image → https://aws-observability.github.io/observability-best-practices/assets/images/Why_is_Observability_Important-88ac959abe0712c1a548537e274501f4.png

Conclusion

• Now, we know what is the use of Grafana Tempo. How can it be useful when we integrate it with our microservice.

• Let me know, how you guys are handling observability and what tools you’re using?

• Lastly, I would like to express my sincere gratitude to Rushabh Shah. Without his support, this blog would not have been possible. Thank you very much.

Hello Everyone,

In this Blog, I will tell you about my experience at C4GT's Dedicated Mentoring Program 2024. So, let us get started with the first question,

What is C4GT?

C4GT stands for Code for GovTech. It provides an opportunity for undergraduate students to work on a real Digital Public Goods project, which is in production. Not the dummy one. To know more about what is DPG? Please follow this link. The public goods can now be accessed digitally.

Now, let us understand What is Mojaloop Foundation. What's the mission and vision of this company? And What problems is it solving? And Who are its target users?

What is the Mojaloop Foundation?

To make it short, in Swahili language Moja means One and the meaning of loop is...google it. So, yeah, mojaloop creates a loop that connects DFSP(Banks), Mobile Money Operators, and much more. Why loop? Because, if there's no loop the mesh architecture will get created and ultimately the complexity will rise, which no one wants to witness. For another question use LLM models or google it. If you're curious.

What problem is being solved by Mojaloop?

Mojaloop targets unbanked individuals. The goal is Financial Inclusion. I hope this answers some other questions too.

Now, let us dive into the projects I worked on. Contributors, Assemble. If you're interested in the concept of Mojaloop and want to contribute to Mojaloop docs, architectures, or code. Pay Attention.

Following are my tasks and my experience with them. If you want to go in-depth What I learned from that task? I also have documented them here.

My First Task

My first task was to bring down the time to generate the final report, which was 150s earlier in core-test-harness the repository(What is Core-Test-Harness Repo?) and I successfully brought it down to 20 to 50 seconds after making the following changes.

• Everyone, I manipulated a few files in, the directory.

• And a few changes in the Dockerfile.

The changes were minimal but the efforts to reach the root cause were time-consuming. But, with the help of my Mentor, I successfully did this task.

A question might arise in your mind, Devarsh, What is Core-Test-Harness?

Everyone, Core-Test-Harness is a tool that can be used by the DFSPs(say Banks for now) and developers. DFSP deploys this core-test-harness repository to know how the transfers are being done and how the APIs are playing a major role during the transactions. Developers use CTH if a new feature is introduced or a new API is introduced. So, for the cross-checking purpose also. The CTH is used. Because, in CTH we have TTK(Testing Toolkit).

Watch this video to see, what changes I have made: Drive Link

My Second Task

Everyone, during the tenure of my internship I explored a lot of repositories at mojaloop which deepened my understanding of mojaloop and its vision to add value to society. Talking about the second task,

I was responsible for adding the test coverage to the Mojaloop Repository and I have added it to a few repositories like management-api, experience-api, and mcm-client. I was supposed to add the coverage summary to the management and experience API only. Because, this coverage part was essential and I learned how to add the test coverage using the jest framework, with the guidance of my other mentor I added coverage to the mcm-client repo too.

What was this contribution was all about?

Everyone, this coverage I have added just increases the code quality. So that, the code will be leveraged in the production if it passes the 90% coverage-summary threshold.

What is this Management and Experience API repository all about?

Everyone, these repositories are the subsets of a project called Payment Manager.

Simply, if I speak in language A and you speak in Language B, we both will need a mediator, right? All, that mediator/translator is “Payment Manager” and this Management-API manages the security aspect, whereas the Experience-API manages the enhancement of user experience I was responsible for introducing the test-coverage-summary using a jest framework and making the code quality good, so that, it will pass the 90% threshold. Refer to the following.

My Third Task

My third task is a little bit challenging for me. Here, I am responsible for configuring the Kubernetes in such a way that all the requests will reach the API endpoint. The whole project is in here, a miniloop (the name of the repository).

Miniloop is the same as the Core-Test-Harness repository. Here, everything is deployed on a Kubernetes environment.

My Fourth Task

My fourth Task is to work on the IaC section of the Mojaloop. We didn’t have more conversations on this but. I am excited to start working on this section.

My Fifth Task

My fifth task is to add the new API in the core test harness itself. Yes, that’s it. But it’s not as easy as it sounds. Refer this diagram(https://docs.mojaloop.io/technical/transaction-requests-service/)

As mentioned in the diagram. I was responsible for adding the Transaction-Request-Service to the current architecture, which is configured for payer and payee transfers. So, I am responsible for adding this new section named Transaction-Request-Service.

If you are interested in understanding What is Transaction Request Service, please go through the above link itself.

My Sixth Task

My Sixth Task is to integrate the Finance Portal with the Core Test Harness.

What is a Finance Portal(FP)?

Everyone, the FP shows the clients a few this on their screen like, What was the last transaction done? What is the TransactionId of that transaction? What is the Payee-id, and Payer-id associated with that TransactionID? and more other information.

And as discussed earlier we know what is Core-Test-Harness.

Why Integrate it, Core-Test Harness?

If the FP portal is attached to CTH, we can see the complete simulation by just running a command or two. Let us try to understand, suppose, we had done the transaction in the TTK simulation, and the money is transferred from one bank(DFSP) to another, and on the other side, we’ll be seeing the update on the finance portal. That’s why we need to integrate the Finance Portal into the Core-Test-Harness.

Follow this Video to understand, how CTH and FP work after Integration: Drive Link.

Why I am sharing all this with you?

Everyone, I consider myself very fortunate that we 100(approx.) students got selected among 2600(approx.) Students and I got the chance to work with very wonderful organisation Mojaloop, where DevOps Tools and Technologies are used in depth.

Also, most of my connections are in the DevOps Field. I thought I to share this blog with you all. I have made this blog very in-depth so that you get a thorough idea of what mojaloop is by reading this Blog only and can contribute to Mojaloop Foundation GitHub Repositories and Enhance your profile. All the best with your contributions.

Outro

Bas! Bye Bye..

Hello Everyone,

To give a little context we've been working on the series and till now we've dockerized the react application using multi-stage Dockerfiles and Docker-Compose.

Today, we're going to deploy this react website on Kubernetes. So, if fortunately in production, the traffic increases then we can Kubernetes will upscale the react application by increasing the number of pods.

What you should expect from this blog?

1. You'll get your hands dirty on Pod, Deployment, and Services.

2. In this Blog, we're using ClusterIP and NodePort services of Kubernetes.

3. We will be leveraging AWS EKS in the next blog and at that time we will use Load Balancer Service and access our application from anywhere we want.

Let's start

Creating Manifest files

We will be directly creating Deployment files which will ultimately create the pod.

Why we're not creating the pod? OR Why we're creating Deployment?

• Suppose the pod crashed due to heavy traffic or got deleted by human error. The react application will go down.

• So, to prevent this from happening we'll be creating Deployment, which comes with so many benefits like we can scale it if the traffic increases on our website.

• If any pod goes down, the deployment will auto-heal the pod, recreate the pod, and ultimately keep the desired number of pods alive.

As we have a three-tier architecture i.e. frontend, backend, and database. We'll be creating three separate deployment and service files.

Configuring Frontend

Using ClusterIP Service

Let's create a Deployment file for Frontend using the Docker Image (which is public): devarsh10/ocean-web

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ocean-web-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ocean-web
  template:
    metadata:
      labels:
        app: ocean-web
    spec:
      containers:
        - name: ocean-web
          image: devarsh10/ocean-web:0.1
          ports:
            - containerPort: 3000
          command: ["npm","start"]
---
apiVersion: v1
kind: Service
metadata:
  name: ocean-web-service
spec:
  selector:
    app: ocean-web
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000

Here, we're using the service ClusterIP. As this react application is running on the minikube cluster we have to forward the port to access the react app from our PC.

But now when we try to access the react app from the minikube, we will not able to access it because we're not using NodePort.

NodePort forwards the external traffic to the Service. In the case of Cluster IP, the service is accessible within the cluster.

So, we'll be using NodePort. So that, we can access it from the minikube.

Changer in code,

---
apiVersion: v1
kind: Service
metadata:
  name: ocean-web-service
spec:
  selector:
    app: ocean-web
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000
      nodePort: 30007 # we're just adding this two lines 
  type: NodePort # This one too.

Let's try.

• As you all can notice, when we try to access the React web app using port 3000, it fails to connect.

• But when we try to connect on port 30007 we get a response. That's the use case of NodePort.

Checkpoint

• Everyone, till now we've leveraged the services and made the React app accessible using NodePort.

• Now, what we're going to do is,

  • We will create the deployment files of the Database.

  • And the Backend, where the Nodemailer is configured. So, that we can send the OTP to reset the password.

• Let's continue

Configuring Database.

Here, the challenge is, that we've multiple databases running on different ports. So, let's create a Deployment manifest keeping that in mind.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: db-deploy
  name: db-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: db-deploy
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: db-deploy
    spec:
      containers:
      - image: devarsh10/ocean-json:0.1
        name: ocean-json
        ports:
        - containerPort: 3000
        resources: {}
        command: ["/bin/sh"]
        args: ["-c","json-server --watch db.json --port 8000 --host 0.0.0.0 & json-server --watch db2.json --port 8001 --host 0.0.0.0 & json-server --watch 
db3.json --port 8002 --host 0.0.0.0"]
status: {}
---
apiVersion: v1
kind: Service
metadata:
  name: ocean-json-service
spec:
  selector:
    app: db-deploy
  ports:
    - name: port-8000
      protocol: TCP
      port: 8000
      targetPort: 8000
      nodePort: 31008
    - name: port-8001
      protocol: TCP
      port: 8001
      targetPort: 8001
      nodePort: 31009
    - name: port-8002
      protocol: TCP
      port: 8002
      targetPort: 8002
      nodePort: 31010
  type: NodePort

Similarly, we can access the port 31009 and 31010.

Now, configuring the Backend where the Nodemailer is configured.

Checkpoint

Till now, all three tiers are configured. Now it's time to access them from a PC.

Which is possible using port-forward. As of now we're neither using Load Balancer nor Ingress. So we've to use port-forward for now.

We will surely write a blog on Ingress and Load Balancer on Cloud(AWS most probably).

Port Forwarding

I have written a shell script, so that, we do not have to use write same command multiple times.

#!/bin/bash

# Port forward for ocean-web-service
kubectl port-forward svc/ocean-web-service 3000:3000 &
echo "Port forwarding for svc/ocean-web-service on port 3000"

# Port forward for ocean-json-service on multiple ports
kubectl port-forward svc/ocean-json-service 8000:8000 8001:8001 8002:8002 &
echo "Port forwarding for svc/ocean-json-service on ports 8000, 8001, 8002"

# Port forward for ocean-nodemailer-svc
kubectl port-forward svc/ocean-nodemailer-svc 5000:5000 &
echo "Port forwarding for svc/ocean-nodemailer-svc on port 5000"

# Wait for all background processes to finish
wait

1. To execute it, first, give the execute permission to it by using the command,

   chmod +x shell.sh and then execute it using ./shell.sh .

Moment of Truth (Execution)

• http://localhost:3000

• localhost:8000

• localhost:5000

Let's try resetting the password and see if the OTP is getting sent or not.

1. Yes, OTP was sent.

2. Let's check the mail

Success.

Terminal

Outro

So, that's it for the day everyone.

Now, you have hands-on experience in how to deploy a complex three-tier application on Kubernetes.

Next: We will be using a Load Balancer and Ingress. So that we can access our application from anywhere. Stay Tuned. Happy Coding.

Hello everyone, I hope you all are pink of health. Today, I am writing this Blog on "How to install Warp in WSL" and being a Windows user, how can we enhance our experience using the wonderful terminal Warp.
Also, at the end of the video I will share with you one Bonus tip that will help you increase your typing speed in the warp Terminal.

Installation

1. Visit the warp website using my referral link -> https://app.warp.dev/referral/X9K3EV (Here is the original link if you don't want to use the referral link: https://app.warp.dev/get_warp). Install the Debian package. We will "dpkg" it.

2. Open the WSL in Windows. Go to the path where you've installed the Deb Package. Let's say "Desktop".

   • Run sudo dpkg -i <the_debian_package>

   • I had to install these dependencies of warp in my case.

     • libegl1 libxkbcommon-x11-0. So if you also face this issue, use this command,

       • sudo apt-get --fix-broken install -y

       • sudo apt install libegl1 libxkbcommon-x11-0

   • In my case, sudo dpkg -i warp-terminal_0.2024.05.14.08.01.stable.04_amd64.deb

3. Now, open the bash_profile file to add the following lines. Using command,

   nano ~/.bash_profile paste the following and Ctrl+X and then Y.

    export WARP_ENABLE_WAYLAND=1
    export MESA_D3D12_DEFAULT_ADAPTER_NAME=NVIDIA
    export BROWSER=google-chrome
   

   The changes will be saved. Here we're adding Google Chrome because we need to open Google Chrome from Warp to get the Authentication Token.

4. Now, let us install Google-Chrome-Stable in WSL

    wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
    sudo apt -y install ./google-chrome-stable_current_amd64.deb
   

   

5. Now, Sign Up (Sign Up Link: https://app.warp.dev/signup) and log in (Link to Log In: https://app.warp.dev/login/remote).

6. Follow the fifth step in the WSL's Google Chrome, which we just installed. Once you successfully log in. You'll land on this kind of screen

   

   Enjoy Using Warp....

   

Bonus. Improve your Typing Speed in Warp

I just watched the YouTube Short posted by Hitesh Sir Choudhary on his YouTube channel. Where he introduced ttyper.

To install it, we need "cargo". To install "cargo" run the command, sudo apt install cargo

Followed by it, install "ttyper" using the command cargo install ttyper

Now, just write "ttyper" and start practicing

And Enjoy...

Outro

That's all for now Everyone. Happy Coding.

Hello Everyone, welcome to part two of Dockerizing the React Application.

Quick Recap

In the last blog, we've Dockerised the React Application using multi-stage Docker Build and saved around 1.5 GB of space.

Getting Started

In this Blog, we will dockerize the same react application, just using the Docker-Compose file. So that we can increase our grip on writing docker-compose files and all.

Here's the structure we're going to follow.

It is the same structure we've used earlier. Just here we've got a docker-compose file.

Here's the docker-compose file.

version: '3'
services:
  web:
    # build:
    #   context: .
    #   dockerfile: Dockerfile
    image: devarsh10/ocean-web:0.1
    restart: always
    container_name: ocean-app
    ports:
      - "3000:3000"
    command: npm start
  json:
    # build:
    #   context: .
    #   dockerfile: Dockerfile-json-server
    image: devarsh10/ocean-json:0.1
    restart: always
    container_name: json-server
    ports:
      - "8000:8000"
      - "8001:8001"
      - "8002:8002"
    command: bash -c "json-server --watch db.json --port 8000 --host 0.0.0.0 & json-server --watch db2.json --port 8001 --host 0.0.0.0 & json-server --watch db3.json --port 8002 --host 0.0.0.0"

  nodemailer:
    # build:
    #   context: ./server
    #   dockerfile: Dockerfile
    image: devarsh10/ocean-nodemailer:0.1
    container_name: backend-nodemailer
    ports:
      - "5000:5000"

Let us traverse line by line to understand the Code.

• Version: We're using the latest version of Docker-compose here.

• Services: Services are the way we can tell Docker that we've to run three Docker Containers.

  • Web: Here and in every service we are using a few parameters. So let us understand one by one.

• Build: If we've got a Dockerfile present with us then we can also avoid pulling the image from DockerHub. Using the Build parameter using context i.e. path and name of the dockerfile.

• Image: The reference to the image

• restart: If you keep restart: always, it'll start the container when you open Docker Desktop. If you have not used restart:always. After opening Docker Desktop the container will not be running. (I have written in the easiest way possible)

• container_name: The name we want. If we do not specify this parameter, it'll take the name of the service as a container name.

• ports: The port we are mapping from our host machine to the container.

  • command: The command we'll be using is like, 'npm start'.

• Now, use to command docker-compose up and the container will start running and the app will be accessible.

Thank You

Yeah, so that's all for this blog if I learn some new parameters I will add them. If you want to contribute willingly. Do let me know in the comment section. I'll reach out to you.

• In this blog, I am going to show you, how to dockerize the application in different containers on the same network

• In the second one, I will demonstrate how to dockerize the React application using docker-compose.

• Finally, we'll be deploying it on Kubernetes and if time permits I will create a fourth blog on troubleshooting and a fifth blog on deploying the app on AWS EKS.

• So, I hope I can show you the clear-cut architecture and agenda.

• So now, without any further ado let's get started.

What should you expect from this?

• After reading this blog, you'll get to know where to see how to debug the Dockerfile.

• How to write multi-stage Dockerfile.

Getting Started

Firstly, let me show you what the registration page looks like when I run the command npm start.

Now, let me give you some context.

• Here we have our react app which is running on port 3000.

• Besides we've 3 different databases, which are supposed to run on 3 different ports.

• Following that, we have a Nodemailer in the backend which is responsible for sending the email to the user who forgot the password.

Hence, 3 containers are required.

My file structure

Note: Here, I have kept the Dockerfile of the frontend and the Dockerfile of the Database in the root directory, and the Dockerfile of Node.js is in the server directory. That is why I have written two ".dockerignore" files.

Dockerizing Frontend

Here, we're going to create a multi-stage dockerfile. Using the Single-stage Dockerfile may reach a GB of space which is not at all space-efficient.

Also, keep in mind one thing. Create a '.dockerignore' file and add the directories or files that you do not want to copy to the Docker container. In our case, it is "node_modules". It looks something like this,

As you can see in the following when we use the single-stage Dockerfile. The size of the image will reach more than 1 GB. In our case, it's almost 2 GB.

Dockerfile used,

FROM node:20 AS builder

WORKDIR /app

COPY package.json .

RUN npm i

COPY . .

Now, see the following change,

Changes made,

FROM node:20 AS builder

WORKDIR /app

COPY package.json .

RUN npm i

COPY . . 

FROM node:alpine

# Creating a new directory named app
WORKDIR /app

#Copying all the content copied in app to current directory
COPY --from=builder /app .

# Exposing port which will be mapped with the host's port.
EXPOSE 3000

# This will start the development server.
CMD [ "npm", "start" ]

Hence, we've saved something around 1.5 GB of Space which matters a lot when dockerizing an app in production.

Now, let us run our app in a container in a separate network.

First, create a network using the command docker network create react-net.

Now, run the establish a Docker container using the command,

docker run -d -p 3001:3000 --network react-net --name react-app <image-id>

In my case, it looks something like this,

Also, all the routes configured are giving responses. That means our app is Dockerized.

Dockerizing Database

For the database, we're going to use node-slim Image. Because the slim comes with minimal packages to run a node, which is sufficient for us to run the JSON server.

As discussed earlier, we've three databases. Here's the Dockerfile we're going to use.

# Use the official Node.js image as a base
FROM node:slim

# Set a working directory
WORKDIR /app

# Install json-server
RUN npm install -g json-server

# Copy your JSON files to the working directory
COPY ./src/db.json .
COPY ./src/Components/Admin/ReferralList/db2.json .
COPY ./src/Components/Agent/Dashboard_Com/Product/db3.json .
RUN ls

# Expose the ports for each json-server instance
EXPOSE 8000
EXPOSE 8001
EXPOSE 8002

# Command to run each json-server instance when the container starts
CMD json-server --watch db.json --port 8000 --host 0.0.0.0 & json-server --watch db2.json --port 8001 --host 0.0.0.0 & json-server --watch db3.json --port 8002 --host 0.0.0.0

Here, there's a catch. We have two Dockerfiles in the same directory. So, here's the command we're going to use to build the Dockerfile.

There's nothing we just have to use the '-f' tag. The command will be like docker build -f Dockerfile-json-server -t json-dbs .

Now, let us run the database in a container.

Here's another catch. What command will you write if you've exposed multiple ports? The command will be

docker run -d -p 8000:8000 -p 8001:8001 -p 8002:8002 --name json-servers <image-id>

Okay. So, now we've react app and json database is running on the individual container.

Dockerizing Backend which contains Nodemailer.

Now, dockerizing Node.js' "app.js" is pretty simple.

To give the context of the code, this code consists of a Nodemailer configuration HTML, and CSS template which will be sent to send the OTP to the user who forgot the password.

Now, what will we need to run the node.js application? node or nodemon, right?

So, our Dockerfile will look something like,

FROM node:slim

WORKDIR /app

COPY package.json .

RUN npm install && \
    npm install -g nodemon

COPY . /app

EXPOSE 5000

CMD [ "node", "app.js" ]

Now, build the Docker Image by running the command docker build -t nodemailer:v1 .

Now, running the nodemailer image in a container using the command, docker run -d -p 5001:5000 --network react-net --name nodemailer-cont <image-id>

Here, the OTP is sent to the user which means the node.js is working perfectly.

Note: Here I have mapped port 3001 of the host and 3000 port of the container & in the backend, I have mapped 5001 of a host to 5000 of the container (which is not at all recommended. Because the API running in backend interacts with each other based on the ports only.)

Okay then everyone. Now, you all know how to write a multistage Dockerfile and add multiple containers in the same network.

To Debug

If your Dockerfile build fails. Then to debug what can you do is, in Docker Desktop, we've a new section named "Build".

We can just click on the build that is failing,

In the error section, it'll tell us what's the exact error we're facing. Though, we'll see this error when the docker build command fails. But, the build feature comes with a lot of pros.

Bonus (Want to get your hands dirty?)

Let us jump into the networking part.

When we run the command docker network ls

Let us inspect react-net using the command, docker inspect <network-id>.

Now, when we use, docker inspect react-net, we'll see the following output.

[
    {
        "Name": "react-net",
        "Id": "75db8f96dcd8e483d5af7758f7227f22ffd023abc4e636ff0b683733d052e151",
        "Created": "2024-05-02T06:41:04.575805992Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.20.0.0/16",
                    "Gateway": "172.20.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "45bad3687dc8e953188ec4ee6555e5a1503cc92263ceaecb4959bef7d57900a0": {
                "Name": "json-servers",
                "EndpointID": "8e0f23511fefc8b389e4e6baca9d06035e30cf05cce9af42a8d36f5aff31dc08",
                "MacAddress": "02:42:ac:14:00:04",
                "IPv4Address": "172.20.0.4/16",
                "IPv6Address": ""
            },
            "8231a9d3ff3d74f0e06709936bfb0cc143127f51f652e47ac25bbe129a9365b0": {
                "Name": "react-app",
                "EndpointID": "2ee55bc4302fe72bfdb1541abd4123c1255a1174285e9e166e4d8e4d38791e45",
                "MacAddress": "02:42:ac:14:00:03",
                "IPv4Address": "172.20.0.3/16",
                "IPv6Address": ""
            },
            "a05e90ce75eb5bf23bfde0a1af8c8af70892e48a49aafd55009914fdfa12d490": {
                "Name": "nodemailer-cont",
                "EndpointID": "fb097380d1d385253d686d4cc3d222f0fa5ed46d48b4ae3ac58d69ecf833f8d8",
                "MacAddress": "02:42:ac:14:00:02",
                "IPv4Address": "172.20.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

Here, in the container section, we can see, the three containers we added in the same network.

Interview Questions

1. Can we write "from" instead of "FROM" and "copy" instead of "COPY"? and you get the idea.

   1. Answer: Yes we can write. We are just following the convention to differentiate our command from the Dockerfile Keywords. We write FROM, COPY, CMD, ENTRYPOINT, and many other keywords in caps.

2. How to build the Docker Image, if there exists two different Dockerfile in the same directory?

   1. Using the '-f' flag.

3. Command to inspect the network?

   1. docker inspect <network-name/id>

Thank You for reading this blog till here. Happy Coding.

Hey Everyone, in this project we will understand Docker Networking by deploying a three-tier application on a custom Docker network.

So, Let's get started.

Kickstart Your AWS EC2 Instance

Step 1 -> First of all, launch an EC2 Instance by keeping all the settings by default.

Step 2 -> Keep the HTTP and HTTPS Protocols checked. So, that we do not have to add it while working with HTTP port 80.

Note: The default port of HTTP is 80 and the nginx's default port is also 80.

Step 3 -> SSH into the instance from your local device.

ssh path/to/pem/file ubuntu@<public-ip-addr>
# OR if you want to connect from ec2 instance, just click connect and connect
# to EC2 Instance.

Step 4 -> Right now, we have only two kinds of networks in our ec2 instance. Loopback and eth0. Loopback is useful to test network connectivity. It is also known as localhost. Whereas, eth0 helps us to connect to the internet i.e. outside world

Docker Installation

Step 5 -> Let us install Docker on our machine and get started with the project.

sudo apt install docker.io -y

Step 6 -> Now, let us run the command, ip addr show You'll notice that the docker network is added to the networks group and as we create containers the veth network will be added to this group of networks.

Giving superuser access to Docker

Step 7 -> Now, let us check for the containers running on Docker.

Damn!, we faced an error.

Now, to watch the container running we can use sudo before using our command. But, how many times we will use sudo. Better, we can add us(i.e. ubuntu in my case) to Docker Group.

We can use the command,

sudo usermod -aG docker $USER
#we can use ubuntu instead of user. But, this command will run in all machines.

Step 8 -> After adding the user (ubuntu) to the Docker group. We must have to reboot our system. Using the command, sudo reboot

Step 9 -> Now, if we use docker ps we should not face any error and this should be our output.

Docker Networking

Step 10 -> Actually, there exist 7 types of Docker Networks. But, when we use the command docker network ls three types of networks can be seen.

Type of Network: Default Bridge

Definition: Whenever we create our container, it is created in the bridge network. That is why the bridge network is also known as the default bridge network.

Step 11 -> Let us create a nginx container and see where it is getting created. Run the command to pull the nginx image,

docker run -d --name nginx nginx
#Here,
# -d is specifying that the container will run in detatched mode.
# and --name is used to specify the name of the container.

Here, we are using --name nginx to name our container and the last keyword nginx is the image name, which is nginx in our case.

Step 12 -> Now, let us go inside the container and check if we can ping the outside world or not.

Oh! we can connect to the outside world. Right now we are in the Container we created. But this container can ping the internet outside the container, but unable to listen.

So, let us delete this container and create a container, which can listen to the outside internet.

Step 13 -> To delete the container, we can use the command,

docker kill <container id> followed by docker rm <container id>

Step 14 -> Now, we will create a container by specifying the port number on which nginx listens. We can use the command,

#Try writing the command by yourself.
docker run -d --name nginx-cont -p 80:80 nginx

Here, we are mapping our host to our container (host:container) on port 80. So, the network is going through the host to the outside world and listening on port 80.

Now, if you check on the browser. The nginx message will be reflected. Which means our container is listening on port 80.

Okay, so till now we have covered the default type of network which is the Bridge network.

Step 15 -> Now, if you want to verify that the container lies in the default network. Run the command, docker inspect bridge . You'll get the output like,

Type of Network: Custom

Let us move ahead with the second type of network, the Custom Network.

Definition: It is the network created by the developer which is used to create an isolated environment. In which the running container can interact with each other. But cannot interact with a container that lies in the default network OR a container that lies outside the custom network.

Step 16 -> In this section, we will create our own network named "my-net". Using command, docker network create <network-name> .

Step 17 -> Our purpose here is will create two containers and keep them in the "my-net" custom network and on the other hand we will create an nginx container on the default bridge network. And then we will try to ping the container that exists in the default bridge from the container in the custom bridge. That's it. Let's do it.

Right Now, we have one container running which is mapped with the host at port number 80. Now, we will remove this container. And create another container that will not be listening.

The commands required are as follows,

docker ps

docker kill <container-id>
docker rm <container-id>

docker ps

docker run -d --name <container-name> <image>
docker run -d --name nginx-in-default-net nginx

Till now, we have just deleted our container and created a new one.

Step 18 -> Now, let us create two nginx containers in our custom network.

As earlier, we have created our network named "my-net" We will create and run two nginx containers in this network.

To verify that the nginx containers are created in our custom network, we can simply use the command docker inspect <network-name> in our case, docker inspect my-net

And here, we can see that the two containers are added to our custom network.

Let us now do a microservice project and implement a 2 or 3-tier application in our created network.

Clone this repository, git clone https://github.com/LondheShubham153/microservices-k8s.git in the EC2 instance.

Here, we will create a Dockerfile (which is created by Shubham Sir already). Followed by build it, then run it in the container. After cloning the repo, change your directory to flask-api dir. And build the image. Using command docker build -t my-app:v1 . Note: the . is not a full stop here. It shows the path of Dockerfile to Docker.

The success message will look like this.

Now, let us boot up the container, using the command

docker run -d -p 5000:5000 --name microsvc-app-ctr --network my-net myapp:v1

Step 19 -> Now, our app is listening on port 5000. But, we didn't have added the rule to allow traffic on port 5000 on our ec2 instance -> security group -> inbound rules. So, let us set up that.

After, creating the rule, our app must be up. And it may look like the following picture.

Now, it is a moment of truth. Our app is running on port 5000 from the container that lies in our custom network.

And Boomm! Our container are isolated successfully.

Okay, so right now, in our custom network we have nginx-custom-1, nginx-custom-2 and python app running. Now, let us add a Database Container of MongoDB to make it a 2-tier application. By using command, docker run --name mongo-custom --network my-net mongo

Now, as our mongo is added to the custom network. This link should be working. http://<ec2-instance-public-ip>:5000/tasks

Type of Network: Host

Step 20 -> Now, let us move ahead with the another kind of network which is host network. Here, we do not have to map our container port with the host port. Docker, by default uses the host IP address to listen at port we specified. If we were in Bridge(default) network our nginx container can ping to the data lies in outside world like google.com or so. But, the outside world cannot reach to our container.

Whereas in host network without mapping i.e. without "this -> -p 80:80", outside world can reach to our nginx container.

To boot up host container use command, docker run -d --name nginx-host --network host nginx

And this is how Host Network Works.

Type of Network: None

The last type of network of this blog is, None.

When we create a container in the "None" network mode, it doesn't have network connectivity. It is effectively isolated from external network communication.

Let us create a container and try to ping the contents outside the container.

Here, as you can see we are not able to establish the connection.

Similarly, when we want to access this container from the outside world.

Port 3000 is not assigned to the container because the container is in a None type of network.

That's all about Docker Networking from my side. Once I learn the further concepts of Docker Networking I will make Part-2 of this Blog.

Thank you for reading the Blog here.

Reference:

A Huge Thanks to Shubham Sir for providing us the in-depth knowledge of DevOps for free.