ISR stands for In-Sync Replicas. It is the set of all replicas (including the leader replica) that are in sync with the leader of a Kafka partition.

For example, if the replication factor is set to 3, Kafka will maintain 3 copies of the data for that partition across different brokers. (1 leader + 2 followers)

One replica is designated as the leader. All read and write operations for the partition are handled by this leader. Other replicas are followers, which replicate data from the leader to stay in sync.

Importance of ISR

High Availability: If the leader replica fails, one of the replicas in the ISR can take over as the new leader, ensuring no data loss or downtime.

Durability Guarantees: Kafka can be configured to acknowledge writes only when data is written to all replicas in the ISR (acks=all), which ensures stronger durability.

Replication Lag Monitoring: Kafka continuously monitors replicas to ensure they remain in sync. If a replica falls behind for too long, it is removed from the ISR.

Trade-Offs

Increased Fault Tolerance vs. Resource Usage: Each replica consumes additional disk space on the brokers. For example, a replication factor of 3 triples the storage requirement for the same data.

High Availability vs. Performance Overheads: A higher replication factor increases the time to propagate writes across all replicas. This is especially significant when using acks=all, as the leader waits for acknowledgments from all in-sync replicas (ISR) before confirming the write.

Hope this helps to get some understanding of How Replication work in Kafka. Let me know if you have any other questions.

Resources

• kafka.js.org

• kafka.apache.org

Event capturing is not the default behavior of Javascript Events. In vanilla javaScript we need to pass true in the 3rd argument of addEventListener function to register the handler in event capturing mode.

In React.js we can register the capturing handler via onClickCapture function in component.

Now let's see this behavior live in React Application. Here I've created three div element with onClickCapture event handler so that we can console.log if this event handler gets executed on this element.

import "./styles.css";

export default function App() {
  return (
    <div className="App">
      <div
        className="green"
        onClickCapture={(e) => {
          console.log("green!");
        }}
      >
        <div
          className="yellow"
          onClickCapture={(e) => {
            console.log("yellow!");
          }}
        >
          <div
            className="pink"
            onClickCapture={(e) => {
              console.log("pink!");
            }}
          ></div>
        </div>
      </div>
    </div>
  );
}

.App {
    font-family: sans-serif;
    text-align: center;
  }

  .green {
    background-color: green;
    width: 600px;
    height: 600px;
    position: absolute;
  }

  .yellow {
    background-color: yellow;
    width: 400px;
    height: 400px;
    position: absolute;
    top: 100px;
    left: 100px;
  }

  .pink {
    background-color: pink;
    width: 200px;
    height: 200px;
    position: absolute;
    top: 100px;
    left: 100px;
  }

Now UI will look like this. So now we can inspect the console by clicking on these three different div.

If we click on the different color box. we get these outputs in consoles. which clearly shows how click events get propagated from the root ancestor and onClickCapture event handler gets executed for each of these div. We can stop this propagation at any div by calling e.stopPropagation() on any event handler so this event will not get propagated further.

Hope this helps! Let me know if you have any other questions.

Resources

1. https://developer.mozilla.org/en-US/docs/Web/API/Event/stopPropagation

What is Kafka? Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

What is Topic? Kafka topics are the categories used to organize messages. Each topic has a name that is unique across the entire Kafka cluster. Messages are sent to and read from specific topics. In other words, producers write data to topics, and consumers read data from topics.

What is Partition? Kafka Topic is divided into several partitions. Kafka guarantees the order of the messages sent within the same topic partition**.** Since a topic can be split into partitions over multiple machines, multiple consumers can read a topic in parallel. This organization sets Kafka up for high message throughput.

What is Consumer? A consumer is the one client that consumes or reads data from the Kafka cluster via a topic. A consumer also knows from which broker, it should read the data. The consumer reads the data within each partition in an orderly manner.

What is Consumer Group? A consumer group is a set of consumers cooperating to consume data from some topics. The partitions of the topics are divided among the consumers in the group so that each member receives a proportional share of the partitions from a topic.

Implementation in Node.js

Let's create our utility class for using Kafka.

import { Kafka } from "kafkajs";

// Our Wrapper class for using kafka utility
class KafkaWrapper {
  // private variable for holding kafka client admin, producer 
  // and list of consumers.
  #admin;
  #producer;
  #consumers = [];

  // initialization kafka client connected to locally running brokers.
  static #kafka = new Kafka({
    clientId: "kafka-demo",
    brokers: ["localhost:9092"],
  });

  // Function to initialization kafka & producers.
  // passed topics array will be created in kafka cluster
  async initProducer(topics) {
    this.#admin = KafkaWrapper.#kafka.admin();
    await this.#admin.connect();
    await this.#admin.createTopics({
      topics: topics,
    });
    this.#producer = KafkaWrapper.#kafka.producer();
  }

  // Function to initialise new Consumer
  // groupId -> id of group 
  // topic -> topic name for which this consumer is subscribed to.
  // callback -> callback handler whenever this consumber this message on provided topic.
  async initConsumer(groupId, topic, callback) {
    this.#consumers.push(KafkaWrapper.#kafka.consumer({ groupId }));
    await this.#consumers.at(-1).connect();
    await this.#consumers.at(-1).subscribe({ topic });
    await this.#consumers.at(-1).run({
      eachMessage: async (message) => {
        callback(message);
      },
    });
  }

  // Function to send message into provided topic.
  async sendMessages(msg, topic) {
    await this.#producer.connect();
    await this.#producer.send({
      topic: topic,
      messages: [{ value: msg }],
    });
  }
}

export default KafkaWrapper;

Our code is self-explanatory. There are two important methods of class sendMessages(msg, topic) which will send a message to the topic provided via argument and initConsumer(groupId, topic, callback) that will create a new consumer inside the provided group, this will also subscribe to the provided topic and execute the provided callback function whenever a message is received on this consumer.

Now let's create a driver code.

// configuration varaiable for topic & partition of topic.
const topic = "topic";
const partition = 3;

// object for storing recevied messages so that we can later print & see.
const res = {};

// Configuration array for consumers
// if there are N number of element in array 
// then N number of consumber will be created with provided configuration
const consumers = [{ consumer: "consumer", group: "group" }];

// handler for received messages & storing in res object.
const handleMessage = (consumerId, msg) => {
  const { partition, message } = msg;
  if (!res.hasOwnProperty(consumerId)) res[consumerId] = [];
  res[consumerId].push({ partition, message: message.value.toString() });
};

// init kafka broker.
const broker = new Kafka();
await broker.initProducer([{ topic: topic, numPartitions: partition }]);

// init consumers from consumer configuration array
for (let each of consumers) {
  await broker.initConsumer(each.group, topic, (msg) =>
    handleMessage(each.consumer, msg)
  );
}

// sending 10 messages to topic
for (let i = 1; i <= 10; i++) {
  await broker.sendMessages(`Hello ${i}`, topic);
}

// printing received messages after some delay.
setTimeout(() => console.log(res), 2000);

Now we have an interface to test our Kafka communication. we can create as many consumers within the same group or different groups by adding more configuration objects in the consumer array. we can also increase or decrease the partition of topics by changing partition variable.

Examples

Now we run our code as per the below images by configuring the number of partitions on the topic and the number of consumers subscribing to that topic.

Explanation - Here a topic is divided into 3 partitions and a consumer group with 3 members subscribes to that topic, then Kafka will assign each partition to each group member to distribute load and increase message throughput on that topic.

Explanation - Here a topic is divided into 3 partitions and a single consumer subscribes to that topic, then Kafka will assign All partitions to that Consumer so that all messages get delivered and processed by that consumer.

Explanation - Here a topic is divided into 3 partitions and the Consumer group has only 2 members. In this case, Kafka will try to distribute loads and assign each partition to each group member and one member will get assigned to two partition. (Here in code Consumer B gets assigned to Partition 0 & 1 instead of consumer A as in image).

Explanation - Here a topic is divided into two partitions and the Consumer group has three members. Even we have three members only two members will get assigned to partition because we have only two partition and additional members will sit idle without receiving any messages.

Explanation - Here a topic is divided into three partitions and there are two consumer group with different number of members inside group. Here you can see Kafka automatically distribute load so that each consumer group get all message.

In First Consumer Group, All partition get assigned to each member of group.

• ConsumerA get assigned to partition 2

• ConsumerB get assigned to partition 0

• ConsumerC get assigned to partition 1

In Second Group,

• ConsumerD get assigned to partition 1

• ConsumerE get assigned to partition 0 & 2.

Hope this helps to get some understanding of How Kafka works. Let me know if you have any other questions.

Resources

• https://kafka.js.org/

• https://kafka.apache.org/

Communication between Microservices can be categorized into two types, Synchronous and Asynchronous Communication. while REST and gRPC are Synchronous ways of communication, Kafka communications are asynchronous and do not wait for a response. Here we will see the local setup of Kafka in node.js and communications between services with Kafka queue.

Installation

To install Apache Kafka, Java is the only prerequisite. Make Sure at least Java 11 is already installed in your system.

• Download the latest version of Apache Kafka from the official site under Binary Downloads.

• Extract the contents (double click in the Finder) to a directory of your choice.

• Now you have to start two services zookeeper and kafka from downloaded binaries.

• Suppose the extracted directory is kafka_2.13-3.60 (The name of the directory may be different based on your installed Kafka version), Go to the Root Directory where you have extracted Kafka and run the following command to start zookeeper.

    kafka_2.13-3.6.0/bin/zookeeper-server-start.sh kafka_2.13-3.6.0/config/zookeeper.properties
  

• 

• Open another Terminal window and run the following command to start the local Kafka server.

kafka_2.13-3.6.0/bin/kafka-server-start.sh kafka_2.13-3.6.0/config/server.properties

• Kafka is Now Started on localhost:9092 . Ensure both terminals are running always otherwise you will shut down the zookeeper or Kafka.

• you can also make a single executable script to run both commands at different terminals as per your operating system. (In macOS you can use something similar to this to start both as a single command).

• you can read more about downloaded Kafka binaries on the official site.

Setup in Node.js

We must install the Kafka client library for node.js to use our locally running Kafka server. for this tutorial, we will be installing kafkaJS from npm.

To utilize Kafka for microservice communications, we need to write some boilerplate code for different roles such as producer, consumer, and admin in the Kafka client library such that the producer will produce messages in some topic, and then a consumer can consume that message. with the help of the admin client users can configure their Kafka clients.

In the admin.js file, we need to provide the address of the running Kafka broker during initialization, In this case, our local Kafka instance is running at localhost:9092. we can also create a topic and use that topic in producer and consumer files for sending and receiving messages as per the above templates. For now, we are just consoling received messages.

Now our basic template is ready we just need the driver code to use our sendMessage function. for that, we have created index.js to act as both producer as well as consumer based on passed argv from the command line. In producer mode, it will continuously send messages using our sendMessage function. In consumer mode, it will subscribe to the topic and start listening to it using initConsumer.

Now open two terminals and enter the following command.

• npm start producer

• npm start consumer

Note: - you may also need to add the start command in the script section of the package.json file as per your folder structure to run the index.js file

First, start the consumer using the above command and then start the producer and see the message passing between these two services in action as below.

Hope this helps to get some understanding of Kafka with node.js. Let me know if you have any other questions.

Resources

1. https://kafka.js.org/

2. https://kafka.apache.org/

A generator is a special type of function that does not return a single value, instead, it returns an iterator object with a sequence of values. In a generator function, a yield statement is used rather than a return statement.

we can directly loop over returned iterator from the generator function or we can get the next value from the iterator object by calling next() method of iterator which will return an object { value: any, done: boolean } where value is any value that is yield by generator function and done is a boolean flag to indicate if the execution of the generator function is complete.

A return statement in a generator, when executed, will make the generator finish (i.e. the done property of the object returned by it will be set to true).

The implementation

we can create a generator function in javascript by using function* and yield syntax. here is an example of a Fibonacci generator that will generate the next Fibonacci number on the fly.

function* fibonacci(){
    let a = 0;
    let b = 1;

    while (true){
        yield a;
        let tmp = a;
        a = b;
        b = tmp + b;
    }
}

const fibb = fibonacci();

// Generating first 10 fibonacci number on fly
for(let i=0; i<10; i++){
    console.log(fibb.next().value)
}

/* output 
0
1
1
2
3
5
8
13
21
34
------------------------------   */

Advantage

1. Lazy Evaluation: Suppose there is a large or infinite stream of data. we cannot spend our whole life evaluating that data. Hence we can use Generator function to evaluate as and when required.

2. Memory Efficient: Generator functions are memory efficient because only those data and those computations that are necessary are used. eg: Suppose we want to read and process all files in a particular directory, we can always write a generator function to read and process files one by one as required.

3. Time-efficient when compared to overhead in managing intermediary lists as shown below image.

Hope this helps! Let me know if you have any other questions.

Resources

1. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/function*

Now let's see this behavior live in React Application. Here I've created three div element with onClick event handler so that we can console.log if this event handler gets executed on this element.

import "./styles.css";

export default function App() {
  return (
    <div className="App">
      <div
        className="green"
        onClick={(e) => {
          console.log("green!");
        }}
      >
        <div
          className="yellow"
          onClick={(e) => {
            console.log("yellow!");
          }}
        >
          <div
            className="pink"
            onClick={(e) => {
              console.log("pink!");
            }}
          ></div>
        </div>
      </div>
    </div>
  );
}

.App {
  font-family: sans-serif;
  text-align: center;
}

.green {
  background-color: green;
  width: 600px;
  height: 600px;
  position: absolute;
}

.yellow {
  background-color: yellow;
  width: 400px;
  height: 400px;
  position: absolute;
  top: 100px;
  left: 100px;
}

.pink {
  background-color: pink;
  width: 200px;
  height: 200px;
  position: absolute;
  top: 100px;
  left: 100px;
}

Now UI will look like this. So now when we inspect the console by clicking on these three different div.

If we click on the different color box. we get these outputs in consoles. which clearly shows how click events get propagated till the root ancestor and onClick event handler gets executed for each of these div. We can stop this propagation at any div by calling e.stopPropagation() on any event handler so this event will not get propagated further.

Hope this helps! Let me know if you have any other questions.

Resources

1. https://developer.mozilla.org/en-US/docs/Web/API/Event/stopPropagation

The Problem

In Software Engineering, there are scenarios when we want to place a mutex lock in some section of code before executing that section and want no other application can access that code while some service is still in that critical section. There are many scenarios in which distributed locks can help us

Suppose there is a microservice with 3 pods named A, B, and C and a scheduled cron job that runs every day at 12:00 AM for some routine task. Because this cron job is written inside the microservice so always 3 instances of the cron job will run at 12:00 AM every day. This will cause unnecessary complexity and may corrupt data of routine tasks. So where we can place distributed mutex lock so only one pod can execute this routine task while for other pods this section of code is locked for execution.

Redlock

If your software uses Redis then you can use Redlock distributed lock, an Algorithm proposed by Redis. This Algorithm implementation is already available in different programming languages. you can read more about it on the official site.

Sample implementation in Javascript

// Creating Redlock instance for some key with help of any library implementation.
const mutex = new RedLock("keyToLock")

// trying to acquire lock if it is available to acquire.
const isLockAcquired = await mutex.tryAcquire();

// if lock is acquired successfully the execute critical code.
if (isLockAcquired){
    scheduleTask()

    // once task is done, release lock for key
    await mutex.release()
} else{
    console.log("task is already running in some other pod.")
}

Postgres Advisory Locks

Postgres Advisory Locks are a great solution for arbitrary application locks, particularly in scenarios where you are already using Postgres and have a need to block concurrent mutations to a resource (that resource DOES NOT have to be data in Postgres).

There are two types of Postgres Advisory Locks: session and transaction.

Session Locks: Session locks, once acquired, are held until they are manually released or the database client is disconnected:

-- Acquire a lock using an arbitrary BigInt. 
-- if fails to acquire a lock, an error will be returned to the client.
SELECT pg_advisory_lock(123);

-- Unlock
SELECT pg_advisory_unlock(123);

-- return "true" is the lock is acquired, "false" if not.
SELECT pg_try_advisory_lock(123);

Transaction Locks: Transactions locks differ only in that they are released at the end of a transaction. Transaction Advisory Locks differ only in _xact on function names:

SELECT pg_advisory_xact_lock(123);

SELECT pg_advisory_xact_unlock(123);

SELECT pg_try_advisory_xact_lock(123);

Hope this helps! Let me know if you have any other questions.

Resources

1. https://redis.io/docs/manual/patterns/distributed-locks/

2. https://redis.com/glossary/redlock/

3. https://www.postgresql.org/docs/current/explicit-locking.html#ADVISORY-LOCKS

4. https://rclayton.silvrback.com/distributed-locking-with-postgres-advisory-locks

Named means that when you call a function, you attach the argument to a label. Named arguments enable you to specify an argument for a parameter by matching the argument with its name rather than with its position in the parameter list. So you can pass an argument to Function in any order irrespective of its position.

Named parameters are supported explicitly in many languages like Python, Dart, CSharp, Ruby, Scala, and Kotlin. JavaScript, by default, does not support Named Parameters. However, you can do something similar using object literals and destructuring.

Object Literals and Destructuring

The destructuring assignment is a cool feature that came along with JavaScript ES6. It makes it possible to unpack values from arrays, or properties from objects, into distinct variables. Here We are unpacking variables a & b from the object a.

let obj = { a: 1, b: 2, c: 3 }
let { a, b } = obj 
console.log(a, b)

Named Parameter in Javascript

Using Object literals and destructuring we can implement Named Parameters in Javascript. You can avoid errors when calling the function without any arguments by assigning the object to the empty object, {}, even if you have default values set up.

const placeOrder = ({ 
    productId = 'defaultId', 
    productName = 'defaultProductName',
    sellerName = 'defaultSellerName'
    } = {}) => {
    console.log("Order Placed ->", productId, productName, sellerName);
}
placeOrder({ 
    productId: "1", 
    productName: "computer", 
    sellerName: "Amazon" 
})
// Order Placed -> 1 computer Amazon

placeOrder({ 
    productId: "2", 
    productName: "mobile"
})
// Order Placed -> 2 mobile defaultSellerName

placeOrder()
// Order Placed -> defaultId defaultProductName defaultSellerName

Named parameters, by default, are optional. But you can easily make them required by validating parameters before executing the function.

Hope this helps! Let me know if you have any other questions.

Resources

1. https://en.wikipedia.org/wiki/Named_parameter

2. https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/classes-and-structs/named-and-optional-arguments

3. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment

Architecture Design

In This pattern, All Services (A, B, C, and D) can be secured behind a firewall and accessed only by Our Proxy Server. Frontend only needs to call a single service (Proxy Server) for all requests and Proxy Server takes care of calling the correct service for frontend. to All Microservices can still communicate through Synchronized and Asynchronized Messaging Tools if needed.

Implementation in Node.js

We can use any HTTP client library also for implementing our proxy server. Here we are going to use the http-proxy npm package for implementing our proxy server because of its inbuilt capability for handling proxies request.

import httpProxy from "http-proxy";

const proxyServer = httpProxy.createProxyServer();

proxyServer.on("proxyReq", (proxyReq,req,res,options) => {
    const body = req.body;
    if (!body) return ;

    const contentType = proxyReq.getHeader("Content-Type");
    if (contentType){
        const bodyData = JSON.stringify(body)
        proxyReq.setHeader("Content-Length", Buffer.byteLength(bodyData);
        proxyReq.write(bodyData);
    }
})

const handleRequest = ({ target }) => (req, res) => {
    proxyServer.web(req, res, { target }, (error) => {
        return res.status(503).send({ message : error.message });
    });
}    
export default handleRequest;

Here Our handleRequest function returns the middleware function which we can use as a Request handler for incoming HTTP requests but before that let's create our index.js file for separating individual microservices API with some prefix in API.

import express from "express";
import serviceARoutes from "serviceARoutes.js";
import serviceBRoutes from "serviceBRoutes.js";
import serviceCRoutes from "serviceCRoutes.js";
import serviceDRoutes from "serviceDRoutes.js";

const app = express();

app.use("/serviceA", serviceARoutes);
app.use("/serviceB", serviceBRoutes);
app.use("/serviceC", serviceCRoutes);
app.use("/serviceD", serviceDRoutes);

app.listen(3000)

Now suppose our microservices are deployed in some cloud provider with the following address.

Proxy Server -> https://proxy-service.com

Service A -> https://serviceA.com

Service B -> https://serviceB.com

Service C -> https://serviceC.com

Service D -> https://serviceD.com

Now let's create a route file for Service A, Similar file can be created also for other services with API which you want to expose, and make sure these API is available in the respective microservice.

Now our handler just transfers the request to provided target option and returns the same response to the Frontend it receives from that target microservice.

import express from "express";

const router = express.Router();

router.get("/someApi", handleRequest(
    { target : 'https://serviceA.com'}
));
router.post("/someOtherApi", handleRequest(
    { target : 'https://serviceA.com'}
));

export default router;

So Now Frontend can access Api from all services using only the proxy server instead of calling individuals service as shown below

https://proxy-service.com/serviceA/<api> instead of https://serviceA.com/<api>

https://proxy-service.com/serviceB/<api> instead of https://serviceB.com/<api>

Hope this helps! Let me know if you have any other questions.

Sources

1. https://www.npmjs.com/package/http-proxy

2. https://www.npmjs.com/package/http-proxy-middleware

3. https://www.tutorialspoint.com/microservices_design_patterns/microservices_design_patterns_proxy.htm

Have you ever heard of minesweeper? It is a little classic game that debuted back in 1990 as part of Windows Entertainment.

The rules are simple. There is an N X N board in which some of the squares have bombs you have to find all Square which does not have bombs and you have to mark all bombs correctly. A square in the board can either be a bomb or a numerical value representing the number of bombs adjacent to it.

Eg – If a square has 2. It means there are 2 bombs out of 8 adjacent squares. If Square have 0 values means that its adjacent have no bombs and all adjacent will become visible with their values

If you haven’t played this game, you can give it a try with google minesweeper Here to get an idea of this classic game.

After playing this game I realized that This game is nothing but a good example of the famous graph traversal algorithm Breadth-first search (BFS). This game logic can be made with BFS Algorithm.

Here is a very simple version of this game that we are going to build with React.js

The game is over as soon as you click on any bomb.

Generating Board

Here we are considering an 8X8 board with the number of mines that we can manage for the game difficulties level. For generating the board, we need to take care of a few things

• The random distributions of the bomb in some of the squares of the board, so some square contains mines.

• If Square has no bomb then populate its value with the count of its nearby bombs.

Below codeSnap contains one way of doing this, where each square contains id, value, and two Boolean to indicate its state (isVisible, isFlag).

Component

• Square - The building block of this game is Each Square which can contain either a bomb, flag, or numerical value indicating the total number of nearby bombs. Two event handlers for left click and right click we are passing in props to handle the event properly. Left click to show value and right click to mark with the flag for a potential bomb.

• Board - The main component of this game that holds all Individual Square as well as all logic of the game. Here we are using two state variables with useState (board, & noOfBombs). Also, we are using react useEffect to set the newly generated Board whenever noOf bombs change by the user. Second useEffect is used here to handle the wrong move by the user which leads to Game Over or if there are no mines left on the board then the user wins this game and the board is reset., And two functions Related to handling clicks.

The right-click is used to mark the current Square with a flag, and the Click is doing a BFS search from the current Square if it has no bomb and makes all Adjacent Square visible which has no bombs nearby. "tmp" Array is used here to mark the visited square.

And At last, we are returning the board with some minor styling and Now we only need to render this Board Component in React App that we are doing here.

Now We Can Run Our React App to see this Game in Action. The full Code can be found in below GitHub Repository.

Edit-

Here is the live clone of the google minesweeper game.

Click Here!