In today’s world, rate limiting has become essential for system stability and fairness. Whether you’re running an AI SaaS platform and want to keep free trial users in check, or protecting your API server from resource starvation, a good rate limiting strategy ensures your service stays reliable under pressure. In this article, I’ll walk you through the most widely used algorithms that power rate limiting in modern systems.

We’ll explore each algorithm in depth, covering its pros and cons. Let’s dive in

Token Bucket Algorithm

Imagine this: we have a bucket, and every T seconds a token is dropped into it. Each incoming request needs to grab a token from the bucket in order to pass through. If the bucket is empty, requests have to wait until new tokens arrive. Because the bucket can hold multiple tokens, clients are allowed to make short bursts of requests — but over time, the rate is capped by how quickly tokens refill.

That’s token bucket algorithm in a nutshell.

When the bucket is empty, incoming requests can’t proceed. Most implementations reject them immediately (think 429 Too Many Requests). Others choose to queue requests until tokens are available, but that’s not part of the core Token Bucket algorithm — it’s an extra design decision depending on whether you value strict protection or smoother user experience.

Pros:

1. Simple concept to grasp and implement

2. Flexible as you can control the refill speed and bucket capacity

3. Throughput of an API is capped by the token refill rate

Cons:

1. Burst allowance can be risky: If many tokens are available (e.g., 50), the algorithm allows all 50 requests in an instant — which can look like a traffic spike (“thundering herd”) and overload downstream systems.

2. Stateful per client: Requires tracking a bucket for every user/client/IP, which can add memory overhead at scale.

3. Queuing isn’t built-in: If you want to delay rather than drop requests, you need extra queuing infrastructure.

Leaky Bucket Algorithm

This algorithm approaches rate limiting from a different perspective. Imagine the same bucket, but this time it has a tiny hole at the bottom, leaking water at a constant rate. Each incoming request is like a drop of water added to the bucket. As long as there’s space, the request goes in and eventually drips out at the steady leak rate. But once the bucket is full, any additional drops (requests) simply spill over and get dropped.

Code wise the steady flow is enforced by a timer (or tick) that “releases” requests at a fixed interval. It is literally like a buffer. You can fully control the rate of leaking and size of bucket.

Pros:

1. Request steadily flow into our system which solves the bursting issue in the token bucket algorithm.

2. It gives a strong back pressure by rejecting any requests once the bucket is full.

3. Simple to undertand and implement

Cons:

1. Increased response time as requests are getting processed in a steady flow

2. Anti burst, this was a pro but can be a con depending on the use case.

Fixed Window Counter

Now we move on to a completely different analogy: this one is all about time. Imagine a window of fixed length — say one minute, from 1:00 to 1:59. Within that window, only a certain number of requests are allowed through, let’s say 100. Once the cap is reached, any additional requests during that same window are rejected outright. When the next window starts (2:00 to 2:59), the counter resets, and the process repeats.

As we can see in the image above once the requests exceed the dotted line they get discarded.

However, this algorithm has one major disadvantage; if we look at the image below

The main flaw of the fixed window approach shows up when requests cluster around the boundary. For example:

• Between 1:30 and 2:00, a client sends 100 requests (the max).

• Then between 2:00 and 2:30, they send another 100.

Both windows independently look fine — each one respects the “100 requests per minute” limit.

But if you look at the overlapping interval 1:30–2:30 (which is still one minute long), the client actually made 200 requests. This breaks the intended rate limit and can overload your system.

Pros:

1. Easiest of all rate-limit algorithms to implement (just a counter and a reset timer).

2. No need to store request timestamps or scan logs.

3. Easy to communicate (“100 requests per minute”) and easy for clients to reason about.

Cons:

1. Requests clustered around window edges can exceed the limit in any rolling interval (e.g., 200 requests in 60 seconds instead of 100).

2. Bursty clients can exploit reset points, while evenly spaced clients are constrained more strictly.

Sliding Window Log

This algorithm fixes the problem with the previous one. Instead of counting requests inside aligned windows (e.g., 2:00–2:59), we keep a timestamped log of each accepted request.

Let’s say we set a rule: maximum 5 requests per 10 seconds. Instead of fixed one-minute blocks, this window is always moving forward in time.

Each time a new request arrives, we do two things:

1. Prune old requests → remove any logged timestamps that are older than now − 10 seconds.

2. Count the remaining requests → this gives us how many requests were made in the current rolling window.

If the count is still below 5, the new request is allowed and added to the log. If the count is already at 5, the new request is rejected.

In this way, the algorithm enforces the rule across any 10-second span, not just neat boundaries like 0–10 or 10–20.

Pros:

1. Enforces “no more than N requests in the last W seconds” with no fixed-window edge cases.

2. Boundary exploits (end-of-window spikes) don’t slip through.

Cons:

1. Memory heavy as it stores one timestamp per accepted request (hot clients = big logs).

2. CPU overhead where it must prune old timestamps on every request.

3. Heavy per-client load can bottleneck a single store key.

Sliding Window Counter

Think of it as a compromise between Fixed Window (too sloppy) and Sliding Log (too heavy).

Instead of tracking the whole window which can be heavy (store a lot of logs for 60 second window for example)

So we split it into buckets, so a 60 second window will have 60 buckets each of 1 second.

For example if we require maximum of 10 requests per 60 seconds;

• At 12:00:00–12:00:01, we got 7 requests (in bucket A).

• At 12:00:01–12:00:02, we already have 6 requests so far (in bucket B).

• A new request comes in at 12:00:01.5 (halfway into bucket B).

And we want to know: “How many requests happened in the last 60s?”

In the Sliding Window Counter, we look at all the buckets that overlap with the rolling window.

• Buckets that are fully inside the window are counted with a weight of 1.0 (their full value).

• Buckets that are only partially inside the window — usually just the oldest and the newest — are counted with a fractional weight equal to how much of them overlaps with the window.

So if it overlapped 4 buckets 2 of them being fully inside the window; the calculation is as follows:

1.0 * (fully inside bucket A request count) 
+ 1.0 * (fully inside bucket B request count) 
+ oldest fraction with window * (oldest bukcet request count)
+ newest fraction with window * (newest bucket request count)

If this total sum is less than the threshold (10 requests for 60 seconds) we allow it otherwise we reject.

At first glance, the Sliding Window Counter sounds neat — split time into buckets, weight the edges, sum them all up. But think about it: if your window is 60 seconds with 1-second buckets, that’s 60 buckets to check on every request. Bump that to a 5-minute window with millisecond precision and you’re suddenly tracking 300,000 buckets. Looking at every single bucket quickly becomes wasteful. The math is overkill, and the per-request overhead will crush performance at scale.

In practice, we don’t need to scan all the buckets. Notice that:

• Middle buckets are either fully inside or fully outside the window → no weighting needed.

• Only the oldest and newest buckets overlap partially and require fractional weights.

So instead of summing hundreds (or thousands) of buckets, we can keep:

1. A running total of all requests.

2. Adjust only the two edge buckets for overlap.

This way, each request check is O(1) — constant time regardless of how long your window is or how fine-grained your buckets are.

Pros:

1. Much lighter than Sliding Log no per-request timestamps, just bucket counters.

2. Fairer than Fixed Window smooths out boundary spikes by blending across buckets.

3. O(1) runtime per request, scales very well

Cons:

1. Approximate, not exact as counts are close but not perfect (depending on bucket size).

2. Complexity as implementation is trickier than Fixed Window or Token Bucket.

Summary

Rate limiting is essential in modern applications, and the algorithm you choose should depend on your business model and what you expect your limiter to achieve. Whether you need strict fairness, lightweight performance, or burst tolerance, there’s a strategy that fits. Choose wisely — and happy coding!

What’s going on everyone! I stumbled upon a very interesting project & have been playing around with it for the past couple of days, we all know about dumping and restoring data right? whether its databases or even raw files. However did you know that there was a way to checkpoint a running process and restore it later on? 🤯

Imagine this a long running task and we wanted to move it to a different VM for example, we could simply pause the execution saving every single detail of the process from instruction pointers to stack pointers, all memory, etc and restore it so it continues as if nothing happened. Here’s where CRIU shines.

CRIU

CRIU (Checkpoint and Restore In Userspace) lets you freeze a running Linux process, save its entire state to disk, and restore it later — like nothing ever happened.

It works mostly in userspace, and supports complex features like open files, memory, TCP connections, and more.

🔗 criu.org has all the docs and examples to get started.

Demo

I wanted to try something using this tool and i’ll demo it here, Imagine having a web server that takes in a request and takes some time to process the request. Will dumping & restoring whilst the request is processing work & actually return the response back? The answer is yes but let’s go into detail at what exactly happens to achieve this.

Installing CRIU

CRIU doesn’t work on macOS because it relies on Linux kernel features that macOS doesn’t have — and likely never will.

However I just spun an Ubuntu VM using digital ocean to get this demo done.

CRIU supports until Ubuntu 22.04, anything after that not yet. If you have an ubuntu version above 22.04 it won’t work.

We can use apt package manager to install it as follows

sudo add-apt-repository ppa:criu/ppa
sudo apt update
sudo apt install criu

Once installed we can verify using criu —version

Simple counter program to test the commands

Before moving on to the web server thing, I wrote a bash script that basically prints counts & sleeps between each iteration.

#!/bin/bash

i=1
while true; do
  echo "Count: $i" >> /tmp/count.log
  sleep 2
  ((i++))
done

Run this via ./count.sh & and the & is to make it run in the background.

If we tail /tmp/count.log we can see that it prints counts

Now let’s try the command to checkpoint from CRIU

First we get the PID via pgrep -f count → 11786

Then sudo criu dump -t 11786 -D /tmp/checkpoint --shell-job

Make sure a directory exists at /tmp/checkpoint

The command above will checkpoint & save the process status as files in the directory /tmp/checkpoint

it will Freeze, Checkpoint & kill the process.

—-shell-job flag is used here because It allows CRIU to checkpoint and restore processes that:

• Are attached to a terminal

• Were started from a shell

• Have a controlling terminal

This is to do with detaching it from the terminal’s process & session groups so it doesn’t forward any signals to the process.

Now the process stops completely, in fact it gets killed. But we can restore using:

sudo criu restore -t 11786 -D /tmp/checkpoint --shell-job

On restoring the count will begin to pick back up again. However if you restore multiple times it’s always going to start from the checkpoint it initially took the first time (at count 50 for example) even if it goes further on continuing it (you’re going to have to dump again).

Now let’s create a simple python web server than listens to a request and takes 10 seconds to process it and see what happens 👀

Python Webserver

First install python3

sudo apt install python3 python3-pip

from http.server import BaseHTTPRequestHandler, HTTPServer
import time

class DelayedHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(10)
        self.send_response(200)
        self.end_headers()
        self.wfile.write(b"Done after checkpoint!")

server = HTTPServer(('0.0.0.0', 8080), DelayedHandler)
print("Starting server on port 8080...")
server.serve_forever()

Run using python3 webserver.py &

Let’s check if its working using curl

curl  http://<VM-IP>:8080
Done after checkpoint!%

Now let’s run a request, dump & checkpoint and see what happens

We can dump & restore whilst in the middle of a request! When dumping the tcp-established flag make sure to preserve anything related to the tcp sockets in the process. It pauses them and the client has no idea what Is happening.

This is easy on the same host because they have the same IP address of course but if it were two different hosts we need to make sure that the addresses resolve to the other host otherwise it will fail.

Summary

This opens up room for endless ideas that could happen & docker uses CRIU for container migration where you would want to migrate a container from one place to another. This has been a small but dense article hope you enjoyed & see you in the next one!

What is happening everyone! Hope you’re all having an amazing start of the summer. In this article i’m diving into something I recently spent time reading about and was fascinated about how cool & useful it is.

In systems that produce logs whether it’s a web server, database, background job, etc the logs tend - overtime - to grow in size. For example a web server that writes to a log file every request would grow linearly as the users grow.

Over time this would significantly affect the disk size which overtime could cause the whole system to be slower. (peep the next article to know why)

The disk needed a hero, enter log rotation

Log rotation basically controls the whole process and prevents files from getting bigger (makes everything more controllable)

You can fully control how you want the logs to be rotated. Either every fixed period of time or once the file reaches a certain size.

The older log files then can be compressed to reduce its size & maybe send to some archive to free up that disk space. You can also control how many log files you can have before it starts deleting them. So to summarize;

Log rotation is a process or strategy that:

• Limits log file size or age

• Archives old logs (optionally compressing them)

• Starts fresh logs for continued logging

• Enforces a retention policy (e.g. keep last 7 logs)

Now that we understand what rotation is, let’s look at one of the most famous tools used for log rotation

LogRotate

Log rotate is one of the most popular and widely used log rotation tools, especially on Linux systems. It’s the de facto standard for rotating log files created by system services, daemons, and applications.

We’ll dive into a quick demo rotating logs for a service that produces logs nonstop

• To install on Mac brew install logrotate

• Ubuntu: sudo apt install logrotate

Once installed. The main entry point for the binary is the logrotate.conf file. On Mac It’s located in /opt/homebrew/etc/logrotate.conf

If we take a look at it we’ll find the following

# see "man logrotate" for details

# global options do not affect preceding include directives

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# packages drop log rotation information into this directory
include /opt/homebrew/etc/logrotate.d

# system-specific logs may also be configured here.

• weekly rotates log files every week (not by itself a cronjob is required to run the logrotate binary but it looks a week behind and rotates), There exists different options daily hourly & more

• rotate 4 keeps the last 4 rotations and deletes the rest.

• create creates new log files after rotating, be careful when using this because any process dumping logs has an open file descriptor to the log file, when a new one is created you’re going to have to manually update the process to point to the new one. copytruncate is a life saver here because it copies the logs to a new file and empties the original file. Meaning the file descriptor still points to that file.

• compress uses gzip to compress the rotated files to reduce size

• include /opt/homebrew/etc/logrotate.d basically is for application management where in the logrotate.d directory you can create different configurations for different applications each customized with their own set of settings. (We’ll dive into more below)

• Example of a simple app I created /opt/homebrew/etc/logrotate.d/myapp

    /var/log/myapp/*.log # specify the path of the logs to operate on
  
    {
    size 5k # Rotate when size is 5 kilobytes, K is kilobytes, M is megabytes, etc (was testing that's why its low) 
    copytruncate # copytruncate basically instead of creating a new file copies the data from the file to another file then empties it
    compress # gzip old logs
    rotate 3 # keep last 3 rotations
    missingok # if no logs are found don't panic
    notifempty # don't rotate if the log is empty.
    }
  

For all the configurations you can visit the man page for log rotate here

But with the setup above I have a simple script that writes logs

#!/bin/bash

logfile="/var/log/myapp/production.log"

# Function to generate a random log record
generate_log_record() {
    local loglevel=("INFO" "WARNING" "ERROR")
    local services=("web" "database" "app" "network")
    local timestamps=$(date +"%Y-%m-%d %H:%M:%S")
    local random_level=${loglevel[$RANDOM % ${#loglevel[@]}]}
    local random_service=${services[$RANDOM % ${#services[@]}]}
    local message="This is a sample log record for ${random_service} service."

    echo "${timestamps} [${random_level}] ${message}"
}

# Main loop to write log records every second
while true; do
    log_record=$(generate_log_record)
    echo "${log_record}" >> "${logfile}"
    sleep 1
done

Now we have:

1. Log rotate setup for our application

2. Logs being created

But we need a way to actually invoke the logrotate command because it doesn’t on its own and here we have two options:

• A cron job that runs every specified period

• Using a file watcher that watches file sizes and acts accordingly

In my case since I added the size config i’ll use a file watcher. On Mac I used fswatch

fswatch -0 /var/log/myapp/production.log | while read -d "" event
do
  size=$(stat -f%z /var/log/myapp/production.log)
  if [ "$size" -ge 5120 ]; then
    sudo logrotate -f /opt/homebrew/etc/logrotate.conf
  fi
done

Basically watch for the size of production.log once its more than 5 KB (small for testing) we rotate.

We can watch it in action to:

Watch how production.log reaches 5KB and rotates to be production.log.3.gz and the existing production.log.3.gz gets deleted

You can do so much more with rotations, add scripts that run on every rotate maybe upload the rotated files to some cloud storage, etc. Endless ideas can be made here.

Summary

Rotation is cool.

Hello guys! This is going to be a quick but interesting one. In design sometimes for faster response times & aggregation purposes we step away from the traditional relational databases and head over towards more analytical processing optimized databases (e.g Clickhouse which Is a columnar data store)

Maintaining a source of truth database is important and that will most probably be the relational one. We opt into syncing both either synchronously or asynchronously where data in Postgres for example is always passed over to Clickhouse which we can directly read and do aggregates and such.

I’m not here to talk about how to sync them together but the main goal is to keep both databases in sync. Both have the same exact data either right now or eventually but that’s a different story.

Sometimes they’ll deviate. Meaning Postgres has more data than Clickhouse. That could happen because either some messages weren’t processed as they should have or network errors, etc

Now you figured out the reason and pushed a fix, but you want to re-sync the missing parts again and the next question arises..

How do we sync the missing data again?

The answer of this will be depending on the scale of the data provided. And let me tell you from experience that that’s the most important part necessary to answer the question.

I’ll explain the next parts in a series of questions and answers and hopefully the answers help you reach a conclusion by the end of the article

How big is the data?

Data sizes can range from a few hundred thousand rows to a few million to almost billions of rows.

Smaller ranges have more options than larger ranges. Meaning it’s easier and takes less time to resync when you don’t have a lot of rows.

Larger ranges are where things get tough. You’ll reach a point where you start questioning yourself but hey everything is a learning curve I guess.

After figuring this out we opt into asking the second question

How big is the delta?

The difference between both databases also and the ratio between the delta and the size of the data is important to know.

Let me tell you before moving on that having a difference of 4000 records in a data size of 1b+ rows is something that’ll cause you headaches. This article is more aimed at that scale of data.

Now knowing the answers of the above two questions can already derive you to a solution

Small-ish data sizes

Small amounts of data (up to a few hundred thousand) can be easily re-synced. The most straight forward option is to Fetch the ids (any unique key really) here and there and just get the difference between both. Will take some memory but not a crazy amount. Once having the delta just re insert them into your analytical database and move on with your day.

Larger data sizes

Here is where the above solution will NEVER work under normal circumstances (unless you have a whopping 100gb RAM machine or something)

I’m going to stick with the 4000 records over 800Mil + records problem here. (We love extremes) And the solutions provided for the questions below are steps I took to actually solve this problem.

Now you’re going to have to play it carefully and ask new questions and these questions depend on the database optimization level you have and setup.

The setup we’ll assume is that there exists no partitioning no sharding just indexes and vibes. (Not the best setup for this amount of data)

The main goal here is to stay away from memory. And by staying away from memory I mean any application level logic you attempt will end up failing miserably.

Some questions to ask

1. Is there a possibility to close the searching gap down? for example do I have to search from the first record to the last or can I close that gap a little bit? Indexes will help but still take a long time.

For example if I figured out that the 4000 records got dropped only in the past month then I can utilize some timestamp (hopefully indexed) and work around that for a start.

So the first task is to minimize the search field as much as you can

2. What are the strengths of each database? Knowing this can help us make better decisions

a- For example Clickhouse is very very strong in aggregating and crushing numbers due to its columnar nature (data is stored physically together in columns not rows). Knowing this information, querying the ids over the date range specified in step 1 is key (a few seconds query) but instead of moving them to memory we write them to a csv file on disk. Something like this

# Clickhouse query
SELECT id 
FROM xxx 
WHERE xxx_date BETWEEN '2025-01-01' AND '2025-01-31' 
INTO OUTFILE 'clickhouse_ids.csv'
FORMAT CSV;

This will output all the ids that exist in that data range. They are missing 4000 records though but this is a solid step towards solving the problem.

b- Now in Postgres the goal is to take the ids from above and exclude them, finding the missing 4000.

Here comes a really cool trick I learned which is creating a Temporary Postgres table

Temporary tables are tables that get removed once the connection/session is closed. For example executing psql and creating a temp table then on exit it gets deleted

The idea here is to create this table, dump the ids in it and use it to query against the larger table. Offloading the memory overhead to Postgres. Memory use is offloaded to Postgres’ shared buffers, disk, and planner.

Even if the table is big, Postgres manages the spillover efficiently, avoiding app memory pressure.

Now executing something like this will be ideal

# Postgres psql
\copy (
  SELECT p.id
  FROM xx p
  WHERE p.xx_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND p.id NOT IN (
    SELECT id FROM clickhouse_ids
  )
) TO 'missing_ids.csv' WITH CSV

Giving that we have properly indexed the table on the date column and the id. Everything falls in the hands of the planner anyway.

Once this query finishes though you’ll have a 4000 row csv of the missing ids. Now all that’s left is just inserting them into Clickhouse.

Summary

Dealing with large data is definitely an experience. The application level solutions have no power here and thinking outside the box is definitely a must. This was an approach I personally used that worked better than I expected so thought i’d share. Thank you guys for tuning in & see you in the next one!

Hey everyone, Recently i’ve been working on migrating 1.5TB PostgreSQL worth of data to AWS from another cloud provider. I wanted to document the journey on the different attempts that were made and what did not work and what did. This is going to be a short one because honestly I’m out of ideas at this moment of time so forgive me 😅

Context

Migrating the data because the goal was to move everything to AWS. So part of it was migrating the database essentially.

Using AWS DMS 💀

The first attempt was to migrate the data using AWS’s data migration tool, and let me tell you a mistake that happened before everything is we should have spiked it out a couple of days before making this decision.

We started moving data using the DMS tool, so basically just migrating data from one place to another. At first we had no idea it doesn’t copy the schema so prepare yourself for the first headache

Headache 1: There’s no schema 💀

After everything finished we tried it out and it had the data, but we couldn’t create anything new (no sequences for ids defined), there was no indexes no constraints nothing. Then we realized that the DMS tool doesn’t copy schema. (wish I read about it before lol). But anyways continuing the story we decided to take a schema only dump —schema-only using pg_dump and apply it on the new db. Aaaand we ran into another problem

Headache 2: Default values for columns

Turns out that the DMS tool creates the bare minimal schema for it to be able to transfer the data (gotta have tables to be able to insert in them lol) and when it does this it skips the default values identified in columns. When we apply the schema the create table commands (with correct default values) don’t get triggered because the tables already exist. So indexes, sequences, constraints and triggers work but the default values are gone.

Unless you decide to go through every single table and do alter statements to add the missing default which really isn’t best practice if you think about it.

This problem spiraled into us trying different techniques to make it work, some of them were the following:

1. Say screw it and add the schema first and make it slower but integral.

2. Add the schema but disable indexes then enable afterwards

3. Try pg_dump using a .sql format, a .dump format, I don’t even remember the rest tbf

And congrats guys we reached our 3rd, 4th and 5th headache of the day

Headache 3: Invalid command galore in .sql dumps

So we took a .sql dump, tried it and well a spam of invalid command \N started to appear. Which means the file couldn’t be parsed correctly when attempting to restore. I couldn’t care less at this point and decided to just do a .dump format see if the error still persists and it went completely. But another one appeared (surprise)

Headache 4: pg_restore doesn’t work if you have generated columns

A generated column in PostgreSQL is a column that is physically stored on disk but is generated from other columns in the table, something like this

CREATE TABLE users (
  first_name TEXT,
  last_name TEXT,
  full_name TEXT GENERATED ALWAYS AS (first_name || ' ' || last_name) STORED
);

And apparently when restoring the .dump using pg_restore it worked for all the tables that don’t have generated columns but for the ones that did

pg_restore: error: could not execute query: ERROR:  column "xx" is a generated column
DETAIL:  Generated columns cannot be used in COPY.
Command was: COPY foo (id, name, ..., xx, ...) FROM stdin;

And the copy command fails completely. Turns out u can’t copy generated columns with pg_restore

Thought of dropping the columns, transferring the data and re adding the columns but that too didn't work.

What ended up happening

We decided to go through with a .dump of the whole database & schema. Optimizing the database as much as we can regarding insertions. Also parallelizing with pg_restore

Some of the optimizations that were made were: (all in postgresql.conf file)

1. Bumped shared buffers

2. Bumped maintenance_work_mem (Specifies the maximum amount of memory to be used by maintenance operations, such as VACUUM, CREATE INDEX, and ALTER TABLE ADD FOREIGN KEY.)

3. Bumped checkpoint_timeout which is the time we go okay let’s sync actual database with where WAL is committed at (kinda goes deeper but tldr)

4. disable fsync and synchronus_commit Which is not production friendly, this basically skips committing WAL to disk on every write (sometimes it can get batched when on but that’s not the case)

5. disable full_page_writes which is not production friendly too, basically instead of copying the whole page to memory to alter it it just partially updates it. (basically for crash recovery safety)

   This makes WAL much smaller and writes faster.

That’s it I guess, needed to rant about this so I wrote it as an article.

Thanks for coming to my tech talk guys and hope you enjoyed! till the next one

What’s happening everyone! In today’s article we’re going to be diving deep into creating a scalable top K list for the most liked videos in a configurable time. Top k questions are widely used in system design interviews and in real life as they provide really valuable insights depending on what domain they are used in.

We’re going to be going through a brief explanation of how Flink works, what the aim of the article is and we’re going to code everything up (I’ll leave the link for the GitHub repo at the end of the article). Let’s begin.

Before moving forward we’re going to imagine a scenario where we were tasked with engineering a top 5 most liked videos problem which moves us to the project requirements section.

Project Requirements

We’ve been asked by the business team that for insights the following is required:

1. View latest Top 5 most liked videos on our platform (Last 5 minutes for example, refreshes every minute)

   • We have a lot of likes coming in per second (around 200~400k/sec)

2. Display them in some fancy frontend

3. It’s okay if data skews a couple of seconds maximum

Out of Scope for now:

1. Keeping track of historical data for periods of time (we just want the live right now)

Now before moving into how we’ll design this thing, let me talk briefly about Flink

Flink Brief Intro

I’ve wrote an article on Flink before if you’re interested in the deep dives here

But for now i’ll give you what you need to know for this article.

What is Flink?

Apache Flink is an open-source system for processing streaming and batch data. It’s highly scalable and fault tolerant and superior in handling massive amounts of data. It’s a great tool for real-time analytics and continuous streams.

How does Flink work? (Very so much simplified)

Every Flink job has what’s called a Job manager, The job manager can have many task managers, the task managers have what is called slots. Every slot can execute some stage of the pipeline so basically a slot is like a unit of execution. It looks something like this (very much abstracted)

Notice the arrows back and forth from Task managers as they communicate and can send and receive data from each other

When you write some code (usually Java) and submit a Flink job. (Let’s say you want to aggregate counts of videos by their id)

Flink takes your code and creates what is called a Dataflow Graph which basically it then uses to know how to execute the given job.

For example aggregating some data stream and counting can look like this

stream
  .keyBy(video -> video.getId()) # Group by id
  .window(TumblingProcessingTimeWindows.of(Time.minutes(1)))
  .sum(1);

Behind the scenes, Flink translates this into a directed acyclic graph (DAG) where each node represents an operation (operator) — like keyBy, window, or sum — and the edges represent the data flowing between them.

This Dataflow Graph becomes the blueprint for execution. It helps Flink decide:

• How to parallelize the work (e.g., how many tasks should do the aggregation)

• How to shuffle or partition the data (e.g., based on the key) (data flowing between task managers)

• Where state needs to be managed (e.g., window state, accumulators) (in the example above a tumbling window every minute to the minute)

• And how to recover from failures by tracking operator state and checkpoints

This is what enables Flink to scale your job from running on your laptop to a 100-node cluster with minimal changes to your code.

The Flink Web UI even lets you inspect this graph visually, showing each operator and its parallelism, helping you understand exactly how your job flows end to end.

Scaling Flink is a whole different story. Depending on your scale, you’ll need to decide on things like the parallelism of your job, the number of TaskManagers, and the slots per TaskManager.

The parallelism defines how many parallel instances will be created for each operator in your job’s Dataflow Graph. Think of it like this: for every step in your pipeline (e.g., map, keyBy, window, etc.), Flink can spawn multiple subtasks to process data in parallel. The higher the parallelism, the more throughput your job can handle — assuming your hardware can keep up.

You can scale horizontally by:

• Increasing the number of TaskManagers (i.e., more JVMs on more machines)

• Assigning more task slots per TaskManager (i.e., more threads to run subtasks)

Flink’s job manager then maps the parallel subtasks of each operator onto these slots across TaskManagers, distributing the work evenly. If our stream source is Kafka then at least the partition count of the topic would be a good start.

Now that we got a high level on how Flink works, let’s talk about how we’ll design this thing

Technical Design

Let’s say we have 2 Kafka partitions and our parallelism is set to 2. (2 subtasks for every single operator/step), The goal is to somehow aggregate these into counts by video id and find a way to get the top 5 only. How can we do that?

Well a popular data structure for calculating top K from a bunch of elements is a PriorityQueue (Min-heap) but how can we leverage it in our design?

We’re actively reading from 2 Kafka partitions data like this {video_id: 12332}

We need to do the following:

1. Aggregate (Count) all same video ids

2. Push them into a Priority Queue of size 5 (fixed size/memory)

3. Return the result

However the steps above are missing something, we have 2 subtasks for aggregating the counts. This means we have one of two approaches here:

1. After aggregating push them all to a single node and generate a top k

2. generate local top k’s for every part and then send them to a node to generate a global top k

The first approach while being simpler but can overload the final destination node with a lot of data which can eat up the memory. Imaging having millions of aggregated records sending them to a single node.

The second approach is more efficient, we’ll generate a local top k for every aggregated part, send these over (only fixed size of 5 per node), and generate a top K out of the received local top K’s

The design looks something like this (This design is the technical thinking not how Flink will physically execute this)

Now since our requirements are not strict and we don’t need history, then every output of Global top k will be send to Redis, so the user can fetch it easily and display it. The key in Redis gets updated every 5 minutes in a Sliding window pattern (The Flink approach we’ll talk about in aggregation), so the design we’ll go with finalizes to this

Stages in Flink

Code here is highly abstracted the repo link will be at the end of the article

Define source

In Flink it goes Source → some operations → sink (destination)

We need to define Kafka as our source of data and this can be done with something like this

        props.setProperty("bootstrap.servers", "kafka:9092");
        props.setProperty("group.id", "video-likes-consumer");
        FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<>(
                "video_likes",
                new SimpleStringSchema(),
                props
        );

        DataStream<String> stream = env.addSource(consumer);

We’re now consuming from the topic video_likes

Steps (Aggregation to Top K)

Now The best thing about Flink is the different API’s it offers, out of the box API’s that do different type of aggregations, Since our requirement is the latest 5 minute most liked and refreshes every minute we can use a sliding window approach (e.g from 0-5, 1-6, 2-7, etc) as the window moves we get the latest 5 minute window only. Before doing that we would need to JSON parse the stream output to extract the video id.(view in repo). The important part here is the aggregation steps.

        likes
        .keyBy(value -> value.f0)
        .window(SlidingProcessingTimeWindows.of(Time.minutes(5), Time.minutes(1))) 
        .aggregate(new LocalTopKAggregator(5))

It starts off like this, keyBy would group and potentially reshuffle between task managers data so that all ids land and are processed in the same subtask.

Then we specify the window we’ll be working on (in our case sliding window for 5 minutes and moves every minute)

Now aggregate executes continuously as stream data comes in, It’s a custom class that inherits the Default Flink aggregation and adds the local priority queue logic on top of it. The code for this you’ll find in the repository for a more in depth look.

LocalTopKAggregator class has a method getResult which executes per window

This is the method that generates the top K per window.

    public List<Tuple2<String, Integer>> getResult(Map<String, Integer> acc) {
        PriorityQueue<Tuple2<String, Integer>> pq = new PriorityQueue<>(Comparator.comparingInt(t -> t.f1));
        for (Map.Entry<String, Integer> entry : acc.entrySet()) {
            pq.offer(Tuple2.of(entry.getKey(), entry.getValue()));
            if (pq.size() > k) pq.poll();
        }

        List<Tuple2<String, Integer>> result = new ArrayList<>(pq);
        result.sort((a, b) -> Integer.compare(b.f1, a.f1));
        return result;
    }

Now our output per subtask is the local top K, now we need to group these somewhere to generate the final top K

        likes
        .keyBy(value -> value.f0)
        .window(SlidingProcessingTimeWindows.of(Time.minutes(5), Time.minutes(1)))
        .aggregate(new LocalTopKAggregator(5))

        .map(list -> Tuple2.of("global", list))
         // Java erasure ***not business logic***   
        .returns(new TypeHint<Tuple2<String, List<Tuple2<String, Integer>>>>() {})
        .keyBy(t -> t.f0)
        .window(SlidingProcessingTimeWindows.of(Time.minutes(5), Time.minutes(1)))
        .process(new GlobalTopKMerge(5));

Now we map over our generated local Top K’s and give them all the key name count. Skipping the TypeHint returns which is Java specific to make sure the return type of the above map

We group by the key global

So now We have the key global along with an array of local top k’s

We only need to process the latest local top k’s so we add another window

If we don’t add this we’ll keep feeding the global top k with all the windows (it’s going to keep history of previous windows not remove them)

Now GlobalTopKMerge merges all the top K’s respectively and then pushes the result to redis.

    public void process(String key,
                        Context context,
                        Iterable<Tuple2<String, List<Tuple2<String, Integer>>>> elements,
                        Collector<Void> out) {

        PriorityQueue<Tuple2<String, Integer>> heap = new PriorityQueue<>(Comparator.comparingInt(t -> t.f1));

        for (Tuple2<String, List<Tuple2<String, Integer>>> localTopK : elements) {
            for (Tuple2<String, Integer> entry : localTopK.f1) {
                heap.offer(entry);
                if (heap.size() > k) {
                    heap.poll();
                }
            }
        }

        List<Tuple2<String, Integer>> finalTopK = new ArrayList<>(heap);
        finalTopK.sort((a, b) -> Integer.compare(b.f1, a.f1));

        pushToRedis(finalTopK);
    }

Sink

Finally pushToRedis sets a simple key on redis where the consumer consumes from,

    private void pushToRedis(List<Tuple2<String, Integer>> topK) {
      try {
        String redisKey = "trending:5min";
        String json = new ObjectMapper().writeValueAsString(topK);
        jedis.set(redisKey, json);
      } catch (Exception e) {
        System.out.println("Error pushing to Redis: " + e.getMessage());
      }
    }

Summary

Data has become an integral part in today’s modern world, The insights data gives can be a dealbreaker in making millions especially for startups. The best and most fun thing about system design is there is never a one size fits all, different approaches have different trade offs and business is an integral part of knowing which direction to head. Scaling this project can be in a separate article if you like & If you made it to here thank you & hope you learned something valuable today. See you all in the next one!

Github

• https://github.com/amrelhewy09/topK_kafka_flink.git

Today is all about probabilistic data structures, probabilistic data structures are data structures that use randomization and approximation to achieve efficient storage and processing of large-scale data. These structures typically trade off perfect accuracy for lower memory usage and faster performance, making them useful for handling big data, streaming data, and distributed systems.

Examples of using something like this would be when you want to get some count of millions of records, when you want to get the cardinality level in a set of data, when you want to check if something exists amongst millions of records. The aim is to sacrifice perfect accuracy for scalability and that is of course if the business is okay with something like this.

I’ll be going through the ones i’ve used and some more too, they are as follows;

1. Count-Min Sketch ✅

2. Bloom Filter ✅

3. HyperLogLog ✅

4. Skip Lists ✅

5. K-Minimum Values

6. LogLog & SuperLogLog

7. Top K & Heavy Hitters

Count-Min Sketch

Let’s say we are receiving a continuous stream of data {"video_id": 1} which are view counts, and we’re required to sum them up. The most straightforward way of doing this is by using a HashMap where we’d have the video_id as a key and the value being the total count.

The problem here is that the videos are a lot and the hash map size would increase significantly potentially being memory inefficient. If the business requirement was not strict on showing the exact view counts then count min sketch is the way to go.

Count-Min Sketch is a probabilistic data structure that provides an approximate frequency count of elements in a data stream using constant space.

Instead of storing each video’s view count in a HashMap, which grows as the number of videos increases, we use a 2D array (matrix) of counters with multiple hash functions to track approximate counts.

Step by step breakdown:

1. Initialize a matrix (w × d)

   • w columns (width): Represents the number of counters in each row.

   • d rows (depth): Each row corresponds to a different hash function.

   • All counters start at 0.

2. Updating the Count

   • When a new view event { "video_id": 1 } arrives:

     • The video ID is hashed using d different hash functions.

     • Each hash function maps the video ID to a column in its respective row.

     • The counters at those positions are incremented.

3. Querying the Count

   • To estimate the view count of a specific video ID, hash it with the same d hash functions and mod it with the column length. The more columns you add the more accuracy but also more space.

   • Look at the corresponding positions in each row and take the minimum value across all rows.

   • This minimizes the effect of hash collisions (hence "count-min").

The reason we pick the minimum value is to minimize the possibility of hash-collisions where different video ids might hit the same row & col. That’s why when always picking the minimum would minimize the collisions. Sometimes also we can have 5 hash functions instead of 3 which would increase the accuracy. But increasing the number of hash functions wouldn’t always be beneficial as there is a point where the more you add won’t offer any significant improvement in accuracy. Below is a gif example of how it works.

Bloom Filters

Another cool probabilistic data structure now, bloom filters is a space efficient data structure that tells you if an element is 100% not in a set of elements or if maybe is.

Meaning it may give false positives but never gives a false negative so it either definitely doesn’t exist or maybe exists.

Similar to the idea of count min sketch a value gets passed in multiple hash functions and the results would then map to different indexes in a fixed size bit array and the bits at these indexes are set to 1.

So if we’re looking for a value we pass it through multiple hash functions and check if the bits in the respective indices are 1 or not. If there exist any zeroes then its 100% not in the set of elements but if they’re all ones then it maybe in the set or maybe just collided with other members in the set.

Here’s how they work

Skip Lists

Skip lists are really efficient data structures that help optimize searching, insertion and deletion in linked lists. In sorted linked lists finding elements will always take O(n) time because you have to traverse to reach the required node. However a way to optimize that is by using skip lists.

They are simply levels above each other (the levels are called express lanes (faster routes)) on top of a basic linked list, something like this

And when traversing we start from the top left 1 and check the next station in the express lane. So for example if we’re looking for the value 5

1. Check the first top left value and its next station in express lane 4

2. since 5 > 4 we go through the express lane

3. we check the next express lane station 6

4. since 6 > 5 we traverse through the basic linked list till we find 5

This can go for multiple levels having a 1 → 5 layer above the one in the image above. This helps to skip a lot of nodes which would optimize search, insertion and deletion time effectively.

Here’s an example if we’re looking for the value 5 in the skip list.

HyperLogLog

This one is all about cardinality, it would give you an estimation on how many unique items exist in a set of elements and is efficient memory wise.

Think of a lottery where you randomly pick a number between 1 and 100.

• If you only draw 5 numbers, you probably won’t get anything close to 100.

• If you draw a million numbers, chances are, you'll eventually get 100.

🔸 The more numbers you pick, the higher the chance of getting a big number.

HyperLogLog works the same way, but instead of picking numbers, it’s looking at how many leading zeroes appear in a hash.

1. Each an every element gets hashed into a random binary hash (a binary representation of a hashed value)

2. Once we start hashing elements, we look at where the first 1 appears in the binary hash and remember the biggest number we've seen so far.

The more unique values we hash, the higher the chance that one of them has many leading zeroes.

If you’ve seen a hash with a 1 at position 7, that means you must have seen a lot of unique elements to get such a rare case.

HyperLogLog tracks the largest position seen and uses math to estimate how many unique elements must exist for that to happen.

Now that items are hashed, they are split into different buckets, buckets help smooth out extreme cases where for example we have 3 items but luckily a leftmost 1 was found at the 7th position in one of the items. let me explain

Buckets are small memory slots where HyperLogLog stores information separately for different groups of elements.

Imagine you're counting unique people entering a stadium.

• If you only look at one entrance, you might get a bad estimate because not all people use that entrance.

• Instead, you track multiple entrances separately and combine the results to get a better estimate.

Each bucket only remembers the biggest value it has seen.

Now, instead of estimating based on one extreme value, we take an average of all the buckets.

For example:

• If all buckets saw only small values (1, 1, 2), we probably have few unique elements.

• If some buckets saw high values (like 5, 6, 7), that suggests we have a lot of unique elements.

It’s kind of confusing but the more buckets the better accuracy you might have

Then we proceed to take the average of all the buckets to estimate the number of unique elements using a mathematical formula. (something called the harmonic mean)

Hello folks! in this article I’m going to be going through all the needed calculations to properly tune your rails app in production. This article aims to provide a schematic for calculating the threads needed both application wise and database wise so no request or background job fails to acquire a database connection when things get heated. Let’s start.

Puma and its workers

First of all let’s start off by talking about Puma which is the default rails web server and how it operates generally.

When booting up puma. It has configurations related to the number or workers and the number of threads per worker but what are both?

When setting the worker variable to 2 for example. Puma will then fork its operating system process however many times you set worker (in our case 2). This means you will have workers count of your rails code as instances ready to serve http requests.

In each puma worker there will be multiple threads based on the threads configuration. However due to the GIL lock in ruby, only one thread can be executed at a moment of time unless this thread is doing some blocking operation (I/O) then the GIL lock is released and other threads can run safely.

So far that means if we have an instance with 2 workers and 2 threads per worker we will have the following number of threads:

Number of DB threads = Worker count * thread per worker = 2 × 2 = 4

That means for each thread we need to reserve a potential thread from the database connection pool because potentially every application thread might connect to the database at one point depending on the load.

Rails maintains its own database connection pool, with a new pool created for each worker process. Threads within a worker will operate on the same pool.. If a Puma Worker utilizes 5 threads per worker, then the database.yml must be configured to a connection pool of 5, since each thread could possibly establish a database connection.

Since each Worker is spawned by a system fork(), the new worker will have its own set of 5 threads to work with, and thus for the new Rails instance created, the database.yml will still be set to a connection pool of 5.

Rails Database connection pool VS Actual Database Pool

It’s important before moving forward to be able to differentiate between these two as they might confuse a lot of people. In each rails app it maintains its own database connection pool. This is nothing but a pool of threads reserved for the database connection when needed a thread gets picked from the pool, does its job and goes back to be reused again later on. It’s usually by default set by the <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %> This resembles the max threads in a single puma worker. Meaning for every instance of the ruby code invoked we will have a db pool of the thread count specified in this env.

Actual database pool for example the PostgreSQL connection pool (or the connections PostgreSQL accepts) defines how many connections the PostgreSQL server can manage simultaneously across all clients where it limits the total number of concurrent database connections the PostgreSQL server can handle. Controlled in postgresql.conf by the max_connections parameter we can alter it accordingly.

This will come in handy later on so make sure you understand the difference before proceeding.

Sidekiq

Now let’s imagine this scenario.

1. Postgres max_connections is set to 100

2. We have a rails app operated by Puma web server and the config is as follows:

   1. workers is set to 5

   2. max_threads is set to 20

Knowing this information We have 100 max_connections from Postgres side and 5×20=100 threads potentially connecting to Postgres from the Application side. They are the same count

Now comes in a background job process such as sidekiq. And sidekiq has it’s completely separate configuration regarding concurrency where concurrency is the number of threads for sidekiq to operate on.

Threads in Ruby operate under fundamentally different paradigms, largely due to the Global Interpreter Lock (GIL) in MRI Ruby. In Rails (running under a server like Puma), each thread is responsible for handling one request at a time. There is no true concurrency for CPU-bound tasks because of the GIL in MRI Ruby. Threads in Rails can perform concurrent operations when waiting for I/O-bound tasks (e.g., database queries, external API calls). During such waiting periods, other threads can process requests. A thread finishes one request entirely before moving on to the next.

In Sidekiq the GIL lock is still there but the nature of job processing makes it feel more concurrent. Because job processing might have a lot more I/O (db access, external API calls) it has a lot more context switching between threads than the web server (GIL gets released).

When tuning the postgres max_connections we need to make sure that we take sidekiq into consideration.

The equation becomes as follows:

pg_max_connections = (Puma worker count * RAILS_MAX THREADS) + (Sidekiq concurrency * Sidekiq process count)

Meaning if we have 1 sidekiq process with a concurrency of 5 We need to increase the max_connections from 100 to 105 so each thread can potentially grab a connection to the database.

Summary

When stress testing and scaling a Rails application, it’s crucial to understand how all components work together seamlessly. A lack of database connections leading to 500 errors is one of the worst experiences a user can face, and addressing this should be a top priority. By gaining deeper insights into these mechanisms, you can ensure your application is resilient, scalable, and user-friendly. I hope this article has helped clarify these concepts, and I look forward to sharing more in the next one!

Also subscribe to the YouTube i’ll be doing more than Leetcode over there soon 😄https://www.youtube.com/@techstuffrandom

Hello folks! in this quick article i’m going to be talking about how a database like Postgres actually persists to disk and what happens behind the scenes. What is WAL all about and what does it even stand for? Let’s dive in.

When it comes to I/O operations (in our case writing to disk). Optimizations become a must because I/O operations are very expensive especially writing to disk due to a lot of factors. Postgres tackles this problem in a clever manner.

When a write request is made to the database, You’d think that it would persist it to disk right away, but that’s not what actually happens behind the scenes. Introducing Postgres Shared Buffers

Shared Buffers

When a write is made it isn’t instantly flushed to disk, Postgres actually loads the page getting impacted by the write to memory and adjusts it in memory.

Postgres saves rows in pages and whenever we update a certain row we find the page it’s in, load it to memory and update the page with the new record respectively. Same for writes whether its a new page or an existing one that’s not yet full.

This is called a dirty page where it needs to be written back to disk later. Later on Postgres relies on background processes like the background writer or checkpoints to write dirty pages from shared buffers to disk asynchronously. (We’ll get into that)

But the thing is this approach hugely minimizes I/O by batching the flush operation to disk and not have it going on for each and every write.

On the other hand, when a read request is made Postgres first checks the shared buffers for the pages being requested (whether dirty or not). If found it then proceeds to serve it from there, if not it loads the page from disk and adds it to the shared buffers cache. If the shared buffer is full a victim dirty page will be written to disk and the newly read page will be replaced by it.

Now the question that arises is what happens if you lose power half-way through writing to the data files? Let’s say some write was made to memory and during flushing it to disk a power cut happened causing that data to be inconsistent and weird. The client still thinks he made that write when In reality it never persisted to disk completely. This can cause a lot of data corruption and integrity loss. Hence introducing WAL

WAL (Write Ahead Log)

WAL or Write Ahead Log is a mechanism to ensure the consistency and safety of data. It is a technique where every change to the database is logged before it is applied to the actual data files. This log acts as a journal that records all modifications. It is an append only log that logs everything a user writes to the database while simultaneously updating the data pages in shared buffer as mentioned before. Any newly written data can remain in the shared buffer as long as we have a log that tracked the change. In case anything goes wrong we can reconstruct the state from the log.

The WAL is much smaller than the actual data files, and so we can flush them relatively faster. They are also sequential unlike data pages which are random. Disks love sequential writes, which is another benefit of WAL.

Every WAL entry is first written to a WAL buffer in memory. Then when a certain trigger comes this buffer gets flushed to disk. The trigger can either be when the buffer reaches a certain size or when a certain period passes. These are all configurable from Postgres’s side. Once flushed to disk these entries can be announced committed.

The database can crash after writing WAL entries (before flushing shared buffer to disk), that is fine, as long we know the transaction state belonging to each WAL entry we can discard or omit uncommitted WAL entries upon recovery (to ensure data consistency).

For example if you are in the middle of a transaction and the database crashed, we consider the transaction rolled-back by default. I will do another article explaining how WAL actually writes transactions

When all the data files have been flushed and updated to reflect the information on the WAL This is something called checkpointing. Once this happens a checkpoint record in the Write-Ahead Log (WAL) is recorded, marking the point up to which all changes have been flushed to disk.

In the event of a crash, the crash recovery procedure looks at the latest checkpoint record to determine the point in the WAL (known as the redo record) from which it should start the REDO operation.

References

1. https://x.com/hnasr/status/1867253354662920379

2. https://www.postgresql.org/docs

UUIDs

UUID stands for universal unique identifiers. They allow generating ids in a way that guarantees uniqueness without knowledge of other systems. They have many versions some are not used as much anymore let’s discuss each version

UUIDv1

A UUID version 1 is known as a time-based UUID and can be broken down as follows:

UUIDv1: [time_low]-[time_mid]-[time_high_and_version]-[clock_seq_and_reserved]-[node]

UUIDv1 uses a 60-bit timestamp that represents the number of 100-nanosecond intervals elapsed since the Gregorian epoch: 15 October 1582 00:00:00 UTC. This timestamp is used to ensure that UUIDs generated at different times are unique. e.g 13743895347200 represents the number of 100 nanosecond intervals passed since epoch. Using the nanosecond representation allows for more uniqueness and finer granularity. Using 100-nanosecond intervals allows for up to 10 million unique time intervals per second.

The total timestamp is 60 bits:

• Most significant bits represent older times.

• Least significant bits represent the finer granularity within the most recent time interval.

Timestamp and its 60-Bit Division in UUIDv1:

The 60-bit timestamp is divided across three fields in UUIDv1:

1. Time Low: Stores the least significant 32 bits of the timestamp.

2. Time Mid: Stores the next 16 bits.

3. Time High: Stores the most significant 12 bits.

These bits are arranged in a specific format to construct the UUID.

Node is a 48-bit value, often derived from the MAC address of the machine generating the UUID. If the MAC address is unavailable, a random value is used instead.

The Clock sequence in UUID Version 1 (UUIDv1) is a 14-bit value that ensures uniqueness when generating UUIDs in situations where the system clock may not be reliable.

It helps maintain uniqueness when:

• The system clock is adjusted backward.

• The system clock cannot guarantee monotonicity (e.g., due to manual adjustments or hardware issues).

UUIDv2

In V2 the low_time segment of the structure was replaced with a POSIX user ID. The theory was that these UUIDs could be traced back to the user account that generated them. Since the low_time segment is where much of the variability of UUIDs reside, replacing this segment increases the chance of collision. As a result, this version of the UUID is rarely used.

UUIDv3 & UUID v5

Versions 3 and 5 of UUIDs are very similar. The goal of these versions is to allow UUIDs to be generated in a deterministic way so that, given the same information, the same UUID can be generated. These implementations use two pieces of information: a namespace (which itself is a UUID) and a name. These values are run through a hashing algorithm to generate a 128-bit value that can be represented as a UUID.

The key difference between these versions is that version 3 uses an MD5 hashing algorithm, and version 5 uses SHA1.

UUIDv4

Version 4 is known as the random variant because, as the name implies, the value of the UUID is almost entirely random. The exception to this is the first position in the third segment of the UUID, which will always be 4 to signify the version used.

UUIDv6

Version 6 is nearly identical to Version 1. The only difference is that the bits used to capture the timestamp are flipped, meaning the most significant portions of the timestamp are stored first.

[time_high_and_version]-[time_mid]-[time_low]-[clock_seq_and_reserved]-[node]

The main reason for this is because UUIDV1 had problems when it came to two types or sorting,

Lexicographical order is similar to how words are arranged in a dictionary or alphabetical order. It's based on the lexicographic (dictionary) comparison of characters or symbols in a string, from left to right.

Chronological order refers to sorting based on time — from the earliest to the latest (or vice versa). This is the kind of order you'd expect when sorting timestamps, dates, or events. In chronological order, items are compared based on their relative timing.

UUIDV1 is designed to be unique identifiers that can include a timestamp. However, its structure can cause confusion because lexicographical sorting does not always align with chronological sorting, due to how fields are ordered. e.g

UUID1: a1b2c3d4-e5f6-11ec-9abc-123456789abc
UUID2: a1b2c3d5-e5f6-11ec-9abc-123456789abc

• Chronological order would expect UUID1 to come before UUID2 because UUID1 represents an earlier time.

• Lexicographical order, however, would first compare time_low (the first field a1b2c3d4), and since a1b2c3d4 is less than a1b2c3d5, it might place UUID1 before UUID2 — this happens correctly, but in some cases, this alignment doesn’t hold true.

• If the time_high_and_version field were placed at the start, then chronological sorting would naturally match lexicographical sorting, as in UUIDv6.

UUIDv7

Version 7 is also a time-based UUID variant, but it integrates the more commonly used Unix Epoch timestamp instead of the Gregorian calendar date used by Version 1. The other key difference is that the node (the value based on the system generating the UUID) is replaced with randomness, making these UUIDs less trackable back to their source.

Use Cases for UUIDs

• Session Identifiers

• File Storage and Versioning

• API Tokens and Authentication

• E-commerce and Order Tracking

• Event Tracking and Logs

ULIDs

A ULID is a 128-bit identifier, represented as a 26-character string encoded in Base32 (with a specific alphabet). It has two main components:

• Timestamp (48 bits or 6 bytes) it is the first component of a ULID and is stored in milliseconds since the UNIX epoch (1970-01-01 00:00:00 UTC). The timestamp is packed into the first 48 bits of the ULID, which allows it to be lexicographically sortable. This means ULIDs generated in chronological order will sort correctly without needing special sorting logic.

• Randomness (80 bits or 10 bytes) to ensure that ULIDs are globally unique, the second part (after the timestamp) is made up of 80 random bits. This random component ensures that even if two ULIDs are generated at the exact same millisecond, they will still be distinct.

Example:

• Base32 alphabet: A-Z, 2-7

• ULID: 01FZQZ4E0AMK4NK9F7J8N9DAX8

Use Cases for ULIDs

• Distributed systems: When you need globally unique IDs that are generated independently and can be sorted chronologically.

• Database indexing: ULIDs can be used as primary keys because they are lexicographically sortable, reducing fragmentation and improving performance.

• Caching: When creating time-sensitive keys or identifiers that need to be unique and sortable.

• Also all UUID use cases can work here too.

UUID vs ULID

NanoIDs

Nanoid is a small, secure, and URL-friendly unique identifier generator, typically consisting of a fixed-length string of random characters. It is much smaller in size compared to UUIDs and ULIDs, making it more efficient for use in contexts where shorter identifiers are needed. They do not contain timestamps and is completely random.

Key Characteristics of Nano IDs:

• Length: A NanoID is around 21 characters long, but the length can be customized.

• Alphabet: It uses a custom alphabet that avoids characters that might be confusing or problematic in URLs (like /, +, and =).

Nanoids are typically generated from a cryptographically secure random source, and the characters in the resulting identifier are drawn from a custom alphabet.

cryptographically secure random numbers (CSPRNGs) alone do not guarantee uniqueness. They ensure that the values generated are unpredictable and hard to reproduce, but they do not inherently ensure that each generated value is unique across all possible values.

The default alphabet used by Nanoid consists of the following 64 characters:

abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789

e.g V1StGXR8ZtM08l5c5yLyzLq

Snowflake IDs

Snowflake IDs are another form of unique identifier, popularized by Twitter and used in systems like Twitter’s distributed ID generator. They are designed to be unique, sortable, and compact, and are particularly well-suited for high-performance, distributed systems.

A typical Snowflake ID is a 64-bit integer, and the structure is usually broken down as follows:

• Timestamp (41 Bit) Millisecond timestamp since epoch

• Datacenter ID (5 Bit)

• Worker/Node ID (5 bit)

• Sequence (12 bits) Sequence number for handling multiple IDs generated in the same millisecond

• total bits (64) which is the total size of the snowflake id.

Key Characteristics of Snowflake IDs:

1. Time-based: The first 41 bits represent the timestamp, which makes Snowflake IDs chronologically sortable.

2. Machine/Node-aware: Snowflake IDs include the datacenter and worker (node) IDs to avoid collisions in distributed environments.

3. High throughput: With a 12-bit sequence number, Snowflake IDs can generate up to 4096 IDs per millisecond per machine, ensuring high throughput in distributed systems.

4. Compact: The 64-bit integer format keeps the ID size compact, reducing storage and indexing overhead.

5. Unique: The combination of timestamp, machine ID, and sequence number guarantees global uniqueness.

Best Use Cases for Snowflake IDs

• High-Performance Distributed Systems

• Event Sourcing

• Scalable Web Applications

• Logging and Monitoring Systems

A comparison between it and UUIDs

Summary

In this article we went through the most popular unique identifier generation schemes listing the use cases for each and every one. If you’re planning on adding one I recommend understanding the main differences between them before making a decision because choosing a wrong unique identifier scheme in a big scale could affect performance in a negative way. That’s been it for this one see you in the next!

References

1. https://planetscale.com/blog/the-problem-with-using-a-uuid-primary-key-in-mysql#uuidv4

2. https://adileo.github.io/awesome-identifiers/

Naive Approach

Let’s say we need to download a large file in our application code and save it somewhere on disk.

The naive approach someone would do is the following:

func downloadFile(filepath string, url string) error {
    out, err := os.Create(filepath)
    if err != nil {
        return err
    }
    defer out.Close()

    resp, err := http.Get(url)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    data, _ := io.ReadAll(resp.Body) // Whole file in memory btw
    printMemStats()

    _, err = out.Write(data)
    return err
}

This snippet does the following:

1. Creates a new file for the <to be downloaded file>

2. Downloads the file

3. Copies the bytes from the downloaded blob to the created file

I have the function printMemStats that’ll give us info on memory usage

Running this on a 100MB file and monitoring memory we can deduct the following:

Alloc: 117.94 MB (currently in use)
TotalAlloc: 587.91 MB (total allocated memory) (This is the total amount of memory that has been allocated by the program since it started. It includes both the memory that is still in use and the memory that has been released by the garbage collector.)
Sys: 253.92 MB (total memory reserved from the OS)
HeapAlloc: 117.94 MB (heap memory in use)
HeapSys: 247.11 MB (total heap memory reserved from OS to the app)
NumGC: 21 (number of garbage collections)

Looking at the Allocated memory 118MBs were in use and it makes sense because the downloaded file was 100MB alone + some other memory required by Go runtime.

Imagine this file instead being of size 1 GB. Having a single process in the go app honking 1 GB of memory is very bad practice. We can do better let’s now discover the art of streaming data

Streaming Data

So the idea simplified is instead of having the flow look like this

How about we get rid of the red part and go straight to writing to the file!

The write buffer is made internally when calling out.Write(body) Go internally writes to a buffer before flushing to disk to decrease IO calls which are very expensive.

The main goal and what I love the most about this is that it’s just plain creativity! Instead of having to download then transfer we can just skip the whole download part and as the bytes come in we write them to the file having minimal memory overhead. Let’s translate this to our code and check the memory stats after updating.

func downloadFile(filepath string, url string) error {
    out, err := os.Create(filepath)
    if err != nil {
        return err
    }
    defer out.Close()

    resp, err := http.Get(url)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    io.Copy(out, resp.Body) // STREAM THE BODY TO FILE

    printMemStats()

    return err
}

Now io.Copy internally buffers between source and destination

Running the code the memory stats are as follows;

Alloc: 0.549 MB
TotalAlloc: 0.549 MB
Sys: 8.209 MB
HeapAlloc: 0.549 MB
HeapSys: 3.776 MB

Almost a 99% decrease in memory! Not only is it memory efficient it's much faster as well because we skipped a whole step!

Not only is this used for files or downloads, also can be used when passing blobs as function arguments as well. Because they’d copy by value so get duplicated for the function we can pass instead a reader that reads from source to destination and process as you go.

Summary

Most modern open source applications out there use tricks like these to optimize for memory and performance. Skipping buffers and unnecessary overhead operations when they can. It’s the deep understanding of what goes on behind the scenes that helps open gaps for optimzation. When you visualize it as well its much more clearer and gives guidance on what to do.

Hello folks! In this one we’re going all in on authorization and authentication in Kubernetes. Whenever you get access to a Kubernetes cluster in your job do you ever wonder what happens behinds the scenes? The DevOps guy just sends you a kubeconfig yaml file and you start using it. We’ll be going through the basics of Authentication and Authorization in Kubernetes covering 3 main parts:

1. Kubernetes Authentication Workflow

2. Authentication Methods in Kubernetes

3. User vs. Pod Authentication

Authentication Workflow

The Kubernetes authentication workflow serves as the first line of defense, verifying "who" is making the request to the Kubernetes API server which is the main entry point for all Kubernetes operations. Every request to the cluster, whether it’s from a kubectl command, a CI/CD pipeline, or an application, must pass through the API server.

Any request that comes in passes by the first phase which is Identity Assertion. Usually the request would carry any of the following:

• Client Certificates: TLS certificates that the API server can validate.

• Bearer Tokens: Passed in the Authorization header of the HTTP request.

• Custom Mechanisms: OpenID Connect (OIDC) (AWS IAM) , etc .

Kubernetes supports a pluggable authentication mechanism. The API server evaluates the configured authentication plugins in the following order:

• Static Token File.

• Client Certificate Authentication.

• Webhook Token Authentication.

• OpenID Connect (OIDC).

• HTTP Proxy Authentication.

If a plugin authenticates the request, the process stops, and the identity is assigned to the request and it moves to the Authorization phase of the request. Otherwise it returns a 401 Unauthorized.

Once authentication confirms the identity, the request moves to the authorization phase. In this phase:

• Kubernetes evaluates whether the authenticated user has permission to perform the requested action on the specified resource.

• Authorization mechanisms include for example Role-Based Access Control (RBAC)

Once authorized the request is then being processed and returned to the caller.

Authentication Methods in Kubernetes

As mentioned above there exist different authentication methods in Kubernetes, these methods cater to both user (human) and machine authentication. Here’s an overview of the main methods:

• Static Token File where tokens are stored in a static file provided by the Kubernetes API server. These tokens never change until changed manually. Not ideal for production environments as the token could get stolen by an attacker and used for malicious intentions.

• Service Account Tokens where tokens are automatically generated and mounted inside pods, stored in Kubernetes Secrets. These tokens are used with Kubernetes Service Accounts which are tied to specific permissions, Ideal for pod-to-API server communication, especially for applications running inside the cluster. Before Kubernetes 1.21 they used to be static but after they are renewable and bound to stricter audiences and permissions.

• Client Certificate Authentication uses TLS certificates to verify identities, ensuring that only trusted entities can access the Kubernetes API server. Can be used to authenticate humans alone or with service accounts together. A certificate authority generates and signs a user’s certificate and validates against it.

• Custom mechanisms include OpenID Connect where Kubernetes asks an external OID provider for authentication instead of using its native one.

User vs Pod Authentication

Kubernetes provides distinct mechanisms to authenticate users (human operators or external tools) and pods (workloads running within the cluster).

User Authentication

User authentication refers to how human users or external tools (e.g., CI/CD pipelines) authenticate with the Kubernetes API server to perform actions like managing resources. Methods include:

1. Certificate Authentication

2. Static Tokens (Not recommended)

3. OpenID Connect (OIDC)

Kubernetes doesn’t manage users directly; external systems (e.g., certificates, identity providers) are required.

Pod Authentication

Pod authentication refers to how workloads running inside the cluster (e.g., pods) authenticate with the Kubernetes API server to perform actions like reading secrets or interacting with resources.

The main way to do this is by service accounts, where each pod is automatically assigned a service account and the tokens get mounted into the pod.

These tokens are used by applications within the pod to authenticate with the API server.

The benefits of this is that its namespace scoped and really controlled.

Summary

There you have it! a brief introduction about authentication and authorization in Kubernetes, I think as a Backend Developer its important to understand these concepts and appreciate how dynamic and flexible Kubernetes made it. The designs can carry on to your day to day work.

What’s Next?

I’ll be doing a demo where image someone new joined the company and needs minimal access to the cluster. I’ll be walking through the basic steps to do so. See you there!

Hello folks and welcome to the second part of the series I made where I discover DevOps concepts that I wanted to understand as a backend engineer. In this one we dive into Kubernetes AutoScaling where we’ll be going through its basics and testing it to make sure we understand everything going on. Let’s start!

AutoScaler Basics

Metrics Server

An auto scaler scales automatically when certain metrics reach an agreed upon threshold. Simple right? there’s a lot more to it though

AutoScaler relies on a metrics source in order to actually watch for metric changes. Kubernetes uses a component called the Metrics Server to collect resource metrics (like CPU and memory usage) for pods and nodes in a cluster. The Metrics Server aggregates these metrics and makes them available to components like the Horizontal Pod Autoscaler (HPA) and other monitoring tools.

• The Metrics Server is a lightweight, cluster-wide aggregator of resource usage data (like CPU and memory) for nodes and pods.

• It does not store historical data — it only provides the current resource usage (live metrics).

• The Metrics Server collects data from the kubelet (the primary node agent that runs on each node).

The kubelet exposes the metrics of the nodes’ containers/pods on port 10250 /metrics/resource

In managed Kubernetes Enviroments (EKS, GKE) the metrics server by default is installed in the cluster. However if you’re using something like kind or Minikube it isn’t installed by default. To install it to your local cluster

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

And Verify using this command & make sure it’s running.

kubectl get deployment metrics-server -n kube-system

the command kubectl top pods is only available to use once the metrics server is installed and running

Now we have a metrics server pulling metrics from every nodes’ kubelet. We need to do something with this information, yep you probably guessed it, autoscaling!!

Nginx Deployment (Example)

Before moving forward to the actual autoscaling, our example will include a simple nginx deployment where we will monitor CPU usage and add an autoscaler to this deployment. We will do a infinite while loop hitting requests to the nginx server and watch the autoscaling happen.

This is the deployment/service manifest file for nginx

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "200m"
            memory: "256Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: LoadBalancer

Each pod has a request of 100 millicores CPU (0.1 Core) and 128 Mebibytes and a limit of 0.2 Core & 256 Mebibytes

Mebibytes are binary units (unlike megabytes which is decimal). In computing, memory is inherently binary (base-2). For example, RAM sizes are measured in powers of 2 (e.g., 512 MiB, 1 GiB). MegaBytes can cause confusion because its decimal nature doesn’t match binary-based memory calculations. 1 Mebibyte =1024 ² Bytes

HPA Manifest

The basic autoscaling manifest looks something like this (In this context we have an nginx deployment where we’re going to apply HPA to it)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-deployment
  minReplicas: 2
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 20

When the average CPU utilization of all current running pods of the nginx deployment exceeds 20% (Just an example not practical obviously) autoscaling starts.

Let’s say we have 2 replicas already running where both have a cpu utilization of 25%

Thus the average cpu utilization is:

(utilization of first pod + utilization of second pod)/ 2 (pod count) = 25%

In order to know how many more replicas we need so our average utilization becomes under 20% again we can see the ratio between the actual and target utilizations 25/20 = 1.25

Meaning, the actual utilization is 1.25 x the target utilization. (1.25 times higher than the target)

By multiplying the scaling factor (1.25 in this case) by the current number of replicas, you determine how many replicas are needed to bring the CPU utilization down to the target. In our case 2 replicas so (2×1.25=2.5) and we ceil that because we can’t create a fraction of a pod so it’s ceiled to 3 so after scaling up we should have 3 replicas instead of 2.

The manifest file for HPA above is a very simple implementation. There’s a lot of configurations regarding scaling down (how long should we wait to scale down again) and other important configs but for the sake of the article we’ll go simple.

Upon applying the above manifests. We should have HPA installed to our nginx deployment.

Testing

We can test using the BusyBox image where it gives us a shell to execute a while loop where we send requests to the nginx web server.

kubectl run busybox --image=busybox --rm -it -- /bin/sh

Then inside the shell

while true; do wget -q -O- http://nginx-deployment; done

If we execute kubectl get hpa we can find a TARGETS section x%/20% Which means the current utilization over the threshold utilization specified in the HPA manifest.

We can monitor and check that once the current passes the threshold new replicas are created according to the equation mentioned above!

Auto Scaling Down

When a Kubernetes HPA is configured, it monitors certain metrics (like CPU or memory utilization) at regular intervals (typically every 30 seconds). If the current resource usage falls below the defined target utilization threshold, the HPA will scale down the number of pods.

Scaling Down Criteria:

• Target Utilization vs. Current Resource Usage: where if it finds the current less than the target it will scale down

• MinReplicas: where it doesn’t scale down this number

• Cooldown Period (Stabilization Window): Kubernetes doesn’t immediately scale down when resource usage decreases slightly. It has a built-in stabilization periodto avoid flapping, which is when scaling occurs rapidly back and forth.

HPA manifest has a behavior section which allows you to specify custom scaling behavior, including how quickly to scale down.

In our HPA manifest we can update it as follows:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-deployment
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 20
  behavior:
    scaleDown:
      policies:
        - type: Percent
          value: 20
          periodSeconds: 60
# Scale down by 20% every 60 seconds

The behavior here is scale down by 20% every 60 seconds. The default stabilizationWindowSeconds is 5 minutes but can be configured too.

Summary

In this article we took a look at how HPA works in Kubernetes according to metrics such as CPU and memory, we looked at how we get these metrics using the metrics server and how we utilize them in scaling up and down. I’ll make a part two of this where we auto scale based on custom metrics such as response percentiles and load. This will require extra work but worth it for the content I guess. See you in the next one!

Hello folks! This will be a series of articles where I try diving into complex Devops topics simplifying them for us backend engineers and making the article serve as a quick recap for whoever is interested. In this article we’ll dive into Kubernetes operators and we’ll be making our first operator using Go (not from scratch but we’ll use a handy tool called kubebuilder that adds a boilerplate so we can focus on what we should focus on only). Before moving forward let’s talk about what operators even are.

Kubernetes Operators

A Kubernetes Operator is a method of automating the management of complex applications on Kubernetes, a typical operator consists of:

1. Controller: A program that watches and reacts to changes in Kubernetes resources (like Custom Resources) and takes actions to manage the application’s lifecycle (e.g., creating, updating, or deleting resources).

2. Custom Resource (CR): A custom-defined object that represents the application or service the operator manages. It defines the desired state (e.g., number of replicas, configuration) of that application.

It automates the management of complex applications by using a controller to watch a custom resource and ensure the application matches the desired state.

Now if you were like me first time reading this, you probably didn’t understand anything. Now its time to simplify this even further

The operator is an umbrella for 2 main things:

1. A custom resource; which Is a new type of object that Kubernetes doesn’t even know about (for example, pod is a resource) (pod-stalker is a custom resource because it isn’t natively installed in Kubernetes)

   So we create new custom resources that have a defined schema. This will become much clearer when we implement the actual operator.

2. The controller which is the brain of the operator and where the main code lies. The controller watches for changes in the custom resource and does some logic based on what happened. This process is called reconciliation where the goal is always get back to the desired state from the current one. That desired state is specified in something called a Custom Resource Definition. Which is just a YAML file actually initializing the custom resource filling in the schema.

Enough with the theoretical stuff let’s do a walkthrough of a cool project. We will create a custom resource PodTracker which will watch over some pods with a specified name and will monitor any pod creations in the default namespace and send a message on a slack channel notifying that creation happened.

PodTracker Walkthrough

To get started first install kubebuilder

Kubebuilder is a framework for building Kubernetes Operators and Custom Controllers. It provides a set of tools and libraries to help you easily create, test, and manage Kubernetes Operators, which automate the management of complex applications on Kubernetes.

This will give us a great scaffold to start off of.

Once kubebuilder is installed let’s run the commands to create a scaffold

# initialize a new kubebuilder project in Go. 
kubebuilder init --domain lost.backend --repo lost.backend/pod-tracker
# creates the API responsible for our new Custom Resource
kubebuilder create api --group pod-tracker --version v1 --kind PodTracker

Once installed the file structure should look something like this

We’re only going to be concerned with two main files

• `

• api/v1/podtracker_types.go

Let’s start by defining our Custom Resource Schema first

Custom Resource Schema

Inside api/v1/podtracker_types.go

You’ll find a structure that looks like the following

type PodTrackerSpec struct {
}

// PodTrackerStatus defines the observed state of PodTracker.
type PodTrackerStatus struct {

}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// PodTracker is the Schema for the podtrackers API.
type PodTracker struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`

    Spec   PodTrackerSpec   `json:"spec,omitempty"`
    Status PodTrackerStatus `json:"status,omitempty"`
}

PodTrackerSpec is for the desired schema of the pod tracker. We will have the following:

• Name Field that ensures that this pod tracker would track pods with that given name (For simplicity, usually pod names are unique so a better approach is to add a deployment name and monitor changes for pods that belong to a specific deployment)

• Reporter Which is a struct that contains Kind, Key & Channel describing the kind of reporting (in our case slack, the api key of it and the channel to post on)

PodTrackerStatus defines the current observed state of the PodTracker. Which is usually updated in reconciliation. (the controller updates it according to events that occur). Let’s leave this empty for now.

our types file should look like this now

type Reporter struct {
    Kind    string `json:"kind,omitempty"`
    Key     string `json:"key,omitempty"`
    Channel string `json:"channel,omitempty"`
}
type PodTrackerSpec struct {
    Name     string   `json:"name,omitempty"`
    Reporter Reporter `json:"reporter,omitempty"`
}

// PodTrackerStatus defines the observed state of PodTracker.
type PodTrackerStatus struct {
    PodCount int    `json:"podCount,omitempty"`
    Status   string `json:"status,omitempty"`
}

NOTE that it’s essential to add the json annotations otherwise it won’t compile properly.

Now since we defined our Custom Resource. It’s time to actually write a definition for it. Something like this

# tracker.yaml
apiVersion: "pod-tracker.lost.backend/v1"
kind: "PodTracker"
metadata:
  name: pod-tracker
spec:
  name: "nginx"
  reporter:
    kind: "slack"
    key: "slack-api-key (will post link on how to)"
    channel: "C0821GM4602 (channel ID)"

This is a manifest where we can use kubectl apply -f tracker.yaml to apply this manifest and have our first pod-tracker object running!

Before doing kubectl apply we actually need to install the custom resource created in a local Kubernetes cluster. Make sure you have one running using kind for example.

We can install the Custom Resource by executing make install inside the project directory.

If we execute the kubectl apply command above we’ll find out that we have a pod-tracker resource instance already up and running

Check by kubectl get podtracker

However it’s just a resource instance running and it doesn't do anything useful (for now). Now comes time to actually add the brain to this resource using the second important file we mentioned and that is our controller file!

Custom Controller

In internal/controller/podtracker_controller.go we should have a method called Reconcile that looks as follows:

func (r *PodTrackerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {

    return ctrl.Result{}, nil
}g

Reconcile Takes in what is known as a reconciliation request (a request to trigger this method basically) and executes the logic inside the method to reconcile the custom resource to the desired state.

It automatically gets called when events happen on PodTracker Resource (Creating, updating, deleting, etc)

Controlling what triggers the reconcile method basically lies within the second method we have here

func (r *PodTrackerReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&podtrackerv1.PodTracker{}). // Your primary resource
        Watches(&corev1.Pod{}, handler.EnqueueRequestsFromMapFunc(r.HandlePodEvents)).
        WithEventFilter(predicate.Funcs{
            CreateFunc: func(e event.CreateEvent) bool {
                return true // Process only create events
            },
            UpdateFunc: func(e event.UpdateEvent) bool {
                return false // Ignore updates
            },
            DeleteFunc: func(e event.DeleteEvent) bool {
                return false // Ignore deletions
            },
            GenericFunc: func(e event.GenericEvent) bool {
                return false // Ignore generic events
            },
        }).
        Complete(r)
}

In this method we basically watch for changes both in the PodTracker and Pod resources. Only create events are allowed to get processed and we discard the rest.

We pass the Pod events to a method r.HandlePodEvents which finds its PodTracker object and enqueues a reconciliation request for that PodTracker object which in turn does its job and sends a message to slack that a new pod has been created.

This is how HandlePodEvents looks like

func (r *PodTrackerReconciler) HandlePodEvents(ctx context.Context, o client.Object) []ctrl.Request {
    // Check if the object is a pod if not ignore
    pod, ok := o.(*corev1.Pod)
    if !ok {
        return []ctrl.Request{}
    }
    // check if the object lies in the kubernetes default namespace otherwise ignore
    if pod.Namespace != "default" {
        return []ctrl.Request{}
    }

    // get the list of PodTracker objects
    podTrackerList := &podtrackerv1.PodTrackerList{}
    // if none are found ignore.
    if err := r.List(ctx, podTrackerList); err != nil {
        return []ctrl.Request{}
    }

    ctrlRequests := []ctrl.Request{}
    // iterate over the list of PodTracker objects
    for _, podTracker := range podTrackerList.Items {
        // check if the PodTracker object is watching the pod
        if podTracker.Spec.Name == pod.Name {
            ctrlRequests = append(ctrlRequests, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(&podTracker)})
        }
    }

    return ctrlRequests
}

We simply just get the podTracker objects and check which one of them is responsible for managing the currently created pod. When we find it we enqueue a request to reconcile that specific podTracker using NamespacedName which is an object useful for passing to the reconciliation request. It contains the resource name and namespace.

ctrlRequests could potentially be an array of reconciliation requests which means the method would be invoked as many times as the length of ctrlRequests respectively.

Now in the main reconciliation method I added this.

func (r *PodTrackerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    fmt.Println("Reconciling PodTracker")
    podTracker := &podtrackerv1.PodTracker{}
    if err := r.Get(ctx, req.NamespacedName, podTracker); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // send to slack an update that a pod with the name was created
    lib.SlackSendMessage(podTracker.Spec.Reporter.Key, podTracker.Spec.Reporter.Channel, "Pod "+podTracker.Spec.Name+" was created")

    return ctrl.Result{}, nil
}

I check if the - to be reconciled - pod tracker exists if it does I send a slack message using this function I made and return.

func SlackSendMessage(key string, channelID string, message string) {
    api := slack.New(key)
    _, _, err := api.PostMessage(channelID, slack.MsgOptionText(message, false))
    if err != nil {
        fmt.Println(err)
    }
}
// just a simple function that sends a message to a channel

You provide the slack credentials in the custom resource manifest we did earlier. To get these credentials make sure you create a new slack app and follow the instructions here:

1. Install the go library for slack slack-go/slack

2. Go to the Slack API Apps page.

3. Create a new app from scratch

4. Navigate to "OAuth & Permissions" in your app's settings.

5. Add the necessary OAuth scopes for your app based on what it needs to do. In our case chat:write

6. Go to "Install App" under the slack settings.

7. Click "Install App to Workspace".

8. Authorize the app with your workspace.

9. After installation, you’ll see an OAuth token in the "OAuth & Permissions" section.

   • The token starts with xoxb- (for bot tokens) or xoxp- (for user tokens).

   • This will be your key.

10. To get the channel id just check channel details in your slack app for the channel you want to write to it should be at the very bottom of channel details.

If you got all these steps correct execute make install again to compile the code and make run to test the controller logic

if we try to create an nginx pod using kubectl run nginx —image=nginx

we should get a slack notification 🎉

However our current code has a problem where if a new PodTracker is created. The reconcile method will run sending a slack message for a PodTracker resource creation. We only track created pods so this behavior is unwanted.

To be able to solve this we can use annotations! That’s where their power comes in. We can annotate PodTracker objects that actually need reconciliation because of a pod creation and not because the pod tracker itself is created. we can update the HandlePodEvents method to as follows:

func (r *PodTrackerReconciler) HandlePodEvents(ctx context.Context, o client.Object) []ctrl.Request {
    pod, ok := o.(*corev1.Pod)
    if !ok {
        return []ctrl.Request{}
    }

    if pod.Namespace != "default" {
        return []ctrl.Request{}
    }

    // get the list of PodTracker objects
    podTrackerList := &podtrackerv1.PodTrackerList{}
    if err := r.List(ctx, podTrackerList); err != nil {
        return []ctrl.Request{}
    }

    ctrlRequests := []ctrl.Request{}
    // iterate over the list of PodTracker objects
    for _, podTracker := range podTrackerList.Items {
        // check if the PodTracker object is watching the pod
        if podTracker.Spec.Name == pod.Name {
            if podTracker.Annotations == nil {
                podTracker.Annotations = map[string]string{}
            }
        // add annotation to check for in the reconcilation
            podTracker.Annotations["triggered-by"] = "pod"
        // update the kubectl cluster podtracker object with the new annotation
            if err := r.Update(ctx, &podTracker); err != nil {
                log.FromContext(ctx).Error(err, "Failed to update PodTracker with annotations")
                continue
            }
            ctrlRequests = append(ctrlRequests, ctrl.Request{NamespacedName: client.ObjectKeyFromObject(&podTracker)})
        }
    }

    return ctrlRequests
}

func (r *PodTrackerReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    fmt.Println("Reconciling PodTracker")
    podTracker := &podtrackerv1.PodTracker{}
    if err := r.Get(ctx, req.NamespacedName, podTracker); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    // check the annotations if triggered by exists only send a message.
    if podTracker.Annotations != nil && podTracker.Annotations["triggered-by"] == "pod" {
    // delete the annotation for cleanup
        delete(podTracker.Annotations, "triggered-by")
        if err := r.Update(ctx, podTracker); err != nil {
            return ctrl.Result{}, fmt.Errorf("failed to clear annotation: %w", err)
        }
        lib.SlackSendMessage(podTracker.Spec.Reporter.Key, podTracker.Spec.Reporter.Channel, "Pod "+podTracker.Spec.Name+" was created")
    }

    return ctrl.Result{}, nil
}

And voila! now we only send slack messages of newly created pods.

Summary

The main goal of an article like this is that first it’s targeted to backend developers with little to know knowledge about operators. Because let’s be honest it’s something we might finish our career and never touch. It’s just an attempt from me to ease the understanding of these concepts that I personally find myself struggling with. The goal was without diving deep make a simple use case that clearly explains the idea of this. Hopefully I delivered what I wanted. Also I might make these series into a YouTube series instead if anyone wants that let me know! Till the next one

In this chapter we’re going to be diving into the different ways services can read data they do not own, in monolithic systems using a single database, developers don’t give a second thought to reading database tables but when data is broken into separate databases owned by distinct services, data access for read operations become complex.

Data Access Patterns

Below are some of the most commonly used data access patterns so that a service can access data it doesn’t own.

Interservice Communication Pattern

This is by far the most common pattern for accessing data, if one service needs data it doesn’t have direct access to it simply asks the owning service for it by using some sort of remote access protocol.

Column Schema Replication Pattern

Here, columns are replicated across tables therefore replicating the data and making it available to other bounded contexts.

Replicated Caching Pattern

While caching is a well known pattern for increasing performance and responsiveness, it can also be an effective way for distributed data access and sharing. Leveraging in memory caching so that data needed by other services is made available to each service without them having to ask for it.

There exist different models of caching between services but the basic one is the in memory caching where each service has its own cache separate from other services. This isn’t really that useful for sharing data between services because of the lack of synchronization between the services.

Another caching model is distributed caching where data is held externally in a caching server, services make requests to that external server to retrieve or update shared cache. However its not that useful for data access due to the following:

1. No different that the inter service communication pattern (still tightly coupled to a caching server instead of a service)

2. Different services can update data breaking the bounded context regarding data ownership which can causes inconsistencies between caches and the owning database.

3. Latency issues since the way to access the cache is through network calls as described earlier.

Another model is replicated caching where each service has its own in memory data that is kept in sync between the services allowing the same data to be shared across multiple services. Any update made to the cache is asynchronously propagated to the other caches in the services.

From the caching models mentioned replicated caching is the most suitable for addressing distributed data access.

Data Domain Pattern

In a previous chapter, a way to resolve joint ownership was to make both services share the database ownership. This same pattern can be used for data access too.

A solution is to create a data domain combining multiple tables into a shared schema accessible to both services needing the data which makes for a broader bounded context.

Summary

In this chapter we went through some of the popular data access patterns where one service basically needs data from another. The trade offs of each one and the answer to which one should I pick is a big “it depends” as always 🤣

In the next chapter we’re going to be talking about some famous distributed architecture sagas! stay tuned

Introduction

When architects think about transactions, they usually think about a single atomic unit of work where multiple database updates are either committed together or all rolled back when an error occurs. ACID is an acronym describing the basic properties of an atomic single unit of work database transaction (atomicity, consistency, isolation and durability)

Let’s first briefly talk about ACID before moving on to distributed transactions.

ACID Properties

Atomicity means a transaction must either commit or rollback all of its updates in a single unit of work. All updates are treated as a collective whole so all changes either get committed or rolled back as one unit.

Consistency means that during the course of the transaction the database would never be left in an inconsistent state or violate any integrity constraints specified in the database.

Isolation refers to the degree of which individual transactions interact with each other. It protects uncommitted transaction data from being visible to other transactions during the course of the business request.

Durability means that once a successful response from a transaction commit occurs, it is guaranteed that all the data updates are permanent regardless of further system failures.

ACID can exist within the context of each service in a distributed architecture, but only if the corresponding database supports ACID properties, each service can perform its own commits and rollbacks to the tables it owns within the scope of the atomic business transaction. However if the business request spans multiple services the entire business request cannot be an ACID transaction rather it becomes a distributed transaction.

Distributed transactions occur when an atomic business request containing multiple database updates is performed by separately deployed remote services.

Distributed transactions DO NOT SUPPORT ACID TRANSACTIONS

Atomicity is not supported because each service commits its own data and performs only one part of the overall atomic business request. So atomicity is bound to the service not the entire request.

Consistency Is not supported because a failure in one service causes the data to be out of sync between the tables responsible for the business request.

Isolation is not supported because an insertion in any of the services data while being a part of the whole transaction but it gets committed so it’s visible to read.

Durability is not supported because as mentioned before its per service where there are multiple databases and anything could go wrong in any of them. Supported for each individual service though.

Distributed Transactions

Instead of ACID, distributed transactions support something called BASE.

Completely opposite of ACID BASE is;

1. Basic availability

2. Soft State

3. Eventual Consistency

Basic availability means all the involved services are expected to be available when a distributed transaction is pending.

Soft State describes a situation where a distributed transaction is in progress and the state of the atomic business request is not yet completed (or in some cases not even known). Basically meaning partial services commit or there is a wait time until we get an acknowledgment that everything has worked (or not).

Eventual consistency means that given enough time, all parts of the distributed transaction will complete successfully and all of the data is in sync with one another.

Moving on we’ll now talk about the patterns involved in eventual consistency and the caveats of each.

Eventual Consistency

Distributed architectures rely heavily on eventual consistency as a trade off for better operational characteristics such as performance, scalability, elasticity, fault tolerance and availability. There are numerous ways to achieve eventual consistency but there are thee main patterns in use today

1. Background synchronization pattern

2. Orchestrated request-based pattern

3. Event based pattern

Lets’s dive in them

Background synchronization pattern

The background synchronization pattern uses a separate or external service or process to periodically check the data sources and keep them in sync with one another. The time for data to become eventually consistent depends on the nature of the background synchronization pattern whether it is a batch job running at night or every hour, etc.

This pattern has the longest length of time for data to become consistent, However in some cases data doesn’t need to be in sync at this very moment.

One of the challenges of this pattern is that the process must know what data is changed, which can be done with different ways such as querying the source tables, a database trigger or an event stream. The most important thing is that it must have knowledge of all tables and data sources involved in the transaction.

As efficient as this pattern is, it has some serious tradeoffs:

1. All of the data sources are coupled together; breaking every bounded context rule between the data and the services. The background job needs to access different databases to be able to achieve the eventual synchronization process in the distributed transaction. They must have write access meaning that the background process has shared ownership with the service owning the tables.

2. Might lead to duplicate business logic because what the background job does might be already implemented In the services responsible for each table already.

This pattern isn’t suitable for distributed architectures requiring tight bounded contexts (micro services), where the coupling between data ownership and functionality is a critical part of the architecture.

Orchestrated request-based pattern

A common approach for managing distributed transactions is to make sure all of the data sources are in sync during the course of the business request (while the end user is waiting).

This pattern attempts to process the entire business transaction during the course of the business request. Therefore requiring some sort of orchestrator to manage the distributed transaction.

The orchestrator is responsible for managing all of the work needed to process the request, including knowledge of the business process, knowledge of the participants involved, multicasting logic, error handling and contract ownership.

One of the common ways to implement this is to designate a primary service to manage the distributed transaction. Although this approach avoids the need for a separate orchestration service, it tends to overload the responsibility of the designated service as the distributed transaction orchestrator. In addition to the role of an orchestrator the service must perform its own responsibilities as well. Also this approach leads to tight coupling and synchronous dependencies between services.

Using a dedicated orchestration service for the business request is a better approach here.

As efficient as this pattern is, it has some serious tradeoffs:

1. Favors consistency over overall responsiveness.

2. Really complex error handling (if one service fails you’re going to have to reverse what it performed on the others) (compensating transaction)

3. Failures might occur even during compensation which causes data to be out of sync and would need human intervention to repair.

Event based pattern

This pattern is one of the most popular and reliable eventual consistency patterns for most modern distributed architectures. Events are used in conjunction with an asynchronous publish and subscribe messaging model to post events or command messages to a topic or event stream, services involved in the transaction listen to events and respond to them

The eventual consistency time is usually short for achieving data consistency because of the parallel and decoupled nature of the asynchronous message processing. Services are highly decoupled from one another and responsiveness is good because the service triggering the eventual consistency doesn’t have to wait for the data synchronization to be done before returning a response to the customer.

Tradeoffs here are mainly failure handling becomes complex where if a consumer is processing and fails what happens? in most brokers they will try a number of times to deliver a message and after repeated failures they will send the message to a dead letter queue. Which would either be automatically fixed or would require human intervention.

Summary

In distributed systems, traditional ACID transactions don’t work due to multiple services each handling their own data, leading to distributed transactions that rely on the BASE model: Basic Availability, Soft State, and Eventual Consistency. To handle eventual consistency, three main patterns exist:

1. Background Synchronization: Runs periodic jobs to sync data across services but can cause delays and duplicate business logic.

2. Orchestrated Request-Based Pattern: Uses a central orchestrator to ensure all data is consistent during a request, often with complex error handling but favors consistency.

3. Event-Based Pattern: Services respond to asynchronous events, allowing for quick, decoupled syncing. Failures may result in dead letter queues, needing human intervention at times.

Each pattern has its tradeoffs in terms of consistency, speed, and complexity.

That’s it for this chapter and watch out for the next one where we’ll be talking about distributed data access. Hope you enjoyed!

In this part, we’ll go through the changes that happen mainly to data once a monolithic system has been pulled apart into separate services each with its own domain. Every service abides by the bounded context rule in Domain Driven Design where each domain has its own application code and data together.

I made several articles about this book where we discussed different chapters of it. I’d recommend checking them out to make sure you have a complete understanding of what’s going on. In a nutshell we’re pulling apart a huge monolithic application into several coarse grained services, and pulling apart the data as well.

When data is pulled apart it needs to be stitched back together to make the system work. So the main hiccup is figuring out which service owns what data and how to manage distributed transactions. Also how any service can access data they need (not own)

Assigning Data Ownership

The main question that arises here is: which service owns which data?

A general rule of thumb for assigning table ownership is that services that perform write operations to a table own that table. This works well if a single service writes to the table but it gets messy when multiple services have to write to the same table.

A simple example is as follows;

We have 3 services and 3 databases. As we observe all 3 services write to the Audit Table.

The catalog service writes to the product table and the inventory service does too.

These examples make assigning data ownership a very complex task.

There are 3 common scenarios encountered when assigning data ownership to services:

1. Single ownership

2. Common ownership

3. Joint ownership

We’re going to dive into them and explore several techniques for resolving these scenarios

Single Ownership

Occurs when only one service writes to the table. Very straightforward and very easy to resolve. For example above the wishlist service is the only one that writes to the wishlist table making it a single ownership scenario.

Knowing this information makes us conclude that the wishlist table is part of the bounded context for the wishlist service.

It’s a lot easier to address single table relationships first when approaching these kinds of problems before moving on to complex scenarios

Common Ownership

Occurs when most or all of the services need to write to the same table. Looking back at our example we see all 3 services writing to the audit table). Since all 3 write to it’s really difficult to say who owns this table domain wise.

A proposed solution is to put the audit table in a shared database or schema since it receives writes from everywhere. However this has its own set of problems:

1. Change control where if the schema changes you’ll have to revisit all the services writing to it and update them accordingly

2. Connection starvation due to the amount of services connecting to it

3. Scalability and fault tolerance where if it goes down it causes chaos where a lot of services encounter issues related to auditing

A popular technique for addressing this is to assign a service for auditing that owns the audit table.

So any services that need to write audits go through the audit service to write data and not access the database directly. This has a huge impact on the design and a lot of benefits

1. If no acknowledgment is required a buffer (queue) can sit between the services and the audit service. Where it can process audits at its own pace

2. Also becomes more fault tolerant where it’ll be easier to scale the audit service independently.

Applying what was said our design should look like this;

Joint Ownership

Occurs just like the common ownership but the catch is only a couple of services within the same domain write to the same table. Not most of them as previously described in the common ownership.

Looking back at our first example, only the Catalog and Inventory services perform write operations on the product table.

There exists several techniques to solve this ownership problem

1. Table split

2. Data Domain

3. Delegation

4. Service Consolidation

Let’s go one by one discuss them

Table Split Technique

The table split breaks the table into multiple tables where each service owns a part of the data it’s responsible for.

Looking at our example if we were able to break down the product table into 2 tables where the inventory can own the data it manipulates and so can the catalog that means we did a table split technique. This highly depends on the nature of what the inventory service writes so if it updates counts for example we can extract that column and add a product id foreign key as a reference so the inventory has its own table.

It moves the joint ownership to a single table ownership. However the overhead is consistent communication between both services to ensure data is synced correctly between them and that it remains in a consistent state.

If a new product got added for example, the catalog service needs to communicate that to the inventory service sending the id and inventory counts to it. If removed vice versa also.

But a lot of questions arise when syncing data between 2 tables:

1. Should the communication be synchronous or asynchronous between both services?

2. What happens if the catalog service want to communicate to the inventory service and found that its not available? It’s a availability versus consistency question

Choosing availability means that the catalog service must always add or remove regardless if the inventory service is working or not.

Choosing consistency means that adding or removing would fail if one of both services is down.

So it depends on the business requirements and what you need in order to be able to answer or make a decision.

Delegate Technique

In this method, one service is assigned a single ownership of the table and becomes the delegate. Any other service communicates with the delegate to perform updates on its behalf.

The main challenge here is to know which service to assign as the delegate (the sole owner of the table) we have two options;

1. Primary domain priority

   Where we assign the table to the service that most closely represents the primary domain of the data (the service that does most of the CRUD operations for a entity in that domain)

2. Operational characteristics

   Assigning the table to the service needing higher operational characteristics such as performance, scalability, availability and throughput

If we look at the following joint ownership scenario

The catalog service performs most of the CRUD operations on the product table because it creates, updates and removes products. Getting product information whilst the inventory service is responsible for retrieving and updating inventory count as well as knowing when to restock if the inventory count is too low.

Applying the priority domain technique results in the following

The catalog service would be assigned as the single owner of the table. The Inventory service must communicate with the catalog service to access that table.

Delegate techniques always force inter service communication between the services forcing them to communicate to update data. Also type of communication is key here because in synchronous communication the inventory service must wait for the inventory to be updated by the catalog service. This impacts performance but ensures data consistency. Using asynchronous communication boosts performance but makes data eventually consistent.

With the operational characteristics priority option, the ownership rules would be reversed because the inventory updates occur at a much faster rate than static product data. In this case the ownership would be assigned to the inventory service.

With this option updates to the inventory can use direct database calls instead of remote access protocols. Therefore making inventory operations much faster also the most volatile data is kept consistent (inventory counts)

However one major problem here is the domain management responsibility. The inventory service is responsible for managing inventory counts, not creating, deleting and updating the products (and potentially error management too).

Service Consolidation Technique

The delegate approach highlights the primary issue with joint ownership which is service dependency

The service consolidation technique solves this by combining multiple table owners into a single consolidated service.

Combining multiple services into one creates a coarse grained service. Which increases the testing scope as well as the deployment risk (breaking something else in the service when a new feature is added or a bug is fixed). Consolidation might affect the overall fault tolerance of the system since the whole service fails together.

One of the caveats is that they both have to scale equally as well even if it wasn’t necessary for one service to do so.

Summary

This part covered different data ownership scenarios and techniques used to choose which service owns which data.

Summarizing everything we said, the design should look like this now:

1. We used single table ownership for the wishlist service

2. For common ownership we created an audit service with all other services sending messages via a queue (asynchronous).

3. Finally for the joint ownership between the catalog and the inventory with the product table, we chose the delegate technique domain priority assigning the table to the catalog service where the inventory service sends update requests to the catalog service.

That’s it for part 1. In the next part we’ll go through distributed transactions and the caveats that happen when data is pulled apart. Stay tuned!

Introduction

Amazon Aurora is a relational database service for OLTP (Online Transactional Processing) workloads offered as part of Amazon Web Services (AWS).

To be able to grasp the reasoning behind Aurora let’s take it back to what a normal relational database does.

In a traditional relational database system, each database server (the machine running the database) does all the work:

1. Processing Queries: It handles reading and writing data.

2. Storing Data: It saves data locally or on disk.

3. Handling Failures: If a crash happens, the same server needs to recover data, replay logs, and bring everything back online.

This setup means each server is responsible for both compute (processing) and storage, which can create bottlenecks, especially in high-throughput systems. Network traffic is high because servers need to constantly synchronize data between each other to avoid data loss.

The I/O bottleneck usually happens because a single server has to handle both compute(processing) and storage (reading/writing data to disk). This can overwhelm the server's disk, causing performance slowdowns—especially when there's heavy load. But in a cloud environment like AWS Aurora, things work a bit differently.

Aurora’s Architecture

The Skeleton

Aurora separates the storage service from the database instances. This storage service manages functions like redo logging, crash recovery, and backups independently, rather than being tightly integrated with each database instance like in traditional systems.

So in other words it split the processing from the storage completely which now resulted in having processing servers without the overhead of storing the data.

Instead of relying on a single disk or server, Aurora spreads out storage across many servers (called the "storage fleet"). This means that no single disk or server is overloaded, as the storage load is distributed. However, this introduces a new bottleneck: the network.

The database needs to send requests over the network to the storage servers to read or write data.

Even though the data is spread across many servers, the database must communicate with several of them at once, creating a lot of network traffic.

Also, since the database sends multiple write requests in parallel to different storage nodes, if one of those storage nodes or the network path to it is slow, it can cause delays. This means the overall speed of the database can be limited by the slowest node or network path, even if the others are performing well.

In simpler terms: by spreading the work across many storage servers, the disks aren’t the problem anymore, but now the speed of the network between the database and those servers becomes the main thing that can slow things down. Even one slow server in the storage fleet can affect the overall speed.

Now the question is How did they optimize the network problem mentioned above?

Design choices

Reducing Network Traffic with Redo Logs

• Traditional Problem: both compute and storage typically reside on the same server. Large chunks of data, such as full data pages, are written to disk during transactions, generating significant I/O load. As databases grow and scale, or in clustered environments, this can lead to performance bottlenecks. When compute and storage are separated, such as in a distributed cloud system like Aurora, these writes would require network communication between the database tier and the storage tier, further amplifying traffic and introducing latency.

• Aurora’s Solution: Aurora only sends redo logs (small records that track changes made to the database) to the storage layer, rather than full data pages. These logs are much smaller in size and require less network bandwidth, drastically reducing network I/O. This design reduces the overall data that needs to be transmitted over the network by an order of magnitude.

Parallel Writes to Distributed Storage

• Traditional Problem: In a traditional database setup, all writes would go to a single storage device, creating a bottleneck. Even with distributed systems, data replication to multiple nodes increases network load and complexity.

• Aurora’s Solution: Aurora writes the redo log in parallel to multiple storage nodes across multiple availability zones (AZs). This ensures that the system is resilient to node failures and improves performance by distributing the work. Instead of a single node handling all writes, they are spread across many nodes. Additionally, by splitting the I/O operations across a fleet of storage servers, it prevents overloading any single server or network link.

Asynchronous Background Operations

• Traditional Problem: Operations like backups and crash recovery are usually synchronous and happen in real-time, which can spike network traffic and lead to bottlenecks.

• Aurora’s Solution: Aurora offloads complex tasks like backup and redo recovery to the distributed storage fleet, where they are performed continuously in the background and asynchronously. This means the database doesn’t have to pause to perform these tasks, and they don’t generate massive network loads all at once. Instead, traffic is spread out over time and across nodes.

Fault Tolerance and Self-Healing Mechanism

• Traditional Problem: If a single node or network path slows down or fails, it can cause significant performance degradation. In a split architecture, the failure of a storage node or network path can delay the entire system.

• Aurora’s Solution: Aurora’s storage layer is fault-tolerant and self-healing. If a storage node, disk, or network path becomes slow or fails, the system automatically reroutes traffic to healthy nodes. This reduces the impact of outliers (i.e., slow nodes or links), ensuring that performance issues at one storage node don’t bottleneck the entire system.

The image above is a birds-eye view of Aurora’s architecture. Aurora Uses the AWS RDS control plane and the database engine is a fork of “community” MySQL/InnoDB and diverges primarily in how InnoDB reads and writes data to disk mainly in the redo-log part as mentioned above. Backups are stored on AWS S3 blob storage.

This was a quick blog summarizing a refreshing architecture to the relational database realm done by Aurora. The main goal of this article was to have a high level understanding of the differences, highlighting the problems that happen at high scale and the reasoning behind these design choices. Every choice always exists because of a problem. Thank you for tuning in and till the next one :)

References

1. https://web.stanford.edu/class/cs245/readings/aurora.pdf

Introduction

Apache Flink is an open-source system for processing streaming and batch data. Flink is built on the philosophy that many classes of data processing applications, including real-time analytics, continuous data pipelines, historic data processing (batch), and iterative algorithms (machine learning, graph analysis) can be expressed and executed as pipelined fault-tolerant dataflows.

There exists mainly two types of data processing;

1. Data stream processing(real time)

2. Batch processing(static)

Both have their use cases according to different business models. However recently there has been an increase in the processing of real time data whether it be logs, changes to the application state, readings, etc..

However most streams aren't being treated actually as streams, they're being processed in batches (static) where the batch could be over a specific time period for example. Data collection tools, workflow managers, and schedulers orchestrate the creation and processing of batches. However these approaches suffer from high latency (imposed by batches), high complexity (connecting and orchestrating several systems, and implementing business logic twice), as well as arbitrary inaccuracy, as the time dimension is not explicitly handled by the application code.

Apache Flink follows a paradigm that embraces data-stream processing as the unifying model for real-time analysis, continuous streams, and batch processing both in the programming model and in the execution engine. Flink supports different notions of time (event-time, ingestion-time, processing-time) in order to give programmers high flexibility in defining how events should be correlated.

Batch programs are special cases of streaming programs, where the stream is finite, and the order and time of records does not matter (all records implicitly belong to one all-encompassing win- dow). However, to support batch use cases with competitive ease and performance, Flink has a specialized API for processing static data sets, uses specialized data structures and algorithms for the batch versions of opera- tors like join or grouping, and uses dedicated scheduling strategies. The result is that Flink presents itself as a full-fledged and efficient batch processor on top of a streaming runtime

System Architecture

Now that we have a good overview on what Flink does, let's talk about its architecture

Flink consists of four main layers;

1. Deployment

2. Core

3. API

4. Libraries

The core of Flink is the distributed dataflow engine, which executes dataflow programs.

A Flink runtime program is a DAG of stateful operators connected with data streams.

Directed Acyclic Graph (DAG):

• The program is represented as a DAG, where each node is a computation (e.g., a function, a transformation) and each edge represents the flow of data between these nodes.

• The edges indicate the direction of data flow, from data sources through transformations to outputs.

There are two core APIs in Flink:

1. The DataSet API for processing finite data sets (often referred to as batch processing)

2. The DataStream API for processing potentially unbounded data streams (often referred to as stream processing).

Flink’s core runtime engine can be seen as a streaming dataflow engine, and both the DataSet and DataStream APIs create runtime programs executable by the engine.

As such, it serves as the common fabric to abstract both bounded (batch) and unbounded (stream) processing.

Flink bundles domain-specific libraries and APIs that generate DataSet and DataStream API programs, currently, FlinkML for machine learning, Gelly for graph processing and Table for SQL-like operations.

Flink Cluster Architecture

Flink cluster comprises three types of processes:

1. Client

2. Job Manager

3. At least one Task Manager

The client takes the program code, transforms it to a dataflow graph, and submits that to the JobManager. This transformation phase also examines the data types (schema) of the data exchanged between operators and creates serializers and other type/schema specific code.

DataSet programs (batch) additionally go through a cost-based query optimization phase, similar to the physical optimizations performed by relational query optimizers.

The JobManager coordinates the distributed execution of the dataflow, It tracks the state and progress of each operator and stream, schedules new operators, and coordinates checkpoints and recovery.

In a high-availability setup, the JobManager persists a minimal set of metadata at each checkpoint to a fault-tolerant storage, such that a standby JobManager can reconstruct the checkpoint and recover the dataflow execution from there.

The actual data processing takes place in the TaskManagers. A TaskManager executes one or more operators that produce streams, and reports on their status to the JobManager. The TaskManagers maintain the buffer pools to buffer or materialize the streams, and the network connections to exchange the data streams between operators.

An operator is a node in the DAG mentioned above, it's a processing step that the stream goes into.

Streaming Dataflows

Although users can write Flink programs using a multitude of APIs, all Flink programs eventually compile down to a common representation: the dataflow graph.

The dataflow graph is executed by Flink’s runtime engine, the common layer underneath both the batch processing (DataSet) and stream processing (DataStream) APIs.

Dataflow Graph

The dataflow graph as depicted in Figure 3 is a directed acyclic graph (DAG) that consists of the following:

1. Stateful Operators

2. Data streams that represent data produced by an operator and are available for consumption by operators.

Since dataflow graphs are executed in a data-parallel fashion, In a data-parallel execution model, the same operation is applied to different partitions of the dataset at the same time across multiple computing resources (e.g., CPUs, machines in a cluster).

Instead of processing one piece of data after another (sequential processing), the system processes many pieces of data in parallel.

Operators are parallelized into one or more parallel instances called subtasks, and streams are split into one or more stream partitions (one partition per producing subtask). The stateful operators, which may be stateless as a special case implement all of the processing logic (e.g., filters, hash joins and stream window functions).

Streams distribute data between producing and consuming operators in various patterns, such as point-to-point, broadcast, re-partition, fan-out, and merge.

So the main idea is to split the data between the producer and consumer, parallelizing it over the operators.

Data Exchange through Intermediate Data Streams

Flink’s intermediate data streams are the core abstraction for data-exchange between operators. An intermediate data stream represents a logical handle to the data that is produced by an operator and can be consumed by one or more operators.

Intermediate streams are logical in the sense that the data they point to may or may not be materialized on disk.

Pipelined Streams: These are used in Apache Flink to allow different parts of a dataflow (producers and consumers) to run at the same time. Data is sent from one operator to the next without waiting for the entire dataset to be processed first. This allows for faster, real-time processing.

If a downstream operator (consumer) is slow, it can slow down the upstream operator (producer), creating "backpressure." Flink manages short-term fluctuations in data flow using buffers.

Blocking Streams: These are used when you need to fully process and store data from one operator before moving on to the next.

The producing operator finishes its work and stores all its output before the consuming operator starts processing. This separates the two operators into distinct stages.

Since all data is stored before being passed on, blocking streams use more memory and may write data to disk if needed.

There’s no backpressure since the next stage only starts after the current stage is fully complete.

Blocking streams are useful when you need to isolate operators (like in complex operations such as sorting) to prevent issues like distributed deadlocks in the system.

When Flink processes data, it splits data into chunks called buffers before sending them from one operator (producer) to another (consumer).

• A buffer can be sent as soon as it’s full, or

• It can be sent after a certain amount of time, even if it’s not full. (timeout)

Here comes the tradeoff between latency and throughput;

Latency: How quickly data is processed and moved through the system.

Throughput: How much data the system can handle in a given time period.

Low Latency: To achieve low latency (faster response times), Flink sends buffers more quickly, even if they’re not full. This means data moves through the system faster, but the throughput (amount of data processed) might be lower. (or even small buffers)

High Throughput: To achieve higher throughput (processing more data), Flink waits until buffers are full before sending them. This increases the amount of data processed at once but can slow down the response time, leading to higher latency. (larger buffers also)

Flink allows you to balance between how fast data is processed (latency) and how much data is processed at once (throughput) by adjusting how buffers are handled. Shorter timeouts mean faster data movement but lower throughput, while longer timeouts mean higher throughput but slower data movement.

Apart from exchanging data, streams in Flink communicate different types of control events. These are special events injected in the data stream by operators, and are delivered in-order along with all other data records and events within a stream partition. The receiving operators react to these events by performing certain actions upon their arrival. Examples are;

1. Checkpoint Barriers; used to create a snapshot of the data processing at a specific point in time.

2. Watermarks; markers in the data stream that show how far along the system is in processing time-based events.

3. Iteration Barriers; used in specialized algorithms that require multiple passes over the data (iterative algorithms).

Streaming dataflows in Flink do not provide ordering guarantees after any form of repartitioning or broadcasting and the responsibility of dealing with out-of-order records is left to the operator implementation.

Iterative Dataflow

Iterations are important for tasks like graph processing and machine learning, where you often need to repeatedly process data to refine results. In some systems(traditional approaches), you either submit a new job for each iteration or add more nodes to the processing graph.

In Flink, iterations are managed by special operators called iteration steps. These steps allow the processing of data to repeat in a controlled manner.

Flink’s iteration steps use feedback edges to create loops in the data processing pipeline. This enables data to flow back into the iteration step, allowing for iterative processing.

Flink uses head and tail tasks (thought as operators) to manage the flow of data through the iteration steps. These tasks handle the data records that are fed back into the iteration, ensuring that the processing is coordinated.

Fault Tolerance

Flink offers reliable execution with strict exactly-once-processing consistency guarantees and deals with failures via checkpointing and partial re-execution.

The general assumption the system makes to effectively provide these guarantees is that the data sources are persistent and replayable.

Examples of such sources are files and durable message queues (e.g., Apache Kafka).

As mentioned before, Flink uses a system called checkpointing to make sure that, even if something goes wrong, your data processing continues exactly where it left off without losing or duplicating data.

Data streams can be huge and never-ending, so if you had to start over after a failure, it could take months to catch up. That would be impractical.

To avoid this, Flink regularly saves snapshots of the current state of the data processing, including the exact position in the data stream. If something fails, Flink can quickly recover using these snapshots, so it doesn’t have to reprocess everything from the beginning.

The core challenge they faced when saving snapshots is that when dealing with data processing, you need to ensure that all parallel operators (processing units) take a snapshot of their state at the same logical time. This means capturing a consistent view of the entire data processing system without stopping it.

So they introduced something called Asynchronous Barrier Snapshotting (ABS)

• Special markers (called barriers) are inserted into the data streams. These barriers represent a specific point in time.

• When a barrier reaches an operator, it marks that operator’s state as part of the current snapshot. Data before the barrier is included in the snapshot, and data after the barrier is not.

• This process allows Flink to take snapshots without stopping the entire data processing system, thus keeping the system running smoothly.

Each partition of a stream operates independently and will have its own barriers. When a barrier is inserted into the stream, it travels through each partition separately.

The barriers represent the same logical time across all partitions, but they may not arrive simultaneously at every partition due to differences in processing speed and network delays.

How it exactly works in depth

1. Alignment phase: Each operator in the data pipeline receives barriers from upstream operators. Before taking a snapshot, the operator makes sure that it has received all barriers from all of its input streams. This ensures that the snapshot reflects a consistent point in time across all inputs.

2. State Saving: After confirming all barriers are received, the operator saves its current state (e.g., contents of windows or custom data structures) to durable storage, such as HDFS or another storage system.

3. Barrier Forwarding: Once the state is safely backed up, the operator forwards the barrier to the next operators downstream. This continues until all operators have taken their snapshots and forwarded the barriers.

4. Complete Snapshot: The snapshot process is complete when all operators have registered their states and forwarded the barriers. The snapshot captures all operator states as they were when the barriers passed through, ensuring a consistent global snapshot.

Recovery Process:

Restoring State:

• From Snapshots: When a failure occurs, Flink restores all operator states from the last successful snapshot.

• Restarting Streams: Input streams are restarted from the point of the latest barrier that has a snapshot. This limits the amount of data that needs to be reprocessed to just the records between the last two barriers.

Benefits of ABS:

1. It guarantees exactly-once state updates without ever pausing the computation

2. The checkpointing mechanism is independent of other control messages in the system, like events triggering window computations. This means it doesn’t interfere with other data processing features.

3. ABS is not tied to any specific storage system. The state can be backed up to various storage systems depending on the environment, like file systems or databases.

Stream Analytics on Top of Dataflows

Flink’s DataStream API is designed for stream processing, handling complex tasks like time management, windowing, and state maintenance. It builds on Flink’s runtime, which already supports efficient data transfers, stateful operations, and fault tolerance. The API allows users to define how data is grouped and processed over time, while the underlying runtime manages these operations efficiently and reliably.

The Notion of Time

Flink distinguishes between two notions of time;

1. Event time; which denotes the time when an event originates (e.g., the timestamp associated with a signal arising from a sensor, such as a mobile device)

2. Processing-time, which is the wall-clock time of the machine that is processing the data.

There can be differences (skew) between event-time and processing-time, leading to potential delays when processing events based on their actual event-time.

Hence they introduce Watermarks;

Watermarks are special events used to track the progress of time within a stream processing system. They help the system understand which events have been processed and which are still pending.

A watermark includes a time attribute t, indicating that all events with a timestamp lower than t have been processed.

Watermarks originate from the sources of the data stream and travel through the entire processing topology. As they move, they help maintain a consistent view of time across different operators.

Operators like map or filter just forward the watermarks they receive. Operators that perform calculations based on watermarks (e.g., event-time windows) compute results triggered by the watermark and then forward it. For multiple inputs, the operator forwards the minimum of the incoming watermarks to ensure accurate results.

Flink programs that are based on processing-time rely on local machine clocks, and hence possess a less reliable notion of time, which can lead to inconsistent replays upon recovery. However, they exhibit lower latency. Programs that are based on event-time provide the most reliable semantics, but may exhibit latency due to event-time-processing-time lag. Flink includes a third notion of time as a special case of event-time called ingestion-time, which is the time that events enter Flink. That achieves a lower processing latency than event-time and leads to more accurate results in comparison to processing-time.

Stateful Stream Processing

State is critical to many applications, such as machine-learning model building, graph analysis, user session handling, and window aggregations. There is a plethora of different types of states depending on the use case. For example, the state can be something as simple as a counter or a sum or more complex, such as a classification tree or a large sparse matrix often used in machine-learning applications. Stream windows are stateful operators that assign records to continuously updated buckets kept in memory as part of the operator state.

State Management in Flink

1. Explicit State Handling:

   • State Registration: Flink allows users to explicitly manage state within their applications. This means users can define and work with state in a clear and controlled way.

   • Operator Interfaces/Annotations: Flink provides interfaces or annotations that enable you to register local variables within an operator's scope. This ensures that the state you define is closely associated with the specific operator that needs it.

2. Operator-State Abstraction:

   • Key-Value States: Flink offers a high-level abstraction for state management. You can declare state as partitioned key-value pairs, which allows for efficient and flexible management of state within streaming applications.

   • Associated Operations: Along with declaring state, Flink provides operations to interact with this state, such as reading, updating, and deleting state entries.

3. State Backend Configurations:

   • StateBackend Abstractions: Users can configure how state is stored and managed using StateBackend abstractions. This includes specifying the storage mechanism (e.g., file system, database) and how the state is checkpointed.

   • Custom State Management: This flexibility allows for custom state management solutions tailored to specific application needs and performance requirements.

4. Checkpointing and Durability:

   • Exactly-Once Semantics: Flink’s checkpointing mechanism ensures that any registered state is durable and maintained with exactly-once update semantics. This means that state changes are reliably recorded and can be accurately recovered in case of failures.

Stream Windows

Incremental computations over unbounded streams are often evaluated over continuously evolving logical views, called windows. Apache Flink incorporates windowing within a stateful operator that is configured via a flexible declaration composed out of three core functions:

1. Window assigner

2. Trigger (optional)

3. Evictor.

4. Window assigner:

assigns each record to one or more logical windows.

Examples:

• Time Windows: Based on timestamps (e.g., 6-second windows).

• Count Windows: Based on the number of records (e.g., 1000 records).

• Sliding Windows: Overlapping windows that can cover multiple periods or counts (e.g., a window every 2 seconds).

2. Trigger:

Determines when the operation associated with the window is performed.

Examples:

• Event Time Trigger: The operation happens when a watermark passes the end of the window.

• Count Trigger: The operation happens after a certain number of records (e.g., every 1000 records).

3. Evictor:

   Decides which records to keep within each window.

   Examples:

   • Count Evictor: Keeps a fixed number of the most recent records (e.g., the last 100 records).

Batch Analytics on Top of Dataflows

Streaming and Batch Processing: Flink uses the same runtime engine for both streaming and batch computations. This means that both types of workloads benefit from the same execution infrastructure.

Handling Batch Computations:

• Blocking Data Streams: For batch processing, large computations can be broken into isolated stages using blocked data streams. These stages are executed sequentially, which allows for efficient processing and scheduling.

• Turning Off Periodic Snapshotting: When the overhead of periodic snapshotting (used for fault tolerance) is high, it is turned off. Instead, fault recovery is managed by replaying lost data from the most recent materialized intermediate stream, which could be from the source.

Blocking Operators:

• Definition: Blocking operators (like sorts) are those that wait until they have consumed their entire input before proceeding. The runtime does not differentiate between blocking and non-blocking operators.

• Memory Management: These operators use managed memory, which can be on or off the JVM heap. If their memory usage exceeds available memory, they can spill data to disk.

DataSet API:

• Batch Abstractions: The DataSet API provides abstractions specifically for batch processing. It includes a bounded DataSet structure and transformations like joins, aggregations, and iterations.

• Fault-Tolerance: DataSets are designed to be fault-tolerant, ensuring reliable processing of batch data.

Query Optimization:

• Optimization Layer: Flink includes a query optimization layer that transforms DataSet programs into efficient executable plans. This optimization helps improve performance and resource utilization.

• Flink uses advanced techniques to optimize query execution, considering network, disk, and CPU costs, and incorporates user hints for better accuracy.

Memory Management: Flink improves memory efficiency by serializing data into segments, processing data in binary form, and minimizing garbage collection.

Batch Iterations: Flink supports various iteration models and optimizes iterative processes with techniques like delta iterations for efficient computation.

This approach enables Flink to effectively manage and optimize both streaming and batch processing tasks, leveraging a unified runtime and specialized APIs for different types of workloads.

Summary

In this article, we dove deep into Apache Flink, exploring its core functionalities and advanced techniques. Key topics covered include:

• Unified Data Processing: We examined how Flink’s runtime supports both streaming and batch processing, allowing seamless handling of continuous and bounded data.

• Fault Tolerance: We detailed Flink’s checkpointing mechanism, which ensures exactly-once processing guarantees by capturing consistent snapshots of operator states and stream positions.

• State Management: We explored Flink’s approach to explicit state handling, including state abstractions and custom configurations for flexible state storage and checkpointing.

• Windowing: We discussed Flink’s robust windowing system, which supports a variety of time-based and count-based windows, and handles out-of-order events.

• Batch Processing Optimization: We covered how Flink adapts its runtime for batch processing with techniques like blocking operators and efficient data management.

• Query Optimization: We looked into Flink’s advanced query optimization strategies, including cost-based planning and handling of complex UDF-heavy DAGs.

• Memory Management: We analyzed Flink’s memory management practices, including serialized data handling and off-heap memory usage to reduce garbage collection overhead.

Overall, the article provided an in-depth look at how Flink handles data processing, fault tolerance, state management, and optimizations for both streaming and batch scenarios. Hope you guys enjoyed and till the next one!

References

1. https://asterios.katsifodimos.com/assets/publications/flink-deb.pdf

These articles were made to get the important bits from the article or the parts I was interested in the most. I've added/removed and kept some parts the exact same. The end goal is to just understand how it works but all credit goes to the writer 100%.

This paper was written in 2011 so it doesn't mention replication because Kafka didn't have it back then.

Introduction

The paper starts by addressing how log processing has become very critical in this day and age and moves on to say that Kafka was made to tackle some of the log processing problems they faced at Linkedin and that it took ideas from other messaging systems. Then the writer starts vouching for Kafka and how its performance and scalability Is superior compared to other systems.

Then moves to talk about what "log" data really is in companies (user clicks, metrics, etc) and talks about how back in the day it was mainly used for analytics but in this day and age it's used directly into production systems in real time (search relevance, recommendations, ad targeting, security protections)

These types of log data are very challenging due to the volume nature of them. Processing them in a fast and efficient way is an even harder challenge.

Then the writer then mentions that the old ways of processing these logs were scraping them all for analysis, which is very inefficient if you think about it. And that several log aggregators were built in the last years (Scribe, Flume ,etc) which normally offload the data into a HDFS (Hadoop).

Linkedin wanted to achieve more with log aggregation, they needed to support all the real time applications mentioned above (search relevance, etc) with a delay of no more than a few seconds.

Then Kafka comes in, a combination of traditional log aggregators and messaging systems. The most important thing was that it allowed consuming these logs in real time. In the next section we'll discuss different messaging systems that were available at that time and why Linkedin couldn't adopt them and had to invent Kafka instead.

Related Work

The messaging systems that were available at that time weren't a good fit for log processing where there was a mismatch in features. The messaging systems then were focusing more on delivery guarantees rather than throughput. And that was considered overkill for collecting log data where if a click wasn't registered it wouldn't be the end of the world. The unneeded features increased the complexity of the system but they were made because not every system focuses on throughput as their main primary constraint. Not only this but those systems were very weak in distributed support. There is no easy way to partition and store messages on multiple machines.

Then the writer starts talking about how these systems usually aggregate the logs and dumps them periodically. But talks about how most of these systems usually do this processing offline (not real time) and that they use a "Push" model where they push the data to the consumers which potentially could overload the consumer if it's still processing data. They found that the "pull" model is more convenient to work with at Linkedin so each consumer can have its own rate and avoid being flooded by messages.

So briefing up, here's the summary:

1. Linkedin wanted throughput, none of the existing systems provided that

2. Existing systems weren't that scalable/ real time

3. The push model was not going to work for Linkedin

In the next section we talk about Kafka's architecture and design principles

Kafka Architecture and Design Principles

The basic outlines of Kafka are as follows:

1. A stream of messages of a particular type is called topic

2. A producer publishes messages to the topic

3. The messages are stored on a set of servers called brokers

4. A consumer subscribes to one or more topics from the broker and pulls data from the broker.

In the consumer, each message stream provides an iterator interface over the continual stream of messages being produced. The consumer iterates over the messages and blocks if there doesn't exist any.

Kafka supports two types of methods:

1. Point to Point delivery (just a basic queue where 1 consumer only takes the message and processes it)

2. Pub/Sub model where multiple consumers get a copy of the same message.

Below is a simple diagram visualizing what we wrote above (stole it from the paper 😅)

A topic is divided into multiple partitions and each broker stores one or more of those partitions. Multiple producers and consumers can publish and retrieve messages at the same time. In the next section we talk about partitions and some of the design choices that were made.

Partition Layout and Design Choices

Each partition of a topic has a logical log. Physically the log is a set of segment files where each file is around 1GB.

Every time a producer publishes a message to a partition, the broker simply appends the message to the last segment file. For better performance, we flush the segment files to disk only after a configurable number of messages have been published or a certain amount of time has elapsed. A message is only exposed to the consumers after it is flushed.

This is an example of providing Durability and Consistency over a little bit of latency (configurable). When the messages come in to the broker they are in an in memory buffer and after a configurable amount they get flushed to disk. Only when they are flushed the Consumer can see them. This gives a huge boost in durability and consistency over some milliseconds of latency which is very much worth it.

Unlike typical messaging systems, a message stored in Kafka doesn’t have an explicit message id. Instead, each message is addressed by its logical offset in the log. This avoids the overhead of maintaining auxiliary, seek-intensive random-access index structures that map the message ids to the actual message locations. Note that our message ids are increasing but not consecutive. To compute the id of the next message, we have to add the length of the current message to its id.

A consumer always consumes messages from a particular partition sequentially. If the consumer acknowledges a particular message offset, it implies that the consumer has received all messages prior to that offset in the partition. Under the hood, the consumer is issuing asynchronous pull requests to the broker to have a buffer of data ready for the application to consume. Each pull request contains the offset of the message from which the consumption begins and an acceptable number of bytes to fetch. Each broker keeps in memory a sorted list of offsets that include the offset of the first message in every segment file. The broker locates the segment file where the requested message resides by searching the offset list, and sends the data back to the consumer. After a consumer receives a message, it computes the offset of the next message to consume and uses it in the next pull request.

The image above visualizes the in memory index present on the broker.

Where each index is the first offset of a segment file. With a simple binary search given an offset we can find the segment file.

The writer also mentions that although the end consumer API iterates one message at a time, under the covers, each pull request from a consumer also retrieves multiple messages up to a certain size, typically hundreds of kilobytes. Just so it can have them ready for processing since they're already flushed on disk on the broker.

One of the very smart decisions the writer talks about is depending on the file system page cache instead of in memory caching for accessing recent messages. This has the following advantages:

1. Avoids double buffering, there is no in memory buffer its only in the system file page cache

2. Completely offloads caching to the OS and not the Kafka process, Which is magnificent because there is no overhead for garbage collection

3. In a Kafka process restart, the cache still exists since it's the OS responsibility and it only goes if the device was rebooted.

OS caching plays a huge role too since producer and consumer access segment files sequentially. It was found that both the production and the consumption have consistent performance linear to the data size, up to many terabytes of data.

Kafka also optimized network access for the consumers, we'll have a look at the normal OS process for reading a local file from disk and sending it over the network.

1. Read the file from disk into memory (page cache in OS)

2. Since its in memory we need to copy it to the application cache (still in memory) but in a place in memory where the application can actually access it (the memory reserved for the application)

3. Then as transmission is about to begin, it offloads to a kernel buffer which is going to interact with the underlying socket and send the data.

4. The kernel buffer sends over the data via the socket.

That's quite a lot of copies and system calls being made. Kafka optimized this by leveraging an API that exists in Linux systems called sendfile.

This directly transfers bytes from the file to the socket skipping all these copies and boosting performance.

Stateless broker: In Kafka, the information about how much each consumer has consumed is not maintained by the broker, but by the consumer itself. Such a design reduces a lot of the complexity and the overhead on the broker. However, this makes it tricky to delete a message, since a broker doesn’t know whether all subscribers have consumed the message.

Kafka solves this problem by using a simple time-based SLA for the retention policy. A message is automatically deleted if it has been retained in the broker longer than a certain period, typically 7 days. This solution works well in practice. Most consumers, including the offline ones, finish consuming either daily, hourly, or in real-time. The fact that the performance of Kafka doesn’t degrade with a larger data size makes this long retention feasible.

There is an important side benefit of this design. A consumer can deliberately rewind back to an old offset and re-consume data. This violates the common contract of a queue, but proves to be an essential feature for many consumers. For example, when there is an error in application logic in the consumer, the application can re-play certain messages after the error is fixed. This is particularly important to ETL data loads into our data warehouse or Hadoop system.

As another example, the consumed data may be flushed to a persistent store only periodically (e.g, a full-text indexer). If the consumer crashes, the unflushed data is lost. In this case, the consumer can checkpoint the smallest offset of the un-flushed messages and re-consume from that offset when it’s restarted. We note that rewinding a consumer is much easier to support in the pull model than the push model. Next up is some design considerations governing Kafka being distributed in Nature.

Distributed Coordination

Producers and the Consumers behave in a distributed setting. Each producer can publish a message to either a randomly selected partition or a partition semantically determined by a partitioning key/function.

Kafka has the concept of consumer groups. Each consumer group consists of one or more consumers that jointly consume a set of subscribed topics, i.e., each message is delivered to only one of the consumers within the group.

Different consumer groups each independently consume the full set of subscribed messages and no coordination is needed across consumer groups.

The consumers within the same group can be in different processes or on different machines. Our goal is to divide the messages stored in the brokers evenly among the consumers, without introducing too much coordination overhead.

The writer mentions that first decision was to make a partition within a topic the smallest unit of parallelism.

This means that at any given time, all messages from one partition are consumed only by a single consumer within each consumer group.

Had we allowed multiple consumers to simultaneously consume a single partition, they would have to coordinate who consumes what messages,which necessitates locking and state maintenance overhead.

In contrast, in our design consuming processes only need co-ordinate when the consumers rebalance the load, an infrequent event.

The second decision that we made is to not have a central “master” node, but instead let consumers coordinate among themselves in a decentralized fashion.

Adding a master can complicate the system since we have to further worry about master failures.

To facilitate the coordination, we employ a highly available consensus service Zookeeper. If you're unsure about what Zookeeper is I recommend you check out an article I wrote about it here

Kafka uses Zookeeper for the following:

1. Detecting the addition and the removal of brokers and consumers.

2. Triggering a rebalance process in each consumer when the above happens.

3. Maintaining the consumption relationship and keeping track of the consumed offset of each partition.

When each broker or consumer starts up, it stores its information in a broker or consumer registry in Zookeeper. The broker registry contains the broker’s host name and port, and the set of topics and partitions stored on it. The consumer registry includes the consumer group to which a consumer belongs and the set of topics that it subscribes to. Each consumer group is associated with an ownership registry and an offset registry in Zookeeper. The ownership registry has one path for every subscribed partition and the path value is the id of the consumer currently consuming from this partition.

The offset registry stores for each subscribed partition, the offset of the last consumed message in the partition.

The paths created in Zookeeper are ephemeral for the broker registry, the consumer registry and the ownership registry, and persistent for the offset registry.

Once a new consumer is added or changes are sent to consumers (broker/consumer changes) that consumer does a rebalance process to determine the subset of partitions it should consume from. I recommend reading the rebalance algorithms directly from the paper as it's explained best there.

Delivery Guarantees

Kafka guarantees at least once delivery, Exactly once delivery can be very complex to achieve (two phase commits). Most of the time, a message is delivered exactly once to each consumer group.

In the case when a consumer process crashes without a clean shutdown, the consumer process that takes over those partitions owned by the failed consumer may get some duplicate messages that are after the last offset successfully committed to zookeeper.

If an application cares about duplicates, it must add its own de- duplication logic. This is usually a more cost-effective approach than using two-phase commits.

Kafka guarantees that messages from a single partition are delivered to a consumer in order. However, there is no guarantee on the ordering of messages coming from different partitions.

To avoid log corruption, Kafka stores a CRC for each message in the log. If there is any I/O error on the broker, Kafka runs a recovery process to remove those messages with inconsistent CRCs. Having the CRC at the message level also allows us to check network errors after a message is produced or consumed.

In Apache Kafka, CRC (Cyclic Redundancy Check) is a mechanism used to ensure data integrity. Specifically, Kafka uses CRC to detect errors in the data being transmitted or stored.

Kafka uses CRC checksums in various parts of its architecture to ensure that the data remains intact from the producer to the broker and from the broker to the consumer.

When the data is read or transmitted, the checksum is recalculated and compared with the original checksum. If they don't match, it indicates that the data has been corrupted.

The last and final section of the paper the writer talks about the usage of Kafka at Linkedin, I won't be writing it here but it's a good read I'd recommend reading it 100%.

Reference

• https://notes.stephenholiday.com/Kafka.pdf