http://www.viveksyngh.com/ Recent content on Vivek Kumar Singh Hugo -- gohugo.io en Fri, 20 Sep 2024 00:00:00 -0600 http://www.viveksyngh.com/posts/controller-runtime-bytes-1/ Fri, 20 Sep 2024 00:00:00 -0600 http://www.viveksyngh.com/posts/controller-runtime-bytes-1/ This is a series of mini blogs for documenting some of the controller runtime features. In Kubernetes, some objects are owner of other objects. For example, a ReplicaSet is the owner of a set of Pods. These owned objects are dependents of their owners. The dependents objects has references to all owners objects under metadata.ownerReferences field. Read more about owners and dependents A kubernetes custom resources can set itself as owner of the other resources, it creates as part of it’s reconciliation logic. http://www.viveksyngh.com/paper-notes/diving-into-amazon-dynamo-paper/ Sun, 19 Feb 2023 00:00:00 -0600 http://www.viveksyngh.com/paper-notes/diving-into-amazon-dynamo-paper/ In this blog post, we are going to look into Amazon Dynamo paper which describes a set of techniques that can be combined to form a highly-available, distributed key-value data store. This paper then became foundation for databases such as Cassandra, Riak etc. Dynamo addresses the need for an incrementally scalable, highly-available key-value storage system. The technology is designed to give its users the ability to trade-off cost, consistency, durability and performance, while maintaining high-availability. http://www.viveksyngh.com/paper-notes/merkle-tree-and-its-applications/ Sun, 01 Jan 2023 00:00:00 -0600 http://www.viveksyngh.com/paper-notes/merkle-tree-and-its-applications/ A Merkle tree, also known as hash tree is a hash based data structure. It is a tree data structure in which each node leaf nodes are cryptographic hash of data blocks for example a file or set of files, and each non-leaf node is cryptographic hash on its children nodes. A hash tree allows efficient and secure verification of the contents of a large data structure. Merkle trees are used in distributed systems for efficient data verification. http://www.viveksyngh.com/posts/converting-kubernetes-go-types-to-yaml-and-vice-versa/ Mon, 07 Nov 2022 00:00:00 -0600 http://www.viveksyngh.com/posts/converting-kubernetes-go-types-to-yaml-and-vice-versa/ In this blog post, we will see how we can encode/decode Kubernetes Go API types to/from Kubernetes YAML. When we work with Kubernetes Go API types (custom types or core types), we often need to convert the Go API types into YAML manifest or vice versa. For example, A CLI application Which output Kubernetes objects in YAML Which takes Kubernetes objects YAML as input For demonstration purposes, we will be creating a simple application that creates a Kubernetes ConfigMap object and encodes/decodes it. http://www.viveksyngh.com/posts/kubernetes-operators-fundamental-concepts/ Sat, 08 Oct 2022 00:00:00 -0600 http://www.viveksyngh.com/posts/kubernetes-operators-fundamental-concepts/ In this series of blog posts, I am going to talk about Kubernetes operator patterns, Kubernetes controllers, Custom resources, and many more related topics. We will also build a Kubernetes operator in Go and look at different tools available for building a Kubernetes operator. What is Kubernetes? Link to heading From the official documentation of Kubernetes Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. http://www.viveksyngh.com/posts/understanding-kubernetes-container-probes/ Fri, 01 Jul 2022 00:00:00 -0600 http://www.viveksyngh.com/posts/understanding-kubernetes-container-probes/ A probe is a diagnostic performed periodically by the kubelet on a container. To perform a diagnostic, the kubelet either executes code within the container or makes a network request. There are three types of probes in Kubernetes Liveness Probe indicates whether the container is running. Readiness Probe indicates whether the container is ready to respond to requests. Startup Probe indicates whether the container is started. Liveness Probe Link to heading The kubelet uses a liveness probe to know when to restart the container. http://www.viveksyngh.com/posts/running-highly-available-apps-on-kubernetes/ Wed, 11 May 2022 00:00:00 -0600 http://www.viveksyngh.com/posts/running-highly-available-apps-on-kubernetes/ As Kubernetes becomes the de-facto standard for deploying applications, many of us are either running our applications on Kubernetes or trying to migrate to Kubernetes. In this blog post, I will go through some tips for running highly available applications on Kubernetes. Kubernetes: Link to heading Kubernetes, also known as K8s, is an open-source system for automating the deployment, scaling, and management of containerized applications. It is designed on the same principles that allow Google to run billions of containers a week, Kubernetes can scale without increasing your ops team. http://www.viveksyngh.com/posts/whats-in-kubernetes-client-go-part-1/ Sun, 21 Nov 2021 00:00:00 -0600 http://www.viveksyngh.com/posts/whats-in-kubernetes-client-go-part-1/ Kubernetes project provides a number of the third-party consumable repository under kubernetes or kubernetes-sigs organization on Github. In the series of blog, I will go through such upstream k8s repositories and code, which are needed most frequently to write extensions for Kubernetes. Also provides some examples for using different k8s packages for developing Kubernetes applications in Golang. First covers famous kubernetes package called client-go. client-go Link to heading client-go is one of the most useful packages for writing Golang based Kubernetes applications. http://www.viveksyngh.com/posts/linux-virtual-ethernet-devices/ Sat, 25 Sep 2021 00:00:00 -0600 http://www.viveksyngh.com/posts/linux-virtual-ethernet-devices/ Linux has rich virtual networking capabilities and in the series of posts, I am going to dig into these virtual networking interfaces, how to use them and what are their use cases. In this post, we will be looking into VETH (Virtual Ethernet) virtual networking interface. VETH devices are virtual ethernet devices. They can act as tunnels between network namespaces to create a bridge to a physical network device in another namespace, but can also be used as standalone network devices. http://www.viveksyngh.com/posts/python-collections-module/ Sun, 19 Apr 2020 00:00:00 -0600 http://www.viveksyngh.com/posts/python-collections-module/ Apart from built-in general purpose container data structures like list, dict, set and tuple . Python provides collections module which implements some specialized container data types. Following container data types are present in collections module for python 3.6. namedtuple() : factory function for creating tuple subclasses with named fields. deque : list-like container with fast appends and pops on either end. ChainMap : dict-like class for creating a single view of multiple mappings Counter: dict subclass for counting hashable objects OrderedDict : dict subclass that remembers the order entries were added defaultdict: dict subclass that calls a factory function to supply missing values. http://www.viveksyngh.com/paper-notes/mapreduce-overview/ Wed, 23 Oct 2019 00:00:00 -0600 http://www.viveksyngh.com/paper-notes/mapreduce-overview/ In this blog, I will be discussing MapReduce programming model and how it works. This blog is based on the original MapReduce research paper MapReduce: Simplified Data Processing on Large Clusters from Google. MapReduce is a programming model and an associated implementation for processing and generating large data sets. User specifies a map function that processes a key/value pair to generate a set for intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. http://www.viveksyngh.com/system-design/key-characteristics-of-distributed-systems/ Mon, 29 Apr 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/key-characteristics-of-distributed-systems/ A distributed system is a system in which components are located on different networked computers, which can communicate and coordinate their actions by passing messages to one another. The components interact with one another in order to achieve a common goal. Key characteristics of distributed systems are Resource Sharing Link to heading Resource sharing means that the existing resources in a distributed system can be accessed or remotely accessed across multiple computers in the system. http://www.viveksyngh.com/system-design/database-sharding/ Mon, 22 Apr 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/database-sharding/ Database sharding is the process of splitting up a database across multiple machines to improve the scalability of an application. In Sharding, one’s data is broken into two or more smaller chunks, called logical shards. The logical shards are then distributed across separate database nodes, referred to as physical shards. Database shards are autonomous and they don’t share any of the same data or computing resources. In some cases, though, it makes sense to replicate certain tables into each shard to serve as referenced tables. http://www.viveksyngh.com/system-design/event-driven-architecture/ Mon, 15 Apr 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/event-driven-architecture/ Event Driven Architecture Link to heading The event-driven architecture pattern is a popular distributed asynchronous architecture pattern used to produce highly scalable applications. It is also highly adaptable and can be used for small applications and as well as large, complex ones. The event-driven architecture is made up of highly decoupled, single-purpose event processing components that asynchronously receive and process events. It is suitable for applications or systems that transmit events among loosely coupled software components and services. http://www.viveksyngh.com/system-design/long-polling-vs-websockets-vs-server-sent-events/ Tue, 09 Apr 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/long-polling-vs-websockets-vs-server-sent-events/ Web applications were originally designed as a simple client-server model where the web client initiates an HTTP request requesting some data from the server. For example, a basic web application with the client-server model flow will be as follows. A client makes an HTTP request requesting a web page from a server. The server calculates the response The server sends the response to the client As developers began to explore ways to implement more “real-time” applications. http://www.viveksyngh.com/system-design/microservices-architecture/ Mon, 01 Apr 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/microservices-architecture/ Microservices are small, focused on doing one thing well and autonomous services that work together. Microservices were not invented. They emerged as a trend, or a pattern from real-world use case. The benefits of microservices are many and varied. Technology Heterogeneity: Link to heading With a system composed of multiple, collaborating services, we can decide to use different technologies inside each one. This allows us to pick the right tool for each job, rather than having to select a more standardized, one-size-fits-all approach that often ends up being the lowest common denominator. http://www.viveksyngh.com/system-design/sql-vs-nosql-databases/ Mon, 25 Mar 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/sql-vs-nosql-databases/ SQL Database is also known as a relational database. It has the following properties. It uses a table or relation to store the data. Table stores data in row and columns. Relational databases are used to store structured data. Data are stored in the form of relationships. Uses Structured Query Language ( SQL) for defining and manipulating data. Define the schema before you add any data to the system. All your future data must follow the same structure. http://www.viveksyngh.com/system-design/consistent-hashing/ Fri, 22 Mar 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/consistent-hashing/ Consistent hashing idea was introduced in paper Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web which was released in the year 1997. Akamai distributed content delivery network uses the approach described in the paper. In 2007, consistent hashing was used in two published works Dynamo: Amazon’s Highly Available Key-Value Datastore Last.fm Memcached client After that it has been used in many other distributed system like Cassandra, Riak etc. http://www.viveksyngh.com/system-design/eventual-consistency-vs-strong-consistency/ Mon, 11 Mar 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/eventual-consistency-vs-strong-consistency/ Eventual Consistency Link to heading Eventual consistency is a theoretical guarantee that, provided no new updates to an entity are made, all reads of the entity will eventually return the last updated value. The Internet Domain Name System (DNS) is a well-known example of a system with an eventual consistency model. DNS servers do not necessarily reflect the latest values but, rather, the values are cached and replicated across many directories over the Internet. http://www.viveksyngh.com/system-design/cap-theorem/ Mon, 04 Mar 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/cap-theorem/ CAP theorem also known as Brewer’s theorem was introduced by computer scientist Eric Brewer at Symposium on Principles of Distributed computing in 2000. In CAP theorem, C stands for Consistency, A stands for Availability and P stands for Partition tolerance. Consistency: Every read receives the most recent writes or an error. For consistency, any read operation that begins after a write operation completes must return that value, or the result of a later write operation. http://www.viveksyngh.com/system-design/what-is-api-gateway/ Mon, 25 Feb 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/what-is-api-gateway/ In microservices architecture, there are several services running each designed for a very specific component of the system. When clients (Mobile Apps, Web Apps or Third party applications) communicates directly with these microservices then many problems arise. The granularity of APIs provided by microservices is often different than what the client needs. Microservice API’s are very generic and granular in nature where each returns only a portion of data for functionality. http://www.viveksyngh.com/system-design/load-balancing-and-load-balancer/ Mon, 18 Feb 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/load-balancing-and-load-balancer/ [Source: https://medium.com/containers-on-aws/using-aws-application-load-balancer-and-network-load-balancer-with-ec2-container-service-d0cb0b1d5ae5] Load Balancing Link to heading Load balancing is the process of efficiently distributing network traffic across multiple servers also known as a server farm or server pool. By distributing the load evenly load balancing improves responsiveness and increases the availability of applications. It has become a necessity as applications become more complex, user demand grows and traffic volume increases. Load balancing is the most straightforward method of scaling out an application server infrastructure. http://www.viveksyngh.com/system-design/what-is-proxy-server/ Wed, 06 Feb 2019 00:00:00 -0600 http://www.viveksyngh.com/system-design/what-is-proxy-server/ Word proxy defines, someone or something acting on behalf of something else. In computer science proxy means one server acting on behalf of other servers. A proxy server is a server ( or computer system or an application) that acts as an intermediary for requests from clients seeking resources from other servers. A client connects to the proxy server, requesting for some service (file, connection, web page) available from a different server and the proxy server evaluates the request as a way simplify and control it’s complexity. http://www.viveksyngh.com/posts/getting-started-with-golang-interfaces/ Mon, 15 Oct 2018 00:00:00 -0600 http://www.viveksyngh.com/posts/getting-started-with-golang-interfaces/ Interfaces in Go provides a way to specify the behavior of an object. Interfaces allow us to write more flexible and scalable code in Go. It is also a mechanism to achieve polymorphism in Go. type Interface interface { Method1(int, string) (int, error) Method2(int) bool Method3(string) } Interfaces specify methods (or behavior) but do not provide an implementation for those methods. Types implementing the interface provide an implementation of methods. http://www.viveksyngh.com/posts/object-oriented-programming-and-golang/ Fri, 14 Sep 2018 00:00:00 -0600 http://www.viveksyngh.com/posts/object-oriented-programming-and-golang/ Go has picked some good concepts from different kind of programming languages like procedural, functional and object-oriented programming language and combined them together to create a unique flavor programming language. Go isn’t an object-oriented language like C++, Java, C# etc. but it supports some of the OOPs concepts like Encapsulation, Type Methods, Composition etc. In this blog post, we will see how can we implement those concepts in Go Type Methods Link to heading Go does not have classes but it has types. http://www.viveksyngh.com/posts/pass-by-value-and-pass-by-reference-in-go/ Fri, 07 Sep 2018 00:00:00 -0600 http://www.viveksyngh.com/posts/pass-by-value-and-pass-by-reference-in-go/ In Go, everything is passed by value. When we pass arguments to a function, a function always gets the copy of the data passed. For example, passing an int value to function makes a copy of the int, passing a pointer value makes a copy of the pointer but not the data it points to. When we pass a value to function it is called pass by value and when we pass a pointer to a function it is called pass by reference. http://www.viveksyngh.com/posts/http-response-in-golang/ Sat, 01 Sep 2018 00:00:00 -0600 http://www.viveksyngh.com/posts/http-response-in-golang/ In this blog post, I am going to talk about HTTP server and how to return different types of HTTP response(text, JSON, html) in Go. Plain Text Response Link to heading Creating a web server in Go is very simple and we can do it by writing just a few lines of code.We need to use net/http package to create an HTTP server. This is how a simple HTTP server code looks like in Go. http://www.viveksyngh.com/posts/timeout-commands-in-golang/ Tue, 28 Aug 2018 00:10:34 -0600 http://www.viveksyngh.com/posts/timeout-commands-in-golang/ Sometimes in our applications, we need to invoke external processes or commands (e.g CURL, Ping, SSH etc.) to perform some tasks. We can use os/exec Go package to invoke external processes. Most of the time we want to invoke these commands with timeouts. In this blog post, I am going to talk about different ways in which we can invoke a command with a timeout. Timeout with timer Link to heading In this method, we use a timer for the timeout. http://www.viveksyngh.com/about/ Mon, 01 Jan 0001 00:00:00 +0000 http://www.viveksyngh.com/about/ My name is Vivek Kumar Singh and I am a Staff Engineer at VMWare where I am building softwares for infrastructure. My expertise lies in harnessing the power of Kubernetes, Golang, and cloud-native technologies to build robust and scalable solutions. I have also worked in multiple open-source communities and contributed to many open-source projects such as OpenFaaS, Kubernetes etc. I have a strong passion for distributed systems, cloud-native technologies, and open-source software. http://www.viveksyngh.com/resources/ Mon, 01 Jan 0001 00:00:00 +0000 http://www.viveksyngh.com/resources/ My Favorite Resources Link to heading Welcome to my collection of favorite resources! Here, I’ve curated a list of blogs, books, tech talks, and more that I find particularly valuable and inspiring. Explore the categories below to find something that piques your interest. Golang Link to heading Talks Link to heading Google I/O 2012: Rob Pike - Go Concurrency Patterns Google I/O 2013: Sameer Ajmani - Advanced Go Concurrency Patterns