Art of "Unsettled" Science in Distributed System

Merriam-webster defines baseline "as a line serving as a basis", or you can think of it is a starting point used for comparisons. In site reliability engineering (SRE), establishing a performance baseline for workloads is a common practice. This baseline serves as a reference point for monitoring and also improving system performance over time. In my previous experience as a SRE, particularly in supporting critical e-commerce websites, I learned that it's beneficial to establish more than one baseline for a more holistic view and accurate analysis. I have found two key types of baselines are quite useful: [A] Performance Baseline Influenced by External Factors: This baseline takes into account real-world conditions such as network variability, user device performance, and other external influences. It reflects the performance metrics as experienced by the end-users, providing insights into how these factors impact the user experience. [B] Performance Baseline Detached ...

"Without observability, you don’t have ‘chaos engineering’. You just have chaos. ” Charity Majors I have always been very intrigued by Sherlock Holmes' "school of thought" and read many of them. There are two stories, that kind of stand out when I think about modelling his thought process, called Abductive reasoning in the context of observability and principles of chaos engineering. Sounds amusing, but when you think deeply ("slowly") about the "Adventure of the silver blaze" it's kind of takes us to realm of "known unknowns" and "unknown unknowns'', "Known knowns" are things we know that we know, in the context of distributed systems, these are the things we understand fully and can predict how they will behave. "Known unknowns" are things we know that we don't know, these might be edge cases or untested code that we know exist but have never fully explored. "Unknown unknowns" are t...

               It's science but...not all variables are "settled"... Welcome to "Art of Unsettled Science" – our little corner of the "real world computing" we talk about topics in the realm of distributed systems that are little more than a textbook full of facts. Here, we dive into these grey areas that are not quite "settled" yet, where theories are still in the making, and everything feels a bit like a work in progress. We are all about exploring the unknown with a sense of wonder and a healthy dose of curiosity. So, if you're into adventures where the answers aren't clear yet, you've found your crew.