Engineering Experiments

Experiments.

Break things. Investigate why. Understand deeper.

DONIVBYTES runs deliberate engineering experiments — we take a technology, push it until something breaks, then document exactly what happened and why. The failures are the point.

Why experiments?

Documentation tells you how a system is designed to work. Experiments show you how it actually behaves — under load, at the edges, when configuration is wrong, when dependencies fail.

Every experiment here follows the same pattern: define a hypothesis, run the test, observe what happens, explain the result. No hand-waving. No skipping the interesting part.

The investigations are documented so they're useful to anyone who hits the same situation — not just a log of what we did, but a proper explanation of why the system behaved the way it did.

experiment format
HypothesisWhat we expect to happen
SetupExact environment & config
TestWhat we actually ran
ObservationWhat happened
ExplanationWhy it happened
TakeawayWhat to remember
Coming Soon

Experiments in preparation

Breaking Docker bridge networking

Docker

What happens when you exhaust the default bridge network subnet? We found out.

DNS TTL vs reality

Networking

TTL says 60 seconds. We measured what actually happens across resolvers.

Git's object store under load

Git

How git actually stores 10,000 commits and what pack files look like.

AWS Lambda cold start anatomy

AWS

Breaking down exactly where the time goes in a Lambda initialisation.

Areas we'll cover

Experiments will span the full engineering stack — from the Linux kernel up to cloud architecture.

Linux

  • Process scheduling
  • File descriptors
  • Kernel namespaces
  • cgroups
  • Signal handling

Networking

  • DNS resolution
  • TCP handshake
  • Packet inspection
  • Routing tables
  • Firewalls

Git

  • Object model
  • Pack files
  • Ref storage
  • Rebase internals
  • Hook system

Docker

  • Network modes
  • Layer caching
  • Image internals
  • Volume mounts
  • BuildKit

AWS

  • VPC networking
  • IAM evaluation
  • S3 internals
  • Lambda cold starts
  • EC2 metadata

Infrastructure

  • Terraform state
  • IaC drift
  • Provisioning failures
  • Secret management
  • Config drift

Backend Systems

  • Connection pooling
  • Query planning
  • Cache invalidation
  • Rate limiting
  • Timeouts

Cloud Architecture

  • Multi-region failover
  • Cost anomalies
  • Latency spikes
  • Cold start chains
  • Noisy neighbours

Want to learn the underlying concepts first?

The learning section breaks down the fundamentals before we break the systems.