It's about time that I did this, all these years of cobbling together and duplicating this code and now I've formalized it into a crate!

So, if you've been following my coding adventures, you know I'm a big fan of Rust. And like many developers, I often find myself needing a fast, reliable local database. Enter RocksDB – it's powerful, battle-tested (used by giants like Meta!), and integrates nicely with Rust thanks to the rocksdb crate.

But... here's the thing. While RocksDB is great, using it directly ofte...

Yes, another application configuration library, but I assure you this is way better than any one of them out there. This is the Provider section of the Config Triad (Provision, Management, Provider).

C5Store is a comprehensive answer to the "what config library should I use for dynamic runtime configuration" question. Library is available here: https://github.com/normano/c5store

Why?

I've scoured around for configuration libraries in multiple languages and find them to be lacking in how they are defined, integ...

C5Store for Rust

C5Store is a Rust library providing a unified store for configuration and secrets. It aims to be a single point of access for your application's configuration needs, consolidating values from various sources (like YAML files), handling secrets securely via built-in decryption, and allowing dynamic loading through providers.

The core idea is to simplify configuration management in complex applications by offering a hierarchical, type-aware, and extensible configuration layer.

Key Features

This is a common challenge, and i...

Lately I've taken an interest in facilitating Remote Procedure Calls (RPC). Spawned from service management (mgmt) application I've been building in Rust. Along with that service management system, I built an in memory db for it in Rust as well. Very interesting stuff for me espcially Rust. More on that some other time.

What is an RPC?

RPCs are used to communicate across applications. Like if I wanted to get a user's email then I would submit a GetUserEmailRPC Request and get back a response at some point (...

You've got this awesome project you built internally. It started small, maybe lived in a giant monorepo, or perhaps its own repo that also accumulated some... internal peculiarities. Now, the time has come: you want to share it with the world! Open source it! 🎉

That initial excitement often hits a speed bump: how do you actually get just that project out without exposing the entire monorepo, confidential history, accidental secrets, or internal configurations? And maybe even trickier: once it's out, how do ...

Like many developers, I've spent my fair share of time dealing with database schema changes across different environments. Keeping development, staging, and production in sync can feel like herding cats. Traditional migration tools often use tracking tables to remember what's been run, which works great, but I found myself wanting something a bit different for certain workflows.

A while back, I built a small Node.js tool called agogo based on a specific idea: what if the tool didn't track state? What if, ins...

A.K.A. Secrets. Secrets are not terribly interesting topic outside of servers and databases being compromised by exposed details where we raise pitchforks at those for not keeping their servers behind proper network access security and credentials encrypted. Configuration refers to values provided to software initially and while it runs by a file descriptor (file, socket) or shared memory. Environment variables are provide only at boot and cannot be changed at runtime, so I'm a fan only for constant values.

...

Couple of months ago I wanted to talk about the Configuration Triad: Configuration Provisioning, Distribution and Provider.

If have not seen my Open Source Config Provider, C5Store, well you can find it on github.

Going to talk about the triad in reverse in kind of the way an application developer works with configuration.

Provider

Provider is the interface to a configuration

Hard coding is the simplest form, the values are provided in code.

Typically a developer uses a file, reads from the file, deserializes it...

Yeah, so I left my job today so I could finally focus on some of the ideas I've had full time. I've always thought that I would have come up with a consumer business on the side while I was working, but that hasn't panned out as I would like. Mostly just focused on work and creating my own infrastructure software and libraries.

What Have I Done Already?

Let's talk about that non-exhaustively. I write mainly in Rust, Java and JavaScript. Any other language is because of third party software, which is very much...

Introduction

Service Discovery, the phonebook of applications, is a foundation for services oriented architecture. It answers the question of how does one service dynamically discover the endpoints of another service.

It is not a new thing that came with microservices. Zookeeper is a famous application for implementing service discovery on regardless of requirements. I give it good praise for the things you can do with it. If you don't have a lot of time to spend I recommend it.

Use Case

Building on top of ...

If you are like me, you just want to run your application in SystemD with no Docker or Kubernetes involved. I made a build/deployment application for my own applications, private closed source, that builds applications using docker/podman, extracts distribution files from the container and deploys by talking to SystemD. I respect SystemD and grown more appreciation of it 😀.

Goal

The goal of the application was to be able to use commands like `deployer build <project> <build name>` to produce an archive that ...

Terrible Idea. Fuck Complexity.

Proxy for every service? I would fire you for this. Oh, you made managing the proxies manageable? Oh, you can load balance properly? Canary routing? Metrics? OMG, service-to-service authentication? Right, I can do that with clients with far less overhead and time spent and still manage the clients like a proxy. Bye bye.

Service Mesh.... a terrible idea.

What makes me want to write this post? Ah, I've been working on my own product and scouring around for a good answer to authorization. Particularly, authorization libraries in NodeJs. Some are really bad just looking at their GitHub pages and others are pretty good. All of them make an assumption that authz rules are stored somewhere (code, memory, db), which is great. Some are flexible about where policies are stored offering plugins. A lot tend to be rigid on the idea of Role based authz (RBAC) and not en...

MySQL's Binary Log is nice to have, but not easiest to use. Change data capture (CDC) is the primary reason why I have to do it. When a user performs and action that causes an INSERT/UPDATE/DELETE, I want to know about it.

Typical way is to plug it into your application, but that is prone to issues when your CDC statement doesn't run because fatal error caused your application to quit in the middle of the data manipulation statement. The entry was written to MySQL and its Binary Log though!

I've always read ...

The MessagePack Protocol supports 64bit Integers, but for whatever reason msgpack for js still decided not to fully support 64 bit ints. Javascript supports 64 bit integers through arbitrary bit precision bigint, so it is insufficient to say the language does not fully support the protocol.

So fuck em, I'm forking it and adding support for my own purpose. I've no confidence that the protocol will support 128bit integers anytime soon. Not that that matters to me since UUIDs and ULIDs are serialized to uint8 a...

When I say RPC request and response, that means the request and response were serialized/deserialized with messagepack. It's just message passing in the end.

ZeroMQ is fast I mean really really fast. ZeroMQ is a networking library and should be part of any toolkit where you need extreme high performance and reliability between applications.

Let's do some math

140k requests per second + 140k responses per second = 280k req/response total. That's 2.8 million in a 10 second span. This measly server can handle m...

I was curious about use of ZeroMQ since I use it extensively with my own RPC, PubSub system.

Spotify's Hermes

Hermes is a network messaging stack created at Spotify, on top of existing open source stack such as ZeroMQ, protobuf and gevent. We are migrating backend service from legacy stack to Hermes and this session explains the rationale behind some of the choices. It also gives a brief introduction of some typical network topology for backend messaging at Spotify. Finally we discuss some lessons lear...

You've been there. Need a new microservice, maybe a quick CLI tool. You remember that perfect template you stashed away, or maybe you grab one off GitHub. git clone, run the generator... npm install... FAILED. Or cargo build... FAILED. Dependencies drifted, build steps changed, the world moved on. The template lied. Template rot is real, and it wastes our time debugging boilerplate before we even write line one of our actual project.

I got seriously fed up with this cycle. I've automated the heck out of th...

I used to work on Datacenter Monitoring circa 2015. I automated the heck out of my job. One of the last projects I delivered there was a Distributed Real Time Monitoring and Alerting Platform, written in Java. I will build it once again with the help of AI in Rust.

It was an incredibly complex system with 5 different roles, technically 4 layers. I spent around 6 months building the system and was incredibly proud of it. It worked like a charm, very fast, high performance, exceeded my wildest dreams, when ...