6 Projects form the basis of my super charged Infrastructure. All built in Rust.
- Project #1: Time Series/Log Database with Streaming
- Project #2: User/Group/Certificates/Domain with Acme Renewals. Repository Manager (Git/Yum/Maven/Docker/Cargo).
- Project #3: Infrastructure Database storing Server, Server Groups and Metadata
- Project #4: Service Database storing service deployments that have host:port, instances and etc. information.
- Project #5: Infrastructure UI to view and configure the infrastructure
- Project #6: Deployment
With this supercharged infrastructure lays a hardened bedrock for making sure applications get deployed, operate and can be maintained easily.
So Project #1, I've always wanted to do a TSDB. I created it without AI, then used AI to build the Log and Streaming portion. I vastly improved the Incremental FST that is used for the tag index. Before I did it naively where every write resulted in a merge, now it's been improved to be a streaming incremental FST which is unheard of. This IFST was incredibly hard to pull off. I talked about it in a previous post.
Project #2 was weird. What started off as a repository manager for Yum/Maven/Cargo experienced the dreaded feature creep. I wanted to add authorization, but to have authorization you need authentication, IAM essentially. I built a Zanzibar lite on top of my Schemaless Hierachical Database then used that as the basis for the AuthZ portion of the project. I split the project into two applications under the same project once I realized IAM can be reused, doesn't depend on repository management and needs to be highly tested and highly available. IAM should basically just never go down for any reason. As part of the IAM portion, I decided well I need automation for certificate renewals, so let's add that too hah.... What came out of the project is pretty crazy.
I can't even believe I went the custom Rust Git Server route, but I just deleted my git server since this project replaces it. Soon, I'll delete the docker registry to replace it with this.
Project #3 is an Infrastructure Metadata store. It is actually where the Schemaless Hierachical Database was first spun out from. The new metadata structure I had built was inspired by C5Store and I had that ready to go until I figured welllllll shit we could make a Hierarchical database from this and add code generation to generate the structs and client operations that would operation on the data. It's crazy, the Infrastructure database is now basically run on the Schemaless Hierarchical database now instead of only the metadata operations.
Project #4 provides information about what is running (Actual State) as well as (Desired State). Application instances need to be allocated to servers and that data needs to be somewhere, this was a natural place. Any application can query for this data. This is especially useful for clustering or traffic networking (load balancing). This is updated by the deployment application, but also presence/health data is updated by itself. This project will come in handy for my Redis Protocol Compatible CasPaxos DB for clustering.
Project #5 is a boring UI to show the infrastructure. Nothing special, just data aggregation and configuration. A way for a human (me) to make decisions about infrastructure. It's just part of system administration. It's very important, so it has to be boring.
Project #6 spawned out of my own desire to run application on servers without running them in containers. I had written it in NodeJS and used it, but NodeJS always had quirks. Now it is written in Rust. It uses podman to build the application then creates artifacts from what is in the container to be deployed on any bare metal machine. It facilitates reproducible builds and smooth deployment. With AI, I added all the things I wanted to add like automatic instance allocation and instance lifecycle operations (start/stop/restart/undeploy). I still have more that I want to add, but some require another project to be built first before I can add those features.
All of these projects depend on a project before it. Project #6 used to be configured through files, but as I built the lower number projects, it depended on them. Project #5 is a UI, of course it depends on other projects, soon it will depend on Project #6 which will lessen the use of the command line tool I use to control Project #6. Project #4 is 3 applications in a cluster. All the projects depend on Project #1 to capture logs and metrics.
This is the most complex system I've ever built. There's so much going on behind the scenes to get to this point. I'm glad I was able to replace the existing system without interruption to the applications themselves and can easily deploy as if nothing changed on the backend (trust me it was a huge overhaul).
After all of that Excerion Sun LLC Redesign is live and it features Excerion Sun LLC Open Source. Took 17 seconds to deploy the application. It's a simple SnapFire rust application. This is the easiest application that I wanted to deploy and make sure to get right. 17 seconds deployment time. 😮💨