You are viewing...

Encrypted Configuration

Updated on November 14, 2019 at the 13th hour
Posted under:

DISCLAIMER: All views are considered my own and you should not draw any conclusions on associates.

A.K.A. Secrets. Secrets are not terribly interesting topic outside of servers and databases being compromised by exposed details where we raise pitchforks at those for not keeping their servers behind proper network access security and credentials encrypted. Configuration refers to values provided to software initially and while it runs by a file descriptor (file, socket) or shared memory. Environment variables are provide only at boot and cannot be changed at runtime, so I'm a fan only for constant values.

Centralized Secrets aka Don't worry about it

I was working with secrets management where I worked previously and Hashicorp's Vault was used for it. It is a pretty good product that focuses only on core product. I'm not too big a fan of centralized storage of secrets or configuration, but it is only one way to manage secrets without worrying about encryption and keys and etc. You could store many kinds of things considered secret such as credentials like API Keys, database passwords and etc. One could rotate passwords using database integration where Vault will automatically create user/passwords. Same for AWS. It does have shortcomings such as how to notify of rotation/change, but tries to mitigate it with grace periods. You still have to ping Vault and check if it has a value changed. Imagine doing that for all of your secrets. Probably don't have many secrets at a small company, but you can at a larger one. It is manageable to do.

Anyway, Vault is a good overall product with limited provisioning, sucky UI, high learning curve (not completely vault's fault), full secrets storage and management and client provider libraries.

In the end, it is a product that needs to generate cash for its masters/dictators.

Decentralized Secrets

Centralized configuration is okay, but my belief is that configuration is inherently decentralized. Configuration should be stored anywhere secret or not. I'm not the only one who believes this either.

I was researching the idea of storing secret configuration into files and checking them in (Encrypted at rest). PGP provides a natural way to do this via key rings/multi-key encryption, so you could actually store everyone's public keys who has access to a secret and use them to encrypt secrets. You could remove users who had access and re-encypt the secrets to block access. Basically, there isn't any authority other than your repository team that controls access to secrets in the repository. This is great as the less configuration servers then the better.

Honestly, why store database user and passwords in centralized store if they never change? If they needed to change then it would require a commit, which makes it inherently auditable. One could create a bot where users could subscribe their projects to that could update the secrets and submit a pull request describing the change. That same bot once all of the pull requests are completed could delete the old secret. The bot itself does not store secrets, only where they are located and a vector clock (version), so they can be updated.

Interesting things that can be done to make decentralized configuration easily manageable without a central authority. Remember, decentralized configuration only means that configuration can be anywhere.

Encryption in yaml

On the topic of keeping secrets in files, Mozilla created SOPS (https://github.com/mozilla/sops), which I think is a great tool for managing secrets across multiple environments inside of files with different keys. It uses PGP, AWS KMS, GCP and Azure and publishes to AWS, GCP or Vault if desired. This all assumes that your developers have access to one of those key management systems. As a team of one, you probably don't need cloud IAM, so PGP it is. I can pretty much use my public PGP keys to encrypt secrets and use a server's PGP key to encrypt secrets. Whenever I need to modify, decrypt and re-encrypt. You see, I won't always remember to do that and that's where git hooks come in hand.

As you can see, it requires more work than Vault, so it is not my ideal solution yet.

If you are interested in secrets in without a dedicated service: https://opensource.com/article/19/2/secrets-management-tools-git

What is my ideal encrypted configuration solution?

PGP encryption/decryption

Encryption at rest, partially, so certain pieces of the file is encrypted (a way to define patterns)

Decryption in the application runtime. Values could be provided via shared memory, private mount point, etc.

Encryption verification to make sure that secrets are encrypted. Uses the patterns defined above.

Editing of secrets without observing values. Just set key = value. in order to set a value here, there must be a decryption underlying to make sure one has access.

That's about it. This is all about encryption of configuration not a full featured configuration provisioning.


I'll talk about a full featured configuration solution in the future after I've built it and tried it out. Likely open source, so I don't have to deal with crappy, inflexible systems at smaller companies.


Tangent. I am thinking about pre-filled PGP keyrings with non-revocable/operational pub keys. Organization master key (constant), runtime pub key (changes per environment), CI/CD pub keys (build keys) should never have to be removed. Organization can provide the keyring for each project where maintainer keys are already imported, so each maintainer can add or remove other maintainers, but could never remove the keys required to operate an application in the environment. Cheap way is the verification of the project keyring at build time.

You just read "Encrypted Configuration". Please share if you liked it!
You can read more recent posts here.