An environment without Secret Management
I arrived on site at the client and things could’ve been a lot better. There were password protected zip files containing more passwords, sticky notes stuck to monitors and desktop applications with names that sounded like saying “psychotic” with a lisp. When an application needed a password to connect to another service it would be promptly written to a config file and checked into version control. Rotating a password meant RDPing into a machine with an administrator account and typing in the new password, trying to remember which server was updated first. Databases were configured with admin accounts and the passwords were left untouched with numbers like '123' slapped on the end. They need a credential/secret management service and starting on a new cloud provider was a good time to create some good habits going forward.
An environment with Secret Management
The best passwords are no passwords, if you can't do that, then passwords with a limited lifespan are the next best thing. The reason being that if someone does something silly like commit the password to Git and push it up to a public repo, by the time anyone notices, hopefully it has expired. Also because they change, saving them into config files is discouraged because it doesn’t stay working for long. Using a service ensures that each time a set of credentials is accessed, it can be recorded, monitored and possibly even denied where that access is not appropriate.
What is Hashicorp Vault?
- Secret store
- Dynamic credential generator
- Secret access auditability
- RBAC to Secrets
Vault is not just a password manager but a complete system to authenticate, authorize and account the usage of secrets. It's configured to be a service that can be programmatically accessed and in some sense is like a proxy to other sources of secrets. For example a user could authenticate with Vault using valid LDAP username and password, then request time limited database credentials. Vault will then connect to the database, create a temporary user with limited permissions, return those credentials to you and later cleanup the temporary user. All of the actions performed are logged for auditability and the permissions granted are defined in a role based access control system.
How can Vault be accessed?
Vault provides a RESTful API for systems to interact with and the Vault binary for CLI access is really just a wrapper for those APIs. For the authenticated resources, the user’s vault token is supplied as a Header value and the typical CRUD actions are performed with PUT/POST, GET and DELETE commands. An example of retrieving an element from a Generic secret backend would be;
curl -H "X-Vault-Token: b1c95b61-5a72-4c3f-a10a-da234efd2c3d" https://127.0.0.1:8200/v1/secret/bar
This would return a JSON serialized response. Adding
?help=1 in most cases to the end of a URL will also return help text related to the URL. The advantage of using a simple RESTful API is that even if your platform of choice does not have an official library to integrate with Vault, it's easy enough to write the integration yourself. I've personally used bash scripting to connect a CI platform without needing the Vault binary installed.
Use consul-template to keep dynamic credentials up-to-date on a filesystem for a service
For the times you do not have access to modify the code of your applications to directly integrate Vault, there are two useful tools provided by HashiCorp that can help you out. envconsul will set environment variables based on values stored in Consul and Vault and consul-template does a similar job but for modifying config files based on templates. The latter can also be configured to execute commands in response to changes allowing you to notify a service of updates to its config or even trigger a restart of that service.
Write a Vault-aware application which has knowledge of how to manage its own credentials
Of course when you do have access to the source, it’s fairly easy to use Vaults RESTful API to retrieve secrets as needed. There are libraries for Go, Ruby, C#, Python, Haskel, Java and more. Failing that, the API is well documented so it shouldn't be too hard to integrate with it. Once your application is using Vault, rolling credentials will be automatic; the old credential expires with a timer and the application just re-requests new ones as needed.
Generate dynamic credentials for database access
A specific example of why you would want to integrate directly with Vault would be to have dynamic credentials for a database. The process would be to start with your application authenticated against Vault via an AWS IAM instance role, so that the application can call '/v1/auth/token/lookup' to see if the token is still valid. Then call '/v1/database/creds/readonly' to retrieve read-only credentials to the database and then connect. When the credentials are about to expire, the library then it requests a new set of credentials. As a result of going through a service, the requests are logged and anomalous activity can be used to trigger an alarm and the activity be shut-down.
Pass credentials from Vault to an application as part of a deployment
An alternative to using AWS IAM instance roles to identify machines is to use the AppRole authentication backend. It can be used where an equivalent service does not exist and using it provides some benefits to supplying just a standard username and password. It also comes with some flexibility in the ways to utilise it. One way is to assign a deployment of an application and a role ID and request a secret from Vault associated with that role. Instead of receiving the secret in plain text, you ask for it to be a wrapped response so the deployment mechanism never sees the secret itself, just the accessor to it. That is then passed to the application as an environment variable. Where the application then uses the accessor as a one time code to retrieve the secret, authenticate to Vault and use it’s token for ongoing access.
Generate certificates for service identification
Another useful service of Vault is that it can act as your Public Key Infrastructure. Serving both as a root certificate authority and as an intermediate certificate authority. When a web server is spun up it can simply request for a valid TLS certificate that will be trusted by the users within an organisation and between the machines too. There will be no excuse for a web service to use a self-signed certificate - the applications connecting to it can’t verify the validity of the certificate and as a result disable verification on the connections.
Isn't it "turtles all the way down"?
If there was a single globally recognised credential system you wouldn't need Vault. Also if you fit perfectly within a single ecosystems credential system you could quite happily operate without Vault too. But we're never that lucky and the way Vault glues so many different systems together just doesn't exist anywhere else. The apex of Vault comes down to the master key but to stop that being a single point of weakness it's split into pieces for safe keeping and never present in a usable form except in non-pageable memory. Depending on the backend type, the secrets you retrieve are protected by one time use, TTLs, usage policies and even wrapped by another layer of protection.
In the process of uplifting the client’s processes and people, we implemented a unified system to build, deploy and configure their applications. Code would be tested as part of a continuous integration pipeline and deployed in an automated fashion. In-house application could be built to incorporate Vault credentials natively and the more "Enterprise" applications needed a lot of sticky tape and chewing gum scripts to get them to work with a dynamic system. One of the key features in the new setup is the ability to audit credential usage and rotation of credentials is completely automated. Taking advantage of AWS IAM instance roles we could almost eliminate the need to store any credential outside of Vault. Also users of the system could authenticate with their existing Active Directory, granting permissions where appropriate. There are still a few outliers but we're working on that. It definitely could’ve been better when we came in, but now I think the client understands that with this new setup, it could be (and had been) a lot worse.
If you need assistance with your authentication processes, or implementing and utilising Hashicorp tools, Vibrato is Hashicorp's exclusive training partner for APAC, and will happily provide your business with a free consultation. Just click below!