HN user
https://pic.dev
Former Co-Founder & CTO at Doppler (W19)
I wonder what impact this will have on Mozilla's OpenSSH configuration guide[0], which currently specifies `chacha20-poly1305@openssh.com` as its primary cipher. Should that be dropped to rely solely on AES ciphers?
I'm the CTO of a popular Secrets Management platform. It's fair to say that I personally have a lot of experience with secrets and requirements around them, based on conversations with customers.
The primary missing feature here seems to be multiline support. That's super common for keys, certificates, and other JSON configuration. Based on the Node PR, they appear open to adding that later (big +1 on shipping incrementally).
The other missing feature that folks tend to heavily rely on is variable expansion. For that to truly work well, I recommend using a holistic platform like Doppler. That allows for expansion/referencing across environments and projects, like when you have multiple independent services that need access to the same set of secrets (e.g. database creds, error reporting tool, stripe key, etc). You can then update the secret once and have the change propagate to all the places it's used.
Lastly, I'd be remiss if I didn't mention the Doppler CLI and our own fairly unique support for .env files. We've traditionally taken a dim view of .env files because they represent a static, long-lived collection of sensitive information that lives offline. Often, these get checked into a git repo. This is probably fine for personal projects, but a major issue for companies and the security aware. However, .env files are a pseudo standard and folks want a way of continuing to consume their secrets via them. Our CLI's approach is to mount a named pipe that we can write secrets to when a reader attaches. That allows us to limit the amount of times the "file" can be read (e.g. once), it guarantees that the file's contents are unavailable once the application process dies, and it uses the same open/read interface as a standard file.
In all, this is an exciting development for Node. I'm glad to see more standard features make it into core and hope that multiline support is a fast follow.
When this deal was originally announced, Adobe's stock took a ~10% hit. Now that the deal is getting blocked, their stock is again taking a hit? I certainly can't claim to understand the public markets.
You're exactly right about those two problems, and they actually go together quite well. If a user who previously had access to a secret suddenly has that access revoked, you should rotate that secret. Technically that's only really necessary if the user ever actually saw that secret value, but most secret managers don't expose that information.
At Doppler, we're currently in the process of building out our rotation engine[0]. It allows you to automatically rotate secrets with third party providers (including your database) on a schedule you define. We also provide access logs to see exactly who saw what and when. The ultimate goal is to automatically rotate your secrets as users' access changes. And if you get notification of a breach, you can rotate everything with a single click.
Our process never includes ghosting anybody, even when we pass on a candidate, so something must have gone very wrong here. Could you email me at thomas@[company domain]? I'll look into this further.
At Doppler we have a strong culture of engineering excellence, thorough PR review, security, automation/CI, and helping each other succeed. We give engineers 30% time to work on things they think are important and are remote first (but also have an office in SF). Our engineering team hangs out in Discord all day to make pair programming/debugging easy. It helps that our CEO used to be an engineer.
Co-founder/CTO of Doppler here. I'd agree that Doppler currently focuses very heavily on UX. We aim to fit into developer workflows, regardless of where devs run their code. This includes local development (macOS/Windows/Linux), CI/CD, Kubernetes, Vercel, and just about anywhere else. We've found that security tools that are difficult to use end up being worked around by developers, which ultimately decreases an org's security posture (see HashiCorp Vault). This does come with some tradeoffs- specifically that you must trust Doppler with your secrets, given our tokenization model[0].
We do recognize that the current security tradeoffs of Doppler aren't going to satisfy everyone. For Enterprises, we offer Enterprise Key Management (EKM), which allows orgs to encrypt their secrets using a cloud KMS. Of course, this still doesn't satisfy every concern. And so, for customers requiring additional security guarantees, stay tuned!
This is another good read buried in the body of https://news.ycombinator.com/item?id=29135559.
Doppler solves this problem by storing your secrets in the cloud *hand wave*.
In actuality, the Doppler CLI (a Go binary) fetches your secrets from Doppler's API and injects them as environment variables into your specified process. That looks something like `doppler run -- printenv`. This prevents your secrets from being written to the filesystem in plain text, and prevents the environment variables from being available more broadly. In the case of docker, you would bake the Doppler CLI into your image, thereby sidestepping the documented `docker inspect` pitfall.
Of course, the CLI still needs a way of authenticating to Doppler's API. You authenticate and authorize the CLI by running `doppler login`. This initiates a login flow that you complete in your browser. Once completed, your newly generated auth token is sent back to the CLI. The CLI then saves the auth token to your system keyring for later use. The identifier needed to access that keyring entry is then stored in plain text in the CLI's config file (~/.doppler/.doppler.yaml), which is only readable by the user.
We're exploring other means of injecting your secrets into your application, as some users are wary of using environment variables. This is a challenging problem though as there are few means of injecting secrets that don't require substantially changing your application's logic.
This solves a very real problem that some services like GitHub [0] have started to address. Auth tokens are being committed to public repos at an alarming rate. Detecting this and ideally preventing it as early as possible is key to avoiding account compromise. There are two components to this: identification of a secret and attribution. Identification is non-trivial and requires determining if some text really is a secret and not just a random hash, uuid, or other high entropy string. Most tokens today are generic, alphanumeric patterns; false positives abound. Attribution is tricky too, currently relying on either parsing the variable name (`AWS_SECRET_KEY=XYZ`), commit message, file name, or some other metadata. In the rare case, a service will have designed their auth tokens with this in mind, prepending a unique, static prefix to their tokens.
The URI scheme proposed in the linked RFC will squarely solve the first problem. It will allow for highly accurate CI scanners and pre-commit hooks. The scheme doesn't appear to address attribution, assuming all service providers use the same `secret-token` scheme. However attribution is a nice-to-have, allowing for automated revocation once the secret has gone public. If done right, identification alone could be used to prevent most of the token leakage that occurs today.
[0] https://docs.github.com/en/developers/overview/secret-scanni...
Really interesting development in the section titled "Crash Monitoring?":
An interesting side effect of the new processing pipeline is that imagent is now able to detect when an incoming message caused a crash in BlastDoor (it will receive an XPC error)... As can be seen, imagent is apparently informing the iMessage servers that the message with the UUID 0x3a4912626c9645f98cb26c7c2d439520 (fU key) has caused a crash in BlastDoor. It is unclear what the purpose of this is without access to the server’s code. While these notifications may simply be used for statistical purposes, they would also give Apple a fairly clear signal about attacks against iMessage involving brute-force and a somewhat weaker signal about any failed exploits against the BlastDoor service.
In my experiments, after observing one of these crash notifications, the server would start directly sending delivery receipts to the sender for messages that hadn't actually been processed by the receiver yet. Possibly this is another, independent effort to break the crash oracle technique by confusing the sender, but that is hard to verify without access to the code running on the server.
Apparently Apple has added the ability to detect crashes when parsing iMessages, report that data back to the mothership (with some attribution), and then mislead the sender of the likely malicious message with fake delivery receipts. This is cool on so many different levels.
I haven't gone as far as emailing support, but my single tweet asking if there would be a postmortem after a previous incident did not receive a reply. My other tweets to them, such as inquiring about a recent increase in spam, received replies.
I've admittedly put in very little effort to obtain postmortems from Fastmail. But I also posit that I shouldn't have to. Postmortems should be made readily accessible, just as the outage was. I reserve my detective skills for figuring out my own service's outages.
I'm a happy Fastmail customer, but the number of outages recently has been a little concerning. There has been at least one outage per month for the past several months. And what's worse is that I've never seen a postmortem published. It's hard to maintain confidence in a service that isn't being transparent about why they keep having outages, or what they're doing to prevent future outages.
This is fantastic! I often workout specific muscle groups and cycle through different exercises to target each, so this tool very nicely aligns with that. The videos are a great touch to help me make sure I have proper form. And overall seems to entirely avoid "broscience."
One feature request: list what complementary muscles each exercise targets (e.g. list "triceps" when displaying dips as a chest exercise). This would help me plan my workout when targeting multiple muscle groups, like a chest/tri or back/bi day.
We fully support GitHub Actions and have a native GitHub Action[0] that you can use to install our CLI.
[0] https://github.com/marketplace/actions/install-doppler-cli
Lots of really great questions in here! Starting at the top-
The process demoed in the video is for user-based auth. Service tokens can be issued programmatically and provide read-only access to one config. You can pass these into your environment via puppet and other configuration tools, or even via LDAP, in much the same way you bootstrap other OS config like SSH keys.
I really like your thoughts around more niche access roles, and it's definitely where we're headed. Admittedly our RBAC is currently more limited with a basic Member, Admin, Owner model. We do have existing audit capabilities for monitoring secrets changes (without revealing the actual secrets) and support shipping those logs to Slack or a webhook.
Audit history is based on your specific plan[0], but we currently offer up to a year. Some customers do have requirements for longer time periods and we can easily configure that for them.
Regarding your last point, this is what we refer to as secret referencing. You can absolutely reference secrets across different configs and projects to avoid repeating common values. Here's our announcement from when we shipped this feature back in August[1].
The local fallback file is updated each time new secrets are fetched. It is possible for your Doppler secrets to change between your last successful fetch and your next unsuccessful fetch, and in that scenario you wouldn't be operating off the latest secrets. We don't see the fallback file as an excuse for availability; our service needs to be up.
The fallback file is encrypted using AES-GCM with a key derived from the auth token and other metadata using PBKDF2. It is theoretically possible for an ex-employee to store the fallback file and retain a copy of the auth token and other metadata, even after the token has been revoked. In this case, they could construct the key using their privileged information. However, this attack would require active malfeasance by an authorized party during the period in which they're still authorized. It would be easier for the bad actor to store the raw secrets (by logging process.env, for example) than storing the encrypted file. To really solve this issue, the secrets would need to be time-bound and dynamic. Dynamic secrets are on our roadmap, but the issue you point out is a really hard problem. (And if you'd like to help, we're hiring![0])
We fully support using Doppler on bare metal servers, VMs, and just about any other cloud service via our CLI [0]. This would be identical to how you'd access your secrets locally when developing. You'd wrap your command with our CLI so that your secrets are injected into the process's environment (e.g. `npm start` becomes `doppler run -- npm start`).
Client certificates are an interesting use case that we'd be open to, but admittedly the request hasn't come up a whole bunch yet from our customers. Same with Kerberos tokens- this would definitely be used in the Enterprise space, which is an offering we're still in the early stages of. Other forms of identity-based auth would fall into this category as well.
Thanks for pointing this out, you're absolutely right. I'm linking to our Security page[0] and our security docs[1] below, but we'll definitely be updating our marketing site to place more of an emphasis on this. I hope the rest of this post and our comments here illustrate that trust and security are things we think a lot about, despite our oversight with the landing page.
Thanks for that feedback, I completely agree. I've updated the linked page to mention this more explicitly.
Great question! We tokenize all secrets and then store the tokens in our database. The raw secrets are stored with our tokenization provider, VGS. When you fetch your secrets, either via our dashboard or CLI, we exchange the token for the raw secret value and then relay that value in our response. This ensures that our infrastructure never persists raw secret values. You can find more information about this process in our Security docs [0].
[0] https://docs.doppler.com/docs/security-fact-sheet#data-flow-...
Server access is an interesting scenario to explore. If we're considering an attacker gaining server access, what's to stop that attacker from shipping a modified EnvKey binary that steals your customers' secrets and their encryption keys? If the security of your binary is predicated on GitHub repo access, what happens in the event of GitHub account takeover? At some point, no system is infallible, and I think our Threat Mode adequately addresses this. I appreciate your point of view on this though.
Hey Dane- totally fair points. Agreed that Doppler and EnvKey have different threat models. Regarding your point about malicious JavaScript- I'd encourage you to take a look at our Content Security Policy. We've gone to great lengths to explicitly disallow all unsafe-inline JavaScript (technically in report mode, but will be moved to block mode within the next few days). This is just one of many things we do to help keep our users secure, but I figured it was worth addressing directly. We've also undergone extensive internal and external audits and pentests to ensure we're not susceptible to XSS and CSRF (and employ development practices to prevent us from introducing these vulns).
I'd also argue that EnvKey might be a bit too absolutist about security, in that we think the user experience greatly suffers as a result. We have a different tradeoff that emphasizes secure defaults and best practices while also allowing for necessary features like audit logs and syncing with different infra providers. We spent quite a bit of time considering the tradeoffs of zero-trust, but our user experience would suffer as a result, and so we have taken a different approach.
Absolutely, we use Doppler ourselves in this way with puppeteer for our E2E tests. You would simply wrap your orchestrator (like puppeteer) with our Doppler CLI [0] to make the secrets available via environment variables. For example, `npm start` would now be `doppler run -- npm start`.
Thank you! Full props to our design guru and CEO @bvallelunga for that one.
You can indeed! You can manage all of your secrets from the Doppler CLI [0]. Specifically, you'd want the `doppler secrets update` command.
Great question! We address this in detail on our Security page [0], but I'm happy to give a high-level overview here:
1. Don't go down! We run two independent compute clusters on different managed infrastructure products (GKE and GAE) and route between them at the DNS layer to help avoid downtime
2. We store local encrypted fallback files on your infra via our CLI [1]. These local fallback files are fully managed and allow the CLI to continue to serve your secrets, even if our API or your internet connection is down. I use this heavily whenever I'm on a flight
3. More coming soon! (really)
Your point about not being an early adopter is well taken. We fully expect a segment of the market will be disinterested in using a relatively new product like ours. If we were a customer considering a new secrets manager, we would likely weigh similar concerns. I think the onus is on us to prove to these larger customers over time that we can be trusted (from a security/availability/longevity standpoint) and we certainly intend to do so.
Great point about dynamic secrets. This is an area we currently don't address, but it is definitely on our roadmap. There is a segment of the market for which dynamic secrets are an absolute requirement and we fully acknowledge that.
We were waiting to see how long it took someone to notice that :) That key has already been rolled and instantly redeployed via our Heroku integration, but nice catch!