aeijdenberg

https://github.com/continusec/htvend/

htvend is a tool to help you capture any internet dependencies needed in order to perform a task.

It builds a manifest of internet assets needed, which you can check-in with your project.

The idea being that this serves as an upstream package lock file for any asset type, and that you can re-use this to rebuild your application if the upstream assets are removed, or if you are without internet connectivity.

Has an experimental GitHub action to integrate within your GitHub build, archiving assets to S3.

I've been thinking a lot about this kind of thing recently - and put a prototype up of htvend [1] that allows you to archive out dependencies during an image build. The idea being that if you have a mix of private/public dependencies that the upstream dependencies can be saved off locally as blobs allowing your build process to be able to be re-run in the future, even if the upstream assets become unavailable (as appears to be the case here).

[1] https://github.com/continusec/htvend

Thanks for taking a look.

The intent was to support basic build systems accessing package eco-systems that tend to always serve the same response for the same URL.

Docker registries do this reasonably well, as do Maven repos.

It wasn't intended to be a full on proper archiving proxy (and I'll admit I hadn't heard that term - I'll look into it and see what else exists in that space).

The main use-case I had in mind for this is private projects, that are developed on workstations which have internet access, but are deployed to other environments using CI/CD systems with less network access. If both systems have access to a common blob store, then that can be populated with htvend build on a workstation and replayed at build time with htvend offline.

For that, there's no need to capture additional request information, as the focus wasn't to support getting a manifest file and being able to reliably re-download all the blobs from internet (and often those responses may have changed in the interim). And for the same reason, would expect to only need to one response per URL, per assets.json file.

Does that make sense?

The TS doesn't seem to provide for a way to modify return values for the function. For example the following is a common pattern in Go using defer to ensure that errors closing a writeable file are returned:

    func foo() (retErr error) {
        f, err := os.Create("out.txt")
        if err != nil {
            return fmt.Errorf("error opening file: %w", err)
        }
        defer func() {
            err := f.Close()
            if err != nil && retErr == nil {
                retErr = fmt.Errorf("error closing file: %w", err)
            }
        }()
        _, err = f.Write([]byte("hello world!"))
        return err
    }

We use Terraform a lot too - and most of the time it's great, but not infallible.

Our team managed to screw-up some pretty major DNS due to a valid terraform plan that looked OK, but in reality then deleted a bunch of records, before failing (for some reason I can't remember) before it could create new ones.

And of course, we forgot that although we had shortened TTL on our records, the TTL on the parent records that I think get hit when no records are found were much longer, so we had a real bad afternoon. :)

Slight, but important (if you don't want to run out of fuel) nit, indicated airspeed (KIAS) is not the same as true airspeed (KTAS).

To calculate ground speed (as required for navigation and fuel planning) you need true airspeed (not indicated) as well as wind direction and speed.

See some discussion here: https://www.quora.com/In-aviation-what-is-the-difference-bet...

False only for very small values of code.

ie if your code itself is split into modules, they won't work (as they are imported by their full path, not relatively), and anything in your vendor dir is also ignored when used outside of a GOPATH entry.

You can certainly claim something is centralized and tamper-evident. ie demonstrate proof that something has not been mutated over time.

See RFC6962 Certificate Transparency logs and their consistency proofs for a widely used example.

Not quite as simple as a cryptographic hash alone - remember that if the set of possible inputs can be easily enumerated, then it's trivial to find the input data by brute force.

There are ways to work around this, for example objecthash[0] describes a small modification that prepends the input data with 32 bytes of random data before hashing in order to prevent this.

[0] https://github.com/benlaurie/objecthash#redactability

Glad to see any doc published that gets developers thinking more about security...

One "trend", or rather bad habit that I've noticed a lot in discussion with other developers recently, and this doc also falls into, is that there seems to more focus on "input sanitisation" rather than "output escaping".

Regardless of what's been done to input, if the result is that you have a string that you need to embed into another string, then you need to know how to escape that appropriately for the context in which it's being used. Whether the data is user generated, or taken from your database, always assume that it's trying to break your app, and always escape it on output.

Making a hash of the release is just a small part of it (and is the first part of what they are doing).

The trick is to be confident that you're getting the same hash as everyone else - and that's what requiring a proof that it be added to a CT logs gives you some level of assurance about.

CT significantly pre-dates the recent Symantec issues, but yes, it does provide an excellent tool for providing evidence of misissuance [0] [1] - and that's the crux of it - in order for a certificate to be considered valid in a CT world, it must present proof that it has been publicly logged.

[0] https://security.googleblog.com/2015/09/improved-digital-cer... [1] http://searchsecurity.techtarget.com/news/450411573/Certific... [2]

CT logs are designed to be able to handle queries from all web browsers on a daily / more frequent basis, and the output from queries is easily cacheable (and the logs can be mirrored in a read-only manner).

If FF is already doing any log inclusion proofs for certificates, then I think including one more (for the FF release itself) would be pretty much line noise.

I think an interesting question arises as to how well with the CT logs themselves would scale to handle the same kinds of certificates for all binaries, if this ends up taking off as a good idea in general. They've had to handle quite an explosion in X509 certificates over the past year or two due to Let's Encrypt. Some of Google's logs now show more than 80,000,000 certificates [0] in there - IIRC 2 years ago it was a low single digit million.

[0] https://crt.sh/monitored-logs

If I'm understanding correctly, the plan is to piggy-back on top of the existing Certificate Transparency [0] infrastructure by issuing a regular X509 certificate per Firefox release, but for a special domain name that includes a Merkle tree hash for the files in that release, with a known suffix (".fx-trans.net").

In that manner they can piggy-back on top of the CT ecosystem (including existing logs, including existing search / monitoring tools, and presumably gossip if/when that's solved).

This seems like a really cool hack! The state of binary software distribution is really pretty scary when you think about it - techniques like this have the potential to restore a lot of confidence.

[0] http://www.certificate-transparency.org/

As I understand it, Chrome (unlike Firefox) does not ship its own root CA store - rather it defers to the root store of the operating system that it's running on. It does however apply some form of blacklist / additional restrictions over what the OS may allow.

If you're looking to be able to consistently hash JSON objects you might want to look at Ben Laurie's objecthash: https://github.com/benlaurie/objecthash

It describes a consistent way to hash an object without defining a new format.

I'd always assumed the primary goal of LinkedIn endorsements is the same as every other email I get from LinkedIn, to keep user engagement with the site - and based on the number of colleagues I see accepting them, I'd say it's been pretty successful at achieving that goal for LinkedIn.

What benefits does this approach give above those offered by Certificate Transparency [0]? I was surprised to see no mention of this as related work in either the linked paper or site FAQ.

[0] https://www.certificate-transparency.org/

The problem I've found is that when we start using cute tricks like the old "xxx" % locals(), or the newer f-strings, is that we tend to start forgetting basics like escaping the strings that we're substituting in.

For the example above, this might be fine, but most examples that I see using this kind of thing are things like:

"select * from foo where id = '%s'" or "<input type='hidden' name='foo' value='%s'/>" or "rm -rf /path/to/customer/%s"

I tend to think that in general appending strings is more often than not quite a dangerous operation, and it behooves the caller to need to have a long think about exactly how the strings being substituted in need to be escaped - as it's pretty rare that they don't need to be.

If I recall correctly, don't some older versions of Windows broadcast (as an adhoc network) the SSID of the last hotspot they successfully connected to? [0]

If so, wouldn't it be quite plausible that an owner of a Galaxy Note 7, who had previously used tethering elsewhere (with that SSID), would then accidentally broadcast that SSID when using their Windows laptop on the flight, even though the Galaxy Note is not there?

[0] https://www.techdirt.com/blog/wireless/articles/20101011/031...

Probably no reason other than that it takes time and resources to go through all the steps necessary to launch in a region (e.g. localization, legal etc), and it's more effective to get feedback on a beta product early in one region than to delay for many.

Should such a requirement (CT for private CAs) exist, wouldn't the said laptop vendor simply ship embedded SCTs in their fake certs signed by their own fake log key, also baked into Chrome? (that doesn't even need to correspond to a log, at least in the case of static SCT validation)

When a laptop vendor is building the device that's being shipped, I don't think it's practical for a browser vendor to be able to expect to win that arms race.

So that you (or other interested parties) can verify the correct operation of the centralized authority, and so that the central authority can prove that they aren't hiding anything.

For example, in the case of certificate transparency, that the CAs aren't mis-issuing certificates, and further that the logs aren't conspiring to hide entries (such as via split views).

If you can accept centralization (which for many non-crypto-currency use-cases actually makes sense, since it mirrors the way the real world works) - you don't actually need proof of work.

You can still get the benefits of verifiability, append-only, detect misbehavior etc without it. See for example Certificate Transparency (RFC6962) for a standard that implements those properties using similar Merkle Tree constructions.

Disclaimer, I have a startup that generalizes the same Verifiable Logs (and also allows for Verifiable Maps): https://www.continusec.com/

Congrats to Adam, Steve and team! Flight search is a tough industry to break into, and they did a great job against the odds to get there.

(founder here)

I recently launched Continusec, which provides hosted Verifiable Data Structures. These are verifiable logs and verifiable maps based on the same principles as Google's Certificate Transparency logs, generalized for any use, wrapped up in a hosted cloud service. You might of them as "S3 meets blockchain".

Would appreciate any feedback from the HN community.

Would be interested in your feedback on Verifiable Maps as implemented at https://www.continusec.com/ which can give verifiable answers to specific questions such as what is the current balance as well as a full history log of all mutations to the map.

Have you considered transferring to a different team with a different project in the same company? Generally this is much easier than moving to a new company, you get to keep any accrued benefits of tenure, and in a big tech company the culture of different projects can be so vastly different that a change like that can feel pretty much like you're working at a new company.

Also - don't worry about failing interviews. Having done hundreds on the other side, and seeing many good candidates be rejected, it's clear the game is stacked against the candidates, erring on the side of caution (no hire), knowing we'd miss out on many.

And finally, if you are depressed, do seek help - others on this thread point to great resources. Use them.

I think the single best thing an API designer should do is write a getting started guide with working sample code for the API they are producing.

On my last project I found myself revising the API surface twice as a result of doing so.

If you accept that for many uses of publicly verifiable data structures that centralization is actually OK (and actually often a good thing, as it matches well how most democratic societies function), then there are other simpler alternatives to proof-of-work that still give many of the same benefits.

See for examples the append-only logs used in Certificate Transparency (https://www.certificate-transparency.org), and a more generalized hosted form provided by Continusec (https://www.continusec.com) that builds on the same principles. (disclaimer: founder of Continusec)