itwasntandy

Thank you Nick.

As a recurse alum (s14 batch 2) I loved reading this. I loved my time at recurse and learned lots. This highlight from the post really resonates:

“ Real growth happens at the boundary of what you can do and what you can almost do. Used well, LLMs can help you more quickly find or even expand your edge, but they risk creating a gap between the edge of what you can produce and what you can understand.

RC is a place for rigor. You should strive to be more rigorous, not less, when using AI-powered tools to learn, though exactly what you need to be rigorous about is likely different when using them.”

I wasn’t talking about food safety.

The lice from the farmed fish devastate the wild salmon populations, so ecologically they are awful.

That to me is a pretty clear reason to avoid farmed salmon.

Keep reading.

For example:

A recent study in British Columbia links the spread of parasitic sea lice from river salmon farms to wild pink salmon in the same river

Sure. Our numbers are smaller, as our org is much smaller but I think they hold up.

We have under 100 engineers at BuzzFeed, so for us we reduce time spent supporting applications by a single percentage is almost equivalent to having an extra engineer!

On a typical day we'll do 180+ deploys. That'll be roughly 60% stage, 40% production. To deploy outside of continuous deployments (e.g. to push a branch build to a stage environment), is as simple as browsing to our deploy UI, selecting the service you want to deploy, the version, and the cluster to deploy too. It takes an engineer about 40 seconds to do that (time measured from typing URL, to hitting the deploy button)

Continuous deployments means for production deploys for master, that step is automated. So that's a saving (in engineer time) of around 40 seconds per deploy. Over the last month over 1400 deploys were made via continuous deployment.

This means from a developer productivity perspective roughly 2 engineer days per month (1400 deploys * 40 secs per deploy / 3600 to get into hours / 8 hours per day) ) are being saved as a result of this change.

However this is really a side benefit. Our real motivation for making this change was to ensure that all master builds are deployed, and that each of our ~500 micro services is always on latest master.

This is to reduce developer anxiety of deploying over a branch build and thus minimize stress around making change.

The productivity side benefit is pretty nice though!

I am part of the team that worked on this project. AMA

This is somewhat intentional I think to guide you toward buying Enterprise edition, which since 11G includes plan baselines ( see https://docs.oracle.com/cd/B28359_01/server.111/b28274/optpl... ) - which allows you to lock a SQL queryID to a specific plan.

BuzzFeed | Principal SWE, Core Infra | London and New York | Onsite Full time | https://boards.greenhouse.io/buzzfeed/jobs/1895087?gh_jid=18...

BuzzFeed | SWE, Core Infra | San Francisco | Onsite Full time | https://boards.greenhouse.io/buzzfeed/jobs/1855419?gh_jid=18...

We’re looking for engineers who understand how good tools can shape company culture, and who are passionate about creating tools and automation to develop and improve the platform that underpins BuzzFeed.

Email me - andrew.mulholland@buzzfeed.com - with any questions, or apply directly to the job specs above.

Indeed. And considering the hundreds of millions[1] of people who pass through london’s stations each year, that would means hundreds of thousands of false positives.

[1] https://en.m.wikipedia.org/wiki/List_of_busiest_London_Under...

While I agree with the aspiration, I don’t see how it’s possible to use facial recognition to flag “america’s Most wanted” or to look for missing children without mass surveillance though?

It only works if it scans _everyone_ .

Additionally what happens when the technology isn’t perfect and innocent people get mistakenly flagged as persons of interest?

The other thing is once it’s installed and in operation, what’s to stop it being used for other purposes? - being used to target people peacefully protesting against the government or whatever.

It’s a slippery path from there into a surveillance state - China is already pretty much there with their social credit system - https://en.m.wikipedia.org/wiki/Social_Credit_System

I for one don’t want to wake up someday soon and discover it’s 1984...

Read “never let me go” by Kazuo Ishiguro, or actually even watch the movie that was made of it (same title)

That illustrates fairly well why having an underclass who provides healthy organs to the rich is a utterly barbaric idea.

That is insane that that is the case here in the US. ( and I just checked as I could not believe it).

In Europe it’s different - https://ec.europa.eu/agriculture/organic/eu-policy/eu-rules-...

“When the animals are ill, chemically synthesised allopathic veterinary medicinal products including antibiotics may be used where necessary and under strict conditions. This is only allowed when the use of phytotherapeutic, homeopathic and other products is inappropriate.”

My own experience ( North east US) has been somewhat different recently.

Yes routes from Apple maps, may appear longer or more convoluted at first glance. However after using it ( due to CarPlay) for a while on routes I had previously regularly done using google maps, I inferred a reasoning for that.

On the ~90 minute journey to my in-laws, the predicted journey time, is generally advertised as being quicker on google, but in practice the time difference is marginal.

What was different in my experience anyway, is that Apple maps seems to try to minimize left turns where appropriate. The benefit being a noticeably less stressful journey.

Has anyone else noticed this?

That's configurable - https://docs.aws.amazon.com/elasticloadbalancing/latest/appl... - default `HealthyThresholdCount` is 5 and `HealthCheckIntervalSeconds` is 30 seconds.

We adjust those down - somewhere 10-15 seconds for HealthCheckIntervalSeconds and 3 for HealthyThresholdCount works pretty well.

I miss Boundary.

Their traffic flow monitoring with per second granularity was utterly amazing. It had almost no overhead, it gave the equivalent of netflow data for instances in EC2, way before AWS gave VPC flow logs. It was incredible.

Alas their sales team were incredibly difficult to deal with, managing to both seem to not know how or what they were selling, whilst at the same time being incredibly heavy handed, escalating to my VP over a minor delay in contract signing and causing trouble to the point I had to almost beg my VP not to tear up the contract because of their actions.

As a result of what I perceive to be the sales org's failure, the company was forced to pivot into a generic monitoring product, which lost its identity and got borged by BMC.

There's nothing like boundary on the market today, if there was I'd be lining up to buy it.

Indeed. I saw a brilliant presentation in 2012 by Michael Mitzenmacher from Harvard on verifiable computing in the cloud. It was based on this paper:

https://arxiv.org/pdf/1202.1350v3.pdf

Maybe - we've not tried it, but I just asked Justin who architected sso, and he suggested that maybe https://github.com/buzzfeed/sso/blob/master/internal/proxy/o... could be used.

However there would be some caveats - like for example `skip_auth_regex` (see https://github.com/buzzfeed/sso/blob/master/docs/sso_config.... ) would not work with this, because the original URI would not be checked.

Golang is pretty beloved at BuzzFeed.

It’s one of our two standard languages - the other being Python - and whilst the vast majority of our services are Python, Golang is being used for growing and significant number too.

Touching on my first point, we have observed people enjoy writing Go apps, and it is a great fit particularly where performance and scalability are needed.

Therefore when engineers have moved to another team internally, they often will evangelize Golang to their new team members.

So we expect it to continue to grow and thrive here!

It’s not. We acknowledge that in our blog post, and our approach was definitely influenced by the BeyondCorp philosophy.

However google IAP requires that your infrastructure is all in Google cloud.

Whilst we do use GCP, most of BuzzFeed’s infra is in AWS, we needed a solution which worked for both.

We did talk in the blog article about engaging with Security Innovation too.

Right now we have a dependency on Google as an OAuth2 provider, as that's what we use internally at BuzzFeed. However we've designed sso to allow us to easily add other providers.

For example, there's this task ( https://github.com/buzzfeed/sso/issues/9 ) to create a default provider without Google dependencies for test purposes.

We'd also welcome PRs adding other providers and believe any OAuth2 provider should be straight forward to add.

thank you - We know from our own experience using other open source projects that having documentation, and guides to get started really helps.

We know there is more to do there too (some of the feedback in this HN post has helped highlight areas we need to improve the docs) and we will be adding to the docs so over the next while.

We will also welcome PRs improving the docs!

Hey,

there were a few reasons behind this.

- ease of migration was a big one, we had 100+ instances of bitly's oauth2_proxy, and were able to seamlessly migrate them to this, without any changes to the services being protected.

- ease of deployment was also important. Our solution doesn't have any datastore dependencies, and is stateless, so was straightforward to deploy into our PaaS ( https://tech.buzzfeed.com/deploy-with-haste-the-story-of-rig... ).

- when we built this, there were far fewer solutions than there are today. For example, Ory's Oathkeeper ( https://github.com/ory/oathkeeper) was released after we were already using sso internally at BuzzFeed.

thanks!

That is understood, and is always why we engaged with some of the top researchers who contribute to our bug bounty program, from the start with this project.

For example offering increased bounties during certain windows, or providing early access to the source code.

We highly value our bug bounty program, and find it to be a very effective mechanism for continuous security validation.

I'll write a tech blog post in the near future about how we facilitate our program.

I’ll correct my post above. I meant to say `mobile devices`, not `mobile browsers` . My bad.

The other reason, which I didn’t mention above, but is talked about in the blog post, is we decided to use bitly’s oauth2_proxy as a basis for our solution. This had been widely used in BuzzFeed (we had over 100 auth proxies in place prior to rolling out sso), and so the OAuth flow was something everyone was familiar with.

Great question. We found that SAML doesn’t typically have great support on mobile devices [edit: had originally written browser here, hence the comments below], and since BuzzFeed has many remote employees around the world, we needed to support those workflows, so OAuth2 made more sense.

Yes, as mentioned in the blog post, we worked with Security Innovation to do a week long security assessment with full access to source code, design documents and endpoints.

We also have a long term consulting arrangement with a widely respected security architect, and they helped review our design and implementation.

Additionally, BuzzFeed has a bug bounty program on hackerone (https://hackerone.com/buzzfeed), and have invited partipating researchers to report on any issues found. We’ve paid out bounties for a number of minor issues, which were addressed prior to open-sourcing.

Additionally, knowing that security is never done, we continue to make it eligible for bounties -- see https://github.com/buzzfeed/sso/blob/master/README.md#securi...

GitHub repo: https://github.com/buzzfeed/sso

This is our identity aware proxy, which we've been using internally for a year.

The blog post explains our motivations behind creating it, and open-sourcing it. It's available today, under MIT license.

We'll be keeping an eye on the thread, and happy to follow up to any questions!

Weird thing is it just a few months ago that AWS contributed what I would consider a pretty significant feature to Redis - encryption in transit:

https://aws.amazon.com/blogs/opensource/open-sourcing-encryp...

While it’s not merged yet, antirez at the time seemed pretty onboard with the approach. ( first Point in https://news.ycombinator.com/item?id=16943289 ). The delay in merging seems understandable based on the comments and discussion on the PR.

So leaves me wondering what’s changed. Obviously antirez != redislabs and their focus is on the OSS side of redis, but the announcement seems to suggest that none of the cloud providers contribute back which isn’t the case?

If you just put a unique subject each time - perhaps include a date or time stamp, this problem goes away.

Aetion | New York, NY | Systems Engineer (DevOps) | ONSITE | http://www.aetion.com | https://boards.greenhouse.io/aetion/jobs/189540?t=6ngjm3#.V0...

Aetion provides a leading-edge analytics platform for rapid-cycle analytics of real-world healthcare data that our customers use to generate real-time evidence on the effectiveness, safety, and value of healthcare interventions. With this essential evidence in hand, our customers can make the smart, value-driven decisions needed to transform healthcare.

Aetion practices DevOps - where both systems and software engineers are involved in the day to day management and support of Aetion's platform. The primary difference in the roles is one of focus - the systems team builds the infrastructure and tooling to support the platform, and the software team builds the Aetion application platform.

Our distributed platform is hosted both in AWS and also on customer premises. This poses some challenges and part of the role is helping with application design to ensure suitability for all environments.

We use Chef for automation to ensure that a small team of operational staff can easily handle our large and growing estate and use Docker for containerization to ensure consistency across deployment environments.

With health data in our platform, security is paramount. Whilst our platform uses de-identified data, we still target technical compliance with HIPAA regulations, as a basis for our security polices

More detail here: https://boards.greenhouse.io/aetion/jobs/189540?t=6ngjm3#.V0...