HN user

jw887c

57 karma
Posts0
Comments20
View on HN
No posts found.

Avoid using Poetry for new projects. Poetry predates many standards for Python tooling. This means that it uses non-standard implementations of key features, such as the dependency resolver and configuration formats in pyproject.toml files.

What? This is the first I've heard of this.

Multiprocessing is great as a first pass parallelization but I've found that debugging it to be very hard, especially for junior employees.

It seems much easier to follow when you can push everything to horizontally scaled single processes for languages like Python.

Airflow's Problem 4 years ago

We were on AWS Managed Airflow, but to stay on it and have a solid platform, I would have been writing Github Actions for CI/CD, standing up ECR and IAM roles with Terraform, setting up EKS to run Kubernetes jobs, managing infra monitoring with Datadog, etc., etc.

This sounds like an issue not with Airflow but with integration.

This is the case for most column oriented data warehouses (including BigQuery, but Snowflake does allow for 1 PK). It's just the nature of the technology.

My favorite flaw of averages isn't even mentioned in this article. It's the aggregation of averages across covariates. The more covariates (higher dimensions) your problem has, the less likely the population will exist "in the average".

This was explored in a famous study of Air Force pilots and when measuring across 10 different dimensions, found that 0 pilots were "average" across all 10.

https://www.thestar.com/news/insight/2016/01/16/when-us-air-...

There was no such thing as an average pilot. If you’ve designed a cockpit to fit the average pilot, you’ve actually designed it to fit no one.

edit: wrong link

A $40B manager once told if I could deliver 6% annual return with 1% volatility, he would give me all his money. Yeah, they want as little volatility as possible.

GCP is behind. They're are behind in niche services to be fair but behind nonetheless. Nothing that prevents building similar services, but coming from AWS, I routinely feel frustrated by lack of finesse and comparable offerings. IMO, GCP's greatest benefits over AWS are:

* BigQuery

* Tighter k8s integration with GKE

* Single message service (PubSub)

Unless BigQuery is your first class citizen, I would avoid GCP.

If you prefer yaml > Python or if you prefer installing one k8s app instead of managing all of the airflow dependencies (scheduler, webserver, workers, etc)

Oh definitely. I have TONS of buildup after a few days. There is a (anecdotally) significant difference in hearing ability post cleaning via q-tip. If I don't use the q-tip, Airpods get very gross within a few days.

Ex-aerospace engineer here, now software developer. This is one major reason why a lot of younger folks at Boeing leave (at least the ones with a more broader view of their industry).

A lot of "engineers" at Boeing are actually project / program / product managers and can pivot to similar roles at tech firms. The stress engineers have it a lot tougher.

I view it as:

Explainable models = statistical models (regression / multilevel models).

Black box models = machine learning (tree based models).

AI = black box models for computer vision / NLP.