xLaszlo

6TB - Snowflake

Why?

That's the boring solution. If you don't have a use case, what kind of queries you would run then opt for maximum flexibility with the minimum setup of a managed solution.

If cost is prohibitive on the long run, you can figure out a more tailored solution based on the revealed preferences.

Fiddling with CSVs is the DWH version of the legendary "Dropbox HN commenter".

That's Kent Beck's 3X: Explore Expand Extract (there is a good video on it about his time at FB)

It's a question of "What is the economy for?"

Disruptive tech monopolies upset entire existing industries for the sake of their owners/shareholders/employees, which is usually tiny compared to the said industries.

(Think of FB/Google advertisement centralisation as the end of other media providers.)

I think feast is sunsetted

Hungary had and still has a state sponsored "elite" education for gifted children. From a relatively early age kids are regularly tested (both materially and curiosity wise) if they are interested in "better" Maths. So they are prepared to a STEM trajectory.

You don't need to be a genius just reasonably good and interested. I would think 10% of all kids take part.

Check out UKMT for the equivalent in the UK or Kangaroo competitions.

Secondary schools has a 130! year old journal (KOMAL) for competition Maths (again various difficulties) but those who go on to become mathematicians (an extremely selective subject in Hungary) pass the hardest tests as well.

So Hungary has various levels of gifted children support programs and has it for the last 100+ years (across 3-4 different government structures)

Hmm, I had to check this and came across this thread on Reddit. NIST has a standard for it, but the exact point is that _that_ can be influenced by them.

Also, somewhere in the thread, they mention that it is probably too hard to verify how it is actually implemented in the hardware.

https://www.reddit.com/r/crypto/comments/klcx4p/is_true_rand...

I guess in security only the paranoid prevails...

Or it is indeed just a cool gimmick...

Until it turns out that the NSA was involved writing said RNG and explicitly weekend it with a backdoor...

It's beer, isn't? Completely makes sense...

If they find evidence that the next step was caffeine, I have no more questions...

Neither does Tadatem ;)

There should be a balance of short term and long term thinking and project management is the optimisation between the two.

My experience is that managers only think short term which makes SWEs defensive and be dogmatic about long term.

The the project fail.

If you are looking for low discrepancy (fills the space "smoothly") quasy random numbers check out Sobol and Niederreiter sequences.

I found that the best to think of graph implementation is sparse matrices of the adjacency matrix. CSR/CSC format has fast lookup abilities, building and switching between formats is "relative" efficient. Most graph algorithms need primitives that can be built on top of this.

For distributed computing one can look int GraphLab or its smaller version, now largely abandoned GraphChi.

This is called "Goodhart's Law"

Thanks for the comments.

I don't have anything against debuggers, but I feel more natural to write into the code and this simplifies it.

TBH I haven't tried this or snoop heavily and the project was more of a curiosity of how the internals of python work after I found a relevant SO post.

Honestly I was surprised how easy it was.

If you actually want to use something like this use snoop: https://pypi.org/project/snoop/ it is more feature rich and maintained.

Ahh, I never heard about it, but it looks promising.

I will reach out to the author.

Thanks for bringing it to my attention.

WIP: suggestions most welcome.

I don't remember response speed mattered until at least ten years after Google's start.

Certainly not when they won.

They were better. Basic PageRank was better than anything else. And once they figured out advertisement, they kept making it better to seal their dominance.

He should find a SWE and partner up. As more clients enter the ML space there are more less sophisticated ones who has less of a concept of cross functional teams. Rather than educate themselves these clients have unrealistic demands.

While I am a huge advocate of DSea writing better code going fullstack is unrealistic for them. What does it even mean to integrate? Fastapi? Docker? Helm charts? Monitoring and observability? SRE? The list is endless.

This is a classic case of "the client doesn't care the want business value".

If as a DS you want to get better at writing code join our Code Quality for Data Science (CQ4DS) discord:

https://discord.com/invite/8uUZNMCad2

- Find out what you want to do

  - Think about what are your strongest applicable skills

  - Talk to as many DSes/MLEs/MLOps engineers you can

  - Experiment with various fields (watch videos, OSS work etc) 

- Find out what DS means at those places and what do they do and if that's what you want to do. Ask as many questions about the details of the job as possible at the interview, the hiring manager will be glad you want to avoid getting a job you are not committed.

Essentially: Do your homework then make a decision that is good for _you_ rather than trying to fit to an abstract idea.

This video of him from 1996 is amazing with hindsight. A must watch: https://www.youtube.com/watch?v=98LdFA-_zfA

RIP Christopher Alexander

If in 2022 you still don't know about design patterns (not using them, just knowing about them) I feel sorry for you.

This is one of those posts that exclusively caters for reassuring the audience regardless of whether they are doing right or not just to make them feel better.

Refactoring is part of a cycle and you do it before you publish your code to others. If you don't clean it up, you are wasting someone's time.

His name is almost certainly Samuel G. Szabó

Overmentioned not overrated. Estimations usually say that breaking the Engima shortened the war by a year. That's hardly "winning the war".

Organising labelling is a core part of productionised ML and the Domain Team has the most knowledge to do it. But are they the best suited to do it?

Our experience shows that because incentives differ in production and off-line labelling, the Domain Team might create biased (subjective) dataset in Human-In-The-Loop settings.

Adding an extra team that was trained by the Domain Team but can only influence end-to-end performance through the deployed model helped to keep the training dataset objective while improving the productionised pipeline and helping the Domain Team.

First time HN poster, please share your recommendations in the comments. We are working on a "Machine Learning Product Manual" ebook about best practices to productionise and operate ML from a product perspective. Follow me on twitter at https://twitter.com/xLaszlo for updates.