bartwr

I have most likely outdated info, I left Google Research 4y ago. Back then, available TPU instances were plenty and GPU scarce. Nobody wanted to mess with an immature crashing compiler and very steep performance cliffs (performance was excellent only if you stayed within the guardrails, and being outside was supported and not even resulting in a warning - as it was so common in code). But I believe most of it has changed for the better for TPUs.

My experience cycling regularly in NYC: bike lanes separated by curb, stoppers, or poles are more dangerous as cars stop at their entrances/exits and I am literally trapped or cannot enter them before/after an intersection. I'm not against them in principle, but without extremely strict enforcent of laws (let's say a ticket 5% of someone's annual income and a loss of DL on a repeated offense - this stuff endangers people's lives), they are sadly counterproductive. :(

I love using AI tools and they are changing my work and life in amazing ways. I cannot imagine going back. And yet, I am more concerned about the social damage due to their widespread use and the amounts of slop they generate. Just this week: - There was an article about a news company faking polls by asking LLMs for answers. - My wife told me that she stopped watching any funny pet videos because 99% now is AI slop - start normal, but then turn into someone's slop idea. - A friend told me their big tech company uses AI-generated metrics as part of performance evaluation. Nobody checks them. - Another friend told me their big tech company requires engineers to use AI-generated commit messages with terrible signal-to-noise ratio and making version control and history useless for engineers. But directors and PMs love them, they are so descriptive! - My neighbor uses LLMs to create some neighbor meeting plans/agendas, plausibly looking PDFs citing contractors etc. It's impossible to read through it, mixed hallucinations and real information, all wrapped in thousands of slop words. What is real and what made up? I'll spend 10x more time double guessing. - Encountering more and more articles and general "content" that is AI generated and looks ok at the first glance, but slop upon inspections. Why would I read LLMs output on a webpage with ads, if I can ask it myself and get better, personal answers and style?

And I am not even talking here about other ethical issues, training data, less junior job positions, job replacement of journalists with LLM-equipeed contractors, etc.

LLMs make my personal and work life so much better, but social life unbearable. Is it worth the trade-off? I guess it doesn't matter at this point.

It's hilarious that people describe their anecdotal experience of being in a calorie deficit as a proof that "intermittent fasting works". This was never the question, but whether intermittent fasting brings additional weight loss benefits as compared to calorie deficit with frequent meals.

My anecdotal experience from 20y of bodybuilding and doing ~3 cuts a year: for cutting, I tried IF, 6 meals a day, low fat, low carb, high fat true keto, balanced... everything works. And works equally well - this is backed by numerous studies. The only difference is the impact on health parameters (different will get worse on low fat vs high fat), satiety, and how easy it is for someone to sustain the diet and stay in a deficit. This will depend on the lifestyle and personal preferences. So my preferred way to cut is high protein, low carb, essential fats, a ton of fiber. When building muscle I go high everything but balanced.

Anything else and more is sectarianism and people bragging about their choices not having verified their true claimed efficacy or benefits.

I share the sentiment. I haven’t used it in a while (at work use different languages and in the last few years my personal coding is only Python script/Jupyter notebook bite-sized), but anytime I hop into it, it immediately "clicks" and gives a comfortable feeling, despite changing over years. A perfect language for medium sized relatively clean and mature personal or small team projects.

Frictionless, pleasant, not thinking too much how to express things (and still managing to write them reasonably idiomatic), tends to support clean encapsulated code, quite rich environment/libraries, great tools (debuggers, profilers), safe, relatively fast, not many foot guns, zero build setup, on small project zero build times, trivial to create good functional simple UI, can get fancy dynamic with reflection when I need "magic".

Basically not many pain points that would make me rage quit and almost everything I'd want is simple to achieve.

Original author here and it's been a while since I have read such word salad nonsense, sorry. Why people who have no idea or expertise comment on articles? GenerateMips API constructs a mip chain by using box/bilinear (equivalent for factors of two) log N times.

Trilinear interpolates across three dimensions, such as 3D textures or mip chains. It is not a method for downsampling, but a method for filtering that interpolates two bilinear results, such as two bilinear filters of mip levels that were generated with "some" downsampling filter (which can be anything from box to Lanczos).

Anisotropic is a hybrid between trilinear across a smaller interpolation axis under perspective projection of a 3D asset and multiple taps along the longer axis. (More expensive)

It wasn't an exaggeration! :) The shock of many of my colleagues (often not even junior... sometimes professors who decided to join the industry) "wait, I need to talk to product teams and ask them about their needs, requirements, trade-offs, and performance budgets and cannot just show them my 'amazing' new toy experiment I wrote a paper about that costs 1000x their whole budget and works 50% of time, and they won't jump to putting it into production?" was real. :) They don't want to think about products and talk to product teams (but get evaluated based on research that gets into products and makes a difference there), just do Ivory tower own research.

One of many reasons why Google invented Transformers and many components of GPT pre-trainint, but ChatGPT caught them "by surprise" many years later.

You don't understand the Silicon Valley grind mindset :) I personally agree with you - I am happy working on interesting stuff, getting a good salary, and don't need a promo. Most times I switched jobs it was a temporary lowering of my total comp and often the level. But most Googlers are obsessed with levels/promotion, talk about it, and the frustration is real. They are hyper ambitious and see level as their validation.

And if you join as a PhD fresh grad (RS or SWE), L4 salary is ok, but not amazing compared to costs of living there. From L6 on it starts to be really really good.

If someone believed they will earn 2-5x better than in academia, with full freedom to work on whatever interests them, and no need to deliver value to the employer... Well, let's say "ok", we have all been young and naive, but if their advisors have not adjusted their expectations, they are at fault, maybe even fraudulent.

Even being in elite research groups at the most prestigious companies you are evaluated on product and company Impact, which has nothing to do with how groundbreaking your research is, how many awards it gets, or how many cite it. I had colleagues at Google Research bitter that I was getting promoted (doing research addressing product needs - and later publishing it, "systems" papers that are frowned upon by "true" researchers), while with their highly cited theoretical papers they would get a "meet expectations" type of perf eval and never a promotion.

One unique and new feature of Slang that sets it apart from existing shading languages is support for differentiation and gradient computation/propagation - while still cross-compiling generated forward and backward passes to other, platform-specific shading languages. Before, the only way to backpropagate through shader code (such as material BRDF or lighting computation) was to either manually differentiate every function and chain them together, or rewrite it in another language or framework - such as PyTorch or a specialized language/framework like as Dr.Jit, and keeping both versions in sync after any changes. Game developers typically don't use those, programming models are different (SIMT kernels vs array computations), it's a maintenance headache, and it was a significant blocker for a wider adoption of data-driven techniques and ML in existing renderers or game engines.

It does! Both platform-specific compute shaders as well as cross-compilation to CUDA. The authors even provide some basic PyTorch bindings to help use existing shader code for gradient computation and backpropagation in ML and differentiable programming of graphics-adjacent tasks: https://github.com/shader-slang/slang-torch (Disclaimer: this is the work of my colleagues, and I helped test-drive differentiable Slang and wrote one of the example applications/use-cases)

It's interestingly disingenuous that many claim of GLP-1 agonist miraculous effects on all kinds of health problems, where the same problems are "simply" solved by getting on a calorie deficit and lean. Liver, kidneys, heart, etc. If you have a non-alcoholic fatty liver disease and are obese, getting leaner will heal it. All those impressive results are on obese or diabetic people. So it is not only not a surprise, but also dishonest marketing or ignorance.

Don't get me wrong - those are miraculous drugs. First real non-stimulant low side effect appetite suppresion that will help millions. But let's wait for honest research on lean people before spreading marketing on how it improves overall health.

Also, how nobody mentions the need for increasing the dosage and tolerance build-up (just check reddits how much people end up having to take after months of continuous use). You cannot be on it "for life".

Yes, I am an (amateur) photographer for the last 27 years, from film, DSLRs, mirror less, mobile. And I worked on camera ISPs - both hardware modules, saving RAW files on mobile for Google Pixel, as well as software processing of RAWs.

But I guess you know better than me. ¯\_(ツ)_/¯

It's not. a) compression can be lossless. b) RAW is not about storing literal photons ADC measurements. It always has "some" processing as those always go through an ISP. We can obviously discuss which processing is the cutoff point and it will differ for different applications, but typically this would include things like clipping, sharpening, or denoising. And even some pro DSLRs would remove row noise or artifacts in supposedly "RAW" files!

If you can change the exposure or WB - it is what is the minimum practical/useful definition of a RAW.

It's the other way around - in hearing, phase is almost irrelevant. At medium frequencies, moving head by a few centimeters changes phase wand phase relationships of all frequencies - and we don't perceive it at all! Most audio synthesis methods work on variants of spectrograms and phase is approximated only later (mattering mostly for transients and rapid frequency content changes).

In images, scrambling phase yields a completely different image. A single edge will have the same spectral content as pink/brown~ish noise, but they look completely unlike one another.

Seems you have not worked with ML workloads, but base your comment on "internet wisdom", or worse, business analysts (I am sorry if that's inaccurate).

On GPUs, ML "just works" (inference and training) and are always order of magnitude faster than whatever CPU you have. TPUs work very well for some model architectures (old ones that they were optimized and designed for) and on some novel others can be actually slower than a CPU (because of gathers and similar) - this was my experience working on ML stuff as an ML Researcher at Google till 2022, maybe it got better but I doubt. Older TPUs were ok only for inference of those specific models and useless for training. And anything new I tried (fundamental part of research...) - the compiler would sonetimes just break with an internal error, most of the time just produce terrible and slow code, and bugs filed against it would stay open for years.

GPU is so much more than a matrix multiplier - it's a fully general, programmable processor. With excellent compilers, but most importantly - low level access that you don't need to rely on proprietary compiler engineers (like TPU ones) and anyone can develop something like Flash Attention. And as a side note: while a Transformer might be mostly matrix multiplication, many other models are not.

Yes, my experience with academics is that there are a lot of very dishonest people. They are political bullies who also lie in their research.

Chances of being caught are close to zero (I have contacted many times authors of papers who's work I was unable to replicate - most of the time zero reply, sometimes "yeah it was a honest mistake, oops"), super high competition (only a few tenured positions in all world's high visibility institutions per year), full control over student's future and being able to force them to do fraud (and later blame on them).

Obviously, not all, blah blah - but many academic scientists are the last people that should be doing science.

I lived in SF for ~1.5y, and it's not NYC, and I did not like it too much, but it certainly has some of the city conveniences and is not a car-hell suburb. (I lived in a building with ~10 units in Castro, which was cool)

But my question remains - how do you scale up your approach to the already-full SF? How do you make it more affordable, as prices are insane due to demand >> supply? Or do you just envision a more sprawled, but similarly dense SF as the solution?

I know this is an unpopular opinion in the US, but the tenements can be pretty great.

I grew up in Eastern Europe (Warsaw) in "commie" blocks and there was a lot of valid criticisms and problems (like poor quality of buildings, small apartments, or thin walls - but consider that they rebuilt whole Warsaw after it was completely grounded in WW2 in a decade or two!), but also a lot to love. Extremely walkable, safe, all amenities (cinemas, stores, cultural centers, playgrounds) in the walking distance, lots of trees and green, easy access to public transit. As a kid or teenager they were great. I preferred it 100x over suburbs where my parents moved later, and to typical American cityscapes. (And this is why I moved to NYC and love it)

Here is a fun and a bit provocative/exaggerated video https://youtu.be/1eIxUuuJX7Y

Everyone is different so I'm not forcing my perspective onto anyone, just worth considering - especially if you have not had such first hand experience (and the main objection to tenements comes from how depressing they look or American association of "projects = crime", which misses a lot of "why"). Feel free to disagree!

And apart from that, I don't think "more of small houses" solves anything. It has to create more car dependence and social isolation. And it does not really scale, where would you fit more of smaller homes in SF?

I enjoyed the post and appreciate the author sharing their perspective. It's one of many valuable datapoints for anyone considering such a transition.

I agree with a lot, but like others - disagree with some.

My background: I've worked in tech for ~14 in all kinds of roles - from pure junior IC, through a "team lead" (something between an expert IC, a tech lead, and a manager), "tech lead," company's "technical architect" (highest level tech lead, peer to the technical director, but without any direct reports), and something akin to a tech director. Now I'm back to IC. Companies from small gamedev ones (80 people total, 15 engineers), medium gamedev (30 engineers and coding technical artists), huge gamedev (Ubisoft where you can have 100+ engineers and 1500 people total on a project), and for the last 7y "big tech".

The idea I would like to push back the most is that "your words have more weight". I have never had trouble getting my opinions heard, even if I didn't push them. Being the expert IC and "problem solver" sometimes I'd have even CEO asking me directly for advice and how to solve some issues (both technical and non-technical!). Not always following them, but I didn't expect that. Having an official title, in theory, you can use some "authority" to formally push those ideas. But... in practice, it does not work better. People will still go directly to the most technical experts. And if you abuse the position/authority/title (I hope I never did that, but that's not for me to judge...), it can cause resentment, pushback, and more disagreements. You will also hear less of gossip and honest feedback, to some engineers you becomes "not one of us anymore".

It can also destroy friendships. I had a great friend (meeting socially with our wives once a week, sharing interests) who was my peer and was then promoted to my lead. I still really liked them and wanted to stay friends. We always had technical disagreements (which were fine for peer ICs). Later, some of those technical disagreements and my bringing up issues publicly caused him to get bitter with me (as the upper management saw those and took my side on some occasions), and eventually stopped the friendship completely; after I left the company, they started ghosting me. :(

Similarly, on a few occasions, I agreed to lead/manage formally (in one case, it came from me - in other cases, I was asked to). I agreed because I thought, "Things are f-d up; I can solve them by being closer to the upper leadership and helping the team succeed." Man, I was so naive. :( I didn't have any more authority or power with the higher-ups, and there were more disagreements. They expected me to enforce policies I disagreed with. As you can imagine, this didn't last long, and I always ended up leaving the team/company and being super burnt out.

So now I'm happy to be a staff-level IC, an expert, and a "hacker," playing with problems hands-on and building my expertise further. The field grows so quickly that there is always something exciting and new to learn and do. I would happily be a tech lead of some project close to me (luckily, at Google and similar, it's flexible, per-project, and not formal), but I probably do not want to manage again. Maybe it will change, depends.

I'm a former game dev and I used ImGui a lot and don't think it's used because those reasons.

It's used for quickly hacked debug tools to interleave UI and regular logic and not do a logic/view separation (as it would result in code bloat and a necessity for a refactor). You want UI code do some logic (like modifying properties of some game entity or renderer) and prefer to inline it. Lots of game code is effectively YOLO without even a single test. It's also typically guarded by IfDefs and compiled out of release versions.

But as soon as it stops being just hacky debuggers and people try to write proper tools in it, it becomes much more of a pain - people try to (poorly) emulate a retained mode in it, hold state, cache - and it becomes unreadable mess.

Wait, what? This is literally not the point of research. The point of research is to form some theory and gather data that is relevant, and then analyze statistically. Plus propose follow up theories and studies.

This is how almost every research publication in every field - from medicine, through psychology, to computer science looks like.

I'm always for people over financial entities and capital holders, however in this case it's pretty simple - you sell control to someone for a ton of money, so why on Earth would you expect to retain it forever? You literally got millions in exchange for having less shares and it not being "your" company.

Maby startups don't even need to take on seed investor money (could be self funded, smaller team, longer time before funding etc) - but founders obviously prefer to have a nice salary and soft landing. Or sometimes literally just as a marketing strategy "we are backed by x/y/z, this proves our value". This is the price to pay. What am I missing?

I shipped a dozen products with them (mostly video games), so there's nothing "wrong" that would make them unusable. But programming them and setting up the graphics pipe (and all the passes, structured buffers, compiling, binding, weird errors, and synchronization) is a huge PITA as compared to the convenience of CUDA. Compilers are way less mature, especially on some platforms cough. Some GPU capabilities are not exposed. No real composability or libraries. No proper debugging.

If they create a better tool chain, ecosystem, and programming experience than CUDA and compatible with all computational platforms at their peak performance - awesome! Everyone wins!

Until then, it's a bit funny claim, especially considering what a failure OpenCL was (programmer's experience and fading support). Or trying to do GPGPU with compute shaders in DX/GL/Vulkan. Are they really "motivated"? Because they had so many years and the results are miserable... And I don't think they invested even a fraction of what got invested into CUDA. Put your money where your mouth is.

How do I know they didn't use them? They already did something with my biological samples (storing for a different purpose than when they drew my blood) without my consent nor informing me.

And also - could eg. police use it?

Something super creepy that happened to me recently: a hospital where I've been to a few months ago called me and asked me to participate in some DNA analysis program. They said "oh and the best part? You don't need to do anything! We will use blood samples we collected the last time." I obviously declined, but it was a huge wtf to me - they stored biological samples associated with me without informing me and can do a post hoc DNA analysis. This is just insane and a proof of how non existent any privacy laws in the US are. (In EU they cannot freeze any samples without consent and unfrozen ones are ok for at most a few days)

As an engineer, you generally don't know which one applies. Even qualified lawyers don't decide on the actual criminality (a jury or judge after a valid trial does).

So I assume that "reporting wrongdoing" encompasses reporting up your management chain anything that "feels off," from mildly misleading marketing to actual witnessed or experienced harassment. Probably, in the vast majority of the cases, you are not qualified to judge it.

And in any case, even if you are wrong and the reported thing is "ok", there should never be any retaliation for just raising concerns and speaking up.

Side note and something hilarious, creepy, but kind of making sense: at one workplace (not SV big tech, but a big company), I had a manager-only training about "illegal harassment" that explicitly taught us what kinds of harassment are illegal (against protected categories) vs. "just" breaking the company's code of conduct, with quizzes on hypothetical situations and asking us "is this bad behavior potentially illegal harassment?".

"non-profit nature of OpenAI would not allow for it."

How so? Seeking more efficiency and cost-effectiveness absolutely does not conflict with a non-profit mission.

"Conflict of interest" is not defined by a bad outcome or malice. It is defined as the potential of those, human nature, and various cognitive biases. "Conflict of interest" is disclosing anything that could lead to a biased decision or lack of transparency. Can a CEO of 2 companies be objective about a contract between the two and when claiming that his company no2 is better than the competition?

And in the tech-specific case: As much as junior engineers would love to believe in "superior" solutions, tech decisions are seldom clear-cut. There are many trade-offs: cost, efficiency, memory use, throughput, latency, ease of use, cost of switching, and many more. You always have a pile of pros and cons. Sometimes, one is strong enough, but most of the time, it feels almost like guessing/intuition. And then the conflict of interest becomes especially concerning.