fwilliams

Personal website: https://fwilliams.info

I also own https://stonks.money and am looking for good ideas for what to do with it

Ha! I had a side project during my PhD to build a segmenting and straightening program for this project!

Source code is here: https://github.com/fwilliams/unwind

Paper is here: https://arxiv.org/abs/1904.04890

It got accepted to Chi 2020 which was cancelled so the paper never got presented, sadly!

I have personally gotten a lot of mileage from just writing the compute heavy parts of my code in C++ and exposing it to Python with a tool like PyBind11 [1] or NumpyEigen [2]. I find tools like numba and cython to be more trouble than they're worth.

[1] https://github.com/pybind/pybind11 [2] https://github.com/fwilliams/numpyeigen

It’s crazy seeing this on the front page of HN!

I grew up in Stanstead. I have fond memories of story time as a child in the library, borrowing movies and comic books, and playing age of empires 2 with my best friend on the two shared computers in the front room.

There’s also a street in the town (aptly named Canusa st.) which is half in the US and half in Canada. Interestingly, the houses on one side have flags reminding you where you are, while there are no flags on the other. Figuring out which side is left as an exersize to the reader ;)

Tried this on chrome and it's pretty awesome!

Unfortunately it doesn't work on Firefox on Ubuntu 20.04 with an NVIDIA GTX 3090Ti. :(

Then who is "they" in this situation? You need a citation!

If you look at the plagiarized language in the article, it seems as if the BM paper authors are claiming contributions (emphasis mine). Credit is a major currency in research, and it's important to give it where it is due. If someone did this with one of my papers, I'd be quite upset.

For example (Emphasis mine):

The risks of data memorization, for example, the ability to extract sensitive data such as valid phone numbers and IRC usernames, are highlighted by Carlini et al. [41]. While their paper identifies 604 samples that GPT-2 emitted from its training set, we show that over 1 of the data most models emit is memorized training data. In computer vision, memorization of training data has been studied from various angles for both discriminative and generative models Deduplicating training data does not hurt perplexity: models trained on deduplicated datasets have no worse perplexity compared to baseline models trained on the original datasets. In some cases, deduplication reduces perplexity by up to 10%. Further, because recent LMs are typically limited to training for just a few epochs

To quote the article:

But even putting aside the fact that claiming someone else's writing as one's own is wrong, the value in survey papers is in how they re-frame the field. A survey paper that just copies directly from the prior paper hasn't contributed anything new to the field that couldn't be obtained from a list of references.

Good survey papers can be important contributions in their own right (e.g. [1]). A good survey should contextualize works within a subject area with respect to each other and identify high level trends/ideas in that subject. These connections are not only useful for learning a topic, but also for positioning novel work or identifying under-researched areas to focus on.

If the authors felt that one of the papers they plagiarized concisely expressed what they wanted to say, they could simply quote and cite that work. Otherwise, it could be construed that the authors are claiming to be the ones drawing the conclusions they wrote. Moreover, from the article, the survey in question seems to be pretty egregiously plagiarizing, which deserves to be called out/shamed.

[1] https://arxiv.org/abs/2111.11426

I want to echo the other comments here that low expense ratio (<0.25%) funds from Vanguard, Fidelity, Schwab, etc... are all great stable investments.

My time horizon is longer than 5 years, and I buy broad market index funds split up as follows: 55% US large cap (e.g. VIIIX, VTSAX, SWTSX), 15% US mid cap (e.g. VMCPX), 10% US small cap (e.g. VSCPX), and 20% international (e.g. VTSNX, SWISX, VXUS).

I also highly recommend dollar cost averaging. i.e. buying a fixed amount of your portfolio at fixed periods. I have my bank do this automatically every 2 weeks. The benefit of dollar cost averaging is (1) it takes the emotion out of investing, and (2) over a long time window, more of your assets will be purchased at a low prices than high prices (because you're buying a fixed dollar amount of assets every N days, fewer you will buy fewer assets when prices are high and more assets when prices are low).

I was a software engineer briefly before starting grad school. During that time, I found I didn't have the time to sit down and learn about topics that interested me. I also wanted to be in research-y roles where I could build things that were more experimental and less well understood.

During my PhD, I got to spend time learning, and attending talks/seminars/conferences. Gaining deeper background knowledge in my field as well as learning how to quickly evaluate and explore new ideas gave me the tools to have the type of job I wanted. I'm a research scientist at an industrial lab now and quite enjoy it.

That being said, I agree with the grandparent post that doing a PhD can be a grueling experience. I had to carry the bulk of the work for many of the papers I submitted. If I took a day off, nobody would pick up the slack. Tight deadlines meant the only way to succeed was putting in long hours. My advisors were also spread very thin so it was difficult to get a lot of time with them. There were times when I felt very alone. This was a really stark contrast to how collaborative engineering in industry was and I don't think I ever fully adjusted to it. My current job feels like a happy middle ground. I publish papers alongside other people and we split the work.

It’s not the site you’re looking for, but I found https://poolside.fm recently and it’s become one of those quirky corners of the internet that I have come to enjoy. I definitely miss the days of discovering weird specialty sites, and poolside gave me a bit of that new site discovery rush (also the music is great).

No mention of conda for environment management? In my experience, conda is by far the best tool for this. Especially when using packages which have non python dependencies.

It's worth noting that the primary result of this paper has only to do with the error on the training data under empirical risk minimization. Zero training error =/= a model that generalizes. For any optimization problem, you can always add enough parameters to achieve zero error on a problem over a finite training set (imagine introducing enough variables to fully memorize the map from inputs to labels).

The major contribution of the work is showing that ResNet needs a number of parameters which is polynomial in the dataset size to converge to a global optimum in contrast to traditional neural nets which require an exponential number of parameters.

Link (from the article) to the paper with details: https://www.ilyaraz.org/static/papers/spectral_gap.pdf

Okay so I went and read the paper. They discuss generative modeling in section 5 and in the appendix (section 7.2).

Section 5 claims "the corresponding CoordConv GAN model generates objects that better cover the 2D Cartesian space while using 7% of the parameters of the conv GAN". There isn't really quantitative analysis beyond a couple of small graphs discussing this any further. Section 7.2 and 7.3 visually compares the results between the generator's output of interpolated noise vectors in the latent space. The results look good but without quantitative analysis, they are very preliminary.

Generative modeling is tricky and I think in your first comment, the jump from a few nice images to CoordConv can "significantly improve the quality of the representations" is a big one given the sparsity of evidence in the paper. I'm not saying that you're wrong but your original comment seemed a bit misleading to me.

I haven't read the paper so I can't comment on the success of the method, but most applied ML research will show their best results in the publication and leave out failure cases.

These images look impressive, but without doing a proper in-depth analysis, more general claims of improvement on a task are hard to make And while it's totally possible that, in this case, the improvements are significant, it's dangerous to extrapolate from just a few examples in a paper.

Somewhat tangentially, some recent work showed that a lot of problems with images (e.g. denoising, upsampling, inpainting, etc...) can be solved very efficiently with no training set at all: https://dmitryulyanov.github.io/deep_image_prior

This work shows that the network architecture is a strong enough prior to effectively learn this set of tasks from a single image. Note that there is no pretraining here whatsoever.

More to your point, I think a big problem with toy tasks are not so much the tasks but the datasets. A lot of datasets (particularly in my field of geometry processing) have a tremendous amount of bias towards certain features.

A lot of papers will show their results trained and evaluated on some toy dataset. Maybe their claim is that using such-and-such a feature as input improves test performance on such-and-such problem and dataset.

The problem with these papers often comes when you try to generalize to data that is similar but not from the toy dataset. A lot of applied ML papers fail to even moderately generalize, and the authors almost never test or report this failure. As a result, I think we can spend a lot of time designing over-fitted solutions to certain problems and datasets.

On the flipside, there are plenty of good papers which do careful analysis of their methods' ability to generalize and solve a problem, but when digging through the literature its important to be judicious. I've wasted time testing methods that turn out to work very poorly.

You could build a peer-tube compatible YouTube-like website. With control over the server, you could collect user data and probably stream ads over peer-tube. So there is possibly a profit incentive to hosting content.

I took the author of this book's course during my undergrad and quite enjoyed it. It's a good theoretical introduction to machine learning principles. The video lectures are available here: https://youtu.be/b5NlRg8SjZg

As others have mentioned this is a fairly theoretical take on machine learning which may not be useful if you just want to use a deep learning library. That said, I think there is a lot of value in having a deeper theoretical grasp of a topic even when practicing.

Most researchers in my field make all their publications freely available online after they have been submitted.

In machine learning and computer vision, the default is to put your work on ArXiv immediately upon completion. There is a lot of openly available research depending on the field. I find most fields in computer science are good for this.

In the case of the Mario example, I very much doubt that the author was not able to find other work because of closed journals.

Research can seem non-transparent to non-researchers because when problems are new, they are often poorly understood. Academic papers discuss novel problems and contextualize them based on other cutting edge work. Reading and understanding these papers requires a lot of context and takes time. Research is a skill that takes years to learn.

After some time has passed and we collectively gain a better understanding of a problem, academic papers may seem abstruse and overly complicated, but when these works were first published, this was the best way to understand them. For somebody looking for a recipe solution to a problem, an academic paper is likely not an the ideal place to look, which is why we write books, blog posts, etc... as we come to better understand a problem and its solutions.

This is hardly a discussion of previous work. The author of the video states the paper he implemented.

In any semi-decent peer reviewed venue, you would cite a wide variety of papers that solve a similar or related problem or introduce a concept related to the method in the paper.

The related work section of a paper is one of the most important parts since it puts the research into context. By citing other work, the authors explain what has already been done, and what contribution their work makes.

A related work section should also illustrate the downsides, limitations, and differences of other cited research. Limitations of other works are often poorly understood since very few people have had the time to evaluate them beyond the initial experiments done before first publication.

Research is not simply about presenting new techniques but also understanding the trade-offs that arise when choosing different solutions to a problem.

Caustic reflections are very high frequency in space and for non-trivial geometry likely won't work well in real time (unless you fake it somehow).

That said, just rendering mirror surfaces or refractions doesn't require more than an extra ray per interaction (one in the reflection direction and one in the refraction direction). You can add a texture and get bumpy reflectons or refractions as well.

The difference between these easy effects and the high spatial frequency caustic effects is indirect versus direct lighting. It's very easy to ray trace a a simple transparent model but a bright spot on the floor (e.g. [0]) below a reflective objects requires a lot of ray samples to capture directly.

[0] http://news.povray.org/povray.binaries.images/attachment/%3C...

Having first class ray tracing primitives on the GPU gives us an easy way to implement a wide variety of effects, even with a relatively small number of rays. While we are certainly far from getting real time performance in generating images comparable to those made by offline rendering techniques, being able to trace rays into the scene efficiently does allow for new degrees of realism in real time renderers.

For example, diffuse indirect radiance tends to have be low frequency in space, so we can construct good approximations of it using only a few samples (see for example, voxel cone tracing). Tracing rays is an easy way to sample the geometry of the scene and construct a diffuse GI approximation.

Doing proper reflections and refractions are also not really possible with rasterization pipelines. Game engines do a lot of hacks to approximate reflections and don't really do proper refractions at all. With a ray tracing primitive, these effects become possible. For a good example of what a modern rasterization engine does see http://www.adriancourreges.com/blog/2016/09/09/doom-2016-gra....

Rasterization is very good at generating the visible geometric information in the scene which serves a purpose, but for general purpose lighting we need to be able to sample what's going on in non-visible parts of the scene.

I think having GPU support for ray tracing is exiting as a complement to existing rasterization techniques: You can generate the geometric information and attributes of your visible objects with rasterization and use ray tracing to sample your scene and get accurate light transport effects like reflections and global illumination.

I'm not a frontend developer and don't know really know JavaScript frameworks. That said, it seems like something very well established like React would choose to have a virtual DOM for a good reason.

Surely there must be some tradeoffs that the author makes for his design to get better performance. Could somebody explain the cost/benefit of doing things one way versus the other?

You could certainly come up with something better than this. At the very least for such a comparison to be generalizable, the raw data should be a set of timed routes, app estimates, and traffic/weather conditions. Given this information, you can extract similar routes and do an apples-to-apples comparison. If you want to extract general facts, you can group routes by certain attributes that you care about (e.g. high traffic areas, time of day, etc...) and do an error analysis.

But all that is beyond the point of my original comment, which states that the "benchmark" the author uses likely does not generalize to very well for the following reasons:

* The distribution of all possible routes is large and depends on many variables that I mentioned (traffic, weather, location, etc...). The author's sample of this distribution is tiny and biased towards routes frequently taken by him and his wife. You could probably choose a different 120 routes and compute completely different results.

* Averages can be misleading if you don't know the underlying distribution you are sampling from. As a trivial example, the average value of a set of samples from a Bernoulli distribution with p=0.5 is 0.5 even though all the samples are either 0 or 1. In this case, the average is not a good tool to summarize the data (unless you know it's a Bernoulli distribution). So even if the author had used a million routes, simply taking the average error doesn't say anything meaningful about the error without also understanding the error distribution.

So I think the data and methodology presented in the article are not sufficient to draw any general conclusions about how good the error estimates are in each app.

I think the conclusion in the article is too strong given the experiments and the data. Estimating average error over a small number trips is misleading. We don't know how the error distribution varies with time, destination, departure location, traffic conditions, etc... Also the 120 trips are definitely biased to the usual routes the author and his wife take regularly. It doesn't seem reasonable to draw general conclusions from what is presented in this article.

Sympy is great (especially the ability to automatically codegen numerical expressions from symbolic ones), but I've found it to be orders of magnitude slower than Mathematica for some relatively simple problems. I also find the sympy documentation to be somewhat sparse and hard to navigate.

It's worth pointing out that particle-based fluid simulators are not physically accurate. They compute the positions of zero-volume particles in space at every frame (fluid has volume). This is convenient for performance purposes since you can evaluate each particle position mostly in parallel and the results look convincing enough for a movie or a game.

In contrast, mesh-based (and grid-based like in the article) simulators approximate the Navier-Stokes equations and converge to the exact solution as the mesh element sizes approach zero.

Mesh-based simulators solve a global system of equations in every frame. This system has variables equal in number to total mesh elements over the entire domain. Solving such a large system (especially in 3D) is very expensive. Furthermore, the size and shape of the mesh elements drastically affect the numerical stability of the solution, and generating good meshes is not a solved problem in all cases. For these reasons, we really only use mesh-based simulators when we need a high-degree of scientific accuracy.

There is literature on Quasi-Newton and Krylov Subspace methods for training Neural Networks. For example, https://dl.acm.org/citation.cfm?id=3104516.

I think the primary reason that such methods are not used much in practice is memory and computational cost: each function evaluation is expensive and you need to solve a very large system at every iteration.

Also to reply to a sibling comment, you can add momentum and step length adjustments to second-order methods in much the same way as in steepest-descent to help escape saddles. The only difference is how the descent direction is chosen for the optimization.

I think it means 1024 kB of addressable virtual memory.