darsnack

Can I use PyTorch or JAX comfortably in Julia?

There is https://github.com/rejuvyesh/PyCallChainRules.jl which makes this possible. But using some of the native Julia ML libraries that others have mentioned is preferable.

How you interact with parameters.

Lux is similar to Flax (Jax) where the parameters are kept in a separate variable from the model definition, and they are passed in on the forward pass. Notably, this design choice allows Lux to accept parameters built with ComponentArrays.jl which can be especially helpful when working with libraries that expect flat vectors of parameters.

Flux lies somewhere between Jax and PyTorch. Like PyTorch, the parameters are stored as part of the model. Unlike traditional PyTorch, Flux has “functional” conventions, e.g. `g = gradient(loss, model)` vs. `loss.backward()`. Similar to Flax, the model is a tree of parameters.

You are also allowed to have bike share, buses, subway/light rail, and taxis. No one is saying trains are the only mode of transportation. I’ve lived in midwestern states with miles of farmland. In the town center, a bus network and bike paths meant I almost never needed a car. Folks who lived outside of town had the option to drive in and park their car at commuter lots.

Also this is what railway lines looked liked in the US when we cared to build out our train infrastructure: https://i.pinimg.com/originals/3c/57/ee/3c57eeffb7e1a3c78691.... Just because the US is big doesn’t mean we can’t build tracks.

You would have to add a learning method to tell it how to encode/decode graph data, but the framework is agnostic to the model choice. So any Flux model is supported.

I’m not the main dev on FastAI.jl, but I work on the Julia ML community team that supported this project.

Since FastAI.jl uses Flux, and not PyTorch, functionality has to be reimplemented.

We are looking to offer a high level API for ML in Julia similar to fastai for PyTorch. The goal is to enrich the Flux ecosystem, so just calling into Python fastai wouldn’t be appropriate. FastAI.jl is built on top of several lower level packages that can be used separately from FastAI.jl. These packages help build out the ecosystem not just for FastAI.jl, but any ML framework or workflow in Julia.

What does this mean for the development of fastai?

FastAI.jl is “unofficial” in that Jeremy and the fastai team did not develop it. But Jeremy knows about the project, and we have kept in touch with the fastai team for feedback. FastAI.jl doesn’t affect the development of Python fastai in any way.

FastAI.jl has vision support but no text support yet.

What is the timeline for FastAI.jl to achieve parity?

We’re working to add more out-of-the-box support for other learning tasks. Currently, we have tabular support on the way, but the timeline for text is not decided.

Note that the framework itself could already support a text learning method, but you’d have to implement the high level interface functions for it yourself. We just don’t have built-in defaults like vision. You can check out https://fluxml.ai/FastAI.jl/dev/docs/learning_methods.md.htm... for a bit more on what I mean.

When should I choose FastAI.jl vs fastai?

It depends on what you need. PyTorch and fastai are more mature, but Julia and Flux tend to be more flexible to non-standard problems in my experience. If you’re interested, then give Julia/Flux/FastAI.jl a try. If we’re missing a mission critical feature for you, then please let us know so we can prioritize it.

You should check out Makie. Getting it set up can be a bit frustrating if things don’t go right, and there is a small learning curve for using `@lift`, but it is an absolute joy to use once you ramp up.

I use it for my research by default. You can pan, zoom, etc. The subplot/layout system is frankly a lot better than Matlab (and I enjoyed Matlab for plotting!). The best part is that I can insert sliders and drop downs into my plot easily, which means I don’t need to waste time figuring out the best static, 2D plot for my experiment. I just dump all the data into some custom logging struct and use sliders to index into the correct 2D plot (e.g. a heat map changing over time, I just save all the matrices and use the slider to get the heat map at time t).

On the NVidia A100, the standard FP32 performance is 20 TFLOPs, but if you use the tensor cores and all the ML features available then it peaks out at 300+ TFLOPs. Not exactly your question, but a simple reference point.

Now the accelerator in the M1 is only 11 TFLOPs. So it’s definitely not trying to compete as an accelerator for training.

In a more complex example where you actually take a variable, do some operations to it, then reassign it, Pluto.jl encourages you to separate that into multiple cells. The reason is each cell marks a distinct node in the dependency graph. If you prefer to use cells, then the notebook can be smarter about what lines actually need to get re-run and what don't.

A downside to using multiple cells is vertical spacing/visual noise. This is something that the package authors are currently thinking about addressing.

Most of that is not fundamental to Julia or Flux itself. It’s the difference between a monolithic package like TF and source-to-source AD in Julia. The former allows the designers to use their own data structures and external libraries to do optimizations. Source-to-source relies on the underlying IR used by Julia, making optimizations challenging without some compiler assistance. But all of that is in the pipeline with stopgap solutions on the way.

As with most things in Julia, the code developers don’t just want to hack changes that work, but make changes that are flexible, extensible, and can solve many problems at once. So, Flux isn’t ready for prime time yet, but it is definitely worth keeping your eye on it.

I have been using Flux for a year (or more?) and I have never found it to be slower than PyTorch or TF. Granted I am training at most ResNet-20 and mostly smaller models, so maybe there is larger training routines where people have issues. Every single one of these deep learning libraries is mapping to CUDA/BLAS calls under the hood. If you wrote the framework correctly, the performance difference should not be drastically different. And Flux doesn’t have much in terms of overhead. My lab mate uses PyTorch to train the same models as me and his performance is consistently the same or worse.

As for features, I think this is because people coming from TF or PyTorch are used one monolithic package that does everything. That’s intentionally not how Flux or the Julia ecosystem is designed. I’ll admit that there are a lot of preprocessing utility functions that could be better in the larger Julia ML community. But for the most part, the preprocessing required for ML research is available. This is mostly the fault of the community of not having a single document explaining to new users how all the packages work together.

Where the difference between Flux and other ML frameworks is apparent is when you try to do anything other than a vanilla deep learning model. Flux is extensible in a way that the other frameworks are just not. A simple example is with the same lab mate and I trying to recreate a baseline from a paper that involved drawing from a distribution at inference time based on a layer’s output then applying a function to that layer based on the samples drawn. I literally implemented the pseudo code from the paper because in Flux everything is just a function, and chains of models can be looped in a for loop like an array. Dumb pseudo code like statements where you just write for loops are just as fast in Julia. And it was! Meanwhile my friends code came to a grinding halt. He had to resort to numerical approximations for drawing from the distribution because he was forced to only use samplers that “worked well” in PyTorch. This is the disadvantage of a monolithic ML library. I didn’t use “Flux distributions,” I just used the standard distributions package in Julia.

This disadvantage to TF and PyTorch will become even more apparent when you do model-based RL. Flux was designed to be simple and extensible from the start. TF and PyTorch were not.

Great summary! At the end you mention the difficulty extrapolating beyond an HH neuron model. I think curious readers will find the work of Jim Smith (https://fcrc.acm.org/plenary-speakers/james-e-smith-plenary) interesting in this regard. His work starts with the possible information representation scheme (temporal coding <=> binary coding) and a compute unit (SRM0 neuron <=> transistor) and builds up the equivalent of Boolean logic/algebra from there.

As opposed to a neuroscientist understanding a processor, Jim is a computer architect using his techniques to understand the brain.

They even address what you are saying in the article. They mention the looming danger of weaponized AI which is exactly when someone has altered the program to do harm.

That’s the entire point of the article. The AI itself will not develop a need to harm. Users/creators of the AI will program that into it. But AI by itself even with creators of the best intentions has real problems that affect society today. And talking about a hypothetical future has distracted the public narrative from the real problems with AI that we need to solve now.

Just to be clear, I’m not trying to put down Clash. I disagree with your statement about it being an alternative to Verilog or VHDL. It’s like saying Java is dead because Scala exists.

I guess my point is that Clash is just a high level language that transpiles to Verilog. There a plenty of languages that already do this. So your original claim that it is a Verilog or VHDL replacement isn’t accurate. We still use Verilog.

Companies have been autogenerating Verilog from high level languages for decades.

I just took a look at clash. HDL describes state in a very real way. Trying to represent stateful hardware using a functional language is just too complicated. I hate these toy examples of “look how easy an FIR filter is.” An FIR filter isn’t hard to write in Verilog to begin with. The real headache of Verilog is generate statements and multidimensional array indexing. And both of those problems are readily solved by System Verilog, System C, or any of the myriad of Verilog generators.

I’ll leave this here: https://t.co/jA1kz8QMvt

You are right about qutrits. Essentially, anything that exhibits quantum behavior is in infinitely many levels at once. It is our ability to measure discrete levels that leads to qubits, qutrits, etc. Generally speaking, the more levels the more unreliable the measurement.

As for Monte Carlo with respect to ML, are you referring to the “random walk” aspect when you say “use randomness.” This refers to the fact that the method samples one possible sequence of events and uses that sequence to update its model. As opposed to dynamic programming methods where the value of all possible sequences is estimated and used to update the model. Not sure that the randomness from quantum is useful here. Where it could be useful is to encourage exploration of the state space. So, normally Monte Carlo methods have various tricks to make sure every sequence is sampled infinitely many times. These tricks could be implemented by the error in quantum computing, but I don’t know enough about the field to really be sure of that. And of course it would help in that you could compute the results of many sequences at once which is critical bottleneck in dynamic programming.

Tried setting up an IPFS website using Hearth, and I ran into the links issue. Gave up pretty soon after in favor of GitHub Pages.

If a distro is committed to modifying the style sheets to look like macOS, then it should be the distro’s responsibility to maintain the applications. Precisely because the distro market for Linux is so varied, it is hard for Gnome app developers to support all possible theme configurations.

Read the banner at the top of the page again. It doesn’t ask users to stop customizing, and it doesn’t say there shouldn’t be distros with custom theming. It just asks distros to stop shipping broken apps as a result of theming, and it asserts that if that’s the route you want to go, then you are on your own.

You’re right, but that is the motivation for the blog post. Learning is a form of fitting that performs better in practice than the curve fitting mentioned in the blog post when the required fit is nonlinear and/or high dimensional. The author is making the point that if the model is simple, since learning is just a form of fitting, why not directly apply standard curve fitting techniques?

I don’t think if you see someone with an iPhone, you assume they have the money to buy nice things. I think the image Apple has built up works in the opposite way. If you see someone without an iPhone, you assume they only bought that product because they couldn’t afford an iPhone. That must be the only reasonable explanation, you think, because who would willing buy another brand (due to the perceived difference in quality).

Most neuromorphic chips are general purpose in that you can tune the parameters of the neuron model. You can run a variety of learning algorithms by tuning these parameters. The only thing that is fixed is the model of a neuron itself.

To have hardware that allowed any neuron model would boil down to having an array of MAC units, or even more generally, a DSP. At that point you’ve lost sight of your original goal with a neuromorphic chip — to build an energy-efficient, scalable neural emulator.

With respect to the academic research aspect — graduate labs can get their hands on neuromorphic hardware. GPUs weren’t made ubiquitous to process scientific algorithms. It was the researchers who had the ingenuity to use GPUs for general purpose compute. Access to a powerful GPU then was as straightforward as access to a neuromorphic chip is now. The issue is when you try to scale the problem you are solving. In this vein, a GPU cluster of that size is not currently available to academics. That’s why you see a bias towards deep learning research in industry. In other words, it doesn’t make sense to compare a server of neuromorphic chips (which is what is required to simulate realistic neural behavior) to a single GPU. Compare it to servers of GPUs like Google’s or FB’s. You’ll see that academics have poor resources in both cases and the research keeps moving on.

Great news! Question for the fellow Jupyter Hub users: how do I expose users’ Conda environments to them?

I don’t. I have the 2016 15” model, and I’ve never had an issue. Most of the people in my graduate program don’t seem to have issues either. It surprises me how much i hear about this issue, because it directly contradicts my anecdotal evidence.

Could you comment on these “aha” moments?

I agree. Technology always makes progress, so it is inevitable that clean energy will be dominant in the future. The question is whether that will be too little too late.

gasoline cars will be worthless scrap heaps over night

This is true if the market is allowed to operate freely — “free” should not be confused with “unregulated.” Unfortunately, when it comes to energy in the US, the market is cleverly controlled to push the inevitable rise of clean energy back several decades. It has been this way for a while now. The regulation here might seem heavy handed, but that’s the kind of push that clean energy needs to win the uphill battle it’s fighting. If we hadn’t rigged the market for so long, then the natural progress of technology would have addressed climate change in time. But now we’ve waited to long, and we need hard hitting policies to peddle back the pace of the damage.

It feels like a fight and will disrupt many lives, and it doesn’t need to be that way.

This is where you won me back over. The truth is that it is a fight. But it is important to note for those of us who want to address climate change that how we fight is important. Unlike some other markets, we haven’t really had the opportunity as a country to see what a regulated, climate-focused market looks like. So, if we push some policies that take away jobs from people who have been working in coal mines their entire lives, then that’s going to be the headline. Everyone for so long has been saying that clean energy is too radical, and it will kill the economy. We have the opportunity to implement a comprehensive plan for a gradual transition. Bring clean technology to the forefront while helping those that will suffer to slowly transition into retirement. If we can address climate change, create a new sector of jobs, and let an old market die gracefully, then the naysayers will have nothing of substance to complain about.