HN user
https://keybase.io/jpference
Yes. this one is fun. Recursive, memoized Fibonacci, the 155th integer precisely.
{{(-&2 +&$: -&1) ^: (1&<) M. y}} 155x
It'll run in your browser in 0.003 seconds.
https://jsoftware.github.io/j-playground/bin/html2/#code=%7B...
J recursion is $: You can do flow control in both scripting style and array style. Loops are loops.
Group is (was) a function in various array languages, including K, APL, BQN, J, and Shakti.
I suspect the J package Jd is probably the most non-trivial public codebase. I don’t love the coding style (functions are long and scripted) and it doesn’t make use of newer lambda functions (“direct definitions”) which are easier to read. https://github.com/jsoftware/data_jd
Possibly so. We do simple IO primarily from parquet or csv data, and I am working on the Apache Arrow/Flight package currently. Our data sets typically fit in RAM but either if you have enough RAM or a file amenable to memory mapping, you shouldn’t have a problem. J has a memory mapping utility. Numerical data is straightforward. We do ETL, QC, and normalization in J. Avoid transposes where you can, but reshapes are trivial and done with metadata, so fast. Overall what you describe sounds like a fun project to try in J.
I cannot share specific projects. J is particularly good for numeric/algorithmic experimentation. It is interactive/REPL-based. Types and boilerplate are minimal. There is no compilation and minimal packages. This is the language I reach for when step 2 is not going to be “install these packages” or deploy to share code. It is most remarkable for data hacking, as it is trivial to manipulate structures and maintain performance. I use python, R, and clojure frequently, but the ability to move quickly in J is without parallel. Weaknesses include namespacing and deployment, although I have seen deployment of substantial codebases both on desktops and servers. Multi-threading and AX512 instructions in _your_ (not package) code, from the REPL are some of what you get with j904.
Yes. Ask questions if you like.
I did some work to compare Jd to data.tables and found that it was more performant in some instances such as on derived columns, and approximately equally performant on aggregations and queries. Jd is currently single-threaded, whereas multiple threads are important on some types of queries. I tried to further compare with Julia DB at the same time (maybe a year ago) and found that was incorrectly benchmarked by the authors and far slower than both; that might be different now. Jd is more equivalent to data.tables on disk; Clickhouse is far better at being a large-scale database.
Rules of thumb on memory usage: Python/Pandas (not memory-mapped): "In Pandas, the rule of thumb is needing 5x-10x the memory for the size of your data.” R (not memory-mapped): "A rough rule of thumb is that your RAM should be three times the size of your data set.” Jd: "In general, performance will be good if available ram is more than 2 times the space required by the cols typically used in a query.”
Re CSV reading, Jd has a fast CSV reader whereas J itself does not. I have written an Arrow integration to enable J to get to that fast CSV reader and read Parquet.
J FFI lib call: https://code.jsoftware.com/wiki/Scripts/CallJ and https://code.jsoftware.com/wiki/Guides/DLLs/Calling_the_J_DL... database: odbc, jd, sqlite, mysql, [arrow bindings] network services: sockets, websockets, zmq, & http 1.1 text: unicode support, names limited to asci https://wiki.jsoftware.com/wiki/Guides/UnicodeGettingStarted
The claim here is +80% reduction in time, +5x throughput relative the current standard.
Nice to see illumination of the benefit of high-level algorithmic design associated with the first-principals aspect of APL.
This is also an interesting claim presently, given there's an active conversation in the Jsoftware community currently on how to handle tree structures.
Since you supposedly know something about this topic, please contribute it.
Companies that use APL and other array languages in production:
A bunch of questions: Is multi-threading happening at the level of user-defined functions? How are threads scheduled? What's the underlying method or library for enabling multi-threading (Cilk, openMP, some LWT library...)? To what extent has this level of granularity been tested against other levels of nested parallelism (e.g. SIMD or otherwise parallel operators)? Have you tested performance by OS, and if so, have you noted any necessary OS-level modifications related to thread management? Is this part of a broader roadmap for accelerator integration?
Without general theories, the only reasonable question is "Faster for What"?
What are generalizable theories of 'fast'? Fewer bits moving less with fewer instructions, and using the most suitable, specialized hardware available.
Even once you have a reasonably well-defined goal, if it's complex enough you may start hitting subsidiary questions, like numerical precision and stability.
Can you comment on experience (or contact me) regarding implementation efficiency? We have recently implemented task-based parallelism in the J language with openMP[0]. Improvements or critiques are appreciated. SIMD instructions there have been coded directly rather than via pragmas.
LLVM has an openMP implementation
XLA is domain-specific compiler for linear algebra. Triton generates and compiles an intermediate representation for tiled computation. This IR allows more general functions and also claims higher performance.
obligatory reference to the family of work: https://github.com/merrymercy/awesome-tensor-compilers
I'd be interested in hearing of any experience or recommendations. If you have them available, please do send me a message.
While MATLAB and Numpy were influenced by the APL family of languages, they contain a small subset of the design available in the J/K/APL world. Informally most users of array languages would not consider them close family members, though perhaps they're cousins.
Congrats on your work with APL. Please do not engage in flaming on obsolete business concerns from 30 years past, on a repeated basis.
J is fine, great, easier license, has a larger and more powerful set of primatives.
If I understand correctly, Nx.defn effectively calls a backend compiler for that function, and that will presumably result in only one round-trip of data to the accelerator hardware (e.g. GPU). If you have two defn functions, do you 1) lose compilation between them and 2) round trip twice, or can you nest defn functions and get compilation of the composed function?
In practice, we've experienced 20 microsecond-100 millisecond thread interruption issues on Windows. The resources we've found are inadequate to address the questions. No such problems on Linux. Windows scheduler experts appear to be rare. We wonder if Windows is simply not amenable to certain types of high-performance applications.
Check out Monument (https://www.monument.ai) (shameless plug)
We specialize in tabular data and are building a pipeline-based approach for creating and serving models.
There are interesting ongoings in array language-world.
People who do things are recognizing this.
...
We use J in all ML and data handling functions, everything except UI and viz.
We primarily focus on algorithms that require rapid sequential iteration rather than parallelism. However: 1) we run an enhanced version of J that is implicitly parallel on CPUs, and 2) it is possible to call GPUs (both CUDA and openCL) through J (e.g. j-fire). We do not call GPUs or other accelerators currently, though we may do so in the future.
Our goal with Monument is to create an Excel-like substitute for ML / AI. In comparing across languages, we selected J as "the right level of abstraction" for ML / AI because we wanted to write fast-running code, quickly.
J runs on virtually all CPUs and OSs, and it is highly tuned for minimized cache misses and mis-predicted branches. It is well-documented, well-supported, and the community is highly supportive.
Like Python, J makes it possible to run highly-optimized code without a great deal of boilerplate in an interactive development environment. Unlike Python, J is composed of reusable functions that readily express math. Our dev productivity is extremely high, and often we're able to engage with mathematicians who have little or no programming background.
Our team comprises polyglot programmers, and I strongly avoid "the language wars," but overall we have found J to be well-suited to our needs.
I lead a company that has a large portfolio of AI/ML/optimization algorithms in J (Monument). Happy to answer questions.