acidflask

A port of Andrej Karpathy's llama2.c to pure Julia. Works on Llama2 and Andrej's tinyllamas models. REPL mode allows interaction from the Julia console

How long until we can have Julia wrappers?

This question keeps me up at night, as I presume it does for my colleagues at Julia Computing. Not that switching to a fully commercial model is necessarily a bad thing, but Julia Labs, like any academic group, always has to worry about where funding will come from.

See Slide 4 for why it's important to be open source.

TL;DR: Researcher A finds things he wants improved in Magma (closed source) but can't. Researcher B tries to write improved FOSS implementation, but lost his job, likely in part by spending too much time writing said code and not doing other things like writing papers. Researcher A moves on and has a successful academic career. Moral: writing FOSS can cost you your academic job; it's safer to find something else to do.

I'll be there for moral support, if I'm allowed to crash!

And the problem with academic funding is that it's really hard to pay people to improve the documentation with those funds.

As Stein explains in the slides: because he couldn't sustainably pay people to work with him on SAGE, and there is only so much you can do alone.

Thanks for the input!

Yes, there are at least two separate stages of historical development here. The first is when people realized it was useful to repeat the same operation on different data in memory and viewed the collection of data as a variable in its own right. The earliest term I can find for this concept is "subscripted variable" (many examples prior to 1954, e.g. Rutishauser, 1952 in the "for" loop paper; Laning and Zierler, 1954)

but the idea appears to go all the way back to Burk, Goldstine and Von Neumann in 1946. Quoting p. 9, paras. 3.3-4:

"In transferring information from the arithmetic organ back into the memory there are two types we must distinguish: Transfers of numbers as such and transfers of numbers which are parts of orders. The first case is quite obvious and needs no further explication. The second case is more subtle and serves to illustrate the generality and simplicity of the system. Consider, by way of illustration, the problem of interpolation in the system. Let us suppose that we have formulated the necessary instructions for performing an interpolation of order n in a sequence of data. The exact location in the memory of the (n + 1) quantities that bracket the desired functional value is, of course, a function of the argument. This argument probably is found as the result of a computation in the machine. We thus need an order which can substitute a number into a given order-in the case of interpolation the location of the argument or the group of arguments that is nearest in our table to the desired value. By means of such an order the results of a computation can be introduced into the instructions governing that or a different computation. This makes it possible for a sequence of instructions to be used with different sets of numbers located in different parts of the memory.

"To summarize, transfers into the memory will be of two sorts:

"Total substitutions, whereby the quantity previously stored is cleared out and replaced by a new number. Partial substitutions in which that part of an order containing a _memory location-number_-we assume the various positions in the memory are enumerated serially by memory location-numbers-is replaced by a new _memory location-number_.

"3.4. It is clear that one must be able to get numbers from any part of the memory at any time. The treatment in the case of orders can, however, b more methodical since one can at least partially arrange the control instructions in a linear sequence. Consequently the control will be so constructed that it will normally proceed from place n in the memory to place (n + 1) for its next instruction."

https://library.ias.edu/files/Prelim_Disc_Logical_Design.pdf

The language is of course archaic, but the idea described clearly is that of indexing in 3.3 and arrays in 3.4. They use the word "sequence" but arguably this usage is in its ordinary mathematical sense.

The written historical evidence, at least, would confirm your strong guess that the idea of arrays itself is older than index registers. There's a missing etymological link though: when did a sequence of data stored consecutively in memory become associated with the word "array"? Still, the earliest written reference I can find for this second stage of historical development is the 1954 preliminary report on FORTRAN.

Maybe the word "array" is somehow derived from the advent of RAM, which even in its earliest form in Williams tubes had memory locations arranged physically in two dimensions. So right from the start we have two dimensions physically, but only one dimension logically, since the earliest computer instructions only dealt with (one-dimensional) offsets, if at all. Furthermore, popular science accounts of magnetic core memory describe them in terms of arrays. To give one example, the June 1955 issue of the Scientific American (no 192, pp 92–100) writes about "magnetic core arrays".

http://www.nature.com/scientificamerican/journal/v192/n6/pdf...

Would you happen to have any insight into the question that brought me to go dig through old letters in a library in the first place?

https://news.ycombinator.com/item?id=11799143

The paper is about, among other things, the history of the array data structure. It's far too early to advertise, but you can see a very early version on my GitHub account. :)

I was surprised to discover recently that the word 'array' prior to 1950 was used exclusively to describe two dimensional tables of numbers that one might find in a matrix or determinant. But by the advent of FORTRAN I in 1957 and ALGOL 58, 'array' now referred exclusively to a one-dimensional entity, as compared with 'n-dimensional arrays'. I was interested in digging through John Backus's papers from this era to see if I could find any clues.

I was able to narrow down the near window to 1952-1954, since the FORTRAN preliminary report of 1954 uses the word 'array' casually in the modern one-dimensional sense as interchangeable with 'subscripted variables', the latter being the more common terminology at the time. By comparison, a virtually unknown paper by Rutishauser in 1952 describing the "for" loop did not use the word 'array' at all, only 'subscripted variables'. (Rutishauser was an accomplished mathematician and quite possibly the world's first computational scientist.) A paper by Laning and Zierler at MIT in 1954 describing a formula compiler also used only the term 'subscripted variables'.

Backus's papers also have evidence showing that FORTRAN I was clearly written specifically to take advantage of the IBM 704's machine capabilities. Not only was the IBM 704 the world's first commercially successful computer, it was also an improvement over the preceding IBM 701 in providing index registers (3 of them) and floating point instructions which were fast for its era. Backus's papers describe how providing hardware support for indexing and floating point was revolutionary, as all programs up to that time had to write in all these instructions by hand (and for many programs was pretty much all they did).

So it is clear to me now that the changeover in the implied dimensionality of the word 'array' must be related to how the array developed as a data structure abstracting away indexing operations. By the time IAL (pre-ALGOL) came on the scene in 1958, the idea of indexable homogeneous containers was already well established. But I still haven't found any strong smoking gun evidence introducing the one-dimensional sense of the word. I suspect further digging into the description of the IBM 704 may be necessary. The 704 was not the first to provide index registers, but it may have been the first to call them as such. (The Manchester Mark I computer of 1948 appears to be the first computer with an index register, but it was called a B line. The [patent](https://www.google.com/patents/US3012724) claiming to cover index registers uses the term "control instruction" - no arrays mentioned - but it very cutely describes numbers as residing in known locations or "addresses" in quotes.)

Thanks for contributing the transcript!

I've started a GitHub repo with a first draft of transcripts for all the letters.

https://github.com/jiahao/backus-dijkstra-letters-1979

Anyone else who is interested is welcome to help proofread and submit PRs.

The title "This guy’s arrogance takes your breath away" is taken directly from Backus's own description of this collection of letters. I've changed the title to make clearer that it is a direct quotation.

Unfortunately, the Library of Congress does not allow scanners without prior approval, so this was the only way I could make my own copies. It did not help that all these letters were written or typed on very thin mimeograph paper.

I'm genuinely surprised that one would say that Julia is "slowing down in development". Perhaps it's because less press is being generated about Julia? Or that the commit rate has gone down slightly, now that the easier issues have been picked off and the remaining work will take longer for the next round of incremental developments? I'm not sure what the OP meant, but from the inside, we are busier than ever.

- Both Julia Computing and the Julia Lab have grown sizably over the past two years. The Lab now houses ten full-time researchers (up from four last year), with five new students coming online over the summer and fall. We also maintain more active research collaborations with more research groups at MIT and off-campus.

- Julia is a grateful recipient of 12 Google Summer of Code slots this year, compared to 8 for 2015's Julia Summer of Code program (sponsored by the Moore Foundation) and 4 for GSoC 2014.

- JuliaCon grew from 72 attendees in 2014 to 225 in 2015 and we are on track to meet or exceed last year's ticket sales for 2016.

- New packages continue to be registered on the central METADATA repository at roughly the same rate since June 2014. http://pkg.julialang.org/pulse.html

By some measures we are still a relatively small project, but I don't see any serious evidence for the imminent heat death of the Julia universe.

Making people think is my favorite part of my job. :)

A big problem is that chained floating point operations are a O(sqrt n) martingale.

Mostly false; see https://www.cs.berkeley.edu/~wkahan/Mindless.pdf

It is true that an arbitrary, independent sequence of n floating point operations can potentially lose up to 2n bits of precision. However, it is false that most scientific computing code has such structure. On the contrary, the loss of precision in numerical algorithms can be estimated quite well using standard techniques of error analysis, especially when there is underlying matrix structure to exploit. The growth of error in properly written numerical code should NOT grow exponentially with the number of floating point operations. It should be bounded by intrinsic properties of the underlying matrix, such as its condition number or spectral norm.

Have you actually profiled the Julia code you wrote and ensured that you have followed the performance tips in the manual? If you did it correctly all the time should be spent in linear algebra operations, and so it shouldn't matter what language you implemented the code in since the performance is primarily determined by your underlying BLAS library.

It would be really interesting if you'd like to implement some quantum chemistry! Rick Muller (of PyQuante fame) has implemented a port to Julia, but I think there's a lot of room to use more language features.

https://github.com/rpmuller/pyquante2/tree/master/julia

Let me know if you do want to do some quantum chemistry coding. It would be good for my soul to revisit some of my PhD roots...

Actually, chemistry and physics do quite well with nonprofit publishers. Compare the profit margins of for-profit publishers with non-profit ones like the American Chemical Society. In 2015, the ACS made a $10m profit on $500m gross revenue, a much smaller profit margin.

http://www.acs.org/content/acs/en/about/aboutacs/financial.h...

The smaller American Physical Society made <$100,000 on $53.5m gross revenue in 2015:

http://www.aps.org/about/governance/annual-reports/

So yes, it is possible to have high quality publications produced on much smaller margins. The question is whether it is possible to provide similar services for similar margins in a for-profit context. Given that for-profit companies, by definition, seek to extract the highest possible profits on what they do, one should really question whether it makes any sense for for-profit publishers to have the stranglehold they have on academic intellectual property.

To cite just one example, Elsevier consistently reports over $1b/yr profit on $3b/yr revenue. Not exactly "almost nothing".

https://en.wikipedia.org/wiki/Elsevier

https://libraries.mit.edu/scholarly/mit-open-access/open-acc...

The issues about the lack of tooling and stability are well heeded. Unfortunately, we at the Julia Lab at MIT, like our colleagues that are the JuMP group at the MIT ORC, are limited to working on things that can be justified as research activities. Developing new algorithms and applying them to scientific computing activities is research; writing a debugger and testing framework is not. For non-research development we have had to rely on the goodwill from hundreds of open source community developers, which have helped improve Julia and its package offerings immeasurably.

Happily, the present situation is more rosy than it has ever been. Funding agencies are beginning to realize that academic software development is an activity worth funding in its own right, not necessarily tied to research. Julia Computing has also been fortunate to receive funding from organizations such as the Moore Foundation to work on precisely the tools that users most need, but are difficult to justify development with research funds.

Most people don't need more than a handful of linear algebra operations (or think they don't), so Breeze and most wrappers of LAPACK or similar libraries don't implement or wrap them. But most people who work seriously on numerical routines will quickly run into performance problems if all they do is call LAPACK routines for general matrices instead of taking advantage of matrix structure.

I have yet to come across any other linear algebra library for any other high level language that provides the depth of integration available in the Julia base library. Want all eigenvalues of a symmetric tridiagonal 10x10 matrix between 1.0 and 12.0? Simply call T=SymTridiagonal(randn(10), randn(9)); eigvals(T, 1.0, 12.0). Or if you want to work closer to LAPACK, simply call LAPACK.stein!. I don't see a wrapper in Breeze or SciPy for this function. Want an LU factorization on a matrix of high precision floats? lufact(big(randn(5,4))). And so on.

Julia may not have everything users want, but its base library really tries to make matrix computations easy and accessible.

Improvements to our LuaJIT benchmarking are currently being discussed in https://github.com/JuliaLang/julia/issues/14222. LuaJIT 2.1 is significantly faster than LuaJIT 2.0, but until now I haven't had a convenient way to get an installation of LuaJIT 2.1 on our test machine. LuaSci was easy to install and we would be amenable to switching over to LuaSci instead of vanilla gsl-shell. See https://github.com/JuliaLang/julia/issues/14222#issuecomment... for a timing comparison.

Your claim is fallacious, because a floating-point value denotes a range of the real number line, whereas you're insisting that, no, it stands for its literal value: the absolutely precise rational number that lies at the center of that range

See Misconception #2 of http://lipforge.ens-lyon.fr/www/crlibm/documents/cern.pdf.

Floating point numbers are _not_ intervals. If you read carefully any formal definition of floating point numbers (The IEEE standards, TAoCP Chapter 4, or Higham's _Accuracy and Stability of Numerical Algorithms_, to name just three possible references), you will see that floating point numbers by definition form an exact rational subset of the extended real line.

All numbers in that range alias to the same double, yet differ wildly in their decimal digits beyond the fifteenth. That "long tail" of digits is a meaningless residue arising from the arbitrary difference between the chosen center-of-range point and the actual number it approximates.

See Misconceptions #1 and #2 on Kahan's list (https://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf).

You are not being clear about the set of real numbers a user may input and their ultimate representation as a floating point number. While it's true that many real numbers round to the same floating point number _f_, that's not the same as then saying that _f_ carries that interval around with it. The latter is false, since the intervals do _not_ propagate in floating point arithmetic. It's also false that the floating point number _f_ is the midpoint of the set of numbers that round to _f_; the precise set depends on the rounding mode and the granularity of the set of floating point numbers around _f_.

Obligatory SMBC reference

http://www.smbc-comics.com/?id=2999

The correct link is https://github.com/JuliaLang/julia/pull/10709

April 1 was a good day in the lab!

Darnit, I missed out on the $43,264,000 postdoc at MIT in 2008.

http://data.jobsintech.io/companies/massachusetts-institute-...

It was either 'go with the title or come up with another one', and I was too tired to argue.

By the way, the paper was really an excuse to document this IJulia notebook.

http://nbviewer.ipython.org/urls/raw.github.com/jiahao/ijuli...

If you run it on your local machine, one of the cells has an interactive slider that lets you control the breadth of the parallel prefix tree you generate.

Agreed, not really a prediction, but rather a projection. Still, it surprises me that few people take the trouble to make the connection between the spectra and the perceived colors. Seems like photochemistry and colorimetry don't intersect as often as they ought to...

No, I put that in the first cell. You can take it out of the notebook if you want.

Is this so unreasonable? I needed to curry the function to sweep over the temperature T. And if you take out the units, it's simply pointwise multiplication of two functions.