marcle

Answering my own question: Umbra and PostgreSQL include "#" as an operator for bitwise xor, whereas DuckDB does not.

This is very interesting code that uses Umbra v0.2-957, PostgreSQL 17.2, and DuckDB v1.1.3.

One minor point: some of the problems could not be implemented in DuckDB due to a missing xor operator - but the documentation includes an xor function (https://duckdb.org/docs/sql/functions/numeric.html#xorx-y) for the same version. Is the form of the function not suitable?

Nice childhood memories: "Thunderbirds are go!"

I always wondered how the palm trees were able to be flattened:).

This is a very interesting IDE to support data science.

Has anyone tried using the Eclipse Theia IDE for data science?

One of these IDEs may finally entice me away from ESS, magit and Org Mode.

"PSA" can also be probabilistic sensitivity analysis. I do health economic evaluations for prostate cancer screening -- which leads to interesting ambiguities:).

The article references the SSJ Java package which is developed by the author and his colleagues. This provides implementations for many RNGs -- and some nice tools for simulations: https://github.com/umontreal-simul/ssj

And see the discussion at https://discuss.ocaml.org/t/owl-project-concluding/14117

See also some discussion at https://www.reddit.com/r/ocaml/comments/1au329z/owl_project_...

ErgoAI is as "an enterprise-level extension of the Flora-2 system" which was recently open-sourced: https://github.com/ErgoAI . It seems to be well documented.

See https://fricas.github.io/, which includes a 990 page book and detailed documentation.

This is a nicely balanced discussion of the two languages - with much food for thought.

I wrote a purely functional AD library in Mercury [0], which adapts a general approach from [1]. I believe that Owl provides a similar approach [2].

[0] https://github.com/mclements/mercury-ad

[1] https://github.com/qobi/AD-Rosetta-Stone/

[2] https://github.com/owlbarn/owl/tree/main/src/base/algodiff

The following homage to Aphyr’s "Typing the technical interview" solves the N-queens problem using types in Typescript. This is both funny and insightful.

https://www.richard-towers.com/2023/03/11/typescripting-the-...

(Edit: this is mentioned elsewhere in this thread by Kerrick)

Waldek Hebisch, who is the maintainer for the Fricas computer algebra software, also maintains a 64-bit version of Poplog for amd64/x86_64:

https://github.com/hebisch/poplog https://www.cs.bham.ac.uk/research/projects/poplog/freepoplo...

This is a free book on Owl. I have only looked at the chapter on automatic differentiation - and it looks good.

I find the solutions from https://github.com/qobi/AD-Rosetta-Stone/ to be very helpful, particularly for representing forward and backward mode automatic differentiation using a functional approach.

I used this code as inspiration for a functional-only (without references/pointers) in Mercury: https://github.com/mclements/mercury-ad

My go-to projects have varied over time:

1. An implementation of Fisher's exact test for 2x2 tables. This adapts the algorithm from R's fisher.test() function, which was implemented with function closures for optimisation, which does not always map well to other languages.

2. A foreign function interface to the Rmath library. I have now done this for Standard ML (MLton, Poly/ML, Moscow ML, MLkit, SML/NJ, SML# and Manticore), OCaml, Ur/Web, Mercury and MonetDB. Code generation sometimes uses cpp, m4 or the language's tree and string facilities.

3. Discrete event simulation using message passing with an application to cost-effectiveness analysis. Someday, I'll get this published.

Another relevant software category is the statistical analysis languages, including SAS, Stata and SPSS.

Old-school SAS included only two data types (floats and character strings), but allowed for SQL and sequential data-steps to live together. Persistence was baked in. The floats could be used to represent dates, datetimes and other formats. I particularly appreciated being able to use macros to define a data-step view to split the follow-up for an individual from a table. Such a view could then be collapsed using SQL. More recently, R tools such as dplyr have brought together data-frames and relational operations. However, I miss the sequential coding in SAS, using macros as higher-level tools to define the logic, including corner cases.

For strictly typed records, I have always wanted to spend more time with SML# [0] this allows for record updating, with close ties to SQL -- an under-appreciated version of SML.

[0] https://github.com/smlsharp/smlsharp

This is a very interesting article.

The author notes: "There is a compiler for KL1 to C available [2], which seems to work, but suffers from bit rot." Note that Ueda and colleagues have been updating this software (KLIC): see https://www.ueda.info.waseda.ac.jp/software.html for details. Ueda's work on LMNtal is also very interesting.

TLDR: Arrow and DuckDB provide fast database aggregates compared with R's RDS format and, to an extent, SQLite.

It is unclear how much functionality is available for Arrow under R: any comments? It would also be interesting to see a similar benchmark for Python, which could include the embedded version of MonetDB -- an R package for MonetDB/e is not yet available.

Edit: amended the TLDR to reflect jhoechtl's and wodenokoto's comments. SQLite provided reasonably memory efficient aggregates.

+1. I was about to post an almost identical comment.

This could also be R, which allows for `function(x) {f(x)}` and (since version 4.1) `\(x) {f(x)}`.

The mail-list has been quiet recently, but Ur/Web is not defunct: http://impredicative.com/ur/

More specifically: https://github.com/fsharp/fslang-suggestions/issues/243#issu...

I particularly like the comment: "I don't want F# to be the kind of language where the most empowered person in the discord chat is the category theorist."

A short answer: this requires a basis matrix for the splines rather than interpolation.

A longer answer: the splines require a basis matrix B(t), do that g(S_0(t))= B(t) gamma for a vector of parameters gamma and some transformation g of survival. A classical choice would be to use M-splines and I-splines with g(S)=-log(S) and a penalised likelihood, with the constraint that the gammas should be increasing. In R, this would use the splines2 package, while in Julia, one could use the splines2.jl package (disclaimer: which I maintain). The computational challenge is that the basis matrices need to be re-evaluated for changes in the coefficients for the covariates (that is, the betas).

There is no free lunch:).

I remember spending a summer using Template Model Builder (TMB), which is a useful R/C++ automatic differentiation (AD) framework, for working with accelerated failure time models. For these models, the survival to time T given covariates X is defined by S(t|X) = P(T>t|X) = S_0(t exp(-beta^T X)) for baseline survival S_0(t). I wanted to use splines for the baseline survival and then use AD for gradients and random effects. Unfortunately, after implementing the splines in template C++, I found a web page entitled "Things you should NOT do in TMB" (https://github.com/kaskr/adcomp/wiki/Things-you-should-NOT-d...) - which included using if statements that are based on coefficients. In this case, the splines for S_0 depend on beta, which is this specific excluded case:(. An older framework (ADMB) did not have this constraint, but dissemination of code was more difficult. Finally, PyTorch did not have an implementation of B-splines or an implementation for Laplace's approximation. Returning to my opening comment, there is no free lunch.

As an older user of SAS (from 1990) and R (from 1998), this may finally push academia away from SAS.

Locally, we continue to use SAS for manipulation of linked health and population registers. The combination of the SAS data step, SQL and macros is powerful for more complex data. However, most of our researchers use R or Stata for data analysis.

That said, younger researchers often prefer to use dplyr or data.table in R for the data management. One challenge is that those researchers often can not read the legacy SAS code - hence they do not understand how to handle the corner cases. This may be a symptom that data management documentation is often poor in academia.

For data analysis of time-to-event data, Python and Julia both lag behind R, Stata and, to an extent, SAS. I would be happy to move to Julia, but some of the basic modelling tools (e.g. safe prediction for formulae with splines) are currently lacking.

Thank you - now edited.

I am often amazed that one can run Maxima on an Android phone (Maxima on Android), including Gnuplot graphics and MathJax math formulae:

  plot3d ([cos(x)*(3 + y*cos(x/2)), sin(x)*(3 + y*cos(x/2)), y*sin(x/2)], [x, -%pi, %pi], [y, -1, 1], ['grid, 50, 15]);

  /* gradient of the log-likelihood for a time-dependent accelerated failure time model */
  assume(t>0)$ H:-log(S(integrate(exp(alpha+beta*x(u)),u,0,t))); diff(delta*diff(H,t)-H,beta);

This is a very interesting addition to the embedded analytics database landscape. MonetDB/e may hit a sweet spot.

I always enjoyed using MonetDBLite (with R), but it is no longer being developed. I understand that CWI has contributed to the development of MonetDBLite, DuckDB and MonetDB/e. It would be useful to know how DuckDB and MonetDB/e differ. The latter may have the advantage of building on a well established server.

For some other embedded databases, see http://embedded-database.com/open-source-embedded-database-s....