drmeister

Awe-inspiring work Shinmera and Charles.

Dang, substitute Lisp for Prolog and this describes me. Seriously though - Prolog is an awesome tool to have in your toolbox. I've implemented Prolog-like logic programming solutions in several places in my 40+ years of programming. Like rules for assigning molecular mechanics force field atom types.

I accidentally a Common Lisp that interoperates with C++ (https://github.com/clasp-developers/clasp.git). We would also like to move beyond BDWGC and the Whiffle GC looks interesting. I will reach out to you, and maybe we can chat about it.

Ah - thank you. "True sharing" makes way more sense to describe the situation. The general problem with locks I've seen again and again - and maybe CAS doesn't improve it, but I thought it did. It is challenging to profile this stuff carefully. I've been disappointed many times by my attempts at multithreaded programming when locks are involved. I've implemented a multithreaded Common Lisp programming environment (https://github.com/clasp-developers/clasp.git) that interoperates with C++. Two locks that cause me massive headaches are (1) in the unwinder when we do a lot of stack unwinding in multiple threads and (2) in the memory manager when we do a lot of memory allocation in multiple threads. Parallel code runs multiple times slower than serial code. In another situation where we do parallel non-linear optimization without allocating memory, we can get 20 CPU working in parallel with no problems.

The big problem I have with locks is under high use, they slow parallel code down MORE than if you ran it serially. This is because of "false sharing" when one core grabs the lock, it clears the cache line of every other core that will grab the lock, and they have to fetch from the main memory to read the lock. Fetching from main memory is very slow. I use compare-and-swap in those cases, and almost every lock I've used I've converted to compare-and-swap. I'd like to explore "restartable sequences" for parallel data structures (https://tinyurl.com/vzh3523y) but AFAIK this is only available on linux and we develop on MacOS and Linux.

Thank you. We do precise GC in C++. I wrote a C++ static analyzer in Lisp that uses the Clang front end and analyzes all of our C++ code and generates maps of GC-managed pointers in all classes. We precisely update pointers in thousands of classes that way. We also use it to save the system's state to a file or relinked executable so we can start up quickly later. Startup times using that are under 2 seconds on a reasonable CPU.

Ah - thank you. We are looking for a GC that supports (in no particular order): (1) conservative stack scanning, object pinning. (2) optionally conservative and precise on the heap. (3) compacting when precise. (3) weak pointers. (4) good multithreaded performance. (5) finalizers. (6) ability to create pools of different kinds of objects - especially cons cells (pairs).

Thanks! I'm very interested in a new garbage collector that works with C++. I have a long list of requirements we need that is currently satisfied by the Boehm garbage collector and were satisfied by the Memory Pool System before we moved away from it. I've been looking at the Memory Management Toolkit (MMTk). Where would you put Oil in that small club of memory managers?

Get `udb` - the reversible/time-traveling debugger from Undo. I don't own any stock - I love their product. You can run your code within it, hit an error, set a watch-point, reverse-continue to where the watch-point memory was changed and then check the stack. udb will turn brain-melting, blood-freezing memory bugs into trivial problems that you can solve in a few minutes.

The GDB JIT interface implementation is seriously flawed. I love GDB - but this causes me a LOT of grief when debugging clasp (Common Lisp implemented using llvm as the backend https://github.com/clasp-developers/clasp.git).

Every time a JITted object file is added using the API, the entire symbol table is sorted. If you have 10,000+ JITted object files as we do - it takes hours to days to weeks to register them all.

We use the Undo time traveling debugger that builds on top of GDB. It's awesome but we are crippled because of the JIT API implementation.

I'd love to see this get fixed - if anyone knows who to talk with about it - drop me a line.

Thanks for that patch - I'll watch this with great interest.

Is there a problem if dynamic libraries invoke dlopen/dlclose they also need to call the sync function - correct?

I'm asking because we've developed a Common Lisp implementation that interoperates with C++ and it uses exception handling to unwind the stack (https://github.com/clasp-developers/clasp.git). We hit this global lock in unwinding problem a lot - it causes us a lot of grief.

This should be the top response. With a time traveling debugger this would take 15 minutes to solve (minus the time to make the crash happen). On linux I use the Undo debugger - I don't own stock - I just love the product.

I like the term "Eternal Language" - programming languages where if you write code now - you will be able to compile and use that code ten years from now (a software lifecycle eternity). Common Lisp, C, C++, Fortran, (edit: Java) are close to eternal languages. After losing a huge amount of Python 2 code (I wasn't going to rewrite it) I implemented Clasp, a Common Lisp implementation that interoperates with C++ (https://github.com/clasp-developers/clasp.git) so that I can write code that will hopefully live longer.

I am one of the few software developers with code that I wrote 27 years ago that is still in active use by thousands of computational chemistry researchers (Leap, a frontend for the computational chemistry molecular dynamics package AMBER, implemented in C).

I had a very different experience. I implemented Common Lisp using LLVM-IR as the backend (https://github.com/clasp-developers/clasp.git).

1. I started with a primitive lisp interpreter written in C++ and worked hard on exposing C++ functions/classes to my lisp using C++ template programming. LLVM is a C++ library, the C bindings are always behind the C++ API. So exposing the C++ API directly gave me access to the latest, and greatest API. That means you need to keep up with LLVM - but clang helps a lot because API changes appear as clang C++ compile time errors. I've been "chasing the LLVM dragon" (cough - keeping up with the LLVM API) from version 3.something to the upcoming 13.

2. I wrote a Common Lisp compiler in my primitive lisp that converted Common Lisp straight into LLVM-IR. I didn't want to develop my own language - who's got time for that? So I just picked a powerful one (Common Lisp) with macros, classes, generic functions, existing libraries, a community etc.

3. I used alloca/stack allocated variables everywhere and let mem2reg optimize what it could to registers. I exposed and used the llvm::IRBuilder class that makes generating IR a lot easier.

4. Then I picked an experimental, developing compiler "Cleavir" written by Robert Strandh and bootstrap that with my Common Lisp compiler. It's like that movie "Inception" - but it makes sense :-).

Now we have a Common Lisp programming environment that interoperates with C++ at a very deep level. Common Lisp stack frames intermingle perfectly with C++ stack frames and we can use all the C/C development, debugging and profiling tools.

This Common Lisp programming environment supports "Cando" a computational chemistry programming environment for developing advanced therapeutics and diagnostic molecules.

We are looking for people who want to work with us - if interested and you have a somewhat suitable background - drop me a message at info@thirdlaw.tech

Came for this.

“Curious how these things happen. The [pencil] chooses the [writer], remember…”

Clasp author here - Clasp uses SICL's Cleavir compiler.

I use the esrap packrat parser in Common Lisp for computational chemistry file formats. I've written half a dozen parsers with it and had a few written for me by others. They are easy to write and easy to maintain - even when I don't look at them for a couple of years. I am happy to not have yacc and bison in my life anymore (no offense intended).

Thank you!

Agreed! There are a couple of really clever xkcd comics about this (google: xkcd lisp). And I also would rather use PATH than default-pathname-defaults and logical pathnames are, uh, crufty. But I see these as minor blemishes on what I think is as close to a perfect programming language as I have seen. Clasp is an implementation of Common Lisp and any differences between what it does and what the standard says is a bug in Clasp that we need to fix. But Cando, Clasp+computational chemistry code, is a superset of Common Lisp and we are adding things to make life a bit more convenient. We even added optional infix arithmetic as a standard part of Cando (I know! I'm going to burn in hell between the ninth nested set of parentheses as a heretic).

We are doing computational chemistry, simulating molecular structure and designing molecules, and we want to use thousands of cores and get as much performance as possible. If my AWS bills are any measure - then yes - the cost savings in electricity and computing resources are very significant.

Also, developing and maintaining Python/C++ bindings for complex libraries is very painful and frustrating. I wrote Python bindings for years using boost::python and earlier Swig and keeping bindings working and dealing with the different memory management approaches of Python and C++... bleh - it's a nightmare. At the same time Python changed from version 2 to 3.x and libraries I depended on and my own Python code was being broken and becoming outdated in ways that I had no control over. It was like trying to build a house out of sand.

I've only been using Common Lisp for the past 6 years - after three decades of writing in other languages including Basic, Pascal, Smalltalk, C, Fortran, Python, PHP, Forth, Prolog... Common Lisp feels great, it feels powerful and every function I write I know will compile and run in 20 years. Common Lisp has real macros (programs that write programs! implemented in one language), dynamic variables, generic functions, the Common Lisp Object System, conditions and restarts... There are many features that haven't made it into other languages. Common Lisp makes programming interesting again.

Contact me using my temple dot edu address - it's on my website. Hacker News doesn't appear to have a messaging feature.

No, I just found out about Pliant from your post - but it doesn't make sense for developing large, stable codebases because it's not a standard language. (Sorry Pliant developers - I love your can-do attitude and I'd love to buy you a beer or a coffee sometime and talk about Sisyphean task management.) But Pliant is a reference implementation of a custom language. Programming language design is really, really hard - I wouldn't dare try and so I chose to go with a language that had literally hundreds of person years of design and testing underpinning it. Regarding FFI's - my approach is the same as the very clever pybind11, luabind and the older boost::python libraries. It works by using C++ template programming and letting the C++ compiler do the heavy lifting of generating wrappers at compile time. I recently updated our binding library to use C++17 std::apply with tuples. Freakin' finally! C++ has an apply construct that can be applied to a heterogenous list of objects - wow - lisp has only had it for 50 years! My point is that only recently has C++ developed the introspective capabilities to implement really powerful FFI's. Also - you have to use C++ exception handling for stack unwinding or you will break C++ RAII all the time.

There was a time when I asked myself that every day (sigh). But I don't think so. The details involved in interoperating with C++ are so intricate and involved. I don't think I could have gotten to the same place. Meanwhile - we are improving our compiler performance and the performance of generated code. Maybe in the far future I'll start a couple of ancestor simulations and do the experiment...

We do have a public website - you could contact me. We are still running under the radar so there isn't much detail but there is a lot in the scientific literature. http://www.thirdlaw.tech and https://www.schafmeistergroup.com/. I also have some talks up on the youtube.

If the question is why not use Julia? The answer is - for several reasons. I started implementing Clasp before Julia was a thing. The Julia language keeps changing and it doesn't have a standard like Common Lisp, C++ and C do. I need tight C++ interoperation and Clasp does C++ interoperation like no other language I've see. Clasp uses C++ exception handling and calling conventions and this allows us to compile and link C++ code with Common Lisp - it's all LLVM under the hood. Clasp stack frames and C++ stack frames are interleaved and try/catch and RAII and Common Lisp unwind-protect and dynamic variable binding work perfectly between C++ and Common Lisp code.

The goals have not changed. The pace of development (in the chemistry) has accelerated by orders of magnitude in the last year. The software is advancing as well. We are still looking for good developers who want to work with us.

Hey - props to the developers of Java for bringing automatic memory management into the mainstream. But it's not quite what I'm looking for when I want to do exploratory programming.

I do have a couple of talks on YouTube and thank you (the most recent one: https://www.youtube.com/watch?v=mbdXeRBbgDM). Yes - we are building molecules using our code. I have started a company and we are several months in to developing a new technology for warding off future pandemics (maybe even doing something about the current one). We literally started compiling the first molecules using an application I implemented in Cando (Clasp + computational chemistry) yesterday.

SBCL is an amazing implementation and it has an amazing compiler. I would argue that it is one of the best compilers around. Javascript compilers in browsers are pretty impressive - but given the far fewer resources that SBCL development has had, SBCL is remarkable. It's a fast compiler that generates fast code. I would be using it (and do for some applications) if I didn't also need my own large C++ computational chemistry code written over decades.

It's working fine. We are developing applications within it for the past year. We have been kind of low key because we are using it as part of a much larger project.