Kina Knowledge, using Common Lisp extensively in their document processing stack

https://lisp-journey.gitlab.io/blog/lisp-interview-kina/

by p_l • 5 years ago

192 60 5 years ago

lisp-journey.gitlab.io

Abhinav2000• 5 years ago

This part is really interesting:

It is fast - our spatial classifier takes only milliseconds to come to a conclusion about a page (there is additional time prior to this step due to the OpenCV processing - but not too much) and identify it and doesn’t require expensive hardware. Most of our instances run on ARM-64, which at least at AWS, is 30% or so cheaper than x86-64. The s-expression structures align to document structures nicely and allow a nice representation that doesn’t lose fidelity to the original layouts and hierarchies.

The article also mentioned they have 3 programmers and 100k lines of code, that sounds impressive.

neilv• 5 years ago

The article also mentioned they have 3 programmers

One of the problems we had with promoting commercial use of Scheme and then Racket was that -- although some companies were using it to great success -- there weren't any job postings for it.

It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams).

And the knowledge of that success wouldn't be well-known. (Because they liked to focus on the work, or because the larger team of business etc. people they were in was also small, or, in at least one case, the business person thought "we use Lisp" would kill business deals even though the code wasn't customer-visible.)

So there would be no success stories, no job postings mentioning it as something people should learn, etc.

Which, I suppose was good for open communities self-limiting themselves to people who were genuine enthusiasts not motivated by money, and with no need to posture as influencers or do SEO, but... not so great for bringing in large developer base, getting lots of startups using it, etc.

pphysch4• 5 years ago

It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams).

Be careful with what you conclude from this observation. Is it that A) Lispers are generally 10x Programmers, or that B) Lisps are generally poor languages for team environments?

csande172• 5 years ago

It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams).

One thing that surprised me when I entered the professional world was just how much this is seen as a downside. At almost every level, companies will choose technologies and techniques that allow them to hire more headcount, even if that results in a worse end product.

Many startups value the appearance of having a large actively-hiring engineering team, and many BigCo middle managers want lots of direct reports. They don't usually care how much work gets done per employee, and sometimes they don't even care about the total amount of work that gets done across the organization. It's all about getting warm bodies into seats, either to impress investors or to gain status within the organization.

agumonkey• 5 years ago

It was the norm for a single programmer to be doing the work that would normally be a team (sometimes multiple teams).

I felt this so very often. There are economical paradoxes and sociological issues at play. People want to be impressed and a big corp makes it look like hard while 3 lispers does not have the glow and even at lower prices you may fail to sell good work.

A lot of this world work this way, if everybody worked smart and efficiently there would be a lot less jobs.

shaunxcode• 5 years ago

Really interested to play with dlisp if/when it is open sourced!

gjvc• 5 years ago

Yes! This sounds really promising.

dang• 5 years ago

Have they said whether it's a complete CL implementation or just a subset? I have the impression that it gets asymptotically more difficult as you approach the finish line.

p_l9• 5 years ago

Looks like it's CL-inspired so kinda like subset.

aidenn02• 5 years ago

Assuming you aren't just talking about how any project becomes harder to fix bugs in once you've fixed the easy bugs, it only gets more difficult as your approach the finish line if you don't plan on making a CL implementation from the start.

It's really easy to make something that kind-of looks like CL by use of clever smoke and mirrors, and many "Lisp in X" projects do so. See e.g. parenscript's LOOP implementation.

Other notes:

- Implementing CLOS is a massive undertaking, but there are two reasonably good portable implementations of it (Closette and PCL).

- The numeric tower is sufficiently loosely specified that you can typically get away with using any existing bignum library.

- LOOP and FORMAT are annoying to implement just because they are big DSLs. I'm also not aware of any portable implementations of them (though SICL might have one).

- Most of the rest of the effort will be spent on implementing things that interact with the system. Streams/files/pathnames are all annoying

pphysch4• 5 years ago

Lisp allows us to scale dramatically and manage a large code base.

Wow, really? How big is your company?

Right now, in our core company we have three people, two here in Virginia and one in Mexico City.

p_l9• 5 years ago

Sounds like scaling developer power all right.

Reminds me of quotes from Google SRE materials about scaling "SRE per managed server count".

thorweur234• 5 years ago

You know Google's flight scheduling software runs on Borg right ?

After they acquired ITA, they have few openings in between, and are yet, apparently, one of the biggest consumers of quotas.

the_optimist2• 5 years ago

What's a quota?

the_optimist2• 5 years ago

Seem's like you're trying for sarcasm with the rhetorical question, but it's unclear. Scale what? Team size?

sitzkrieg• 5 years ago

oh no no, scale only means hiring tons of programmers after winning funding rounds in these parts

pphysch4• 5 years ago

Apparently, scale means LOC of Lisps (or LOL, if you will).

zeIMOr5vRXhVpO• 5 years ago

I liked the idea of distributing binary applications as well, which we needed to do in some instances, and building a binary runtime of the software was a great draw, too.

fwiw this is possible to with clojure using graalvm. Only mentioning this because of the clojure comparison in the earlier paragraph. While I do lament the JVM I haven't found a straightforward way to build statically compiled binaries with sbcl. I would love to be proven wrong though! :D

mark_l_watson• 5 years ago

Give SBCL another try for building compiled applications. If you compile SBCL from source, enable compression of standalone compiled apps.

I own a license for LispWorks Pro, which does a very good job building applications, but SBCL is also good for delivering apps.

brabel• 5 years ago

Doesn't `(save-lisp-and-die :executable t)` [1] create static binaries?

I had a look at one of my SBCL-made binaries on MacOS, it shows this:

    ▶ otool -L  lisp-enc 
    lisp-enc:
 /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
 /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)

EDIT: GraalVM takes several minutes to generate binaries, and the binary may not behave the same as the JVM-based runtime, so I wouldn't recommend using that.

[1] http://www.sbcl.org/manual/#Function-sb_002dext-save_002dlis...

svetlyak40wt2• 5 years ago

Yeah, you've got a dynamically linked binary.

There is a work in progress on SBCL to make it possible to create a really static binaries:

https://www.timmons.dev/posts/static-executables-with-sbcl-v...

I tried this fork and it worked for me!

vindarel3• 5 years ago

You have this way: https://lispcookbook.github.io/cl-cookbook/scripting.html#wi...

and truly static executables are coming: https://www.timmons.dev/posts/static-executables-with-sbcl-v...

Kina Knowledge, using Common Lisp extensively in their document processing stack

Related Stories