Searching a Million Lines of Lisp

http://www.wilfred.me.uk/blog/2016/09/30/searching-a-million-lines-of-lisp/

by pmoriarty • 9 years ago

73 25 9 years ago

wilfred.me.uk

Tim Teitelbaum, one of the first researchers on IDEs summarized problems like this in a quote from one of his papers.

"Programs are not text; they are hierarchical compositions of computational structures and should be edited, executed, and debugged in an environment that consistently acknowledges and reinforces this viewpoint."

It is unfortunate that most major code editors and IDEs today do not store units of code (like Lisp SEXPs) as databases to be updated as you edit, which would make problems like this or other operations like metaprogramming, formal analysis, or documentation much easier to solve.

skybrian2• 9 years ago

I'm not sure why you think they don't? Most IDE's do typically build a language-specific search index and keep it up to date as you edit. (That's the main difference between an IDE and a text editor, though the line is blurrier these days.)

Having taken Teitelbaum's compiler class as an undergrad, I was happy to ditch the IDE he inflicted on us and go back to text editors. Structured code editing is a tricky UI problem and I didn't find a really good IDE until many years later.

There's no one way to edit code. Sometimes refactoring tools work well, but typing text can be quite efficient too. Getting locked into a tree editor at the expression level is no fun.

olewhalehunter2• 9 years ago

I'm not sure why you think they don't?

If they are they either aren't offering user/programmer access to that database, they aren't doing semantic/type binds, or aren't advertising those features well.

I was happy to ditch the IDE he inflicted on us and go back to text editors. Structured code editing is a tricky UI problem

It's reconcilable with normal text editing, just update the structure once its valid. I agree that things like block/visual programming can be absurd.

imtringued• 9 years ago

Especially in statically typed programs you want to "break" your program for refactoring purposes. Usually I look at a function or class and create new functions/classes and then rename the old one and then fix all the errors one by one by using the new code.

skrebbel• 9 years ago

There are editors that do this. Notable, Jetbrains MPS comes to mind. It looks like a text editor but once you use it you quickly notice that you're actually editing the abstract syntax tree directly.

It's cool, but it has some major downsides too. For example, MPS stores the source as XML, not text (since it isn't text, it's a tree). This makes lots of basic tools we've taken for granted a lot harder, such as git merging etc. They've had to make a custom mergetool just to make basic collaborative coding feasible.

I bet there's other ways around that, all I'm saying is that text has major, major upsides because of the enormous ecosystem support.

sitkack4• 9 years ago

It would be nice to have a VCS that could work on the native MPS data structures.

Text is probably the next biggest mistake in programmer productivity after null.

db48x• 9 years ago

For example, MPS stores the source as XML, not text

If only there were a way to write your code using a uniform tree syntax in the first place...

noir_lord• 9 years ago

I thought about this a while ago, it would be nice if you could have all code stored as the AST and the formatting handled on ingestion/export.

It would finally settle the formatting arguments and all the tooling would be able to leverage each others projects.

Even PHP has an internal AST representation these days.

coldtea• 9 years ago

It's cool, but it has some major downsides too. For example, MPS stores the source as XML, not text (since it isn't text, it's a tree). This makes lots of basic tools we've taken for granted a lot harder, such as git merging etc. They've had to make a custom mergetool just to make basic collaborative coding feasible.

Doesn't solving this just require a text-to-AST, AST-to-text input and output step?

Anyplace outside the editor the programmer just sees regular text.

sitkack4• 9 years ago

For those looking for the Teitelbaum paper mentioned above [0]

[0] The Cornell Program Synthesizer: A Syntax Directed Programming Environment https://core.ac.uk/download/pdf/21750999.pdf?repositoryId=14...

pkaye• 9 years ago

I think Microsoft did that in their Roslyn dot.net compiler. There is a server that is fed source code changes and it updates the AST internally incrementally. The text editor can then make queries against that AST. I believe Microsoft also took a portion of this as added it to Visual Studio Code.

alphapapa• 9 years ago

Wilfred is a genius. The OP article is from about a year ago, but it laid the foundation for the Emacs package he just released: http://www.wilfred.me.uk/blog/2017/08/30/helpful-adding-cont...

sooheon• 9 years ago

Ah this is where I've been seeing that domain before. Yeah, the other package suggest.el is also a gem.

vcdimension• 9 years ago

He should try speeding it up further by compiling to C-code using this: https://github.com/tromey/el-compilador

tyingq• 9 years ago

Github's search went the other direction, where it's decidedly less useful than grep. Case insensitive, and drops search string characters that provide context. Like quotes, =, $, etc.

https://help.github.com/articles/searching-code/#considerati...

Searching a Million Lines of Lisp

Related Stories