ibdknox

There was actually quite a lot of discussion on the mailing list (primary community back then) when we had to shift priorities. https://groups.google.com/g/light-table-discussion/c/XNNi2yx...

The LT blog also had a few updates as the community drove the project forward for a few more releases.

http://lighttable.com/2015/12/10/light-table-0-8-0/

http://lighttable.com/2017/01/27/light-table-roadmap-2017/

http://lighttable.com/2019/03/31/New-year-old-plans/

Huh, not sure why the original link changed. It's still on my blog here: https://chris-granger.com/2012/04/12/light-table-a-new-ide-c...

You mention:

    The solution is to build a "graph" of archetypes to cache these results... If ComponentIds are densely packed, you can use sparse sets to cheaply jump between archetypes.

I wish that a big company like Google, Apple or Microsoft attempted something bold in that space.

Me too. I tried stirring things up a bit when we were nearing the end of Eve, but nobody bit. This is work that _can_ be done, someone just has to fund it and let it grow without too much interference. The big players certainly have the ability to take care of the first part, not so sure about the second though.

I remember the Lighttable editor making a bit of a buzz a few years ago

It's hard to believe.. but that was 8(!) years ago now. Time sure does fly.

The 17 lines don't include the parser or the code generator, which most people would count as "part of a compiler" in a practical sense. They are usually the most mechanical parts of a compiler though, so there's relatively little to be excited about in them.

Yeah, there are many others I'd add to a more general list - Carmack, Bellard, Wirth, the folks from Our Machinery, etc. I was just referencing the people specifically mentioned in the original post.

I'm working on tools/interfaces at Relational AI, which is doing really cool work in the declarative languages space. It was started by several of the folks whose papers were foundational to Eve. :)

I agree with the post, though as others have pointed out, it doesn't really dive into the fact that this problem is systemic and would require a shift in incentive structure.

I think the last quote you have is one of the most important missing pieces for making a meaningful change in this space. A lot of people want something better, but right now, as a community, I don't think we really know what that is. What is the complete story for an ideal version of software development? And by that I don't mean idealized examples, I mean the ideal version of the real process we have to go through. What does perfect look like in the world of changing requirements, shifting teams, legacy systems, crappy APIs, and insufficient budgets? If we could show that - not the simple examples we had for Eve, but something that addresses the raw reality of engineering - I think it would just be a matter of beating the drum.

I might strengthen your argument even more and say it's largely a non-technical problem. We have had the tools necessary to build good software for a long time. As others have pointed out, I think a lot of this comes down to incentives and the fact that no one has demonstrated the tradeoff in a compelling way so far.

I find it really interesting that no one in the future of programming/coding community has been able to really articulate or demonstrate what an "ideal" version of software engineering would be like. What would the perfect project look like both socially and technically? What would I gain and what would I give up to have that? Can you demonstrate it beyond the handpicked examples you'll start with? We definitely didn't get there.

It's much harder to create a clear narrative around the social aspects of engineering, but it's not impossible - we weren't talking about agile 20 years ago. The question is can we come up with a complete system that resonates enough with people to actually push behavior change through? Solving that is very different than building the next great language or framework. It requires starting a movement and capturing a belief that the community has in some actionable form.

I've been thinking a lot about all of this since we closed down Eve. I've also been working on a few things. :)

Depends on your definition of genius, but I definitely agree that these folks don't quite hold up the sentiment that "anyone can do it." I would put Martin Thompson, Raph Levien, and Jonathan Blow at least in the top 0.1% of programmers.

They are great examples for his overall point though. It probably would've been better just to leave out the genius bit and talk about them as folks proving it can be done.

Huh, does MIRI mean that rust is actually doing some form of partial evaluation during compilation? What's the best way to learn more about that?

There were a few directions we went hereafter. One was thinking of the world in terms of simulations, another was statemachine oriented. There are a few things that were never quite right with the set of semantics here: things that really were an ordered set of steps (far fewer things than you think, but still some) were pretty hard to express and the distinction between commit and bind caused some weird things to happen at the edges. The former could be fixed by directly integrating statemachines into the language. The latter remains a bit of an open problem. Some things you want to manually manage the lifetime of, others you want to be tied to their supports. The problem is that you often want to mix and match those freely, which is just a tough thing to reason about.

Taking a stronger stance that Eve was still a bit too much like programming, and not as directly modeling as we meant for it to be, can take you to some interesting places too.

I completely agree that the semantics are probably the most overall useful thing we presented in this version of Eve. Having the world just be views over a datastore that you don't have to manage in any meaningful way is hugely liberating coming from the way we program now. You don't have to worry so much about how things are "placed" - you can always access them wherever and however you want. Bloom gave us a nice way of thinking about how to deal with change in that context so that it's not just some giant pile of scary mutable state.

Removing the fetching and placing of data is probably the biggest source of Eve's efficiency too. It turns out a lot of the code we write is nothing more than plumbing things into the correct location.

yeah, that was a very early version that we never released publicly, but was effectively a cleaned up version "Aurora," which I presented at Strange Loop once upon a time ago.

For what it's worth, I have used Vim since ~2004. When I built Light Table, one of the first things I did was put a Vim mode into it. I understand pretty well why at least I use Vim. I also believe that these days there's relatively little payoff given the cost of investment. Rarely is the speed of text-editing the bottleneck in building something real. Moreover, there are far more powerful ways to interact with code than at the text buffer level.

HARC no longer exists unfortunately.

Perhaps, but it's pretty easy to make judgements from the sideline about what people should or shouldn't do.

I was the PM on Visual Studio. I can assure the things I showed in Light Table weren't there. Nor were they in Eclipse, as I studied that as well. You can pick and choose any number of things from our work over the years and say "Hey, but this looks like that" - I'm sure it does, but the question is does it work like that? Does it enable you the way the things we showed did?

Light Table was also ultimately abandoned before completion.

Unfortunately even after paying ourselves only just enough to live in the area (around 40% of what my peers were paid), we needed to find a way to eat. No software is ever done, but we had more than 40,000 people using it and Light Table was a stated influence on tools at Apple, Google, and Microsoft. We did our best. Was there more to do? Of course, there always is, but at some point we had to make the hard decision to leave LT in the hands of the community. I'm curious what direction you think we should've gone instead?

Perhaps Granger should rein in his ambitions somewhat

Maybe, but at the same time, we had people willing to let us try. You paid literally nothing for access to our work, nor to the effects our research had on others, so I'm not sure why there's this much negativity here. We need people testing the fringes because where we are is so far from where we could be. Lots of things did work in Eve and there were cases where we were so much more efficient it hardly felt like we were "programming" at all anymore. We've shared everything we've done and we'll continue telling more people about it in the hope that others can benefit from what we've learned.

And yet you're the one acting hurt, while I'm the one having to shutdown the project. That's very different from the HN that rallied us to do our kickstarter in the first place - the one that encouraged innovation and trying to do crazy things in the off chance that they work. There's so much more to do and I sincerely hope HN doesn't become so cynical and demeaning that it's not worth sharing people's efforts here.

Hey Michael! Congrats on the launch :)

Providing a one-size-fits-all solution here is probably impossible, but it seems like it would be nice to provide some mechanism to be notified that you're making edits based on stale information. If such a mechanism existed, it would be easy to add a bunch of canned merge strategies. In doing so you can probably teach people a little bit about the pitfalls they're likely to run into (these sorts of bugs are insanely difficult to track down), while not really making them do much work.

The approach we've taken in Eve is that we can't solve all these problems for you, but we can at least let you know that things can go sideways and prompt you to make a deliberate decision about what should happen. It's amazing how helpful that ends up being.

How do you deal with consistency in the offline case? E.g. If I make a bunch of edits while disconnected, and others have made edits while connected, how are the conflicts resolved?

do you have any pointers for learning about it? Looking up "Sun Dev Guide" didn't seem to find me anything related to GUI editors.

Is there anything written about your multi-way join algorithm? We've done some really interesting implementations of Generic Join and the like and are always interested in hearing about other potentially better join algorithms.

That's not the standard meaning in my experience. Regex's are often used for extracting information from larger strings, forcing a full match by default would be pretty counter to their purpose.

That would also make this question specific to python, which wasn't at all what I was thinking about. It's a question about regular expressions, it should be as agnostic as possible to implementation.

Question 9 appears to have two answers that are correct, but only one is accepted.

Yeah, there are a lot of interesting benefits to synchronous programming that haven't been explored in a wider context and we're excited to be able to do so. Figuring out how to actually implement Eve's semantics has been quite a quest and unfortunately the implementations of most of those languages don't really fit us. Fortunately, we've put some really interesting things together lately that have produced some very surprising performance numbers for us, so hopefully that's finally resolved and we can move on to how GALS and the like apply in our world. :)

That must be a different Aurora. The one mentioned here was never really released, only demo'd at Strange Loop. We subsequently went on to turn that into Eve[1].

[1] http://witheve.com

Eve[1] is in the process of becoming a lot more real than it has been up to this point. We've found a great model and discovered a way to build a high performance implementation of it, which means much of the foundational research is finally in place. Over the next couple of weeks, we'll be revamping our website, docs, etc to help people get started building real things with it. :)

There's going to be a lot of really exciting stuff coming over the next few months. We've gathered a set of ideas, evidence, and implementations that have certainly blown us away - we hope others will find it valuable too.

EDIT: I realized I didn't address your initial question. Fwiw, we just recently crossed a really big milestone in terms of usage - more than 40,000 people have now played around with Eve on http://play.witheve.com and we've learned a ton from that experience. Part of the website revamp will be making that workflow simpler and nicer. We have a surprisingly high conversion rate (> 30%), so hopefully we can help smooth out that experience and begin to grow the community more and more.

[1] http://witheve.com

It was indeed, though Eve is a much more grown up idea than Aurora ever was. It's amazing how little we really understood back then and how far we've come. :)

In terms of #3, if you're yielding until the next requestAnimationFrame, the browser is telling you when you have the opportunity to do more work. Is there something that isn't covered by that?

they're already fast enough that most of the time is spent in rendering, not in javascript DOM manipulation

That hasn't been my experience, but it's been awhile since I've benchmarked any of the frameworks in common use. Change tracking, diffing, and then the dom calls have all been the bulk of the work in large updates. Assuming you're doing those in a dom fragment I'm not sure how "rendering time" (I'm taking that to mean compositing and painting?) could be the bottleneck in that scenario.

You could do even better than this if you wanted. Since there are only between 100-200 dom attributes, you can pack that into the opcode itself. Assuming 6 bits for the opcode (64 ops) and 8 bits for the dom attribute (256 attrs), you still have another 18bits to play with in a 32 bit int. If you wanted to allow arbitrary attributes, you could dictionary encode them and then use the full 26 bits for a total of 67M possible attribute names. Alternatively, you can reduce that entire op down to just the opcode by dictionary encoding the operand and packing that in as well. How many programs have more than 262,000 static strings? :)

Compared to the string encoding above we went from 14 chars (1-4 bytes each, we'll just say 2) at 28 bytes to 4 bytes for our 32 bit int.

In this particular case, I believe there are 3 reasons:

1) Size - turning each op into a couple of bytes means that the size of your template is significantly smaller. If each instruction is 4 bytes, I could get ~20 instructions in the space of just one function with a setAttribute call:

    function t(e) {e.setAttribute("id", "bar")}

2) Parsing speed - given the size of the JS that templates produce, you start running into JS parsing performance. Just by volume you're going to eat an insane amount of time not just downloading the JS but then trying to turn it into something executable. A correctly implemented bytecode VM could easily beat the cost of parsing.

3) Scheduling - If you just produce raw JS code, you don't have much room to dictate how it executes. Since glimmer's goal is to never miss a frame, they're going to have to take control of the work that gets executed to make sure that they always pause at a frame boundary. That's a much more straightforward thing to do in a VM, where pausing work is just a matter of yielding the interpreter loop. This gives you complete control over how you schedule the work from the ground up. Have some huge dom tree to render? Split it across 10 frames without doing a bunch of control inversion.

In terms of cost, I haven't looked at their implementation, but I assume these guys did their homework. You can implement interpreters that execute instructions in a few nanoseconds without too much effort. If you really put in the effort, you can do it subnano, but that's outside of the scope of handwritten JS. Even if the overhead was 20x a normal call, the cost of the operations this interpreter is running makes that a rounding error. The DOM is slow and the other benefits almost assuredly outweigh whatever tiny cost they're paying at the per instruction level.

There are lots of other potential benefits as well: opportunities for specialized optimizations over the bytecode (you could basically do your own domain specific jit), ease of implementing the base VM for different targets, and so on. There are relatively few times when writing your own interpreter probably makes sense, but it seems like this architecture would give them a ton of headroom to do some great stuff down the line.