The slow march of progress in programming language tooling

https://earthly.dev/blog/programming-language-improvements/

by kiyanwang • 4 years ago

135 143 4 years ago

Author here. This was fun to write. I think there are counter-examples, but this was my main idea:

    When a new developer tooling innovation is discovered, newer programming languages get a chance to bake that innovation into their language tooling. Doing so gives them an incremental advantage, and these increments add up over time to a better developer experience.

    So newer languages have one clear, well thought out way to do something, and older languages will have either many contradictory ways, or no ways at all, to do the same thing. And this makes older languages feel old.

verdverm2• 4 years ago

I'm surprised you didn't talk about Go's novel dep management system. They have put as much thought into that design as the stdlib

https://verdverm.com/go-mods/

https://go.dev/ref/mod

adamgordonbell4• 4 years ago

I'm not an advanced user of either but both go.mod and cargo.toml seem like great examples of well thought out approaches to a problem, where they've learned from previous languages and came up with a pretty polished solution.

civilized• 4 years ago

Great post. Now, why can't someone just come up with a great tooling system for a "dreaded" language, put it out there, and everyone start using that when they see it's better? Doesn't the answer just go back to the Green and Brown Language split? People who use these languages mostly work on legacy projects where they don't get to decide the tooling used. Because otherwise why would they use that language with all the trendy new options available?

adamgordonbell4• 4 years ago

Thanks for reading.

Yeah true, legacy code must be a factor. If most people in C++ are working in legacy code bases that don't leverage many thrid-party packages then I guess there is little desire to standardize on a third-party package manager.

Another issue is more social. C++ has lots of package managers which is almost worse than declaring Conan the standard. Communities standards are hard to retro fit I guess?

(Not to pick on C++ by the way. Every popular language eventually sees itself competing with newer incumbents. It's like the innovators dilemma for PL. )

lesam• 4 years ago

I’ve worked in C++ and picked my own tools. The tooling experience in Go is much more polished for a few reasons.

First, as the author states, it’s hard to get consensus on a tool if it wasn’t baked in from the beginning. I don’t think there will ever be a dependency manager or especially an option-free formatter for C++ with the same adoption as gofmt or go modules.

Second, it helps a lot to have the compiler/runtime authors think about the tooling. Go runtime developers think about pprof. GCC developers don’t think about the UX of valgrind. C++ standard authors don’t write PR’s for code formatting tools when they change the language spec.

twic2• 4 years ago

I'm trying to think of examples of trying this for C or C++.

* Batteries-Included Standard Library - can't actually be a standard library, but could be the next best thing, a widely-used library; things like NSPR and APR for C, or Boost for C++

* Third-Party Package Repositories - Conan, which seems to be healthy, but hasn't taken over the world

* Documentation - Doxygen, which has been very successful

* Write Once, Run Most Places - little progress towards this for C, but the people in the C++ standards process do at least think about portability; i think there is an inherent tension between portability and being close to the metal which means it is always going to be a lower priority for C or C++

* Package Managers - see under 'Third-Party Package Repositories'; not clear to me to what extent this is a different thing

* Code Formatters - these have long existed, i suspect clang-format is most popular, but it is far from universal; what's missing is a universally-accepted strict formatting standard, as found in Go and Rust, and arguably now Python

So, it seems that these things do get retrofitted, and do get adopted, but not to the same extent as if they were there from the beginning of the language.

cercatrova• 4 years ago

That's basically what the Laravel stack does for PHP. It's great, very ergonomic, I'd use it all the time if not for having to write PHP which I don't wish to do anymore.

saghm• 4 years ago

Now, why can't someone just come up with a great tooling system for a "dreaded" language, put it out there, and everyone start using that when they see it's better?

I think the first two steps sometimes happen, but the last one is where things get stuck. Often times a better experience means incompatibility with the previous solutions, so adoption happens slowly if at all. Then eventually yet another incompatible solution comes out, and things fragment.

Gunax• 4 years ago

It makes me think there isn't any reason old languages cannot adopt these.

Or even, come up with a new, batteries included version.

Eg. keep the Java syntax, but add one standard package manager (instead of ant, mvn, gradle), one standard test framework, standard formatter, etc. Call it 'Java++' or whatever.

setr2• 4 years ago

You need to incentivize codebases to be moved to the new thing, or you have to rely on incremental adoption by new projects. It’s a tragedy of the commons — for any individual codebase it’s hardly worth the trouble (My codebase(s) already has a standard formatter / package manager / whatever; what do I get by changing?) but it’s worth it every other codebase was already moved.

Alternatively you build a thing that every other system could be auto-migrated to, which inevitably means supporting every possible feature of everything in play and slapping an ANSI standard label on it; and then watch as everyone extends the standard arbitrarily

oaiey3• 4 years ago

There are low hanging fruits which would have adoption problems but that can be navigated.

However, there are very painful things like modern compilers which are massive undertakings for an old languages. It cost Microsoft 3-4 years to rewrite the C# compiler to be modern. But the benefit is a language server which keeps pace with the compiler and refactorings on the same AST (aka more stable/reliable).

oaiey3• 4 years ago

There are some absolutism in there which are not correct

Because of this, Go developers lean on the standard library more than many other communities and generally hold it in high regard.

As a .NET developer I can just laugh here. Compared to JS and C/C++ this is accurate, but .NET, Java, PHP, Python this is the normal case. I would say the majority of platforms for application writing nowadays come like that. Which supports your argument, just that it is the norm already for a long time.

To your generalized argument: i do not follow that. Yes, modern languages have the benefit of knowledge, but the old stacks are massively catching up. Package manager and libraries is a solved problem (except for C++). The most interesting aspect this should focus about are modern compilers. The ones like C# or TypeScript which expose their services (AST, type checking, etc) as language servers. Here old platforms (PHP, C++, Java, ...) have massive pains because rewriting the compiler is the opposite of fun and new languages could start like that (e.g. TypeScript).

idoubtit• 4 years ago

> Because of this, Go developers lean on the standard library more than many other communities and generally hold it in high regard.
[...] Compared to JS and C/C++ this is accurate, but .NET, Java, PHP, Python this is the normal case.

This is not my experience with Python, and even less for PHP. There are too many dark corners in the standard libraries of these languages to hold them in high respect. The standard library of PHP is an inconsistent mess, with many documentation holes.

For instance, there are several ways to send HTTP queries using the PHP STL. Apart from very basic needs, the usual way is through the libCURL integrated into the STL. Unfortunately this library is just a port from C, far from a native syntax and barely documented. For any advanced usage, the developer has to read the C documentation, or switch to a non-standard library.

tialaramex5• 4 years ago

Two things this doesn't mention about Rust that I think we can expect to see in new languages:

1. Version control. If you ask Cargo to make you a new Rust project, and you have git (which, if you're a software developer you almost certainly do) it makes you a Git repository for the new project, with its target directory ignored.

As with formatting you might have very strong opinions about revision control, or you might be subject to a company policy, maybe you must use Bob's Special Version Control Program for all Company Projects. Cargo won't stop you doing that, but if you don't have an opinion you get git which is definitely better than nothing. And if you want to, somebody can teach Cargo about Bob's SVCP and make 'svcp' one of the possible choices in the config for the few people who do care.

2. Hello World. If you ask Cargo for a new Application, the project it gives you prints "Hello, World!" because why not. Rust's boilerplate for such an application isn't very heavy, but like a blank piece of paper is intimidating when writing an essay or novel, it's nice to have something to start from. Step #1 remove the part where it prints "Hello, World!". Step #2 write my actual program. Hey, we're off to a great start already.

This also means you have a correct, valid program to check your tooling. A new Cargo Application is a Hello, World program that will compile, and run, showing that your tools work as intended before you write a single line of Rust. It doesn't have any tests, nor any documentation, but since it's a working program the test infrastructure and doc builder work, the results will just be not very exciting.

usrnm• 4 years ago

Any half-decent IDE created in this century gives you both these features and much more. I remember selecting the type of the new project in Visual Studio 6.0 and getting either a console "Hello, world" or a full-blown GUI application. VS 6.0 is literally older than some engineers I work with today

HexDecOctBin5• 4 years ago

Too many people believe in the superiority of Linux as an OS for software development that they would consider it beneath them to even notice Windows toolchains, and would then be flabbergasted by the idea of project templates.

tialaramex5• 4 years ago

Any half-decent IDE created in this century gives you both these features and much more

If you've paid for the expensive "Professional" Visual Studio .NET in 2002, you do indeed get version control. But it's Visual SourceSafe, which is garbage†. Fortunately, it's optional, so you probably just never switch it on (it's not default) and don't use any version control at all, so you're back to square one.

If you're just using the (cheaper) Microsoft Visual C++.NET for example - whose users I imagine would have believed they had an IDE - you don't get Visual SourceSafe. As we saw that's probably no problem since you don't want VSS anyway, but it does mean you do not, in fact, get version control out of the box, you're going to need to pay Microsoft $$$ (or, try using anybody else's offerings which are better and many of them are free...)

Either way, you're not getting that comparable experience.

† It's markedly worse than CVS, which itself is awful. Microsoft actually sold VSS for money rather than giving it away and so there was (until 2005) a team at Microsoft maintaining it. As I understand it the revision control of the Visual SourceSafe software itself was eventually... a Perforce repo (technically "Source Depot" Microsoft's internal-use licensed fork of Perforce's software). This is not an endorsement of your confidence in your own product.

davidkunz• 4 years ago

And with `cargo init --lib` you even get a green unit test!

2143• 4 years ago

blank piece of paper is intimidating when writing an essay or novel

Who starts off of a template when writing a novel?!

armchairhacker• 4 years ago

Tooling is extremely important. However there’s one thing I think is worth mentioning:

Java tooling (IntelliJ and Lombok) is incredible. Yet Java is still on the most dreaded list.

Why is that? Honestly I don’t know, and I actually think Java is a good language exclusively because of its tooling. That being said, current tooling doesn’t fix all of Java’s flaws.

- Gradle and maven are reliable like npm and cargo, but super verbose and confusing

- Standard library has some issues and quirks. Like URL#equals sends a network request, apparently that will never be changed. Some other packages (log4j, JAX) are similarly bloated

- The “billion dollar mistake” untyped null. People tried to fix this with @NonNull annotations but it’s not far enough.

- Java is in a lot of legacy software, which often cannot use Lombok and requires Java 8 or even Java 7.

- I just have to say, IntelliJ and Lombok really do fix Java’s verbosity. Both reading and writing. Indeed, tooling is the new syntax. But maybe some disagree.

ModernMech3• 4 years ago

Because Java is taught to kids as their first language in high school and college. It’s associated with homework, exams, and boring corporate jobs.

celdon25• 4 years ago

No.

Edit: apparently some people think I am disagreeing with the statement in and of itself. Yes, Java is a homework language, but it’s real problems are far worse than that. Worst of all is the generation of developers that it produced.

kabdib• 4 years ago

Java was my first introduction to full bull-goose, consultant-class enterprise architecture, where I think that salaries and hourly rates are proportional to the number of cascaded factory classes your solution requires. At least four or five, or you're bush leagues.

Wait, am I sounding shocked, wounded and bitter again?

"This thing is so enterprise-y that the documentation alone can send email."

shepherdjerred3• 4 years ago

My first introduction to Java was via Minecraft. Afterwards I used it extensively at AWS.

It's not a boring language, but it is often used in enterprise which gives it a bad reputation.

twic2• 4 years ago

Agreed (although i hate Lombok myself, as being altogether too much magic; happily, standard Java language improvements now provide a lot of the comforts that Lombok does).

But other languages have quirks on a par with Java. JavaScript is quirkier, and Ruby is basically a DSL for composing quirks.

I see two areas where Java fell more dynamic competitors.

Firstly, Java does not have a great story about being able to open up an editor, write some code, and ship it. Using a simple text editor is too tiring, and the extremely sophisticated IDEs take a lot of learning. Java has never been well supported by the middle ground of Sublime Text-esque programmer's editors. Meanwhile, the standard tooling (javac and jar) is simple, but using it to get to production is cumbersome, and the more sophisticated build tools are not standard, and either verbose and maddeningly inflexible (Maven), or crammed with arcane secrets (Gradle). And none of them make starting a new project as easy as 'rails new' or 'npm create-react-app' (Maven has archetypes for this, but i don't think they ever got really popular).

Secondly, Java does not have a really good framework for being highly productive at building webapps right out of the gate. By which i mean Rails. There's absolutely no technical reason you couldn't build a Rails equivalent in Java. I am astounded that (AFAIK!) there was not an urgent effort to do a straight port of Rails to Java once it became clear that Ruby was eating Java from the bottom up. Meanwhile, Java had Spring, but until Spring Boot, Spring was nowhere as easy and consistent to use as Rails; Spring Boot has helped, but by slathering another layer of even more complexity on top. I have never used Dropwizard, but maybe that's closer to the Rails spirit. The more 'sophisticated' frameworks like Play and Vert.x are not the answer, because they're so tied up with particular overly clever concepts.

rightbyte2• 4 years ago

There's absolutely no technical reason you couldn't build a Rails equivalent in Java.

Groovy with Grails? Never tried Grails so dunno of it was mature enought.

rr808• 4 years ago

Java is great, I think many wrinkles need some active deprecation. Its interesting that the blog likes the extensive standard libraries, but Java is littered with old APIs and language features that aren't used or liked any more. Newer languages avoid having these old warts, its only matter of time though.

adamgordonbell4• 4 years ago

Newer languages avoid having these old warts, its only matter of time though.

That is sort of the summary of the post.

Banana6993• 4 years ago

Honestly I don’t know

I do, because I'm a Java hater. Here are the reason I hate it for

- Baking the difference between primitives and objects into the language itself : an ugly mistake with far reaching consequences, made by a language designed in 1995 while another designed in 1980 (smalltalk), in 1991 (Python) and 1995 (Ruby) all didn't fall for it.

The difference is an irrelevant VM-level optimization detail, there is no reason to uglify the human-level language with it. Once the initial mistake has been made, the correct response was NOT to make the even uglier hack of wrapper classes, but to make the primitives objects in the newer releases of the language, this won't break old code, as valid uses of objects are a superset of valid uses of primitives, except perhaps that objects need to be allocated explictely with "new", but this can be a special case for primtives (i.e. "int is a special kind of object that you don't need to allocate explicitly"). The compiler can figure out whether it needs to be represented as objects or as primitives, you can leave hooks and knobs for people to tell the compiler they need to the primitives to be represented as primitves, but it shouldn't be mandatory.

- Baking in choices about object representations : Like the fact that objects are always passed by reference, or that they are always allocated on the heap. Why the "always" part ? why not give developers the choice between pass-by-value and pass-by-reference like C# does ? why not give developers the choice to allocate on the stack (and complain as loud as you want when they want to do something unsafe with it, like escaping from methods), which, unfortunately, even C# doesn't ?

Everytime you see something like "foo deepCopy()" that's a failure of the language, forcing you to explicitely pay attention to the fact that foo objects need to be copied deeply everytime they are copied, instead of just once when you define the object by marking it as a "struct" or whatever word to signify that object has value semantics, and then deep copy is just assignment or passing as a parameter. Why make it the default to be inefficient with the heap when it's very easy to give developers the choice to be efficient in situations where it's always safe ?

- No operator overloading : I get the hate, it's a powerful tool. But it's misguided to ban it, operators should not be special, languages like Haskell and Raku go even further and allow you to define new operators entirely and control their predence and other things. You don't need to go that far, why can't objects use the already built-in symbols the language support ? because it might be confusing ? anything can be confusing, you can write assembly in any programming language, and it will be even worse than assembly because of the more powerful and obscure abstractions.

- Generics : The overall theme of forcing you to do things its way seems to a staple with java. Why do I need to use type-erased generics ? why shouldn't I get the choice to specify whether I need a new class generated for runtime efficiency or use the type-erased catch-all for size efficiency? there is no need to bake VM-level support for this, it can all be done at compile time (possibly with help of additional metadata files or special fields in the .class of the generic type).

- Overall verbosity : Why "extends" and "implements" ? do you really need to know whether you're inheriting a class or an interface ? and can't those be lighter symbols like "<" and ":" perhaps ? why is "private/public/protected" a must in front of every method and field ? most people align fields and methods by their visibility, C++'s way is that you declare "public:" and then everything declared below that is public. In the worst case you can always recover Java's way by "public : <method> ; private : <method> ; public : <method>" and so on, but it's nice to at least have the choice of not repeating yourself.

Why aren't any constructors generated ? there are at least 2 very obvious ones : the empty one, and the one that assigns all the non-defaulted fields (and can take optional arguments to override the default fields). Why aren't generated getters and setters available with a small and light request, like C#'s "get ; set ;" ? Java's design is just full of things like this. It feels like a weird sort of disrespect for your time, "yeah you must write those routine 25 lines of code all by yourself, you have anything better to do?", how about actually writing my application instead of pleasing your language with weird and unnecessary incantations ? It's like a modern COBOL.

- Horrible OOP excesses : Not really the language's fault (except that it encourages verbosity and loves it) and already mentioned, but worth mentioning again.

Overall, I treat java as assembly. I write kotlin in my spare time, and whenever I'm confused about the semantics of some construct I make intellij show the bytecode then hit "decompile" to see a Java rendition of the code, the exact semantics will be obvious but verbose. A language that took this literally is Xtend, a high-level augmented java which transpiles to java and is a strict superset of it, but with option that the Xtend compiler figures out all the verbosity for you. Groovy also takes the "Superset and Augment" approach but doesn't transpile. And off course Kotlin is very good with it's interoperability, every JVM language is but Kotlin's mixture of being close to Java semantics (unlike say, Scala or Clojure) and Intellij excellent support for mixed projects makes it at least somewhat special.

I like the JVM and it's cutting edge research and performance, and these days the Java standard writers seem to show signs of finally waking up to reality after years of being behind every mainstream language, and they regularly augment and modernize the language. But you can't undo 20 years or so of bad design, not easily and not painlessly.

Indeed, tooling is the new syntax.

Very much agreed, long long gone are the days when a compiler or an interpeter is the only thing expected out of a language. But it's not a panacea to treat any bad design, at best it's just a band-aid for bad designs that makes them barely berable. The language has to be designed from the start with the knowledge of "this is going to run in an IDE" baked in to make full use of the full range of fantastic things an IDE can do.

[1] https://www.eclipse.org/xtend/

EdwardDiego• 4 years ago

why is "private/public/protected" a must in front of every method and field ?

You can omit them, and then the member is package private.

anonymoushn4• 4 years ago

Everytime you see something like "foo deepCopy()" that's a failure of the language, forcing you to explicitely pay attention to the fact that foo objects need to be copied deeply everytime they are copied, instead of just once when you define the object by marking it as a "struct" or whatever word to signify that object has value semantics, and then deep copy is just assignment or passing as a parameter. Why make it the default to be inefficient with the heap when it's very easy to give developers the choice to be efficient in situations where it's always safe ?

It's not clear that running a bunch of code in the form of a copy constructor and potentially performing a bunch of heap allocations whenever someone returns a value, passes an argument, assigns a variable, or evaluates a temporary as part of an expression is a good thing.

- Generics : The overall theme of forcing you to do things its way seems to a staple with java. Why do I need to use type-erased generics ? why shouldn't I get the choice to specify whether I need a new class generated for runtime efficiency or use the type-erased catch-all for size efficiency? there is no need to bake VM-level support for this, it can all be done at compile time (possibly with help of additional metadata files or special fields in the .class of the generic type).

There ends up being no difference, right? Your generic ArrayList is just an ArrayList of pointers to Object no matter what. You never had the option of declaring an ArrayList of structs.

shepherdjerred3• 4 years ago

It sounds like you just want to write C#. The reason that Java is so successful is precisely because of the points you mentioned. If they broke backward compatibility to fix the warts then it wouldn't be the popular language that it turned out to be.

Modern Java (11+, and especially 17) has greatly improved the language. Sure, there's still type erasure and primitives, but it is adding a lot of features that matter like record types, more efficient GC, deprecation of older classes, and more fleshed out APIs.

I can see why you'd dislike Java. There are a lot of things that could be improved. There are times when it is the right language and times when it isn't.

lenkite• 4 years ago

Baking the difference between primitives and objects into the language itself

This is not unique to Java and this difference is present even in a modern language like Rust. Primitives are copied by value, objects are copied by reference. If you want copy by value use Java records: https://www.baeldung.com/java-record-keyword

Baking in choices about object representations : Like the fact that objects are always passed by reference, or that they are always allocated on the heap

Because the JVM does escape analysis and performs automatic stack allocation. The JVM wants to retain control here.

Generics

Type erased generics were a deliberate design decision implemented for backward compatibility. .net had to introduce brand new collections while java programs could continue to work. To be honest, I felt this was stupid too. A clean break would have been better.

Overall verbosity

Deliberate design decision. Though frankly, I personally prefer extends and implements. This is just done once at class declaration and allows for easy and no-nonsense grepping of sources from the CLI.

Why aren't any constructors generated

Satisfied by records now which are meant as the placement for POJO's. Classes should now be used for services - for which you _don't_ want generated constructors.

Horrible OOP excesses

Looks like you are ~10 years out of date on Java tech and community. Perhaps look at some of the modern java libraries ? Look at GraalVM ?

I prefer coding in Modern Java over Kotlin. Kotlin readability is poor.

raiph• 4 years ago

why can't objects use the already built-in symbols the language support?

In Raku they can. But one can also introduce new ones. More generally, it's nice Raku makes it easy for a dev to write a module that lets other devs write code and get nice output like this:

    use Physics::Measure :ALL;

    # Define a distance and a time
    d = 42m; say d;     # 42 m (Length)
    t = 10s; say t;     # 10 s (Time)

    # Calculate speed and acceleration
    u = d / t; say u;   # 4.2 m/s (Speed)
    a = u / t; say a;   # 0.42 m/s^2 (Acceleration)

    # As a special treat it allows working with measurement errors:
    d = 10m ± 1;
    t = 8s ± 2;
    say d / t           # 1.25m/s ±0.4

(With thanks to SteveP for the module and niner for the above example code.)

puffoflogic• 4 years ago

Bad languages leave lots of room for tooling. Good languages leave less room.

Only the deluded run ubsan/asan/tsan on safe rust. Does that mean safe rust has less tooling than other languages? Well, yes, because that tooling is redundant and irrelevant.

62951413• 4 years ago

* I find it hard to believe anyone with experience on the JVM would choose Java over Scala or Kotlin. The latter actually addresses most of your concerns. The existence of Lombok is a strong argument in favor of a more modern language.

* I'd violently disagree with anyone grouping gradle and maven together. One of them is declarative the other is a mess surpassed only by SBT.

* I don't get why nulls are so hated by people who are not real FP purists. How are they different from ", err := .. if err != nil .." in golang which people do a lot and seem to enjoy (in comparison with exceptions and even Optionals)? NPEs are not common in development and very unusual in prod. I'm not sure I like how much ceremony there's in Kotlin to keep track of nullable things.

* I hate to break the news but log4j was dropped in favor of slf4J&Co a decade ago. There are better restful frameworks such as akka-http and sparkjava.

shepherdjerred3• 4 years ago

* I'd prefer Java (with Lombok) over Scala or Kotlin

* How often do you see both Gradle and Maven in a project? Not often. They're similar tools.

* [0]

* This is completely untrue. log4j is still widely used, even in new projects. sl4j is simply a universal API that allows applications to choose their logging backend. sl4j and log4j aren't mutually exclusive.

* I was going to say that spark hasn't been updated in years, but it looks like they posted a release [1]!

[0] https://hackernoon.com/null-the-billion-dollar-mistake-8t5z3...

[1] https://mvnrepository.com/artifact/com.sparkjava/spark-core/...

Tainnor• 4 years ago

I don't get why nulls are so hated by people who are not real FP purists. How are they different from ", err := .. if err != nil .." in golang which people do a lot and seem to enjoy (in comparison with exceptions and even Optionals)? NPEs are not common in development and very unusual in prod. I'm not sure I like how much ceremony there's in Kotlin to keep track of nullable things.

Disagree that NPEs are a rare occurrence, that Go-style error handling is universally enjoyed (I would say it's one of the top complaints people have about Go) and that Kotlin null handling uses too much ceremony.

And as for that last point, idiomatic Kotlin code (or, indeed, code in any language that has null-safety) will tend to eliminate nullability from types as much as possible, it's not really the case that most function arguments, return values, etc. really need to be nullable.

anonymoushn4• 4 years ago

Java tooling (IntelliJ and Lombok) is incredible

Until a Java IDE can jump from an annotation to all the code that actually processes that annotation in the context of the current project, Java tooling lags substantially behind ctags.

- Gradle and maven are reliable like npm and cargo, but super verbose and confusing

At one recent-ish job, I wrote some scripts to download gradle dependencies via wget and stuff them into gradle's cache because gradle was incapable of resuming downloads and multibillion dollar companies are incapable of running maven repositories on the same continent as their developers or delivering HTTP responses from maven repositories across oceans. The resulting amalgam of bash, wget, and gradle was still quite slow because it takes forever to run gradle long enough for gradle to tell you what it failed to download.

- Standard library has some issues and quirks. Like URL#equals sends a network request

What.

- I just have to say, IntelliJ and Lombok really do fix Java’s verbosity. Both reading and writing. Indeed, tooling is the new syntax. But maybe some disagree.

It may be useful to compare an "idiomatic Spring Boot" implementation of a small collection of related REST endpoints for a moderately complex model to an implementation in any other language and calculate how many times more files and how many times more lines of code you end up with.

mikewarot• 4 years ago

The article states that Rust Docs are the best form of documentation, "state of the art"

I'm all for good documentation, but why embed it in the source code? Why not have good documentation files that stand alone?

Doing it that way repeats the mistake of HTML. Annotation should be separate from content, not made into a sausage with it.

gfody• 4 years ago

I'm not sure if rust does this but C#'s embedded docs are integrated throughout the toolchain so they appear alongside intellisense as your writing or navigating etc. Keeping them with the code makes sense to me because they have the same reasons to change.

notriddle• 4 years ago

https://rust-analyzer.github.io/manual.html#hover

ithkuil• 4 years ago

In rustdoc you actually can include arbitrary markdown files, unlike godoc.

formerly_proven2• 4 years ago

rustdoc may be good for reference documentation (as a Rust noob, I don't really think it is, most docs are pretty messy and the UX isn't very good, it's often better/easier to read the docs in-situ in the code), but it seems to lack any and all facilities for non-refence docs, which most Rust projects basically don't have at all, some put in the top-level module docs (which usually limits the size) and a few have completely separated docs for anything non-reference. (Or one of the creative hacks to get around it, like creating empty public modules or useless macros to have additional pages).

In these areas rustdoc is a significant regression compared to say, Sphinx.

The upside of rustdoc is no settings, no setup, no friction. Which also means that the deficiencies as far as the reference functionality is concerned can actually be fixed.

nsm2• 4 years ago

As an example of documentation at the same or better standard, also look at Racket. I'd say the python ecosystem is also really getting better at this in the past few years.

burntsushi3• 4 years ago

Well, rustdoc lets you include arbitrary Markdown files. So that case is not eschewed.

With that said, they serve two different use cases. So as someone who has probably written more lines of docs than I have code in Rust, it doesn't make sense to me to ask for one and not the other.

API docs with a bit of high level prose and some examples are very likely good enough for small focused libraries.

For bigger libraries or projects, you are absolutely free to write prose that is disconnected from the API. You can do that in rustdoc or use some other tool (mdbook is very popular).

But either way, you pretty much always want at least some API docs.

Doing it that way repeats the mistake of HTML. Annotation should be separate from content, not made into a sausage with it.

This doesn't make any sense to me. The API of a library is both the signatures of items exported by the library and the documentation on those items. The docs might make certain guarantees about behavior, error/panic conditions and more. Those can't be described by the type system, and thus, the items and the docs are intricately connected. Both are required to define a library's API.

tialaramex5• 4 years ago

One crucial thing about Rust's documentation is that your examples are run by the test infrastructure.

This makes logical sense. If the programmer provided an example of how to call some_function() then we should test that example works. If it doesn't work their example was wrong, or some_function() is broken, either way the programmer needs to fix that.

For the test infrastructure to run your examples, they must be code, not some sort of annotation.

The result is now you don't ship documentation which oops, is documenting the old some_function which had different behaviour and so of course that example doesn't work any more. Sorry, we didn't rewrite the docs yet, just go read this Discord thread and the six bug tracker entries it links...

bstpierre• 4 years ago

I first learned this from python docstrings, which can include testable examples.

I think rust improves on how it’s achieved, but I’m more familiar with how go does it.

Whichever language you use I agree that it’s a great way to ensure that the docs match the package. One thing I would love to see more of is annotations in docstrings that specify when changes were made to packages. (E.g. “version 2.3, added FooFrob()”) Python standard lib does a good job of this, but I can’t think of a convenient (for developers) way to enforce this with tooling.

bobbylarrybobby• 4 years ago

My experience with most rust crates is that they do use separate markdown files for their big doc pages like readme and tutorial, getting started etc; and they include (`!#doc["readme.md"]`, or something like that) these files in their source so that they're picked up by `cargo doc`. Only actual Rust items get directly-in-source-documentation -- again, in general.

However I agree (I think) that markdown is not a great format for long form docs; you want something a bit more expressive (say, asciidoctor).

mr_toad• 4 years ago

I'm all for good documentation, but why embed it in the source code? Why not have good documentation files that stand alone?

It usually ends up out of sync with the code almost immediately. And there is a very good chance of the documentation being lost, buried in some some enterprise document management system. Sometimes I feel like Gandalf reading through stacks of scrolls trying to find thousand year old references to magical rings.

jaredcwhite• 4 years ago

The counter argument to this is that developers requiring more and more tooling surrounding what is arguably just a text file with some code in it can cause friction, yak shaving, and cognitive load over time.

Listen, there's nothing wrong with IDEs, linters, formatters, test frameworks, advanced dependency management across multiple projects & repos, yada yada yada. But there's also something to be said for being able to open up a blank file in a blank folder, writing some lines of code, and running a command to execute said code. That's it. That's the stack.

Programmers are constantly tempted throughout the ages to overcomplicate and overburden language ecosystems and stacks such that a newbie coming in has to learn an inordinate amount of knowledge about everything surrounding the language before they can even learn the language itself.

I guess what I'm trying to say is I'll always push back against this idea of "progress" == more complexity. Tools, frameworks, languages, and ecosystem evolve over time. Best practices change and adapt. That's to be expected. Is it progress though? Is it actually better?

I remain entirely skeptical.

jjoonathan• 4 years ago

Yeah, it's definitely possible to make tooling "investments" that never amortize, but I've also seen an awful lot of programmer-minutes become programmer-days when tooling degraded to the point of printf/run/printf/run/printf/run/bisect/repeat.

pixelrevision• 4 years ago

Go strikes a really good balance here.

jamincan• 4 years ago

More opinionated solutions such as gofmt/rustfmt/black seem to be popular in part because they remove some of that cognitive load.

returningfory2• 4 years ago

I don't think this is a counter argument at all? If anything, it's reinforcing the argument in the article.

But there's also something to be said for being able to open up a blank file in a blank folder, writing some lines of code, and running a command to execute said code.

E.g., in Go the tooling is so good that you can just create a file and run it using `go run .`. This is sort of the point the author is making? No need to worry about build systems, dependencies, etc.

nsm2• 4 years ago

This is a good article. It covers a bunch of "ecosystem" stuff.

At the same time I'd generally encourage a lot of developers using plain editors and modern, but not widely used languages, to give IDEs and common languages a try. That addresses the "iteration" experience.

Modern IDEs for java, C++, C# area huge step up from whatever cobbled together setup one is likely using in neovim (saying as something who mainly used vim till about 2 years ago).

VCS integration, build integration, debugging facilities, refactoring, linting etc. are just amazing. Jetbrains IdeaVIM is also really good at getting a fast editing experience. At this point, as soon as I'm beyond a small script, it is worth it to open the project in an IDE.

We still aren't at the smalltalk or lisp machine level in certain aspects, but that is more a limitation of our larger computing platforms.

tialaramex5• 4 years ago

I use an IDE for my day job (because when in Rome...) and I use Vim the rest of the time.

There genuinely are things to like about the IDE. But, they're fewer and further between than I'd anticipated. For example the debugger is more intuitive to work with. If I have no idea what's actually wrong, so that I'd be resorting to just poking around hoping I see something weird to investigate, this makes the IDE a better choice.

I expected several things to be nicer, or to be roughly the same but perhaps with a cute GUI that I'd ignore - but they were generally worse. Test infrastructure is worse: Just tell me what failed and how. Don't waste my time making some sort of chart, I'm not trying to impress some middle manager I have work to do. Revision control is worse: the IDE forgets which files are modified! Sometimes restarting it helps. Buggy software is annoying all the time, but buggy development software is awful.

Now, maybe JetBrains is amazing, I haven't used it. But I have used a famous IDE that got mentioned elsewhere in this HN discussion and it was... not good. Not so terrible that I told my boss I can't put up with it. But definitely not better than Vim.

spc476• 4 years ago

A few years ago I did just that. C (a very common langauge) and a popular IDE (either Eclipse or IntelliJ, I don't recall which one). I loaded up a simple two-file C project into the IDE and it promptly crashed.

I think I'll stick with my text editor I've been using for 30 years. I've never had a problem with editing any language.

skybrian• 4 years ago

One reason that progress is slow is that new language that comes along often have to start over. The article points out how some new languages have benefited from this, but it also means that getting to parity with the best of existing mainstream languages is quite difficult.

An exception is languages that piggyback on an existing ecosystem (for example, Kotlin on Java's ecosystem). But they're not advancing the tooling state of the art.

MBCook4• 4 years ago

There’s something I’ve never quite understood and I wondered if someone who knows this history might be able to explain it.

I’m used to Java and Maven. My package manager (mvn) downloads 3rd party packages into my local repo, and all my Java projects have access to it. So I only have to download a specific version of a specific package once.

Node and NPM came along, and I think were written by a former Java developer. But NPM puts a full (unzipped!) copy of every package in each of my projects. This leads to pointless duplication of files, and in Windows is slow as dirt due to how NTFS works.

Yarn’s PNP mode sort of solves this, and it seems like recent versions of NPM may be changing things. I’m not sure.

But why did it start that way? Does anyone know?

webkike• 4 years ago

The reason is that JavaScript is not compiled and needs the dependency’s source files to run, and those files need to be there when packaged.

wging2• 4 years ago

That's not an obstacle to a central cache - you can solve that with symlinks. A build tool should make node_modules/foo a symlink to ~/.npm/cache/foo-1.23 if foo 1.23 is what you need. A packaging tool should be able to follow symlinks easily.

jonahbenton• 4 years ago

The reason for a centralized local cache in Java is just because Java and jar files date back to the mid 1990s. Maven to the early 2000s. Storage was expensive. I wrote my first Java programs on a machine that had 16mb of RAM and a couple hundred mb of disk space.

Centralized caches of compiled code present all sorts of logistical issues when it comes to deployment. Java has suffered through several generations of uberjar-like designs. Python, also dating to the 90s, as a source based language tried to solve for both local and centralized trees and the result is a horrendous soup. In contrast, PHP- yet another 1990s language- had a model of just dumping files into the directory and serving them, which was comparatively simple and uniform.

Node, invented in the late 2000s by a C++ and Ruby programmer, was intended to simplify all sorts of aspects of writing webservers, leveraging in particular the deployment model of PHP (all needed files in source code in the local file tree) with the superior runtime performance and scalability, including support for async, of the Javascript V8 runtime, and with the benefit of having the same language both client and server side, something that was first tried with both Java and Javascript in the late 1990s but for various reasons failed. The other major envisioned benefit to Node was that it was its own webserver, and this has led to many other innovations.

That Windows still after 25 years of the NT architecture has a file system that doesn't scale is not something most tooling in the web ecosystem doesn't concern itself with. Node, like PHP, was linux first and foremost.

HTH.

wging2• 4 years ago

It's not the storage used but the repeated downloading, if I understand the complaint correctly. Avoiding the network traffic if you have an exact match should be much faster on a clean build, even if you don't care about disk space.

User23• 4 years ago

That Windows still after 25 years of the NT architecture has a file system that doesn't scale is not something most tooling in the web ecosystem doesn't concern itself with.

This isn't exactly true. It's not so much that NTFS is slow, it's that it's not optimized for the usage patterns that Linux's FS is. That being said, it's certainly possible to have high IO performance on Windows if you know what you're doing.

Sankozi• 4 years ago

Distribution problems you mention in Java are not handled by NPM at all and are not related to how you store dependencies (in Java). Not sure why you mention it.

NPM dependency resolution is neither simpler nor faster compared to Maven. You have two commands for installing dependencies:

slow one (npm install) that updates package-lock based on config file developer use

faster one (npm ci) is based on package-lock cache-like file that is hard to read, you cannot be sure package-lock file is correct for current config file without running npm install, different versions of NPM will create different package-lock files

Both commands are slower and more buggy compared to Maven. Clearing dependency directory (node_modules) and trying again installing is a common practice when working with NPM. NPM is fast and correct only if it acts as a script runner - i.e. it is calling JS libraries (that are expected to be installed, NPM will not check that for you).

I find putting dependencies inside project folder to be most stupid design decision of NPM (second one is choosing json for configuration file syntax).

NPM's simplicity seems to be related only to its implementation (but somehow it did not prevent it being the most buggy cli tool I have ever used). It is basically a dependency downloader with script runner capabilities.

waynesonfire• 4 years ago

I'm really wishing that the innovation that language server brought to the editor ecosystem can be expanded to other dev tools.

I'd love for dependancy management and build systems to be a solved problem and not reinvented by every new programming language that has gained enough popularity that some volunteer decides to build these basic dev tools.

Imagine learning one tool and using it for every language. They all solve the same problem. What's the common interface?

doliveira• 4 years ago

Nix sounds like it could be such a tool. But it seems to also suffer from Linux's wizard syndrome

Kaze404• 4 years ago

I agree with this. With Nix I can finally stop using nightmarish tools like nvm and rvm. I have my development tools where I need them: my projects. There's no reason for me to have, say, node or eslint available anywhere else.

ghoward2• 4 years ago

The common interface is dependencies and dependents, but the problem with existing build systems is that they make assumptions about those two things. They also place other restrictions.

For example, the Go build system assumes you set up the project according to the Go standard. Cargo sets things up a certain way too.

They also assume certain build tools, like the Go compiler and `rustc`.

Restrictions for ease of implementation are the reasons why every build system sucks. Figure out how to remove the restrictions and implement that, and you will have a universal build system.

Source: I am building such a build system, so whether I'm right will only become clear over time.

Too• 4 years ago

One difficulty is that dependency resolution need to be language-aware, to understand the semantics of #include, import, require, CLASSPATH, compiler -i flags etc etc.

Best would be if compiler could export this in a standard format for consumption by a build system. Like the language server.

Looking at build tools that try to be cross language, you have Bazel probably being the most serious contender, which often requires you to vendor your dependencies and rewrite the build files entirely in a different mindset than how upstream packaged it.

transfire• 4 years ago

They are dreaded because people have jobs using it.

lumb63• 4 years ago

It’s interesting to me the emphasis placed on tooling. The tooling in Rust’s ecosystem, while good, is far from my favorite part. I very much enjoy how expressive and ergonomic the language is. Coming from primarily C and C++, it was eye opening to me to see all the things I never knew I was missing out on (borrow checking, const by default, ? operator, monadic types, the list goes on…)

phlogisticfugu• 4 years ago

the old language that's a bit of a counter example to this is php.

folks came out with a package manager in composer that was just so much better that everyone jumped to it. and then the IDE's finally got modern features, and then the runtime got much faster in php 7.

It's just by the time all this happened, "PHP sucks" had become"common knowledge" and it really fell off the popularity ladder. but not for lack of tooling.

C++ has also made a lot of strides in developer productivity, it just still has a bad rep.

Shish2k• 4 years ago

Does PHP have a debugger which doesn’t totally cripple performance just by being installed yet?

tored• 4 years ago

xdebug 3 has been released with great performance improvements.

https://php.watch/articles/xdebug2-vs-3-benchmark

And you actually can have xdebug installed without performance degradation if it is not automatically enabled on every request (think that was true for xdebug 2 as well). What you do is to send a special request payload to enable it when needed, can be done with a browser extension.

There is also phpdbg, shipped with php, it does not suffer from the same performance penalties that xdebug prior to version 3 had (not sure how it compares to xdebug 3). However it is a command line debugger only, it does not support remote debugging or IDE integration.

https://youtrack.jetbrains.com/issue/WI-21414

oxff• 4 years ago

It is underrated how hard it was Microsoft with VSCode that made e.g. Language Server a mandatory / must have for any non-joke language.

hutattedonmyarm• 4 years ago

I actually hate the tooling for Rust. VSCode on Windows sometimes refuses to launch the project or the debugger crashes on launch. IntelliJ Community doesn't do debugging in Rust at all, neither does Sublime Text. Nova for macOS does have a Rust extension, but it also doesn't support debugging.

It's pretty good with IntelliJ IDEA Ultimate on Linux though

mamcx• 4 years ago

Another hint on all this, is that the most "loved" languages and special here, Rust, Go, Zig, ... are HOLISTIC.

C, C++, JS are badly designed languages and that is something that will never, ever, for all eternity ( :) ), allows to make great tooling.

"Badly designed" here is related, but not the same, to "bad languages".

All this tooling is a circle that start from the syntax, paradigms, tastes and feedbacks that reinforce the whole thing.

A good example is Rust clipply.

Is amazing how great is (past week it help me turn a mess of logic with 2-3 steps of refactoring that turn like 10/15 lines of code on 5!).

But you CAN'T have that good without a) GOOD design or b) TOO MUCH effort.

For a great linter, Rust design has:

- Cargo and sub-commands, so `cargo clippy`

- A formatter, so it exist some guidance in how "good code" *look like (try guessing a lint that show a compressed line of code: Not helpful!)

- A idea of what are "idiomatic looking apis"

- That is based in traits and some enforced idioms

- That is based in the Rust type-system and checks (so it knows when you are cloning a copy value)

- That is based in a explicit syntax

- That is based in extensive annotations of code

- That has a GREAT error reporting machinery

- That DEMAND a sophisticate parser that is resilient to errors and has full access to the source code

- That run alongside the whole chain so you can start again in each cycle.

And more importantly:

- That is made by the same guys that made Rust, not somewhere else guessing what to do with that!

Too much things to align, and that show how a "well designed" language is a holistic endeavor.

The slow march of progress in programming language tooling

Related Stories