zverok

(Author here)

I don't think this corresponds to my experience.

I am working as a staff engineer in big production codebases (with all the traits of those: some code is good, some is bad, some is unrecognizable legacy, sometimes we are in a hurry and write awful code, sometimes we have time for refactoring etc.)

And my observations about the logic and perception of the features are drawn from the practice of code reviews, mentoring new people, and discussing ways of solving tasks in this environment. Obviously, I am frequently a driver of new code practices, but I am also trying to be a good person and a good colleague and notice how comfortable people are with various parts of the codebase, various idioms, etc.

One of the main topics of this article series is uncovering the language's logic and intuitions (to stop perceiving it as a "bag of random syntactic features you need to learn or should guess") and using those intuitions for code that is, yes, better for the reader, but the code that can be created in a quickly-changing production environment and rewritten fearlessly.

Oh, thanks! Added a footnote to the article.

Thanks, that's the highest praise I could possibly hope for :)

It is one of the things I always enjoyed and tried to achieve myself (when the discussion of design decisions in one language would provide food for thought in the context of other languages/domains).

(Author here)

Well, unfortunately a lot of new stuff becomes controversial on the pure ground of "no new language changes! it is good enough for me!"

TBH, I thought about a post series like this for a long time, but the last trigger was an announce of a special Rubocop addon to "disable useless syntax sugar"[1], and pattern matching was one of the "useless sugars" in the list. So I felt I need to cover it, too. And in the end of the day, I think it is an interesting excercise to analyse it in the same way as other, less significant and more controversial, features (the most intresting stuff would be in the second part, though).

1: https://www.reddit.com/r/ruby/comments/16slc10/announcing_ru...

Ruby didn't always had both. `reduce` as an alias for `inject` was introduced in Ruby 1.8.7; `filter` as late as in Ruby 2.6 (just a few years ago, and most of the codebases I saw are still preferring `select`).

Yes, the dictionary is normalized in the last years towards what became industry/mainstream standard names for the concepts, but the statement that initial names were derived from Smalltalk is correct.

Yeah, should've mentioned that probably (though a post is already a mess of footnotes, aside notes, and parenthicals).

I do believe though that a lot of modern languages got that from Ruby (or, via Ruby, from "JS that Rubyists wrote" and such), when it was a "hot new thing" for a brief moment in 2000s. Or maybe it is my skewed perspective.

Ugh, never thought about that this way!

I believe it is a habit back from my journalistic days, when there were "insets" besides the main text with "things that deserve mentioning here, but kinda out of the main flow... but more related to it than footnotes." I miss the richness of markup of paper magazines!

Does it harm the reading experience significantly?

Well, first of all, I was one of those who fought for method references as first-class objects for a few years (the story is told in my blog before[1]), and "let Ruby eventually become more functional" was one of my main arguments!

That being said, there are a few problems on this road, that make it less probable to be followed. First, as Ruby doesn't have first-class Method objects (e.g. object corresponding to `File.method(:read)` doesn't exist until you call `method(:name)` explicitly), this style being popularized will either bring a lot of subtle performance problems, or will require redesign of Ruby's internals.

Second, if we'll make an example _a bit_ more complicated, say:

  ENV.fetch('CONFIG_PATH')
    .then { File.read(_1, encoding: 'KOI8-U')
    .then { JSON.parse(_1, symbolize_names: true) }
    .dig('metadata', 'created_at')
    .then { Time.new(_1, in: CURRENT_TIMEZONE) }

1: https://zverok.space/blog/2022-01-13-it-evolves.html#taking-...

I don't find the hypothetical Ruby syntax ["hello","goodbye"].map(&reverse) to be offensive or wildly inconsistent.

This syntax wouldn't work in Ruby, because bare `reverse` is already a method call, not a reference to a method by name. Allowing method calls without parentheses is crucial to Ruby's design, where all objects are fully opaque, and every `obj.attr` looking like a getter, is just a call to an instance method `attr`. (This is, as far as I understand, the opposite to similar languages like Python/JS, where the object is a dictionary of attributes, and `obj.method` reference the attribute of type "method", while `obj.method()` is an invokation.)

This design became a huge drawback in the age of functional(ish) programming, because the simplest way for refer to a method in Ruby is `method(:its_name)` (which is also inefficient, because it creates wrapping object of class Method on the fly, there is no pre-existing first-class object), and any attempt of passing/combining methods would be cumbersome due to it.

FWIW, you can do that in Ruby:

  (JSON.method(:parse) >> method(:puts)).call("[1, 2, 3]")

Another important trait of Ruby's design is that most of the important methods belong to their first argument, so it is not `reverse(string)` but `string.reverse`, so even to obtain a reference to a method, you need to have an object it belongs to! So you can't do that:

   method(:>).call(a, b)

   a.method(:>).call(b)

I made a small nod towards Haskell's way of doing things in the last paragraph of "How others do it" section:

We might also start look into the concept of the “tacit programming”, which, from some point of view, is also about “not repeating the arguments,” but this would be a much longer post—while it is obscenely long already.

Note though that my analysis was mostly about "how you would do that in an established language," not "how would I design a language from scratch." And Ruby is very different from Haskell in its syntax, semantics, and typical intuitions of the language writer and reader.

So designing a solution for clearer representation of "default parameters" is quite different in a language which is built around an `object.method(argument) { block }` as its atomic phrase, and one built around `function(arguments)`. (This can turn into a discussion of which phrase structure is more elegant in general... But I believe it was already done quite a few times in a history of programming languages evolution!)

Well, I expected that, but slighlty hoped that ironical quotes will help (which is also part of the name of the series, started with explanatory intro[1]).

But I am afraid it is hard to do something to prevent people communicating with article title on its own. Happens all the time with me, whatever I write about! (I tried boring over-explanatory titles, people still argue with _that_.)

1: https://zverok.space/blog/2023-10-02-syntax-sugar.html

Thanks, that's insightful!

Yes, that's what I referred to. Before 2.7, object&.:method was almost merged (or rather merged and reverted) because Matz had second thoughts about its uglyness... Which is not completely untrue, but not having a concise way for referring to a method is irritating.

That still wouldn't have solved passing additional args, so maybe { object.method_name(_1, args)} is the next best thing. Though it perceives non-atomic due to wrapping block.

Ruby-idiomatic pipe operator is Object#then. After a lot of design proposals and discussions it is more or less evident no solution other than method would integrate naturally with the rest of the code. So it is just `arg.then{ third(_1)}.then{ second(_1)}.then{ first(_1)}`

Would've been a bit more concise with method references, almost introduced in 2.7, but alas.

(But, well, people tend to want "something that looks like operator, preferably something that looks exactly like |>" and reject every other possibility)

In rewritten code: https://zverok.github.io/advent2021/day11.html — I'd say it is relatively probable (grep for this line):

   if_(stories == 1, ()=> chance(2/3, ()=> [1, "Pizza Hut"]))

...and how many of these buildings will be in the picture is more complicated. Anyway, just opened the online picture and scrolled it for a few iterations and met it almost immediately https://www.dropbox.com/s/mr1f0gpjr5zvrsq/pizza_hut.jpg?dl=0

The `times` solution was overhtought, and later I get rid of it. In the end, the code of cycle like this are handled with `range`, for example:

    var points = range(resolution).map(
      i => ({x, y: y - (i * height) / resolution})
    )

As you might see in the final code[1] the initial "cycle of noise generation" you are pointint to was gone at all (because several iterations of "making it one phrase" made it obvious that it has no context of its own, and should just be embedded in a points generation).

With that being said, I explicitly stated many times throughout the diary I don't expect the approach to be for everyones liking.

[1] https://github.com/zverok/grok-shan-shui/blob/main/grok.html...

Wow. Now THAT is impressive!

Yeah, I am mentioning it at Day 11, while debugging the singular house logic: https://zverok.github.io/advent2021/day11.html :)

I actually hope the WikipediaQL, when it will become a bit more mature, to be helpful in parsing Wikipedia data _into_ Wikidata. As of now, Wikidata lacks a lot of knowledge yet (the article talks about that, too).

I am (slightly) contemplating this (see also Reddit discussion: https://www.reddit.com/r/programming/comments/oe9oia/wikiped...). But currently I am mostly focused on deducing atomic selectors that would be useful and choosing their interaction. The particular syntax is kind of afterthought.

Though, I should say that I am still not sure GraphQL's mental model would work here (WikipediaQL for now structured around "which part of the document to choose and how to transform it", with "where to put it" being the final statement just of some branches; with GraphQL, as far as I can understand, each node is "where to put it"-first, and it _may_ lead to clumsy and repetitive queries).

The answer is complicated (I am mentioning it in the midst of plans and possibilities in README: "(maybe?) other MediaWiki-based sites (like Wikvoyage, Wiktionary, etc.)").

In more details:

1. With the current library, you can't (connection to en.wikipedia.org is hardcoded :shrug:)

2. Adding "what MediaWiki instance to connect" is relatively easy (they all have the same API), including Wiktionary

3. For Wiktionary, current (v. 0.0.2) selectors will be useful, but future ones (infobox, wikitable and other high-level Wikipedia-specific stuff) will be not

4. So I might think of some way of specifying "set of selectors relevant for this instance" or something like that.

5. So, in some distance future, Wiktionary definitely would be fully supported; how distant—probably depends on how fast the project will be moving forward, and how it will be (hopefully) used by others

6. But I might add a support for "use this instance of MediaWiki" in the near future, it is easy!

Thanks!

I am mentioning Wikidata/SPARQL in README (TL;DR: less data still, harder to discover unless you are deeply into SemWeb tech), but I expect this project, will it live long (fingers crossed) will definitely has some relations with Wikidata, too. My previous attempt on something similar: https://github.com/molybdenum-99/reality — used both Wikipedia and Wikidata (and OpenStreetMap and...) but however cool were few demos I managed too achieve, it was a dead end for several reasons.

Yeah, it is dated and naive in many ways :) Really don't know how the HN works and why this one made it to the main page while some of the other stuff I write rarely does so ¯\_(ツ)_/¯

(In _some_ defense of the post, at the time of the writing it was targeted _only_ to the Ruby community. But still, probably, if I'd included "how others do it" section it would bear more weight.)

(author here) Me neither! Don't know how or why it suddenly appeared on the HN main :)

Just to provide a bit of the personal context: this article caused creator of DaRu (Sameer Deshmukh) to contact me and propose to work on DaRu together, and so I did (see @zverok here: https://github.com/SciRuby/daru/graphs/contributors). I also was, for some time, SciRuby/DaRu's mentor for Google Summer of Code (and, IIRC, it was my initial idea that daru-view grew from).

Also, since that article, an independent dataframe library https://github.com/ankane/rover was created by Andrew Kane, handling some of API and implementation in a cleaner way.

That being said, I am not sure that DaRu, or Rover (or "dataframe" idea in general) has enough visibility in the Ruby community. It is mostly thought as "some special scientific thing", while I believe in 2021 it should be seen as one of the necessary everyday high-level datatypes.

That's what I'd focus this article on if I'd written it today.

That's a huge topic, which I am planning to cover towards the end of the article series <s>please like and subscribe</s>, but in short: yes, my opinion is that spellchecking is actually a "machine learning problem in disguise", and most of existing dictionaries are more a roundabout way of storing something-not-unlike-models than analytical data.

But ML approach will raise a question of data availability. What good your "deep learning OSS spellchecker" will do if there aren't good (and open) models for it which cover as much languages as existing Hunspell dictionaries do? And what if adding a bunch of new words requires laborous model retraining? It is not unsolvable, but non-trivial.

I believe all the giants have something like this inside (I don't think spelling correction in Google search bar is handled with Hunspell, right?), but it is much harder to do as an open tool, ready to embedding into other software.

There are a notable attempts, though: JamSpell for one (https://github.com/bakwc/JamSpell), which has an open "free" models, and more precise commercial ones; source code is open (maybe also only for using "simplistic" models, haven't dug deeper).

I believe that LanguageTool[0] is the closest open-source counterpart to Grammarly. Though, in my experience, it is not a half as useful... But multilinugal and open-source.

I have a distant dream of doing to it what I did to Hunspell (write a code/series of articles explaining how it works and why it is so hard), but we'll see.

For what I know, LanguageTool is based just on a huge set of rules (you can see them in the repo[1]); and Grammarly is a mix of rule-based and machine-learning suggestions (I heard a rumor that it is 99% rule-based, and talks about ML are mostly marketing, but I don't know how reliable this rumor was).

0: https://languagetool.org

1: https://github.com/languagetool-org/languagetool/tree/master...

Ah, indeed :) I just googled the first one for "symspell+js"

I wish there was a similar project written in Ruby.

Hehe... Actually, Ruby is my primary language, but I have chosen Python for this project for a complicated set of reasons I tried to explain[0] in the first article.

If I recall native hunspell suggestions are quite slow - the same algo must be even slower on python.

Pretty slow, yes. But the current project's goal is to "uncover" how the Hunspell works—so, I implement it the Hunspell's way. The next (several) parts of the series would explain a lot on suggest, including "why is it hard", and "why SymSpell might not be enough" ;)

0: https://zverok.github.io/blog/2021-01-05-spellchecker-1.html...

Thanks!

For what I know (I've mentioned it in the first part[0]), the nspell[1] is the most close to "port (some) of Hunspell", and typo.js[2] ports even less (but might be enough for some, we used it in my previous company: it uses dictionaries for lookup, but uses its own simplistic suggest, which I needed to tweak a lot).

SymSpell algorithm (which is quite different, I'll go into it in the next part to some extent) is much easier to port, so there is a JS SymSpell port[3] (which seems abandoned though).

0: https://zverok.github.io/blog/2021-01-05-spellchecker-1.html

1: https://github.com/wooorm/nspell

2: https://github.com/cfinke/Typo.js/

3: https://github.com/IceCreamYou/SymSpell