What `for x in y` hides from you – From Scratch Code

https://fromscratchcode.com/blog/what-for-x-in-y-hides-from-you/

by rbanffy • 7 days ago

16 29 7 days ago

'for' loops in Rust do the same: they create an iterator and then iterate over that.

You can write the exact same loop with `let mut iter = v.iter(); while Some(x) = iter.next()`.

'for' loops in Rust are purely syntax sugar, and I somewhat wish they didn't exist. They provide you two ways of doing the same thing, but one of them hides the details from you. Having 'for' as a keyword is nice for folks coming from other languages, but then it hides the possibility of other interesting usages, like cloning an iterator inside a loop.

bigfishrunning• 7 days ago

Which is funny, because whenever I encounter a language for which `for` *doesn't* work this way it feels antiquated. I do however wish another keyword was used in many cases, becuase `for` in C and Go is so much different then `for` in Rust or Python. I think the higher-level case (Rust and Python) should really use a word like `foreach` or maybe something compeletely different like `itr`, although I get that they want to "look" more like C

rbanffy5• 7 days ago

I would say Python really wants to look as much not-C as it can. In this case it's not the word "for" but the "for x in" construct.

b40d-48b2-979e• 7 days ago

This is what PowerShell does, it has both the archaic `for` style with a newer iterator-like `foreach`.

adrian_b• 7 days ago

Iterators and the loops of the form "forall X in A do ..." have been invented simultaneously with the appearance of the first version of the language C (i.e. around 1974), in the languages Alphard and CLU.

From Alphard and CLU, loops of this form have passed into one of the versions of the language Algol 68 (1977), and from there into the UNIX Bourne shell (1979), which inherited a few Algol 68 syntax features (because its author had worked at writing Algol 68 compilers).

Iterators and "forall" loops have begun to spread into mainstream programming languages, e.g. into C++, only a couple of decades after their invention.

Martinussen• 7 days ago

Is "hiding" in the sense that you just need to have read the docs or know how the language works at a pretty basic level really a problem, or even a negative? I would certainly say that the readability and clarity on the form of loop being used is a bigger win, either way.

tialaramex3• 7 days ago

It's true that they're just sugar but Rust does explain how the for-in loop de-sugars and you've over-simplified considerably. Your syntax also doesn't quite work.

The value is in idiom, turning everything into loop expressions (The "while" keyword is also just sugar, Rust's only fundamental loop is named loop) makes it harder to discern what's actually going on.

If you want to clone the iterator in some cases rather than consuming it, that should look different so that reviewers will see what you're up to.

kevinmgranger• 7 days ago

It should be `v.into_iter()`, and that distinction matters because of ownership / move semantics.

It just so happens that for most collections, `IntoIter` is also defined for references to them, which typically gives you the same behavior that `.iter()` would give.

tialaramex3• 7 days ago

More specifically for x in y consumes y in Rust.

https://doc.rust-lang.org/std/keyword.for.html explains how a loop is de-sugared

https://rust.godbolt.org/z/5jzhxYM51

... shows that today ranges like 0..5 aren't Copy even if that would be possible, which means if they're consumed they're gone, whereas an array of integers is Copy and so consuming it doesn't mean it's gone, you can just consume it again.

The desire is that Rust 2027 edition will change the nice syntax for ranges to produce new ranges like core::ranges::Range which are Copy if possible and only IntoIterator, the original ranges are never Copy but are Iterator, we now regret this choice.

saghm• 7 days ago

In addition to the points that sibling comments have made about how you're not quite right with the exact semantics of for loops (which also take ownership via `into_iter` and therefore are a bit different from `while let`), it's worth pointing out that if you peek a bit further in to MIR, all loops in Rust just desugar into the `loop` keyword with manual breaking, including `while`. It's not really clear to me why for loops in particular bother you.

anthonj• 7 days ago

I don't really get the point of the article. Even if I knew little about python, would be it surpsing that a language with no real basic types is probably abstracting a lot?

Even a simple i=0, i=i+1 is "hiding" a lot in python then.

PaulDavisThe1st2• 7 days ago

But it's not a Python thing. Rust is noted below, and there's also C++.

  std::container<T> container;

  for (std::container<T>::iterator i = container.begin(); i != container.end(); ++i) { ...} 

  auto iter = container.begin(); while (iter != container.end()) { ...; ++iter; }

  for (auto const & t : container) { ... }

tialaramex3• 7 days ago

Barry Revzin has a talk explaining about different kinds of iterator. The C++ iterator is a maximalist, very powerful and flexible idea which - as a result - also has lots of opportunities to set everything on fire. Their model was the C pointer, we can iterate a pointer, compare it against some sentinel "end" value, dereference it, but also we could index into it, decrement the pointer and go backwards, comparing against a "start" value, and we can write to a reference rather than just read from it.

This is far more than most iterator designs allow, and as I said if you needed more this is very welcome, but if you didn't it's probably bad news.

C++ can sort iterators for example, so long as they're suitable for indexing back and forth as the sort progresses. Rust's sort functions are defined on its slice types, which makes sense in practical terms but if what you've got isn't quite a slice but could meet the requirements in C++ that's a problem for you.

rbanffy5• 7 days ago

Very few people who use Python realize the loop is not just looking into the values but asking the values to produce an iterator. It's only when they outgrow this early stage that they are ready to understand how to make a finite iterator themselves.

courtcircuits• 7 days ago

To be fair, when I started learning CS, the `for x in y` syntax was cryptic to me because I was unfamiliar with concepts such as iterators & generators. `for(int i=0; i<len(y); i++)` made way more sense since there is no hidden logic (besides additions as you highlighted in your comment, but which I think is easier to have a grasp of). So I really wish I had read this article when I started my CS journey a couple of years ago.

bux93• 7 days ago

In the author's mind, it's unexpected/amazing that 'for' can iterate over many types. But it's NOT unexpected/amazing that 'iter' can iterate over many types. I have no idea why.

It's not like 'for' is limited to counting in other languages. The grand-daddy in c does something until some condition is false, and that thing can equally be incrementing/decrementing a number or invoking some function. That's what a loop does in any case, it compiles down to a conditional jump (JNE/JE..)

Maybe his reason for astonishment is obscured by over-use of an LLM to 'enhance' the text.

thaumasiotes• 7 days ago

It's not like 'for' is limited to counting in other languages. The grand-daddy in c does something until some condition is false

C 'for' is a while loop. It's strictly syntactic sugar for an already existing feature. And it's really, really transparent. `for(A; B; C) { do_stuff(); }` isn't just a while loop, it's this while loop:

    A;
    while(B) {
      do_stuff();
      C;
    }

Other languages have treated for as a separate concept from while. C isn't really informative in that case.

goodmythical• 7 days ago

Why would it ever be surprising that I can y= [x,x,y,y] for x in y; x=y; return y;

and get [y,y,y,y]?

orthogonal_cube• 7 days ago

It would be very surprising for somebody without a formal software development background and years of experience.

Looking back at 2015 when Python 2 was still supported, there was a lot of confusion for why Python 2 would create a tuple while Python 3 created a generator for the following statement:

  foo = (x for x in [10, 20, 30])

The blog post is trying to help fill in a gap of knowledge for anyone trying to understand more of what goes on behind the curtains.

mdemare• 7 days ago

AI slop. Flagged.

tantalor• 7 days ago

Compared to i++ from C/C++ or forEach from JavaScript, Python's version just works.

Comparing to forEach in JS is incorrect because forEach is an method of Array.

You should compare it to `for...of` in JS. Both operate on iterators.

Article is missing an important distinction between iterators and other "array like" types (including strings):

Iterators don't have to stop, e.g., they can take from a generator that never ceases.

Both Python and JS are happy to loop forever if the iterator never stops.

quietbritishjim• 7 days ago

There's also the range based for loop in C++. An actual comparison of iterator-style loops might be interesting but this article is a pretty vacuous bit of basic Python.

It's also a confusing title, given that Scratch is the name of a programming language.

klibertp3• 7 days ago

TL;DR: if you want a general looping construct that works no matter the container type, you need an iterator protocol that types can opt into. This is a standard technique adopted by most programming languages that are not explicitly low-level, and has existed for the past 60[1] years. The OP (re)discovered it and thought it was worth blogging about.

Well, it is kind of interesting to see how the very basic programming building block (iteration) gets generalized without incurring syntactic costs. Whether it's worth a place on the HN front page is debatable, though.

[1] Too lazy to track the actual first implementation, but I'd be astonished if the concept wasn't well-known by the 70's.

klibertp3• 7 days ago

I got curious and checked: CLU had iterators in 1975.

Another key feature of the CLU type system are iterators

It also had exception handling and tuple unpacking!

For instance, writing x,y := y,x would exchange values of x and y

(I'm joking, that's "just" parallel assignment, it just looks like what you do in modern Python with unpacking. And yes, it's all from 1975.)

Details: https://en.wikipedia.org/wiki/CLU_(programming_language)

rbanffy5• 6 days ago

The best known programming language in the 1980s, BASIC, had a FOR that was limited to counting. A lot of people without formal CS education would think Python’s for can go over a finite set of things and live happily at that level of abstraction without ever needing to peek under the bonnet and learn how iterators work (in Python).

klibertp3• 5 days ago

Sure. But you know, from personal experience, you can be a happy BASIC user writing simple games without knowing about the existence of arrays. It's a bit hard, but nothing copy & paste couldn't defeat. And while learning about `DIM` was quite a breakthrough for my 8 y.o. self, I'm not sure I'd welcome an article about it on HN's front page.

The same is true for iterator protocols: as noted in my other comment, they have existed since 1975 in almost the same form (with a few closely related variants). It's not easy to find a mainstream language that doesn't support them "since forever" (in "tech years"). They're just not news, no matter how you slice it.

rbanffy5• 6 days ago

How was this flagged?!

What `for x in y` hides from you – From Scratch Code

Related Stories