Pointer Free Programming and the Future of Nim

https://nim-lang.org/araq/destructors.html

by mindB • 9 years ago

205 151 9 years ago

pavlov3• 9 years ago

I’m not a huge fan of the modern C++ style that obsessively avoids null pointers and instead uses object references that are created in an invalid/empty state. With a pointer there is a single way to represent “nothing”, regardless of what the object represents. With a reference I have to go read the API to understand the “nothingness” states.

Also, when you’re thinking in pointers, it’s easy to add levels of indirection with the same mental tools (pointers to pointers, etc.) Personally I find it’s easier to solve problems with a limited orthogonal toolset than a sprawling array of marginally differently balanced optimizations, which is what most of C++ has become.

NiceGuy_Ty• 9 years ago

While null may be the simplest way to represent nothing between a large amount of data types, I think the biggest issue (beyond how far null can propagate from the source in error scenarios) is that it's overloaded with too many possible meanings. Does it mean nothing was found? Was an error thrown? Is the value itself null? At least with a well defined API, you can determine these scenarios from a glance. I think that handling types like, None, Err("reason"), or Some(None) (using Rust types) is far clearer than looking at a null pointer and an int error code (which one may ignore).

pavlov3• 9 years ago

I do like the Rust way because it’s standard (so far anyway). The situation in C++ is more like “a thousand blooms of nullishness”, where each library and API is frustratingly different in terms of something so fundamental. Objective-C with its messageable nil is so much better in this respect.

delinka2• 9 years ago

Creating objects in “empty” states is lazy programming. The point of references is to guarantee an object exists. But don’t pretend an object exists just to satisfy the reference. Rethink your code to take advantage of the guarantee.

quotemstr3• 9 years ago

You've illustrated why I'm against the kind of simplistic programming maxims you'll find in "Effective X" books. It's easy to brand certain syntactic constructs as "bad", but unless you teach people good taste (which is fiendishly difficult to learn), all you do by telling people to avoid certain easily-recognizable syntactic patterns is to get them to come up with new horrors.

jstimpfle18• 9 years ago

Just a reminder that there are many people who prefer not to think in objects.

Rant: Personally since working on my game engine in C, my architecture has become so much simpler, I can easily avoid unnecessecary allocations. My code is much more coherent. Aspect-oriented - which is not at all OO spirit, since OO encourages (trying) to bundle things in self-contained packages.

Yes, that seems hard to believe, since there is so much bad C code out there. With allocations, strings, file system operations etc. all over the place, just as in typical OO code. (Of course some things are really hard to do in C vs C++. The point is forcing myself to have a good architecture and avoid having to make many meaningless decisions).

echelon• 9 years ago

This is where `Option` or `Optional` types shine. They force you to unwrap the nilable value at compile time, and are zero-cost abstractions. It's not difficult to use option types, either--especially if the language features syntactic sugar to promote their ergonomics.

Rust, Swift, Guava, etc. all get this right. Option types need to become a language feature for all statically-typed languages. C++ should adopt it too.

Arcsech• 9 years ago

While I love option types, retrofitting them into languages which already have nulls has problems.

Ask me about using a Java library that could return an option type, which could be null, so you have to check for both the None return value AS WELL AS null.

cwzwarich4• 9 years ago

This is where `Option` or `Optional` types shine. They force you to unwrap the nilable value at compile time, and are zero-cost abstractions.

They aren't zero-cost abstractions in most languages. Every unwrap() potentially results in a dynamic check, even if one isn't necessary. There are a number of ways you could extend a type system to track the potential cases of sum types, which would help alleviate this cost.

klibertp• 9 years ago

C++ should adopt it too.

They apparently got the memo: http://en.cppreference.com/w/cpp/utility/optional

gpderetta2• 9 years ago

std::optional is available in C++17. It has also been available in boost for a veeery long time.

Lack of good sugar for pattern matching sometimes makes its use a bit awkward though.

albinofrenchy4• 9 years ago

Is this really considered modern c++ style? Best practice as far as I've ever known it is to disallow wherever possible the allowance of having an invalid state; typically with constructers locking those details down.

Obviously references are better in most circumstances but only because you don't have to check for an empty or invalid state. They offer no performance or optimization on their own.

tomjakubowski2• 9 years ago

I think GP meant "smart pointers" where they said "object references". At least, that was the only reading that made sense to me, since it would be a little bit insane (and UB) to create a C++ reference (T&) from a null pointer.

d__k• 9 years ago

What is really undesirable is to have both pointers and references in one programming language as two separate constructs.

jstimpfle18• 9 years ago

I don't get the buzz about null pointers. They are not a problem. They become a problem when you start checking for null where null is not acceptable (most of the places null can't be a meaningful input), which is where is where the original intent starts to become unclear.

Let it go. Let it crash. Just assume inputs are non-null (except where null makes sense). Even C crashes safely on null pointer dereference.

IshKebab• 9 years ago

The problem is that the type system doesn't encode whether a point can be null or not, so unless your documentation is amazing (I doubt it) you'll eventually end up getting passed a pointer from someone else, or giving a pointer to someone else who has a different assumption about whether null is allowed. Boom segfault.

There are several solutions in C++.

1. Always check for null. Kind of annoying and lots of people don't for whatever reason.

2. Use references. As you say, annoying because then you can't have null (sensibly) even when it would be really useful.

3. Use std::optional<int&> or something like that. I only just thought of this and don't know if it would work, but I bet it's a pain.

So no great solution. Personally I would use a smart pointer type, document it as well as I can, and always check for null.

dan004• 9 years ago

Let it go. Let it crash. Just assume inputs are non-null (except where null makes sense). Even C crashes safely on null pointer dereference.

If the amount of work done before the crash has some value and is lost by the crash, then crashing just isn't an option.

lmm7• 9 years ago

They become a problem when you start checking for null where null is not acceptable (most of the places null can't be a meaningful input), which is where is where the original intent starts to become unclear.

The trouble is it's not always clear whether null is a meaningful value or not. The type of an expression should tell you what values it might take - that's the whole point of a type. Having the same type for pointers that could be null and pointers that will never be null is like having the same type for integers and floating-point numbers.

Even C crashes safely on null pointer dereference.

False. A null pointer dereference in C is undefined behaviour and can therefore easily be a security flaw (there's been at least one local root vulnerability in Linux due to this).

pjc503• 9 years ago

Let it go. Let it crash

It must be nice to work in an environment where nobody cares about results.

simias2• 9 years ago

Even C crashes safely on null pointer dereference.

Uh, that's a rather optimistic way to look at this. In C NULL pointer dereference is UB so you shouldn't rely on this behaviour. For one thing the compiler is allowed to do weird things if it can statically assert that a NULL reference takes place, for instance by dropping any code that can be proved to lead to a NULL dereference. Not so safe or friendly.

Most of the time for userland apps dereferencing a NULL pointer for reading or writing will cause a segfault but that might not be the case for bare metal/MMU-less applications. Furthermore even when it does crash it simply terminates the process without any unwinding which might leave non-volatile data in a non-consistent state.

The real problem with pointers is that they're effectively an algebraic data type, they can point to something or be NULL. But the type system can't enforce that, there's no concept of non-nullable pointers in C.

Take the prototype of nanosleep(2) for instance:

    int nanosleep(const struct timespec *req, struct timespec *rem);

Turns out that the 2nd parameter is NULLable if you don't care for the reminder. The first one isn't however, so passing NULL there is a big no-no.

In rust the equivalent signature would be something like:

    fn nanosleep(req: TimeSpec, rem: Option<&mut TimeSpec>) -> Result<(), i32>;

Which is a lot more explicit (mind you, that's probably not what the API of nanosleep would look like in rust since output parameters are not very common, you'd probably return rem in the Result).

The problem is the same on the library side. When you're passed a pointer, should you assume it's nullable? Should you check for it? If you want to be safe you do, and you end up with a bunch of redundant checks.

It's also easy to end up with a NULL-ish pointer that's not NULL if you happen to offset a NULL pointer by mistake (like a NULL array or NULL struct pointer). That can be pretty tricky to track down since the crash doesn't occur when the offset is calculated but rather when the bogus quasi-NULL pointer is dereferenced:

    #include <stdio.h>
    
    struct s {
      int a;
      int b;
    };
    
    void bar(int *i) {
      if (i == NULL) {
        printf("NULL\n");
      } else {
        printf("%d\n", *i);  /* i is invalid but not NULL, this (probably) crashes */
      }
    }
    
    void foo(struct s *ps) {
      int *b = &ps->b;      /* ps assumed not-NULL here */
    
      bar(b);
    }
    
    int main(void) {
      foo(NULL);            /* This is where the actual error takes place,
                               foo's argument is not NULLable */
      return 0;
    }

Also consider what happens if we're using a NULL pointer to a very large buffer. Dereferencing the beginning of the buffer would probably crash but higher addresses might end up in mapped pages and you start accessing and possibly overwriting random parts of the runtime.

titzer• 9 years ago

Let it go. Let it crash. Just assume inputs are non-null (except where null makes sense).

That's just asking for a disaster. Null pointer dereferences are undefined behavior in C, and compilers can and do make these assumptions, leading to all kinds of horrors. (https://software.intel.com/en-us/blogs/2015/04/20/null-point...)

DamonHD2• 9 years ago

Not in many important environments, eg especially those without virtual memory where the zero page can be mapped invalid.

Talk to me about how good an idea it is not to check potentially-NULL pointers in embedded systems.

alkonaut• 9 years ago

most of the places null can't be a meaningful input

"Maybe something" is not rare. In fact it's often just as common as "Definitely something" in my experience. A customer maybe has a shipping address. You could bend over backwards and say that either he has an address or else the address reference must point to the special no-address value instad of being null.

But in that scenario you still need to check for whether it's the no-address or an actual adreess. Otherwise you might end up doing things like trying to set the street address of the no-address value which could either throw an exception or (worse) suddenly everyone without address lives at that address.

So the only real solution is to use proper types to model "maybe something". You can use a list of addresses for the customer. That naturally supports 0..N. Or you could use a proper option type (which is just a collection with max count 1).

dkersten2• 9 years ago

Even C crashes safely on null pointer dereference.

Does it? What if you needed to clean up external resources? What if some data was left in an inconsistent state? What if you were in the middle of writing to a file? Etc.

I do agree that crashing is often the right thing to do and of course the machine can die anyway, but sometimes you want to at least attempt to recover.

Also, in much of the software I've written in recent years, crashing was not acceptable because just because one thing (eg request of some kind, message, task, whatever) failed unexpectedly, does not mean that others will too and we still want others to be attempted. Sometimes you can get away with letting the program crash and then restarting it to run the other work, but often things are done concurrently and restarting would lose any in-progress state. If the work takes a long time, restarting all work because one piece failed is often unacceptable.

dan004• 9 years ago

I’m not a huge fan of the modern C++ style that obsessively avoids null pointers and instead uses object references that are created in an invalid/empty state.

If you've objects with an invalid state then you've nothing won compared to a null pointer.

But the point is to make invalid state not representable as much as possible, then the reasoning of code gets a lot easier, because the invalid state can't distribute throughtout the application.

gcc_programmer• 9 years ago

Have you heard of operator bool() ? If the library is implemented by people who know their stuff, each object will define operator bool() and you can then have code like:

Foo foo; foo.init_from_file("/tml/lol"); if (foo) { // do some stuff }

we could be getting Foo objects by value from a factory, which constructs and moves the Foo objects. References are meant for situations where the object CANNOT be null/invalid by contract, so you are missing the point amigo.

pavlov3• 9 years ago

The accumulated corpus of C++ libraries has been implemented by people who may or may not have known their stuff at some point in the past 25 years.

You say operator bool() is the one true way. Many other people define "empty" states for their objects that sometimes make sense only to the library author. Others use class wrappers. Someone else came up with a cool trick for a Maybe-style functor, I'm sure. Yet another bright person has built his very own mind-bending approach using move semantics and some odd corner of the C++14 spec.

With C++, you never know what to expect until you dig into the API. It's an impediment to doing anything with the language unless you work on a huge project where coding standards are rigid and external dependencies come in every once a century (which many C++ projects are, certainly).

smelterdemon3• 9 years ago

Creating objects in invalid states is definitely not idiomatic C++.

I do agree that people are a little too fearful of using bare pointers for optional references, if you're using a linter that will catch possible null dereferences it's often the most sensible technique.

CyberDildonics• 9 years ago

C++17 has std::optional

If you've ever used unstable software that crashes in a dozen different ways you can thank null pointers.

Modern C++ isn't about substituting references for pointers, it is about dealing with values and removing a level of indirection that pointers gives.

The old style of of using pointers everywhere frequently usually implies a lot of small heap allocations which is also a big performance problem. Both these things end up being huge detriments to speed and stability.

duneroadrunner• 9 years ago

First let me say that I'm not familiar with Nim. But I understand that it compiles/transpiles to C/C++. And it sounds like they're now trying to move away from dependency on the GC. In that case, might I suggest they consider switching the transpile target to SaferCPlusPlus[1]. It might make things easier as it already addresses the "efficient memory-safety via scope lifetimes without a borrow-checker" issue. And also the (data race) safe sharing of objects between threads. (Note that the documentation is currently rather out-of-date, and a major update is coming.)

[1] shameles plug: https://github.com/duneroadrunner/SaferCPlusPlus

mratsim4• 9 years ago

Nim compiles to C/C++/Objective-C and Javascript and all memory safety is handled on the Nim side.

The C/C++ code generated by the Nim compiler is then free to use unsafe constructs including the dreaded goto.

seertaak• 9 years ago

I'm a C++ dev and this sounds extremely interesting. I have nim a spin a year ago and really liked what I saw, but GC was a real issue because I'm doing audio development where you end up having to circumvent the GC anyway. I would love to allow my users to create MIDI plugins with nim and this is a key step towards that.

mratsim4• 9 years ago

I'd like to add that "circumventing the GC" is supported by default. There are 3 object types in Nim

- `type Foo = object` --> Value type, on the stack

- `type Foo = ref object` --> Reference type, managed by GC

- `type Foo = ptr object` --> Pointer type, manual memory management (with Nim's equivalent for malloc, free and memcopy)

Also Nim does not have 1 GC but several:

- Default Deferred Reference Counting - Not stop the world

- Boehm GC

- Mark and Sweep

- Memory Regions

- Real-time GC (with tunable max-pause)

- GC disabled

GC can also be deactivated per thread (setupForeignThreadGc) or for a section of the code (GC_ref, GC_unref) even on Ref types to improve interoperability with other languages.

baldfat• 9 years ago

You make Zenaudio ALK! That is a really interesting sequencer and looper. Congrats on a great concept.

tomsmeding6• 9 years ago

That sink concept sounds really cool. I would really want that semantic in my C and C++ code.

Boxxed• 9 years ago

Pretty sure that's the semantics of an `auto_ptr`: http://www.cplusplus.com/reference/memory/auto_ptr/

audunw• 9 years ago

Does this mean that Nim 1.0 will be postponed even further?

That's OK, it's better to get it right, and it's still useful for small projects in the meantime. But Nim seems like a language that is never truly finished.

I like this direction. I tried to avoid "ref" in my last Nim project, but it's too hard to do that for every type the way the language is currently designed.

Tiberium• 9 years ago

It won't be postponed AFAIK

dom964• 9 years ago

Indeed, Araq's plan is to release v1 ASAP (he said by the end of this year on IRC) and implement the ideas described in this article afterwards for a Nim v2.

nimmer• 9 years ago

Nim seems like a language that is never truly finished

True for every language.

"1.0" will freeze the spec and it's not like the current versions are unstable or unusable.

tempodox• 9 years ago

Completely off-topic, bit I feel compelled to mention this: The timestamping of this article is exemplary.

Even if it only showed a date, and no time, the time zone is relevant. The Internet is supposed to be for the whole world, not just one time zone.

ndh2• 9 years ago

Suggestion: Add a date, the author, and possibly some version number that indicate what iteration of Nim this refers to.

userbinator• 9 years ago

The syntax and title reminds me of standard Pascal, which has very constrained pointers that, among other things, you can't do arithmetic with --- eventually leading to code that basically reinvents memory itself by using a large array and lots of indexing operations.

(Look at Donald Knuth's TeX for an example of this style.)

mratsim4• 9 years ago

Nim was written in Pascal at the very beginning.

But you can't do pointer arithmetics in Nim.

Either

- `type Foo = object` -> Value type, on the stack

- `type Foo = ref object` -> Ref type, managed by GC

- `type Foo = ptr object` -> Pointer type, manual memory management (malloc, free, memcopy ...)

dom964• 9 years ago

But you can't do pointer arithmetics in Nim.

Actually you can, by using casts.

pjmlp7• 9 years ago

With the added benefit of bounds checking.

However all Pascal dialects had extensions for low level pointer manipulation.

dom964• 9 years ago

Hello guys. I'm one of the core Nim developers, together with Araq (the creator of Nim) who should also be hanging around this thread. Feel free to ask us anything.

This submission is likely the result of the recent livestreams that myself and Araq have been making. You can check out my past livestreams[1] and Araq's[2] at the links below. If you're into watching us live the next time we create a livestream then follow d0m96 (https://go.twitch.tv/d0m96) and araq4k (https://go.twitch.tv/araq4k) on Twitch.

1 - https://www.youtube.com/watch?v=UQ4RvUlXIDI&index=3&list=PLm...

2 - https://www.youtube.com/watch?v=E2qlDKm_WzE, https://www.youtube.com/watch?v=UV38gQfcb9c

qaq• 9 years ago

You guys are doing amazing work! Found Nim through a mention on HN recently and having a ton of fun learning and playing with it!

athenot• 9 years ago

Same here. Nim has this elegance that is hard to qualify and impossibe to quantify, but it makes it a real pleasure to write code in it. Can't wait to get deeper into it.

jonathanstrange• 9 years ago

I like it, too. Just wish mimx was in a better shape. I need rich text with images and links.

squarefoot• 9 years ago

This one just to thank you for everything, Dom! BTW, your book is on its way to my desk. I briefly skimmed the pdf, but can't wait to get the dead tree version!:)

Don't forget about embedded systems: they could be the killer platform which makes Nim mainstream overnight.

dom964• 9 years ago

Thanks for grabbing a copy and killing some trees in the best way possible :)

I won't forget about embedded systems, but I'm not sure this platform has that much power to change things overnight

jamesu• 9 years ago

The other year I tried to implement a GC on top of an existing scripting language to handle circular dependency cases. A big problem however was the existing system for referencing objects wasn't particularly well designed for this particular case (e.g. no roots, no thread safety), and I couldn't come up with a solution I found satisfactory and fool-proof.

The fact the author had so many problems with GC bugs is somewhat reassuring. Would be interesting to see a GC-less Nim in a production use-case.

Pointer Free Programming and the Future of Nim

Related Stories