C++ Modules Might Be Dead-on-Arrival (2019)

https://vector-of-bool.github.io/2019/01/27/modules-doa.html

by harporoeder • 5 years ago

44 55 5 years ago

vector-of-bool.github.io

MontagFTB• 5 years ago

This post is from January 2019- is there a more recent update on the issues outlined herein?

dang2• 5 years ago

Year added above. Thanks!

Fronzie• 5 years ago

Visual studio support is mostly there: https://devblogs.microsoft.com/cppblog/moving-a-project-to-c...

CMake support is an open issue: https://gitlab.kitware.com/cmake/cmake/-/issues/18355

lelanthran4• 5 years ago

Okay, this was a convincing and persuasive argument.

Did anyone ever respond to this? What do the anti-SG15 (and/or pro-module) people say? I want to see the other side of this argument.

gfxgirl• 5 years ago

Yea, it's a build system. The same problem exists with lots of projects.

    // foo.cpp
    #include auto_generated_file.cpp

The C++ compiler doesn't know how to build auto_generated_file.cpp, the build system does.

klodolph3• 5 years ago

From my perspective, all the complaints are solved by using a more modern build system, like Bazel. If you are already encoding dependencies between modules in your build system, then when you are searching for an import, it's trivial... you just look at the declared module dependencies. Bazel (and other similar systems) already force you to do this for other reasons, and so you theoretically get modules "free".

It's not like the same problem doesn't already crop up with generated sources. Like, if you see this:

    #include "file.h"

If file.h is generated, then your build system needs to generate it before you compile anything that includes it. Generated sources are a major pain point in C++ build systems. It throws a wrench in the typical compilation flow... where you don't know which header files you need until you run the compiler, so you have to manually specify header dependencies in your Makefile if any are generated (instead of getting the dependencies from the compiler output). To keep things consistent (and more reliably cacheable), in Bazel, you have to specify all the dependencies in your build system, not just the ones that the compiler can't figure out for you. This solves some other problems with cached builds--like how you can have multiple files named "file.h", and the insertion of a new file earlier in the include search paths should trigger a rebuild, or else you end up with a poisoned build cache.

Obviously there are going to be some people stuck on build systems that can't handle this stuff gracefully. Too bad. You can keep using headers, it's not like those are going to stop working.

dataflow2• 5 years ago

Does anyone know of a good example of C++ module usage out in the wild? Not Hello World examples, but I mean an actual project with a serious build process using it? I'd love to see how they're currently used.

KyleSanderson• 5 years ago

These are from 2006~, extensions for a language (SourcePawn) took over in 2009. http://www.metamodsource.net/plugins

pjmlp4• 5 years ago

Nothing serious, but you can look at

https://github.com/pjmlp/ppm2png

https://github.com/pjmlp/AStarDemo

I am also in the process of porting Raytracing Weekend to C++ modules.

dataflow2• 5 years ago

Ah ok, thanks!

dang2• 5 years ago

Discussed at the time:

C++ Modules Might Be Dead-On-Arrival - https://news.ycombinator.com/item?id=19015079 - Jan 2019 (193 comments)

Mr_P• 5 years ago

There's a 2016 CppCon talk here about making C++ Modules work at scale for Google's 100Mloc mono repo: https://www.youtube.com/watch?v=dHFNpBfemDI

So, apparently it can work. I don't don't how much Google's internal Module incarnation differs (if at all) from what was ultimately published in the standard, but I'd expect it to be similar.

The article discusses a problem of resolving dependencies between compiling different c++ files. I'm no expert, but I think this dependency graph is implicitly baked into the build system, with tool-assisted user-written bazel rules.

Kranar• 5 years ago

The module system from that video would have been great; there was/is a sound implementation that is based on Objective-C's module system, has been battle tested extensively and works without all kinds of silly quirks, makes packaging and distributing modules relatively easy, and provides performance benefits similar to precompiled headers which are known to improve build times substantially.

Of course, for those reasons it did not get standardized and instead what did get standardized was a mostly theoretical module system that has a great deal of ambiguity and implementation divergence across different compilers, places a great burden on build systems, none of which fully support modules yet, does nothing to help with packaging libraries, and only sometimes provides performance benefits, sometimes provides performance regressions.

Such is the way of C++, a language whose standardization process has been hijacked by politics and corporate stacking of the committee, legacy interests who insist on holding back key features that would improve the language for the vast majority of developers for the sake of ABI compatibility, and whose advancement has been plagued by numerous defect reports and broken or buggy features that get adopted without any kind of battle testing, user experience or broad community feedback.

kevin_thibedeau• 5 years ago

Presumably you're expected to act like Ada and put the interface in a separate TU that can be processed first by the build system.

dgellow3• 5 years ago

Edit: I just notice now that you said "TU", my mind autocompleted your sentence with "file" instead the first time I read it. Still keeping my comment here, module :private is not well known and a nice way to declare a distinct interface Translation Unit while keeping everything in one file.

You can do this, but you can also use the "module :private" fragment if you want to have everything in one file while still keeping the distinction between interface and implementation.

Example:

  // File "mymodule.ixx"
  module;

  #include <string>

  // declare interface
  export module mymodule;

  export namespace mymodule {
      // templates have to be declared
      // in interface to be known at
      // compile-time
      template<typename T>
      T add(T x, T y) {
          return x + y;
      }

      std::string greetings();
  }

  // implement interface
  module :private;

  std::string mymodule::greetings() {
      return "hello";
  }

pjmlp4• 5 years ago

You can do that if you wish, they are called module interface units.

natded• 5 years ago

How do C++ programs / programmers enforce and manage modularity in the general sense if they don't even have simplest module system? I'm not even talking of having module functors like Ocaml, but something like Rust has. It seems to be one of the most obvious steps to reduce complexity and compose program of smaller parts.

pubby• 5 years ago

#include and namespaces.

actinium226• 5 years ago

Mostly we rely on the system package manager. If I'm trying to compile a C++ program and I need somelib I `sudo apt install libsomelib-dev`

Works about 80% of the time. The other 20% I gotta go digging through either github or sourceforge or someone's personal website that hasn't been updated in 15 years.

petmon• 5 years ago

It's not "enforced;" it is managed through various means including namespaces, header includes, and prefixes. For example, SQLite is a library whose functions all start with 'sqlite3'. In practice it works well enough: name collisions are uncommon and can be managed.

xpressvideoz• 5 years ago

Wait, does the C++ module proposal allow modules names to be different from their filenames? This sounds so strange, I've never used such a module system in other languages. I've always assumed C++ modules would somehow restrict the excessive liberty it has had from its headers and preprocessors. How did they arrive at such a weird idea?

dgellow3• 5 years ago

C++ modules have little to do with the concept of "modules" you find in other languages. You can think about it as a more standardized way to define and create what is known as pre-compiled headers. But that's basically it.

klodolph3• 5 years ago

C++ modules are not like pre-compiled headers. They give you freedom to specify what you are and are not exporting from a module. This is unlike header files (including pre-compiled headers), where you are constrained to re-export any dependency your header file has, as well as any code internal to your module which is necessary to declare your exports.

C++ headers are a total mess, and modules are the cleanup effort.

pjmlp4• 5 years ago

C++ is not alone in this regard, take .NET as one of the ecosystems that follows the same idea, D, Ada and Fortran as well.

lelanthran4• 5 years ago

Wait, does the C++ module proposal allow modules names to be different from their filenames?

It sort of makes sense. The calling program specifies `foo.bmi`, but compiling for ARM might look in "armv5/foo.bmi" while targeting x64 might look in "x64/foo.bmi", and that doesn't even take into account that there might be different debug and release versions.

munch1175• 5 years ago

That is very easily solved by not using modules for those things that cause issues, or by using a module facade over traditional C++.

For your example, you could make a 'foo' module that pulls in an implementation from a traditional C++ source file, "armv5/foo_impl.h" or "x64/foo_impl.h", using preprocessor #if/#ifdef to choose the right one. Or you could even use a module backend, pulling in the implementation from either "foo_armv5.bmi" or "foo_x64.bmi".

knorker6• 5 years ago

Go allows it too. Traditionally the module name is the same as the directory it's in, but that's no longer true with versioned API modules.

ncmncm• 5 years ago

There is really nothing to see here. The "(2019)" should clue you in.

There are lots of ways to skin this cat. C++ has what they call a "large installed base" -- i.e., literally hundreds of billions of lines of code in active use. So, the solution for C++ is not what we might have done for a new language. It had to be reasonably easy to move an existing project to modules without breaking everybody that might depend on it, and without needing everything you depend on to be moved already. Everybody has their own upgrade schedule, many of which must be "never".

So, it looks odd compared to a green-field approach, but it works and is not a burden to maintain, which is what matters.

mikewarot2• 5 years ago

Do C++ folks still not have the equivalent of Turbo Pascal Units (which were from 1987)?

I can compile almost anything short of rebuilding my IDE in less than a second, every time, thanks to units in Lazarus/Free Pascal.

lelanthran4• 5 years ago

Do C++ folks still not have the equivalent of Turbo Pascal Units (which were from 1987)?

Well, it's an easy problem to solve when a single compiler is used for everything. I doubt that Turbo Pascal Units could be used with other Pascal compilers.

With C++, your module compiled with Clang might be used in a program compiled with gcc. I don't think that C++ modules specify a binary interface in the spec.

With new languages it is easier to implement modules, because a new language, by definition, only has a single implementation. If the new language specifies a modules system it does not break existing software because, by definition, it is a new language.

I can compile almost anything short of rebuilding my IDE in less than a second, every time, thanks to units in Lazarus/Free Pascal.

I know, I sometimes use Lazarus. I'm not really sure that a FreePascal Unit is compatible with other Passsl implementations (maybe Delphi, because the goal is to be a drop-in replacement for Delphi?).

My understanding of the C++ modules specification is that there is no requirement for modules from different compilers to be compatible.

mikewarot2• 5 years ago

With C++, your module compiled with Clang might be used in a program compiled with gcc.

Why do you need more than one C++ compiler installed? The idea is to compile everything only once, and save all the time when you need to use the code later.

It makes no sense not to use the same compiler against all the source when building an executable.

pjmlp4• 5 years ago

Yes, modules finally, although not without issues, as the stardard is freshly baked and there are 40 year of history to drag.

On the context of Turbo Pascal units (actually they are UCSD Pascal units), C++ Builder did support the concept of C++ packages with a similar purpose.

Or if one wants to delve into the COM dungeons on Windows, that is also a path although naturally the COM ABI only covers the basics of an OOP ABI.

Visual C++ support for modules is already quite good, although still has some warts.

eyelidlessness• 5 years ago

Wild that JS got this right in ESM with its first standard, but the lisper in me is still screaming that both are needlessly complicated by tooling.

boardwaalk• 5 years ago

I was just tonight watching Bjarne’s cppcon 21 talk[1] and he does not seem to think they’re dead.

OTOH, from what I can tell my compiler of choice, clang, has not improved its support since last time I dabbled with it unsuccessfully. But I could be Doing It Wrong. There seems to be very little stuff on the internet about actually using them which is not a great sign.

[1] https://youtu.be/15QF2q66NhU

anonnybonny• 5 years ago

https://www.youtube.com/watch?v=9OWGgkuyFV8

int_19h• 5 years ago

I can't help but notice that of all the names in the linked docs, there doesn't seem to be any from the companies that implement the actual C++ compilers. What's the stance of Clang/gcc/MVSC/... developers on this?

klodolph3• 5 years ago

Eh, the assumption here is that you don't know what the dependencies of a C++ file are before you start compiling. That's not a great assumption.

That assumption allows you to write really lazy Makefiles. You just throw a bunch of C++ filenames in your Makefile, add the correct dep flags, and add an "include" line, and your Makefile will correctly rebuild the object files when header files change, without needlessly rebuilds. Here's how such a lazy Makefile works:

    CXXFLAGS = -MF $*.d -MMD -MP
    -include $(wildcard *.d)

Sure, that's nice, but there are a lot of reasons why you might want a more sophisticated build system than a lazy Makefile. You might have a project with long build times, so you might want a good build cache. You might have generated header files. As your project gets larger and more complicated, the idea of building everything with a Makefile really loses its appeal, and you look for better build systems.

The new generation of build systems is Bazel, Buck, Pants, Please, Meson, etc. Plenty of choices to go around. These systems require that you specify the dependencies up-front, rather than relying on -MF & friends. This means that if you add a generated header somewhere, nothing breaks. If you add a new file earlier in the include search path with the same name as a file later in the path, nothing breaks. (By "nothing breaks" I just mean that your incremental builds and clean builds will still both generate the correct output.) The -MF option is then used to trim the dependency set (in a particular way to keep the build correct... rather than record the header dependencies, you record which headers were ignored during compilation).

With these build systems, modules are going to work just fine.

knorker6• 5 years ago

I think this post has a clearly missing solution that is (essentially) a first stage in a build system.

Yes, C++ can be slow to build. But it's slow to build because of the includes and the code generation (and we still have to do code generation).

How much time does the author think it takes to pipe code through the preprocessor (for defines, not includes since in this case there are no includes), and tokenize?

I just checked the dependency files of a fairly large project (GNU Radio), and each cc file transitively depends on 414 files on average. In total in order to compile its 1339 cc files the compiler needs to open and read over half a million files.

So just in terms of data to read on this first pass it's faster by a factor over 400x. Add to that that you don't have to do anything in this pass except look for module dependencies, I bet it's several thousand times faster than a compile.

So in other words, if it takes you an hour to build your big project, are you saying you don't have one single second left over to calculate the dependency tree as a first pass?

And you only have to do it once (and then re-do files as source timestamps change).

WalterBright2• 5 years ago

D just reads and compiles each module only once, no matter how many times it is imported, or whatever rat's nest the import graph is.

Modules could be precompiled, but I found that it was faster (and easier) to just recompile them.

(And not having a preprocessor eliminates several passes.)

munch1175• 5 years ago

If the compiler and the build system is an integrated product, then right, this shouldn't be too hard. But that's not usually the case.

If they're not, what's the interface for the initial build step? Is it still something like "gcc -MD"? But "gcc -MD" can't just spit out a list of files. It will have to output not only file names but also module names (both referenced and declared), because resolving module names to file names requires global knowledge.

That means every build tool that wants to work with C++ must be changed to take in this extended information and resolve module dependencies. Because otherwise, if there's even a single module in your 1339 file project, then current tooling breaks.

Which will put pressure on developers to not add that first module, slowing adoption.

knorker6• 5 years ago

Sure, probably a little bit more work. But I think building a tree of 1339 items is quick enough even on a PDP-11, right?

Or is there a step to this I'm missing? Can't they be analyzed individually and then joined?

Conceptually this merging isn't much different from the fact that gcc -MD needs to output data formatted in a way that Make understands.

But fair enough. I feel like I'm for the second time am saying "just add another stage to the build". Still. "Dead on arrival" is a bit much.

WalterBright2• 5 years ago

D's module system benefited greatly by getting rid of the preprocessor. I implemented precompiled headers for C++, and so knew that a robust and successful module system wasn't compatible with preprocessing.

Shadonototra• 5 years ago

it is

- no circular dependency allowed

- you have to fw declare everything, yet again

i gave up on C++ personally, for a little while i went back to C only, now i only use D, and some GO for my backend needs

garaetjjte• 5 years ago

How does this design handle cyclic dependencies? Or it doesn't, which would be even worse than headers in which you can at least forward declare some things?

C++ Modules Might Be Dead-on-Arrival (2019)

Related Stories