entelechy

mostly strace and it's macos equivalent; Later we moved to ptrace and ebpf. LD_PRELOAD unfortunately doesn't work for statically linked libc. There are also kernel probes but didn't like that it required root permissions...

Love it! We did something similar using strace/dtruss back in 2018 with https://buildinfer.loopperfect.com/ and were generating graphs (using eg. graphviz and perfetto.dev) and BUCK files on the back of that

Whilst we regrettably never came around to package it as a propper product, we found it immensly valuable in our consulting work, to pinpoint issues and aid the conversion to BUCK/Bazel. We used graphviz, https://perfetto.dev/ and couple other tools to visualise things

Recently we cicled back to this too but with a broader usecase in mind.

There are some inherent technical challanges with this approach & domain:

- syscall logs can get huge - especially when saved to disk. Our strace logs would get over 100GB for some projects (llvm was around ~50GB)

- some projects also use https and inter process communications and that needs ot be properly handled too. (We even had a customer that was retriving code from a firebird database via perl as part of the compilation step!)

- It's runtime analysis - you might need to repeat the analysis for each configuration.

Indeed I didn't remember it correctly:

According to https://tfhe.github.io/tfhe/ states:

Each binary gate takes about 13 milliseconds single-core time to evaluate, which improves [DM15] by a factor 53, and the mux gate takes about 26 CPU-ms

Addition of two bits can be implemented using 5 binary gates (fulladder) Hence to add 2 32 bit numbers ~416ms => 2 additions per second

EDIT: Shame I cannot edit my original post

ohh interesting! Are there any opensource implementations of homomorphic numerical processing?

Were there any efforts of combining both approaches?

Great work! This allows to perform arbitrary computation on untrusted devices.

However last time I checked computation in this scheme is ridiculously slow: on modern machines, cutting edge implementation of FHE manage to get around 100 integer operations per second.

Never the less there have been some brave startups trying to commercialise this technology:

https://venturebeat.com/2020/02/18/enveil-raises-10-million-...

Other interesting things build on top of FHE:

sql database where data and queries are fully encrypted: https://github.com/zerodb/zerodb

fully encripted brainfuck vm: https://github.com/f-prime/arcanevm

doesn't work for me: "IP could not be resolved"

The story behind the fake company: https://www.youtube.com/watch?v=fTHg-tGvlJ8

What's your take on Bazel?

Here a couple key points:

- Buck and Bazel are very similar.

- Buck currently models C++ projects better [1].

- Buck leverages remote caches currently better than Bazel [3] - Bazel is very easy to install

- Bazels "toolchains" make it very easy to onboard newcomers (to any project and language) but also ensure the build will run as expected.

- Bazel is less opinionated and more extensible than Buck.

In fact Bazel is so powerful that you can have Buildfiles that download a package manager and use it to resolve more dependencies. This is great to get things off the ground, but makes things less composable because the package manager won't see the whole dependency graph. As a result you might get version conflicts somewhere down the line.

To summarize: I think having a very opinionated build-system is easier to reason and scales usually better.

Communities with very opinionated packaging and build-systems are proving this by having orders of magnitude more packages that eg. the highly fragmented C++ community where configuration is prefered over convention.

Have you considered offering support in Bazel for your package manager?

Yes we did. As soon as this feature [1] is implemented we will have a 1:1 mapping for C++ Buck Projects and Bazel. Then after a small (automated) refactoring of our Buckaroo packages, you should be able to build any package from the Buckaroo ecosystem with either Buck or Bazel.

Btw. The cppslack community is attempting to create a feature matrix of various build-systems here [2]

[1] https://github.com/bazelbuild/bazel/issues/7568

[2] https://docs.google.com/document/d/1y5ZD8ETyGtxCmtT9dIMDTnWw...

[3] https://github.com/bazelbuild/bazel/issues/7664

No but this will be covered by buckaroo [1]

[1] https://github.com/LoopPerfect/buckaroo/issues/314

The C++ community started a comparison of various buildsystem a while ago: https://docs.google.com/document/d/1y5ZD8ETyGtxCmtT9dIMDTnWw...

Overall Buck and Bazel are quite similar as they are both converging to Starlark DSL.

However there are still some differences: Buck is much more opinionated than Bazel. Buck models slightly better C++ projects[1] and currently it's remote cache is more efficient[2].

Bazel is more extensible and offers also remote execution. Bazel has a bigger community and it's roadmap is public.

There has been more than 350 C++ [4] libraries been ported to Buck for the Buckaroo Package Manager[3].

There are also technical details that manifest in some odd ways but are not significant. There is a nice paper by Simon Peyton Jones (creator of haskell) and others that goes into the design details [5]

[1] https://github.com/bazelbuild/bazel/issues/7568

[2] https://github.com/bazelbuild/bazel/issues/7664

[3] https://github.com/LoopPerfect/buckaroo

[4] https://github.com/buckaroo-pm

[5] https://www.microsoft.com/en-us/research/uploads/prod/2018/0...

Happy to go more into detail if desired

We think Buck is great. It's Deterministic hermetic builds and it's composable and declarative high-level build description language made packaging very easy. We built even built a package manager for Buck: https://github.com/LoopPerfect/buckaroo

Currently we marketing it for C++ but it can be used for any language that is supported by Buck.

I signed already up. Let's see how it goes...

This is amazing!

As one of the creators of a distributed package manager for C++ and friends [1] we made a funny discovery:

Many C libraries that a big chunk of the ecosystem depends on, have not been updated for many years. Some of those can only be downloaded from sourceforge or ftp server.

Even worse, some libraries are copy and pasted from project to project and have no actual home.

We uploaded them to github and started maintaining them.

If you know any abandoned C/C++ projects or C/C++ projects you need a hand in maintaining, we are happy to help.

[1] https://github.com/loopperfect/buckaroo

This is great! This will make bootstrap more attractive and competitive as a framework.

JQuery was one of the most important frameworks in Javascript history. It has enabled us to built real webapps. However since then differences between browsers shrunk significantly and we learned how to build maintainable and scalable apps in a more declarative fashion (hello React, Angular and friends)

We tried to implement coroutines based on the CoroutineTS here: https://github.com/LoopPerfect/conduit

CoroutineTS uses coroutines that are implemented on the LLVM-IR level, as a result they get optimized away in many cases.

Unfortunately the CoroutineTS has also some design flaws around allocations and ownership

Since super optimized build systems also increase complexity of their usage.

That's not correct. Complexity arises from the inability of expressing what you want and lack of abstractions over platform details.

Ninja for instance is to primitive to be productive and does not have any understanding of C++.

Buildsystems like buck allow you to express things in a declarative manner eg.:

    cxx_library(
      name = 'foo',
      srcs = glob(['src/**/*.c']),
      exported_headers = glob(['*.h'])
    )

    cxx_binary(
      name = 'app',
      srcs = ['main.c'],
      deps = [':foo', ':bar']
    )

Furthermore it will use services like watchman (if available) to precompute what things need building if you change some files for fast incremental builds.

Lastly it will strip and sort symbols in your binary to make sure the hash of your binary is always the same for the same set of inputs.

All this complexity is handled for you and all you need to do to run your executable is

    buck run :app

I'd like to add Buck and his siblings Bazel & Pants to this list. Buck is used by large companies like Dropbox, Facebook and AirBnB.

Additionally I want to mention my company released also a package manager that uses buck as a packaging format: https://github.com/LoopPerfect/buckaroo

And so far over 320 libraries have been ported to buck and are maintained by our bots: https://github.com/buckaroo-pm

But all those dependency packages will be resolved against a conan server not against the origin eg. github. Furthermore I'd need to put every version of the package into the conan cache.

This is impractical if a complex dependency graph as it would require the user to solve a SAT problem just to determine which packages and versions to put into the conan cache to avoid pulling from the conan server.

May projects use some document generators or tools to collect metrics. However some build-systems that were really interesting were OpenCV and OpenSSL.

OpenSSL buildsystem is over 6k lines of perl scripts which communicate over pipes.

OpenCV uses cmake, python and prolog for it's build. Until this day, I have no clue why there is prolog involved.

You can read some bits about it in our initial announcement of buildinfer: https://hackernoon.com/announcing-buildinfer-for-c-3dfa3eb15...

Thanks for pointing this out, we mention it now in multiple places:

https://github.com/LoopPerfect/buckaroo/wiki/Installation#te...

https://github.com/LoopPerfect/buckaroo/wiki/Telemetry

We gather this data so we can improve Buckaroo and it's ecosystem.

How do I install a package from git directly? - and can that package depend on another package that lives in another git repository?

Afaik this is not supported by conan

Afaik Conan requires a server and you cannot install from arbitrary sources. Furthermore Conan prefers to download precompiled libraries. Unless you run your own artifactory, this may be problematic for people for people concerned about security or ABI compatibility.

yes and you can also distribute your cache [1] Furthermore it uses git to manage the sourcecode and use shallow&shared clones to have quick downloads.

[1] https://buckbuild.com/files-and-dirs/buckconfig.html#cache

Lastly I'd like to note that Buck and Bazel are converging to Skylark for describing builds. The differences between Buck and Bazel should become smaller over time

we build https://buildinfer.loopperfect.com/ to transpile buildsystems automatically.

One of the authors here. Yes you are totally right, as build-systems are not standardized it is quite challenging.

However we have seen more than 300 ports back in 2017 [1] and saw many emerging patterns. This enabled us to write rules to transpile and optimize 100s of build-systems to buck using buildinfer [2] [3]. We hope that more sane decisions will be made regarding build-systems in new projects. We discovered analyzing over 300 projects that a lot of complexity is not needed and often a non-turing complete language for describing the build is sufficient. In fact we found that over 90% of projects can be described solely by using glob expressions.

[1] https://hackernoon.com/lessons-learned-from-porting-300-proj...

[2] https://hackernoon.com/announcing-buildinfer-for-c-3dfa3eb15...

[3] https://buildinfer.loopperfect.com/

For better or worse, we are not far from "bare metal javascript"

There is wasmjit [1] a kernel module that enables execution of webassembly via the linux kernel.

Furthermore there is AssemblyScript [2] that allows to transpile typed javascript to WebAssembly

[1] https://github.com/rianhunter/wasmjit

[2] https://github.com/AssemblyScript/assemblyscript

Let’s try to replace the only STL component we’re still using, std::vector, with a really simple dynamic array - we don’t need resize or push_back in our code, all arrays are initialized with the right size.

Given those requirements, a custom allocator would speed things up. A custom allocator could allocate even on the stack (small buffer optimisation) if possible or use a memory pool.

Embracing C++ features or reading the docs often gives you more than going back to C or reinventing the wheel.

However I think this demonstrates one of the many lurking problems of C++: It does not enforce the right way of doing things nor makes it easy. Instead it makes the right solution awkward or cumbersome to write.

I strongly agree. Think of Rust as a very opinionated and functionally flavoured version of c++.

Also I recommend looking into F#, Haskell and Clojure.

this statement makes we wonder: what does pythonic exactly mean?