CDE: Easily run a program on any linux without thinking about dependencies

http://www.stanford.edu/~pgbovine/cde.html

by gregwebs • 16 years ago

132 32 16 years ago

stanford.edu

rbanffy3• 16 years ago

Common Desktop Environment? I have a couple old RISC boxes running it...

;-)

Sorry. Couldn't resist...

Now, seriously, this goes a bit beyond package management by bringing along what seemed to be the specific versions of libraries. The goal is to do reproducible results for scientific data but, in this case, I would prefer not to have the original libraries packed along but having the dependencies flagged in a way that any version-specific bug could be easily pointed out.

js4all2• 16 years ago

Common Desktop Environment. That was also my first thought. We had it on our Sun E250.

rbanffy3• 16 years ago

Nice machine. Wicked fast on its day. I bet it still can hold its own despite the 2GB RAM limit.

pyre• 16 years ago

I also though Common Desktop Environment at first. At least from a 'marketing' perspective, it might be a good idea to not use an acronym that is already in use in the Linux/Unix world.

mattew• 16 years ago

This is a very good point. I can't see CDE and not think about all the environment on all the Sun workstations that I have had in the past.

darkandbrooding• 16 years ago

That was my first thought as well. My initial exposure to nix was a purchased copy of Red Hat Linux 4.2, running on an HP Vectra tower. and I remember paying extra for the CDE install CDs. An absolute noob when I started, by the time I'd learned enough to make my modem work I felt like I'd climbed a mountain. sigh* Good times.

mayanklahiri2• 16 years ago

This is fantastic. I haven't tried it out yet, but I'll give it a shot for the paper I'm writing.

BTW, even with a "uniform" Linux environment, I've been finding so many differences in the behavior of libraries that it's driving me nuts. This will be a nice way to get past, for example, inconsistencies in wxWidgets.

pgbovine6• 16 years ago

what research field are you working in? please email me personally if you'd like to chat more about how cde can help your workflow. i'm interested in collecting real-life case studies and anecdotes

mayanklahiri2• 16 years ago

I replied to you via email, but thought I'd post it here as well in case anyone else had suggestions.

I've been thinking about a general purpose experimentation framework for research. Over the last 5 years, I've been re-writing parts of this framework for different projects and wish I had spent some time generalizing and packaging it.

The workflow is:

dataset -> experimental parameters -> code+libraries (this is where the trouble often is) -> result files (plain text, hundreds!) -> analysis (generally in R) -> selectively plotting results

pierrefar• 16 years ago

From my superficial knowledge, this is quite similar to the PC-BSD PBI format: http://www.pcbsd.org/content/view/20/26/ . They both allow you to distribute software in an easy non-hellish way on Linux.

pgbovine6• 16 years ago

hi everyone, i'm the author of CDE, so please feel free to ping me if you have questions, complaints, or bug reports. i'm in the midst of fixing a bunch of silly bugs now and improving the documentation.

rbanffy3• 16 years ago

I find the dependency-capture mechanism very interesting. I would love to see it used to help generating dependency lists for package management systems too.

I will probably have to build my first .rpm packages (I am yet to read the first docs) in the near future and CDE could be a lovely addition to my toolbox.

Thanks for doing it.

Goladus• 16 years ago

This looks awesome. My biggest initial fear (not having used it yet), is that I would inadvertently package up something like my private SSH key.

pgbovine6• 16 years ago

oh good point. the contents of the cde-package is simply a directory hierarchy, so you can feel free to use 'find' to skim thru it. also, i'm now adding a feature where you can tell cde to ignore certain directories (e.g., ~/.ssh/) so that it won't pick up those files.

partition• 16 years ago

Something like this has been sorely needed. I'm wondering if it's possible to generalize this over different OSes, if the code itself is multi-platform, or there is some way to specify hints for it (or should there be). I also go to Stanford so would be easy to discuss this in depth if you want.

pgbovine6• 16 years ago

please email me at the address listed on the website. looking forward to chatting more about this!

sandGorgon• 16 years ago

could you put up a doc or something explaining the difference to chroot jails ?

because you mention: . CDE runs packaged programs in a sandbox similar to a chroot jail, but truly malicious applications can circumvent this sort of isolation attempt.

Do you use similar techniques to chroot - essentially I am asking whether your code is basically chroot (in terms of its behavior AS WELL AS its internals) with some security features removed ?

pgbovine6• 16 years ago

no, it's not using the same mechanism as a real chroot (since you need to be root user to create a chroot). cde uses strace to intercept all the system calls that a monitored child process makes, and re-writes all file access system calls to re-route their paths inside of cde-root/.

InclinedPlane• 16 years ago

If anyone wonders why C/C++ programming is still so popular, here's a perfect case study. Most new fangled languages don't offer the option of static linking. This single piece of uncompromising idiological doctrine has led to billions of dollars in effort expended in engineering systems to workaround that choice and to effectively reproduce static linking in different form.

j_baker• 16 years ago

Most "newfangled" languages don't use linking in the same way as C and C++ do anyway (unless of course you're interfacing with C or C++). For instance, you can't statically link to a python source file. That's not because of ideology. It's because static linking doesn't even have any meaning.

d0mine• 16 years ago

Analog of static linking for a pure python library is including it within the library package. Another example: deploying python application in its own virtualenv environment.

mkramlich2• 16 years ago

dynamic linking is generally a functional superset of static linking because you can bundle the needed libraries with the app, as say loose files in the app's dir tree, for example. This lets you avoid Shared DLL Hell while still allowing you to surgically deploy a critical bugfix to one of those libraries.

mkramlich2• 16 years ago

awesome tool, great job!

no like the name though. sounds like a Unix desktop system.

nitrogen• 16 years ago

Cool idea, and a worthy cause. Is there any provision for copyleft license compliance? Perhaps a distant-future version could use dpkg -S or the rpm equivalent to find which package owns a file, then apt-get source or something to optionally link or bundle a source package with the sandbox.

sfk• 16 years ago

Reposting the comment of chbarts, which has been killed despite being entirely appropriate. Moderating has gone completely out of hand lately:

chbarts wrote:

" I think the potted example on the web page is somewhat poor, as it seems to be a problem existing package management systems already solve.

The more interesting aspect is the sandboxing, and how this is great for one-off software runs where it isn't worth it to install the needed packages on your base system because they're logically just a unit with the software you're testing. You mention this, but I think you'd do well to make it more obvious.

This is also, and this is something you mention, a way of preserving historic software and data. Bundle a full CDE package with an emulator and you can keep running that program for even longer; in fact, it seems a CDE package could gradually grow layers as time demands, from moving a (user-mode) kernel in, to adding device emulation, to going the final step and making the package more-or-less a ship in a bottle running in full hardware emulation.

This could also grow into a way to run untrusted (or minimally-trusted) software in an iron box, not just a sandbox. But that is harder and probably not your preferred direction."

pgbovine6• 16 years ago

thanks for reposting, sfk. i don't know why the original comment would've been killed ... chbarts brought up a great critique about my presentation of cde.

blasdel• 16 years ago

It looks like they got capriciously hellbanned for comments in this thread: http://news.ycombinator.com/item?id=1320949 — everything after that was auto-killed, several dozen productive comments over the course of six months.

Items can be fully deleted or [dead]-ed, and users can be silently hellbanned so that all of their contributions are [dead] on arrival for everyone but themselves. AFAIK there are no other metamoderation mechanisms.

pixdamix• 16 years ago

It reminds me of the Ermine Statifier. http://www.magicermine.com/sesl.html. This tools is so useful when you have to for example run tcpdump or a QT app on an embedded system without installing a TON of libraries or messing with LD_LIBRARY_PATH.

bbulkow• 16 years ago

Wouldn't this cause severe problems with copywrite and licensing? Just because you have a copy of some library on a machine doesn't mean you have the ability to duplicate it to another machine, let alone archive it for future researchers.

I could also see the possible issue of pulling in GPL licensed libraries, and the general theory is if you ship a package with GPL licensed code in it, your code must be GPL. There's a fine line between "sending along someone else's libraries" and becoming a monolithic executable containing someone else's libraries - and between those points is where GPL gets you.

pilif• 16 years ago

There needs to be some kind of solution for the problem of really confidential files to be packaged. The second example in the video packaged the machines /etc/passwd for example (granted, it doesn't contain the passwords, but still).

The third example was showing some real bloat in that it also packaged the kernel and the initramd. I really don't see how these two can be of any use on the target machine.

Still. Very interesting idea and impressively easy to use.

CDE: Easily run a program on any linux without thinking about dependencies

Related Stories