pwang

What if you want to ensure that the same packages are available, built in a similar way, between your Mac and some linux servers? What if you need to share or ensure that your projects work between Linux and Windows? What if you are supporting a lot of less-sophisticated users across a number of different OSes, or even different versions of the same OS?

There are a ton of subtleties in the build toolchain even within a single OS type, and these will lead to downstream frustrations with packages that just bundle pre-compiled versions of C/C++ libraries.

The conda approach treats the underlying libraries as first-class citizens in the package ecosystem; tracks their interdependencies; and most packages are built in such a way that they are relocatable on your filesystem and don't require system privileges to install. conda and the various packages in the conda universe (whether official ones from Anaconda or the community-built ones in conda-forge) all make it so that this baseline hard problem is mostly solved across all major OSes, and solved in a relatively consistent way.

You can think of conda as a cross-platform, cross-architecture, multi-language, userspace package manager. It grew into this because numerical Python's package ecosystem is so horribly complex and the users span such a huge set of install environments, that conda ended up having to be a generic rpm/brew/apt kind of thing.

I posted an update about this: https://www.linkedin.com/posts/pzwang_hi-everyone-recently-t...

TLDR - we are working to clean up this language to leave it clear that educational institutions are exempt, and that these commercial terms do not apply to third-party channels hosted at anaconda.org (which includes conda-forge).

Fun joke but Anaconda has a track record of creating OSS and then turning it over to community governance. This includes the conda tool itself, libraries like bokeh, dask, numba, jupyterlab, and many more. And while PyScript project governance isn't in NumFOCUS, all of the code is permissively-licensed BSD/MIT.

The commercial licenses for the products and commercial repository is what supports all of this OSS development work.

Anaconda Code is what you're looking for! It's Python (via PyScript) running as an Excel plug-in, that has full access to the spreadsheet and can harness a big part of the core PyData stack (including matplotlib, sklearn, pandas, etc.)

The whole thing runs via PyScript/WASM, and lives locally inside the Excel spreadsheet. https://www.anaconda.com/blog/introducing-anaconda-code-add-...

Over 50% of Python users report using it for data-related things, e.g. data science, ML, etc. (Jetbrains poll)

That's numpy, pandas, sklearn, and friends.

I feel you on this. Having done a lot of Python consulting for engineers and scientists, it is absolutely the case that most non-programmers have -zero- model of what I/O latency and bandwidth limitations look like. They are looking at programming APIs, and their mental models can include concepts like files and even byte layouts within files. But they generally have no working model of how a physical computer actually implements those things.

I've definitely seen file I/O in the middle of FORTRAN loops. Entirely correct from a functional perspective, and total disaster from an actual runtime perspective.

The homepage that's linked from Github seems to be down? https://renpy.beuc.net/

Thanks for the feedback... I had no idea that pyenv was hosing conda envs. Do you have more details on this brokenness?

This is one of the reasons people use Anaconda/miniconda for non-data science work: conda environments are self-contained Python installs, so if you conda/pip install packages into those environments, they will not break each other. This design requirement arose from the specific needs of numerical computing (which always drags in a ton of system-level C/C++/FORTRAN dependencies), but is a generically useful design construct.

Anaconda is a distro, and conda is a package manager, that works across OS platforms and hardware architectures, and installs cleanly into userland without requiring admin privileges. The only way we achieve this difficult goal is by creating a distro and build system that creates "portable" packages that can be relocated/relinked at install-time.

Ultimately, Python's challenges in this department come from the fact that it has such great integration with low-level C/C++ libraries. This gives it super powers as duct tape/glue language, but it also drags it down into the packaging tech debt of C/C++. Hmm... maybe I should write that blog post: "Python Packaging Isn't The Problem; C/C++ Is." :-)

Thanks!

FWIW, we are soon going to be releasing a much faster depedency resolver. We are also thinking hard about how best to address the "growing ecosystem" problem, in a future-proof way.

You only need a commercial license under certain circumstances, and it only pertains to accessing the default package repository at repo.anaconda.com. You can still use conda-forge etc. Conda itself (the packaging tool) remains open source.

Part of the proceeds from the cost go to fund the Anaconda Dividend Program, which gives money to various open source projects in the pydata/scipy ecosystem.

You keep reposting this Nature article but it is vacuous. There are many "guilt by association to Trump", "false equivalency to outrageous conspiracy theories", etc. and other non-good-faith argumentation that the author relies upon. This does not give confidence in her motives.

There are a few much more substantive sites with analysis into the genetics and circumstances around the virus, which emerged since the April 2020 which your Nature article cites as its primary source.

Here's a direct debunk of that article: https://harvardtothebighouse.com/2020/03/19/china-owns-natur...

That author has written a more extensive article with much more information around the lab itself: https://harvardtothebighouse.com/2020/01/31/logistical-and-t...

And here is an analysis of RaTG13, the closest relative of SARS-CoV-2, as a "smoking gun":

https://spark.adobe.com/page/7BVPjWfEJgQYB/

Don't forget Summa Technologiae, from 1964, wherein he wanted to example the "thorns of roses yet to bloom": https://en.wikipedia.org/wiki/Summa_Technologiae

It's tempting to say that Lem was way ahead of his time, but then we look at his contemporary philosophers of politics, technology, society like Jacques Ellul, Marshall McLuhan, Gilbert Simondon, and realize that the mid-20th century was awash in brilliant foresight about the unpleasant implications of a technological society.

IMO this nuanced thought was simply lost in the craziness of the late-60s and the sex/drug/rock&roll hedonism of the 70s, which then matured into stockbroker 80s, before a second wave of tech-capital-blindness emerged in the 1990s.

And now as these waves have transformed the entirety of modernity, we are faced with the unpleasant question of "where does it go from here, now that the Boomers -- whose narcissism birthed Consumerism -- are dying off?"

There is a factual inaccuracy in the article: Beaker Browser is not "actually under the control of governments and big corporations". It is an independent open-source project.

Pirsig was more well known as I was coming of age in the 1990s. Seems to be less talked about among the younger crowds now?

The sequel to ZAMM is the far more illuminating book, IMO. His metaphysics of quality continue to be a useful metaphysical frame.

Exactly. Ads are a legacy business model of a legacy distribution system. Too few people have read JPB's excellent "Selling wine without bottles", but that is in full effect here.

The problem is that new content-centric business models have not yet emerged. So, some people remain chained to the old paradigms.

The crisis we have is a gap in vision. People don't realize that driving cost of information distribution to zero means that it no longer has enough scarcity to force the economic transfer of other scarce resources. Basically, what OSS did for commercial software, the Internet did for anything that fits within a 2D screen. Netflix, NYTimes, Fortnite, JK Rowling, and Jenna Jameson are all competitors in a space whose Pixels*Seconds value is commoditized.

Capitalism doesn't thrive unless there is an exponentiating dynamic. The only one available on the Internet is bandwidth capture. Which, for now, translates into attentional capture.

With the imminent arrival of P2P web software (e.g. Beaker Browser) and mesh networking, the tides will turn back towards a creator-centric peer network.

It looks like that paper is about the transformations of visualizations for higher dimensional data, not rendering accuracy, so these two things are being conflated even though they are completely separate concepts.

Actually, no. The paper may not have been explicitly clear about this, but the ENTIRE point of a "data visualization" system is to transform potentially high-dimensional datasets, with a large number of columns, into meaningful images by a series of steps. You seem to be interpreting this narrowly, and imagining that geometry is already pre-defined in the dataset, so then of course this looks like a fairly trivial 2D accumulator.

That is not the intent, nor is the common use case.

For data visualization, the question of "how do I accurately aggregate or accumulate the 25 - 1million points in this bucket" is a deep one. There is NO data visualization system that programmatically gives access to this step of the viz pipeline to a data scientist or statistician. Most "infoviz" tools gloss over this problem - they do simple Z buffering, or cheesy automatic histograms of color/intensity, etc. These are almost always "wrong" and produce unintended hallucinators.

Your first comment - about "not needing all the geometry present" - indicates that you are not understanding the nature of the problem datashader was designed to solve. There is no simple "cull" function for data science; there is no simple "Z" axis on which to sort, smush, blend, etc. At best, your data points can be projected into some kind of Euclidean space on which you can implement a fast spatial subdivision or parallel aggregation algorithm. But once that's done, you're still left holding millions of partitions of billions of points or primitives, each with dozens of attributes.... what then?

Yes, datashader actually gives you the ability to dial-in as much gamma compensation as you want, to account for the human visual system's nonlinear response to luminance.

I think you are deliberately trying to misunderstand what is being done in this project.

It's not about what APIs are being used to render whatever. At that level of analysis, all that anybody is every doing, is just doing memcpy and bitblt. Rather, datashader provides a framework for applying semantically meaningful, mathematical transformations on datasets as they're being accumulated, as those accumulations are converted into aesthetic/geom primitives, and as those primitives are rendered into colors. It really is "renderman for data", along with arbitrary vertex/texture shaders, driven by a dynamic rasterizer that can use whatever bins in data-space (not merely physical pixels).

BTW "Out of core" does NOT come from raytracing; in fact its history in computing is a term for anything that exceeds physical memory. We use it all the time in scientific/HPC and data science because datasets are frequently much larger than available memory.

https://en.wikipedia.org/wiki/External_memory_algorithm

It's actually not merely an accumulation buffer. It's a shader pipeline that allows for arbitrary Python code to be executed at each stage of data processing. It's actually very much like "renderman for data", but with Python (via Numba, Dask for performance).

The pipeline is also built in such a way that it permits front-end JS viewers like Bokeh to drive a very dynamic experience.

You're missing the point of this project. It's not about the feasibility of throwing a billion points at a pile of software, to get an image. I can do that with a simple Python script. It's about doing so to create a meaningful and accurate data visualization, and not just a picture of, say, shiny spheres or a scene from Avatar.

I actually have a background in 3D computer graphics, and it's precisely because of my detailed knowledge of raytracing, rasterization, OpenGL, BMRT, photon maps, computational radiometry, BDRFs, computational geometry, and statistical sampling, etc... that when I came to the field of data science & specifically the problem of visualizing large datasets, I realized the total lack of tooling in this space.

The field of information visualization lags behind general "computer-generated imagery" by decades. When I first presented my ideas around Abstract Rendering (which became Datashader) to my DARPA collaborators, even to famous visualization people like Bill Cleveland or Jeff Heer, it was clear that I was thinking about the problem in an entirely different way. I recall our DARPA PM asking Hanspeter Pfister how he would visualize a million points, and he said, "I wouldn't. I'd subsample, or aggregate the data."

Datashader eats a million points for breakfast.

Since you're clearly a computer graphics guy, the way to think about this problem is not one of naive rendering, but rather one of dynamically generating correct primitives & aesthetics at every image scale, so that the viewer has the most accurate understanding of what's actually in the dataset. So it's not just a particle cloud, nor is it nurbs with a normal & texture map; rather, it's a bunch of abstract values from which a data scientist may want to synthesize any combination of geometry and textures.

I chose the name "datashader" for a very specific and intentional reason: we are dynamically invoking a shader - usually a bunch of Python code for mathematical transformation - at every point, within a sampling volume (typically a square, but it doesn't have to be). One can imagine drawing a map of the rivers of the US, with the shading based on some function of all industrial plants in its watershed. Both the domain of integration and the function to evaluate are dynamic for each point in the view frustum.

This paper on our original ideas of "Abstract Rendering" talks about the kinds of accuracy problems that plague the visualization of large datasets: https://www.semanticscholar.org/paper/Abstract-rendering%3A-...

We renamed from Abstract Rendering to Datashader for affordances of human cognition.

This is a great paper from Gordon Kindlmann and Carlos Scheidegger talk about how to gauge the accuracy of a visualization, as part of an effort to come up with an algebraic process for visual design: https://vis.cs.ucdavis.edu/vis2014papers/TVCG/papers/2181_20...

Using their metrics around "confusers" and "hallucinators", Datashader came out as one of the few things that doesn't suffer from such intrinsic limitations.

It's certainly the same feeling I had back in 2010 when I knew I had to quit my job and start pushing Python for data analysis.

Yes

Because selling stuff requires more effort than just "here's my price, pay me".

By doing that, you've put your code into the realm of "commercial software", which engages with companies/corporate use in a whole different modality. If you've never sold software to a business, you will have no idea how much is involved in this side of things.

In addition to being treated differently by your potential customers, you will also earn the hatred and ire of your open-source-loving colleagues. Some bored college kid will see your dual-licensed software as an immoral act, and spend time building a less-awesome and more incomplete "completely free" version, which will then attract dev mindshare and users and eventually ossify into a de facto standard which you will then have to support.

That college kid will one day graduate, and think he can build a business on top of this amazing software he's made that everyone loves and wants to use, and then sit around trying to figure out how to make money from it. He may even consider making a "free-mium" model or an "enterprise" offering on top of the "open core" of his widely adopted OSS.

These will most likely fail, because - again - SELLING THINGS IS HARD.

Then you and this college kid who disrupted your dual-licensed OSS will one day meet at a symposium for "open source sustainability". It will be awkward. Teeth will be gnashed. You will get lectured about not having just used Patreon. Meanwhile the companies that use your and the college kid's OSS continue to hit their quarterly numbers for Wall Street. Executives earn out bonuses. The circle of life continues.

¯\_(ツ)_/¯

False.

They're trying to create a new marketplace where commercial users of OSS can actually give money back to FOSS devs.

Many FOSS devs live a life of poverty because they have no idea how to actually sell things of value around their free artifacts and free labor. It turns out that selling stuff is hard, in general, and feels almost nothing like coding. ¯\_(ツ)_/¯

Yep, F500 companies bring in all sorts of external tools to look at their source code, for compliance, security, etc.

NumPy and other things in the Python ecosystem are certainly part of what they're considering.

The founder/CEO of Tidelift is actually participating in a keynote panel this morning at the NumFOCUS summit.

I get it, it's fun to be cynical about things, but these are folks that are legitimately trying to help.

The fact that raytracing (onto a 2D plane) and rasterization are essentially isomorphic to one another is a powerful observation.

Hasn't this been a pretty well-known perspective (no pun intended) for a couple of decades now?

I was first annoyed at the amateur level of analysis in this piece. "Um, it's 2018, open source software runs production workflows in every single business; if you don't realize yet that every industry in the world is being reduced to becoming a commodity machine to run business rules & agreements as fluid software, then you're about 5 years behind the curve."

Then I realized it was written by Bill effing Gates. And that made me feel sad.

Firstly, because Bill is smarter than this. His lack of awareness of just how pressing this phenomenon is, indicates that he's either had his head in the sand, or the intellectual circles he swims in hasn't been ringing the alarm bells about this. (See my essay on the Changing Nature of Scarcity: https://medium.com/@pwang/the-changing-nature-of-scarcity-fc...)

Secondly, because of the irony. Bill Gates made his fortune having created a business around selling software. It would be good for him to review John Perry Barlow's "Selling Wine Without Bottles": https://www.eff.org/pages/selling-wine-without-bottles-econo...

As the world's first and most successful wine bottler, billg might find this perspective illuminating.