travisoliphant

Congratulations on the launch and work so far! This is indeed very interesting. I believe a language like this is necessary for future progress. I have also been impressed with Julia and its progress. I would have liked to see more progress on Ahead of Time compilation and using it to write Python extension modules.

For Mojo, I'm interested in seeing how the language can be used as a path forward for the Cython community. This could be a stepping-stone towards reimplementation of Python in Mojo. For the past 3 year, I have been talking about the need for a Python-Steering-Council-recommended extension language for Python. This will be particularly important as WebAssembly keeps progressing and potentially redefining what we mean by virtualization and containers.

We have already been using LLVM extensively in Numba and there have been several explorations around MLIR and related technologies. There are several potential paths forward and I'm looking forward to finding ways to cooperate.

Understanding what will be open-source is of course, critical for that.

This is a very good point. You must support AOT compilation if you are going to have a general purpose system. This was clear when Julia came out and I interacted with core developers communicating our experience with creating SciPy and the critical reliance on the AOT features of Fortran/C/C++ as well as the bindings to Python. I believe simpler spellings can be achieved (i.e. more unification between scripting, dynamic JIT, and foundational AOT use-case), but the ecosystem is not close to a one-language to rule them all scenario.

There is a branch of the 3.9 release that removed the GIL created by Sam Gross that you can read about here: https://gavincyi.github.io/2022-10-03-does-sam-gross-nogil-c...

There is some work to bring it up to 3.12 and some resistance to merge it into 3.X because of the impact on extension modules (they all have to be recompiled and in some cases changed a bit).

If you are interested in it, reach out to Sam. He has done a pretty impressive piece of engineering work.

Thank you for stepping up and taking this seriously and quickly helping to restore the SymPy Documentation to the internet.

I'm sorry about triggering you. I can see your point. I did not mean to focus attention away from what happened and how to avoid it.

I am sincerely interested in seeing if the HackerRank leadership will reach out and discuss. I was not intending to promote anything I'm doing right now. I suspect they won't but I will talk to them respectfully if they do.

Let's promote the SymPy maintainers, though. Let's also promote NumFOCUS, because they do a valuable service to community-driven projects and could help SymPy respond to this sensibly. Let's also promote NumFOCUS because they efficiently and tirelessly work to help the projects they fiscally sponsor (like SymPy) -- providing legal support: https://numfocus.org/.

Perhaps something good can come out of this, still.

Thanks for showing up here and replying. I can understand the net being cast too wide for a real problem you are trying to solve. It does have real consequences for already under-resourced communities, though.

I appreciate the "fix-it-twice" attitude implied by the RCA promise (Root Cause Analysis for those who also had to look it up). Also, consider recognition and restitution for the unnecessary work you created for the NumFOCUS director, a NumFOCUS lawyer, and the SymPy maintainers.

A $25k donation to NumFOCUS would be a good start.

If you are willing to talk about what happened publicly, I'm starting a podcast/video series to discuss business and open-source. This would make an interesting conversation. Perhaps we can turn this into a positive to raise awareness and inspire better behavior in the industry? Ping me.

This is related to the idea of EPython that we are working on (as we have funding): https://github.com/epython-dev/epython

It currently emits Cython for the C-backend (and PyIodide). It is very alpha currently, but if people are interested in helping, get in touch.

This author misses the point that the first problem a business faces is how to get customers and how to please them. Getting customers is the number one priority -- perfect architecture is only important to the degree that helps get customers.

Business owners and managers do need to understand that they may have to rewrite the software from the ground up because the first version will have all the problems the author talks about.

Most business owners and managers actually do understand this already. The goal of the first version is not to be architecturally perfect. The goal of the first version is to see if anyone will care if it exists. This is a good argument for not always building on the first version of a startup idea -- but actually build it again once you know people will buy.

I think this is true. I have used the Python C-API heavily having started SciPy and NumPy and Numba. I have a pre-alpha plan for addressing the C-API by introducing EPython (a typed subset of Python for extending it). It is not usable and in idea stage only, but I welcome collaborators and funders: https://github.com/epython-dev/epython. Here is a talk that describes a bit more the vision: https://morioh.com/p/6db365736476

I didn’t read the entire thread of previous discussion. The referenced article is consistent with the work of Thomas Metzinger who wrote a very accessible book called “The Ego Tunnel” as a follow on to his more detailed and scholarly treatise “Being No One”

He introduced me to the ideas in this paper that he induced from his own observations of his “mystical” experiences.

It is fascinating to realize these self models exist in a wide variety of animals and the modeling of others is what makes some animals different.

In “Sapiens” https://books.google.com/books/about/Sapiens.html

Yuval Harari argues that Homo sapiens took over all other Homo species because of the development of “myth” which may be related to the Hyperactive attribution of consciousness referenced in the article. Humans cooperate at scale much better than any other species (but also fight at scale more aggressively) because we can communicate about abstractions.

Very interesting inter-related ideas.

This is about creating a standard for array computing in Python which has been around for more than 20 years. NumPy (which has been a de facto standard) has been around for 14 years and was based on Numeric which was around for 10 years before that. Pandas has been around almost 10 years now. The parts of the standard that are clear and will emerge are the parts that have been around and used by sufficient numbers for at least a decade.

I agree that interfaces like Datasette are a fantastic idea. But, it's not the same thing. Think of this more like the SQL standard. What would you do if SQL did not exist and there were no common ways to discuss querying databases?

Bias warning --- I'm part of the group doing this work. For an end-user, your arguments are fine. A single developer or team can indeed use whatever tool they want and adapt their code without too much difficulty.

The real challenge comes, however, as you try to build an ecosystem of libraries on top of multiple different APIs. You end up reinventing the wheel or struggling to maintain multiple versions of the same library.

Some cooperation is immensely helpful at the lowest level, and we hope ultimately to be helpful to the different tool builders in providing information and guidance about what is already a standard and agreed upon in the space.

This is not about establishing or forcing behavior. It is about documenting and clarifying best practices that have already emerged as well as potentially giving libraries a way to signal to downstream developers of additional libraries that they adhere to the standard.

I didn't perceive it to be an argument for anything specific. I did not get the sense the author is suggesting either lifting restrictions or not. I saw it as more of a piece trying to get us all to think harder and better and expose the difficult trade-offs and the conundrum we face as humans of the morality vs utility argument of saving someone hurting in front of us at any cost versus saving more people at a lower cost potentially at the risk of not saving the person in front of us. I appreciated the author's parable of the river.

My own thoughts go in the direction of maybe the right thing to do is different at different "scales". The right thing to do in my family is different than the right thing to do in my job which is probably different than the right thing to do at the city level, the county level, the state level, the nation level, and the world level (to use labels appropriate for the USA). Leaders at each level have a different decision to make and things work better the more people we have making "good" decisions at each level. The process of figuring out what "good" decisions at each level is the really hard part that is not easily wrapped up with platitudes, memes, or other projections into a single one-dimensional axis (i.e. left vs. right) of a high-dimensional, multi-attribute space.

Travis Oliphant, here. OpenTeams is much bigger than me, but I have been part of the early team putting energy into the effort. Henry has captured some of the personal motivations for me. They are sincere. There are plenty of ways for me to make money more easily than wrestling with the economics of open-source software. This is my calling and something I care about a lot. You can watch a video of me talking something about this here: https://www.youtube.com/watch?v=MCEUPosJuIY if you want to get to know me better.

I have had the privilege of watching open-source projects I started or participated in get used by millions. I want to see a world where more people are paid a great living to work on open-source software. There are several approaches to improving the funding fabric for open-source. OpenTeams is one of them. Quansight Initiate (http://www.qi.ventures) is another one. Quansight Labs (http://labs.quansight.org) and Quansight (http://www.quansight.com) are others. A group of us are also working on a few other ideas including FairOSS (not going to give the website because it is a behind-the scenes operation for now --- ping me if you are interested in learning more).

I have spent many years writing open-source software, improving open-source ecosystems, hiring open-source developers, and paying people to work on open-source software. I have also spent the past 12 years in particular working in industry watching open-source get used and seeing the disconnect between open-source usage and open-source communities.

There are several business models that can work to improve things. With OpenTeams we are building out a platform and ecosystem to help open-source communities thrive by directing money from industries that use them to the communities that exist while also learning from the basic laws of incentive alignment and local choice making that are at the heart of every economy.

One of my core believes is that not all software will be or should be open source. I don't make this conclusion lightly. However all software will be based on and use open source foundations. Those communities that produce these foundations should be able to thrive and yet retain their freedom and self direction.

OpenTeams is a platform to make that happen by connecting companies that use open-source and want to fund specific features and improvements with organizations that hire open-source developers to work on proposals that have input from community participants and leaders. Some initiatives will be new projects, some will be usability improvements and feature proposals for existing projects. An effect of OpenTeams is also to enable a place for all open-source contributors to build their reputation (not just contributions that lead to pull-requests).

We are in a pre-alpha stage at the moment, but will be continuing to add features and are open to suggestions and ideas.

Thanks for your feedback.

I was also there on Thursday and agree. The talks were actually observing that cloud giants do alter the business landscape for open-source, but there are always innovations and possibilities. There were one or two talks on alternative licensing schemes, but the most interesting talks to me were alternative virtuous cycles that are available for supporting open-source while allowing any company to benefit.

Thank you for doing this. This is a great idea. If every company that used open source committed to supporting the projects they depend on with this kind of approach, it would solve a lot of the current sustainability problems. I look forward to several initiatives over the next few years to encourage this kind of behavior from all companies.

I believe it is true that you can make more money selling non-free software (some of the profits from which should always be used to make free software).

You can generate money only through free software, however. Here are three approaches I have found: 1) Consult on projects that use the free software and use some of the profits to support that free software, 2) Sell enterprise-grade support on the free software to big companies. This is much more than just help-desk and answer the phone. All commercial software comes with a big contract. You provide the same kind of contract just no license restrictions. Others can do the same and so you have to distinguish yourself by either having all or most of the experts on the software or just really good marketing. 3) Dual license using GPL3/AGPL for the free version and a commercial license and then aggressively go after people for GPL violations if they don't get the commercial license.

I don't like the relationships created by the third model --- your sales processes become aggressive and counter-service minded by definition. I don't see it scaling and really providing value.

The other two are really hard to impossible to get investors exited about and therefore you struggle to get the capital together you need to prime the customer pump.

There is a another general model with many corollaries where you basically "do something else" that uses the software as a critical part of the business and let the profits of that activity fund open-source development.

A lot of open-source these days is actually funded by this kind of activity (or from VC's hoping to profit from a promise of great wealth from this kind of activity).

Original NumPy author here. I have a lot to say on this topic, given that it has literally consumed my life over the past 20 years. You can go here for some thoughts about some of this: http://technicaldiscovery.blogspot.com/ There are several articles there that relate but in particular http://technicaldiscovery.blogspot.com/2012/10/continuum-and... and http://technicaldiscovery.blogspot.com/2017/02/numfocus-past...

I knew what I was getting into when I wrote NumPy. I knew there was not a clear way to support my family by releasing open source software, and I knew I was risking my academic career.

I did it because I believed in the wider benefit of ideas that can be infinitely shared once created and the need for software infrastructure to be open-source --- especially to empower the brightest minds to create. I did it because others had done it before me and I loved using the tools they created. I hoped I would inspire others to share what they could.

There have been a lot of people who have helped over the years. From employers willing to allow a few hours here and there to go to the project, to community members willing to spend nights and weekends with you making things work, to investors (at Continuum) willing to help you build a business centered on Open Source.

There are many people who are helping to fix the problem. In 2012, I had two ideas as to how to help. Those who know me will not be surprised to learn that I pursued both of them. One was the creation of NumFOCUS that is working as a non-profit to improve things. The second was the creation of Continuum (http://www.continuum.io) to be a company that would work to find a way to pay people to work on Open Source full-time.

We have explored several business models and actually found three that work pretty well for us. One we are growing with investors, a second we are continuing with, and another we are actually in the process of helping others get started with and ramping down on ourselves.

Along the way, I've learned that open source is best described in the business world as "shared R&D". To really take advantage of that R&D you need to particpate in it.

We call our group that does that our "Community Innovation" group. We have about 35 people in that group now all building open-source software funded via several mechanisms.

We are looking for people to help us continue this journey of growing a company that resonantly contributes significantly to Open Source as part of its mission. If you are interested, contact me --- I am easy to track down via email.

Yes, I agree. I would love to see parts of what is in Pandas actually supported better in NumPy.

No. I'm not aware of this happening.

NumPy built on Numeric which was primarily written by Jim Hugunin while he was a grad student at MIT. While I was a professor at BYU, I wrote the the core of NumPy with a lot of community input -- outside of my regular job. I sold a book "Guide to NumPy" for a while that I used to replace the grants I should have been writing and pay for a grad student to help write iterative solvers for SciPy. Chuck Harris joined the NumPy effort early and has been steadily contributing ever since without direct funding. Many others have contributed volunteer time since then.

A major reason I started Continuum was to help create places where people could get paid to write open-source. I am happy to say we have been doing this for 5 years though mostly outside core NumPy itself (Numba, Dask, Bokeh, conda, etc.) We are working to support many more open source projects more generally -- and our devs have now made additional contributions to NumPy itself. We have a thriving 40 person Community Innovation team at Continuum supporting many open source projects. I expect this funding to help bring more new people to the NumPy development ecosystem.

The community also started NumFOCUS at this same time to be a community-run foundation that could be a focal point for donations and support to projects including NumPy.

Nathaniel Smith wrote a great proposal and put the effort into securing this funding. It is real work to secure funding. I look forward to NumPy getting better for the benefit of all because of this work.

Continuum co-founder here and previous NumPy developer. We received funding for Bokeh and also explorations around scalable "NumPy-like" work. The efforts around Blaze produced Numba, Dask, and Datashape as well as a more limited in scope library called Blaze.

While we continue to develop those things, we are working on taking the core ideas in NumPy and creating a set of lower-level C and Python libraries that could be used by pandas, xarray, Numba, dask, arrow, and potentially NumPy: https://github.com/plures. This work is not directly related to the NumPy funding.

I don't think this is very true. There hasn't been much private funding for NumPy. People working at private companies have worked sometimes on NumPy but very rarely as part of their full-time job.

Kudos on the post and for sharing. I really enjoyed reading the candor and hearing from your experiences. I heard amazing things about RethinkDB and admire your team for what you did.

There may still be a few routines that expect C-ordered arrays and so require a copy be made when given a Fortran-ordered array --- especially as you extend to one of libraries that use NumPy. For the most, part, however, Fortran-ordered arrays should work well. It all comes down to the expectation of the routine writer.

Lua is indeed a pretty nice language for some use-cases and it's nice to see this example. It would also be great to see a comparison to Numba which is a Python JIT particularly suited for fast numerical computing with Python: http://numba.pydata.org

Numba provides speed-ups over NumPy and Python and does so in way very easy for Python programmers to use with decorators on Python functions. Numba can be used to easily build new "ufuncs" which are NumPy's universal functions and it has multiple interfaces for programming the GPU directly from Python for even higher performance.

For some plugins integration just means visual integration (i.e. window-manager integration). And, to be clear, this one is not poorly-implemented. It's based on a very powerful web-application toolkit called PhosphorJS written by very experienced UI developers.

For other plugins there is very nice integration so that you can have interactions in one plugin update outputs in another and vice-versa.

The ease of creating a connected ecosystem of web-applications connected to your workflow is one of the powerful aspects of the new platform.

What would great integration look like to you?

Everyone has their own story around this. For me, the reason I did not go with Octave when I chose to migrate from Matlab to Python was that I wanted to be part of a general-purpose language community. Octave had the nice advantage of being very close to the Matlab syntax --- which has some nice properties (e.g. easier matrix-math expressions and matrix-building expressions)

However, I felt there would be more innovation around the programming language itself (think decorators, generators, futures, compilers, and coroutines). The boundary region between "applied-math-programming" and general purpose computing requires a lot of sometimes tedious work. Having a general purpose computing language would mean that more people would be available for that work.

Even now, one of the principal challenges of NumPy which is the foundation of the Python scientific stack is that it combines applied math (fft, linear algebra, polynomials) with computer-science (type systems, data-structures, and multiple-dispatch functionals). Maintaining all of that with one group of people is difficult.

If we could re-factor the NumPy code base into 1) a data-declaration type system --- i.e. look at datashape.pydata.org which is a generalization of dtype, 2) a multiple-dispatch generic function system (the ufuncs) and then 3) a container object. These could all be maintained by separate groups (and even #1 and #2 could be pulled into the Python language itself). See the libdynd project for a reference example of what it could look like.

Then NumPy could be maintained as a set of math libraries on top of that.

Then, it would be relatively straightforward to build the octave DSL on top of the Python computing stack and we would be able to share work.

Good luck, William. You have my full support! Thank you for all the great work on Sage and associated tools!

Bokeh can be used with both Python 2.7 and Python 3. Additional dependencies are needed to work with Python 2.7. In fact, the "write call-back functions in Python" capability uses a Python to JS compiler called Flexx -- http://flexx.readthedocs.org/en/latest/flexjs. Flexx is currently Python 3 only (though 2.7 support is coming for it).