maartenbreddels

I guess someone needs to build a Solara app running in the Jupyter Notebook to interface with Jupiter (running on Solana).

Yes, with pycafe that is possible: https://py.cafe/snippet/asgi/v1#c=H4sIABN3wGYAA41UTW_bMAz9K4...

Also check out https://py.cafe/ (I work on this project)

It supports streamlit, dash, virzo and solara. Frameworks like shiny, gradio and panel are planned (maybe even fasthtml).

It's also really nice to integrate this into your documentation: https://mkdocs.py.cafe/

We plan to support fastHTML at https://py.cafe/

This also uses pyodide, but supports more frameworks out of the box.

On https://solara.dev/docs/tutorial/streamlit we highlight the main differences:

* Solara will not continuously re-execute your script as Streamlit does.

* Solara will re-execute components instead, only what needs to. * State in Solara is separate from the UI components, unlike streamlit, where they are strongly linked.

* State can be on the application level (global) for simplicity or on the component level (local) for creating reusable components.

* Solara should not block the render loop. Long-running functions should be executed in a thread using use_thread.

Yes, the API is a bit more complex than before but it does give more options!

Thank you

It’s demoed in https://www.youtube.com/live/U9mJuUkhUzk?si=Ko3FZ0Axx7jcZaQ7 @ 35:11 Hope that helps :)

It can be used for general purposes, but we focus on the data science landscape. However, our main website is built using solara, and we know a few startups using solara for their main product (public-facing, not internal).

We (the authors of the Solara web app framework) were inspired by the OpenAI keynote Wanderlust app they demoed and rebuilt it in Python using Solara. It would be good to showcase our framework's power and inspire others to build UI's with AI/LLM elements in it.

GitHub: https://github.com/widgetti/wanderlust App: https://huggingface.co/spaces/solara-dev/wanderlust X: https://twitter.com/maartenbreddels/status/17223244907077020... Solara: https://github.com/widgetti/solara/

Thanks for the shout out.

The follow up of that is Solara: "NextJS, but in Python" :)

https://github.com/widgetti/solara/

Not sure where you found that link, but https://solara.dev/api should work fine!

Let’s take that question to GitHub or discord not to go too much off topic here ok? Thanks!

In Solara you can dynamically put together the UI like in react, in pynecone you need to use their primitives to make lists and conditional rendering. Also, state management is not tied to solara, but could use dataclasses or pydantic, wrapped in a reactive variable.

I think what sets Solara apart from the rest is that we use a 10 year proven paradigm, React, but on the Python side, although with a more Pythonic API. For instance making reusable components in Python is a real benefit. Pynecone uses ReactJS in the front end, but on the Python side it’s their own API.

I do like pynecone, their API is pretty. I do wonder with many of these frameworks. Does the paradigm works for more complex apps, or does it end in complexity hell once you go past the hello world examples. At least React has proven itself.

Co-creator of Voila here. Voila is rather un-opinionated about what you can run in it. While it offers excellent support for ipywidgets, it's certainly not confined to that. One downside of Voila is that it can be resource-intensive, as it creates a new kernel/process for each page view or browser tab.

Mercury, in my view, seems really interesting for quick demos and reports. It's somewhat akin to Streamlit, making it ideal for notebook authors.

On the other end of the spectrum (and pardon my shameless plug here), there's Solara (https://github.com/widgetti/solara/). Solara is a fully-fledged web framework for Python, specifically designed to handle app sizes that surpass what Streamlit and Dash can manage. It's worth noting that Solara can render ipywidgets, using a pure Python React-like framework. This can be particularly useful after the exploration phase, when you're looking to transition a project from the notebook into production.

Correct. We were already doing that with pure/classic ipywidgets based apps.

We helped many clients get their prototypes into production and identifier some friction doing that.

We now have a way to make more complex apps using the same stack, and get better app startup performance.

We're excited to introduce Solara: A pure Python web framework built for large, complex apps.

While there are many Python web frameworks out there, most are designed for small data apps or use paradigms unproven for larger scale. Code organization, reusability, and state tend to suffer as apps grow in complexity, resulting in either spaghetti code or offloading to a React application.

Solara addresses this gap. Using a React-like API, we don't need to worry about scalability. React has already proven its ability to support the world's largest web apps.

Solara uses a pure Python implementation of React (Reacton), creating ipywidget-based applications. These apps work both inside the Jupyter Notebook and as standalone web apps with frameworks like FastAPI. This paradigm enables component-based code and incredibly simple state management.

By building on top of ipywidgets, we automatically leverage an existing ecosystem of widgets and run on many platforms, including JupyterLab, Jupyter Notebook, Voilà, Google Colab, DataBricks, JetBrains Datalore, and more.

We look forward to your thoughts and feedback!

My guess is that should be possible, feel free to hop onto https://github.com/vaexio/vaex/discussions !

Thanks, glad you find Vaex useful.

Indeed, for small data there is not much to gain, at least this is not the focus of this article. Although even with small amounts of the, the automatic pipelines are useful https://vaex.io/blog/ml-impossible-train-a-1-billion-sample-...

AFAIK Modin tries to be the API compatible with Pandas, but then faster/distributed. Vaex tries to be a DataFrame library that is as fast as possible on a single machine to keep things simple and fast (although distributed is on the horizon, we don't need it currently). We're not afraid to break compatibility with pandas, because we care about performance. Both libraries try to hide the laziness from the user.

Yes, Arrow 3.0 has much more string kernels https://arrow.apache.org/docs/cpp/compute.html

For a visual output, voila can give you a dashboard look: https://github.com/voila-dashboards/voila/ see e.g. https://github.com/maartenbreddels/voila-demo for a mobile-ready responsive rendering of a notebook.

Thanks :)

The first commit was Jan 2014, when it between 50-80% of my time until 2018 I think where I mostly developed it myself. After that, it is more difficult to say how much time was spend on it, and it wasn't only my time. Although I do most of the development, ideas and discussions with data scientists can be more important than just pure dev time. But it took some time :)

Indeed, numba, Pythran or cupy can be really useful. In the example of the article, it is 'simple' vectorized math. But in general any function can be added in Vaex. Those that go through numba or Pythran usually release the GIL and can get you an enormous speed boost.

Indeed, numpy for numerical calculations. For strings, we have our own data structure based on Apache Arrow, but we plan/hope to move to Apache Arrow (in combination with numpy), since that's kind of the numpy++ for data science work.

What do you mean by that?

No Python looping happening in Vaex :), otherwise, we wouldn't get this performance.

We are also working on GraphQL support, with a Hasura-like API: https://docs.vaex.io/en/latest/example_graphql.html

I think GraphQL is easier in combinations with front end development, and you can tab-complete your way out. Early days for this sub-project, but I think very promising.

I've never benchmarked against postgres, but would be interested about the results. I once tried monetdb, and it was orders of magnitude slower for simple calculations, so I stopped looking at RDMS'es after that.

I think they solve a different problem, smaller data, data/relational integrity, data mutations. Dataframes can take shortcuts here.

If you want to do a serious benchmark, feel free to contact us. Github: https://github.com/vaexio/vaex/issues Email: (I'm easy to google).

It is not similar to Dask, but similar to dask.dataframe. Dask.dataframe is built on top of Pandas, but that also means it inherits its issue, like memory usage, and performance. (BTW, totally a fan of Pandas).

Xarray is more about nd-arrays, less about ~tabular data.

Vaex is built from the ground up with the idea that you can never copy the dataset (1TB dataset was quite common). We also never needed distributed computing, because it was always fast enough, and thus never had to use dask (although we're eager to support it fully).

Also, vaex is lazy but tries to hide it from you. For instance, if you add a new column to your dataframe, it will only be computed when needed (taking up 0 memory). However, in practice, you're not really aware of that. This means it feels more like pandas (immediate results) than dask.dataframe (no .compute()/.persist() needed).

I would say they are all complementary, with small amount of overlap. Small data: use Pandas. Out of memory error: move to vaex. Crazy amount of data (100TB?) that will never fit onto 1 computer: dask.dataframe, or help us implement full dask support.

Don't expect this performance on a 4GB machine. Most machines now would have 16GB, or more. Let us assume you have 32GB and take a 24GB dataset. Most libraries load this into memory (allocating 24GB, leaves max 8 GB for the OS, including disk cache). The next process that wants to do the same cannot without a memory error. Also, when you restart your program, the OS will not have it in the cache, it will be as fast as your hard drive.

Vaex is much smarter with memory, it memory maps the data, nothing is allocated, all the memory is left to the OS for disk cache. This means you can have 10 users regularly restarting/running their program, without any (or minimal) harddrive activity. So I think it is a fair statement, and actually very conservative (see my other comments). An important message is to make people aware that working with a dataset this large is a pip install away, no need to spin up a cluster (yet).

the 'convert' name is misleading perhaps, maybe we can agree the proof is in the execution time https://youtu.be/TlTcQJPUL3M?t=478 Anyway, let us celebrate a wider adoption of Arrow! :)