Ludic: New framework for Python with seamless Htmx support

by paveldedik • 2 years ago

205 105 2 years ago

paveldedik5• 2 years ago

I've been working on a lightweight framework called Ludic that focuses on web development using Python, htmx.org, and a somewhat React-like component approach. It leverages Starlette for performance and the latest Python 3.12 typing features.

Interested in feedback!

* Docs: https://ludic.readthedocs.io/ * Code: https://github.com/paveldedik/ludic/ * Examples: https://github.com/paveldedik/ludic/tree/main/examples

sesm• 2 years ago

By React-like component approach, do you mean using functions instead of templating language?

paveldedik5• 2 years ago

I think it is somewhat similar to React, for example, you can create a Link component like you can see in the README.md

Now you can use it in f-strings, and well, other components.

The idea from the beginning was to integrate it with htmx.org, so to me, it feels kind of similar when you are writing endpoints using these "components".

But I understand why you are raising this question, I also didn't know how to name it, and React is a completely different framework. I didn't know how to describe the framework in just a few sentences but wanted to somehow. I might change the description at some point.

I still don't know how useful the framework is, I am playing around with it and I like it. Who knows what other people think, though.

// edit - code block don't work here, removed code sample.

rmbyrro• 2 years ago

I don't know, every time I see people sneaking html into Python feels weird and wrong to me...

andybak2• 2 years ago

Do you feel the same about sneaking html into javascript? Just curious if it's a Python-specific objection.

shinryuu3• 2 years ago

same feeling there tbh. Though generally, when it's done in javascript it's done completely in a string, without javascript objects.

legutierr• 2 years ago

In general, this usually results in front-end logic being very tightly coupled with back-end logic. In some of the examples given, you even have database access in the same line that is generating the HTML document.

https://github.com/paveldedik/ludic/blob/main/examples/click...

It's the kind of thing that looks very cool and concise in small examples, but tends to become a nightmare when you are working on larger projects.

paveldedik5• 2 years ago

Yeah, I understand that. Now that Python 3.12 has better support for f-strings, I thought there might be a way to make it possible. But I am still not sure it will work. There is a pretty weird hack to make f-strings work while avoiding the possibility of rendering unsafe user input.

creshal• 2 years ago

It's nice for small hacks, but for anything longer than a couple dozen characters you're probably still better off making jinja2 (or similar) templates as painless as possible.

kissgyorgy4• 2 years ago

Until you try it out and realize the benefits!

- Type safe HTML: you get an exception instead of malformed HTML

- Truly reusable components: any Python web framework, component packages are truly reusable in any other projects. I never seen this with template engines.

- Huge productivity boost: not even close! No hunting for templates, no jumping between files, everything is in Python functions.

- Composable elements: you can nest things as much as you want, refactor very easily. Doing this with templates is not even possible, or such a pain I never did it.

- Storybook: There are a couple ones for Django, but they are clunky and not reusable at all.

I felt the same for a long time, but started thinking about a nice API. It took me months, but I finally got it: https://github.com/kissgyorgy/compone

grrandalf• 2 years ago

Insightful summary. Thx.

mixmastamyk6• 2 years ago

Yes, I quite like the idea of being able to run pyflakes/flake8/black/pytest/mypy over my "html templates", and benefiting from all the power that entails.

icyfox• 2 years ago

I imagine a lot of this comes down to personal preference. In the early days of Mountaineer [^1] (gee, almost two months ago at this point), I played around with the idea of embedding html into python instead of needing a JS layer. Eventually my consensus was:

- The most flexible shim approaches typically end up wrapping JS/React components anyway (like Reflex/Pinecone)

- We really need better IDE support for html strings that are within python strings. The editing problem is a big setback.

The ergonomics of Python + JS in separate code files won out and the user experience has been better than forcing them both into a common language would be.

This has the benefit of leveraging whatever the two languages are best at, in native code, so you have access to all the native APIs without having to learn a shim on top of it. Way more longevity to that approach too. Context switching between two languages isn't that bad if you minimize the glue layer that you have to write between them.

[^1]: https://github.com/piercefreeman/mountaineer

rougemine• 2 years ago

It feels weird at the beginning, but after a bit of practice I found it pretty nice to write HTML in Python.

Here is an example of a HTML page layout written with the DOMinate [1] library for example, in a "JSX-like" way:

https://github.com/olivierphi/zakuchess/blob/main/src/apps/w...

It may hurt your eyes at first sight, for sure... But similarly to technologies like Tailwind CSS, it's mostly a matter of getting used to it - and after a while it end ups feeling very natural to use :-)

1: https://github.com/Knio/dominate#readme

VagabundoP• 2 years ago

Dominate is quite fast as well. I did a few tests vs jinja for my flask app and it beat the pants off it.

whalesalad• 2 years ago

recall python server pages ... good old index.psp

AsthmaBoy• 2 years ago

I agree and I don't fully understand the why of it.

I remember coding PL/SQL to emmit HTML in Oracle around 1999 or 2000 and using functions to code the various elements.

That got old and repetitive very quickly - for instance, everytime I had to correct a spelling error, I had to recompile the code.

To get around it I used one or two tables to hold html snippets to decouple the business/backend logic from the frontend, and stopped using the PL/SQL functions completely.

My speed of developmet skyrocketed, and separating and abstracting the frontend from the backend made so much sense.

A few years later, I was doing web developemt with Python using the Zope framework (not many people know about it tiday, I think).

It uses a specialised serverside templating language called TAL (Template Attribute Language)[1] that basically builds the front end dynamically, and then you feed it data from the backend.

Very neat and allowed me to build reusable compoments as well as collections of a schema definition (basically a dict), html template(s) and the code to validate that the input matched the schema and could be rendered.

Or something like that - its been 2 employers and almost 20 years since I worked with that :)

I did build a small extension for Wordpress using a PHP implementation[2] of TAL a few years ago, and TAL still works like a charm :)

My point is that I still believe there is value in keeping python out of the html-templating, and in keeping the front end logic apart from the backend logic.

There is something I am not understanding about the renewal of mixing HTML/GUI template with code, but I haven't fully found it yet.

[1] https://en.m.wikipedia.org/wiki/Template_Attribute_Language

[2] I believe it was this one https://phptal.org/

sergiomattei• 2 years ago

I’ve been burned by ASGI and asynchronous python so much in the past, I wouldn’t touch it again even if someone paid me for it. It’s prone to bugs, hard to debug and for little benefit.

shinryuu3• 2 years ago

How have you been burned? What kind of bugs did you have, and in what way was it difficult to debug?

timhh2• 2 years ago

Here's a bug I ran into with Python async:

https://stackoverflow.com/q/78036302/265521

I've written about 200 lines of Python async code in total, so to run into a bug like that so soon was not encouraging.

I suspect it's just not a very popular feature and so it doesn't get a lot of use and debugging. And it's Python so it really needs a lot of ton of real world use to detect bugs.

Anyway I'm not going to waste my time debugging Python internals so I just switched to Deno.

PaulHoule3• 2 years ago

Myself I built an RSS reader using aiohttp and then forked the code to make an image sorter.

I had a lot of people tell me I was crazy to write aiohttp servers because of the problems the previous poster mentioned but I had a long run of it working pretty well. On the other hand I’ve frequently had the experience of writing something in Python that falls apart like a tower of Jenga blocks the moment somebody else touches it. (Quite different from my experience with Java, Javascript, C#, PHP, …)

Once I started serving images along with the text and using the app over a 2Mbps link it seemed to me the aiohttp server was getting a little unreliable.

Most basically, an aiohttp server can only use one CPU thread at a time with the consequences that if one request holds the CPU for 5 seconds the server stops processing requests completely.. Although the aiohttp server could be reliable if you are almost entirely waiting for data to get back from the database, the residual amount of CPU load is going to put an upper bound on what the server can handle in the best case. If a coder is not so careful you get burned. My RSS reader has an AI component which runs completely in batch jobs, I don’t do any inference during requests which limits my flexibility and makes the application less “real-time” than it could be.

You can imagine systems where one aio server hands tasks off the another aio server but the blocking situation doesn’t get any better and if you want to make the system scalable there has to be at least one multiprocess or multithreaded server somewhere. At some point I am like “I have a 16 core machine but I am only using 1 core like my laptop in the 1990s” It didn’t help that my aio database drivers aren’t in conda which is another build hassle.

So last week I tore up the image sorter and rewrote it in Flask (my aio framework looks like Flask so it’s not too hard) and I run it out of gunicorn, I put a reverse proxy in front to deal with some Tailscale problems so I have IIS serving the images.

kissgyorgy4• 2 years ago

asyncio makes the codebase concurrent, which is an order of magnitude harder with all the concurrency problems! I agree it shouldn't be the default

j4mie2• 2 years ago

I wrote a library a little while ago which is intended to be the simplest possible expression of this idea:

https://github.com/j4mie/hotmetal/

The core implementation is only 85 lines of Python, and has no imports at all, so works nicely on minimal/alternative Pythons like Micropython.

I'm using it on a medium-sized side project and it's shockingly productive compared to string templates. I don't think anyone is using it except me, though.

eddieroger2• 2 years ago

We've come full circle on naming HTML tools!

https://en.wikipedia.org/wiki/HoTMetaL

j4mie2• 2 years ago

Glad someone got the reference!

slifin• 2 years ago

Search for "clojure hiccup"

egeozcan• 2 years ago

I use gomponents with go, which is very similar. The productivity increase is immense. https://github.com/maragudk/gomponents

tzimo• 2 years ago

This looks really neat. One suggestion is that in your readme, show the webpage output with your examples. It’s hard to image exactly what the page will look like just looking at the hotmetal python code

fermigier2• 2 years ago

Cool idea. I had a similar one a few weeks ago, with this additional idea of turning this into a "real" templating language (e.g. by adding a special construct for conditionals, looping, variables, macro expansion...).

samdudley• 2 years ago

Your project looks great and I like the fact you can use it with MicroPython.

I've also been working on a library with a similar idea:

https://github.com/SamDudley/neat-html

It's still a work in progress so an example might be better to look at:

https://github.com/SamDudley/neat-html/blob/main/examples/fo...

I was inspired by some JavaScript libraries that I've used in the past:

- https://github.com/jorgebucaran/hyperapp

- https://github.com/hyperhype/hyperscript

There is also a working integration with Django that enables the use of neat-html as a template backend, however it isn't up on GitHub yet.

I find the space of HTML generation libraries which can leverage the power of Python, really interesting.

borntyping• 2 years ago

I've been looking for something exactly like this for a while — a small Python library for generating HTML that isn't prescriptive about framework and doesn't introduce any particually weird syntax. Jinja2 macros are... not great when you want to turn your HTML into lots of smaller functions.

baq• 2 years ago

see also: twisted nevow + stan, which I believe predate the term 'AJAX'.

https://github.com/twisted/nevow/blob/master/examples/simple...

elbear• 2 years ago

This looks like a similar model to how Elm handles HTML. Except Elm provides functions for each tag, so it also does type checking to ensure that the passed arguments correspond to the actual attributes of the tag.

mixmastamyk6• 2 years ago

Reminds me of dominate.

polyrand2• 2 years ago

This is wonderful, I love it. I recently wrote a blog post about generating HTML in Python using LXML (https://ricardoanderegg.com/posts/python-build-html-componen...) and I didn't know about your library. I'll update it to add a mention to it.

Apart from the simplicity, I like the fact you can "install" it by copy-pasting the code at the top of your script, or running:

  curl -L https://raw.githubusercontent.com/j4mie/hotmetal/main/hotmetal/__init__.py >> myscript.py

which is nice when you want a self-contained file.

I think I'll be using this in the future.

asa400• 2 years ago

Nice work! There are a few expressions of this idea. There's a reason people keep coming back to it: it's actually really nice to write markup this way.

https://github.com/weavejester/hiccup https://github.com/JuneKelly/sneeze

giovannibonetti• 2 years ago

Elm [1] is based on a similar idea. Build your app from pure functions that return HTML tags.

[1] https://elm-lang.org/

jpgvm• 2 years ago

I encourage anyone looking to do htmx to consider Kotlin and it's excelent kotlinx-html DSL. Lets you do stuff like this: https://github.com/corlaez/kotlin-htmx/blob/master/src/main/...

Disclaimer: Not my code, just an example I found.

Neikius• 2 years ago

This is a lovely use of the DSL..too bad I prefer plain java.

smallerfish2• 2 years ago

I second this. kotlinx-html is poorly documented and has a couple quirks, but is fantastic. As I've been building one of my current projects, I've also been building a framework based on jooq + kotlinx.html + javalin + htmx + flowbite (+kotlinjs for a few fancy bits, e.g. the charting library). Aside from tailwind, you end up with a fully typed and compiler-checked application end to end, i.e. no magic strings to be seen. I'm exclusively using HX-Retarget, so the backend is in charge of where things land, meaning that the frontend can do things like:

   div {
      onClick(loadItems)
   }

I don't know whether I'll ever actually get it in good enough shape to release, but I do plan to blog about how some of the tricks I've found it fitting it together. One thing I definitely need to do before anybody looks at it is clean up how routes and targets are managed.

hjadal• 2 years ago

That sounds like a lovely way to develop, what is your blog? So I can watch for your post.

efilife• 2 years ago

https://youryoure.com/?its

kissgyorgy4• 2 years ago

I also started working on a Python Component library, but it's fully independent of any web framework, and you can generate XML/RSS with it too. I'm using it for months now, I will release the first version soon: https://github.com/kissgyorgy/compone

danpalmer3• 2 years ago

Nice release! This looks good, docs look really nice too. I'd use HTMX anywhere I can to improve a basic HTML site.

One word of warning for people considering HTMX though – it's not a framework, it's an appliance.

In my experience with it, I ran into problems integrating with HTMX very quickly. Drop it into your project, use the HTML attributes, and it'll be great. However if you need custom Javascript, HTMX provides little in the way of an integration path (the lifecycle events are basic), and nothing in the way of structure, and can even get in the way in places where its lifecycle causes issues in the lifecycle of whatever you're adding. To add your own Javascript you'll almost certainly want Jquery or React/Svelte/etc, and with the latter you'll get interactions between their lifecycles.

If you're using it entirely via the HTML API, and you're happy to accept that sometimes it won't do something, you'll be fine. If you're looking to add units of more complex client-side functionality in places, try prototyping with it to make sure it works well, it didn't for me.

unclebucknasty4• 2 years ago

I think those are all good points, but do cringe a little at the idea that "to add your own JavaScript you'll want React". Seems like a lot to bring in if you're really just wanting to add some JavaScript.

danpalmer3• 2 years ago

Ha, fair point. I don't mean so much that for any JS you'd want to bring in React, but more that something like that will likely be needed to manage a codebase of any scale.

My context here is a mostly server-side application using HTMX, where I wanted to add a few small bits of additional client-side code. I tried to do it without any libraries but rapidly realised that Jquery would simplify things a lot, and managed to do ok with that, but was still fighting HTMX throughout, and I ended up with a lot of poorly managed local state quite quickly. It doesn't take much complexity to need some better form of state management, and something like React provides some direction there, but of course there are other options, my point was only really to say that HTMX will not do any of this for you, and may get in the way depending on what you need to do.

jerrygenser• 2 years ago

React can be added by rendering into a dom node that's not necessarily the root of your app.

Yes, it can be used to render the entire UI app with rich state management across all pages, routing, etc.

However it can just as simply be used to render 1 complex part of an app that has an isolated but possibly complex state. In this scenario I think sprinkling react is reasonable.

Epskampie• 2 years ago

True, I can recommend hotwire turbo as an alternate similar to htmx, it has a js framework called Stimulus that integrates nicely for those instances where you need some custom interactivity.

rokkitmensch• 2 years ago

django-turbo is awesome, author abandoned it for reasons that I do not quite understand, as it's entirely functional.

_heimdall• 2 years ago

I've actually found HTMX to work really well with custom elements for basic interactivity.

If most of your logic is server-side and client JS is kept small, custom elements let you lean on lifecycle events built right into the browser specs.

cipherself2• 2 years ago

Generating `HTML` from lisps has poisoned any other approach for me, see for example https://www.neilvandyke.org/racket/html-writing/, https://reagent-project.github.io/, and https://edicl.github.io/cl-who/

callamdelaney• 2 years ago

That's very nice, I wonder if we can write a sort of dsl for this in python.

skgough• 2 years ago

Airium is close: https://gitlab.com/kamichal/airium

explorigin• 2 years ago

The idea of nested function calls to build HTML is not new. Back in the hey-day of JS frameworks, this was a common vdom pattern. I kinda miss [MithrilJS](https://mithril.js.org/#dom-elements)

dkuznetsov• 2 years ago

It is a bit funny. When I started with Python ~18 years ago, the team lead on that project liked this framework "Nevow". It wad plugged into Twisted. Completely async, IIRC JS could have been built in, too. Some ideas are really not new under the sun.

Thanks for the wave of nostalgia.

fermigier2• 2 years ago

I have been working on a (currently half-baked) similar project:

- <https://github.com/abilian/webbits>

Inspirations for the idea of generating HTML code from Python:

- <https://www.yattag.org/>

- <https://tylerbakke.github.io/MarkupPy/>

- <https://github.com/michaeljones/packed>

- <https://github.com/twidi/mixt/>

- <https://github.com/byteface/domonic>

- <https://pypi.org/project/hyperpython/>

- <https://pypi.org/project/PyHTML/>

- <https://github.com/jviide/htm.py>

- <https://viewdom.readthedocs.io/>

- <https://github.com/pcarbonn/fast_html>

- <https://github.com/sanic-org/html5tagger>

- <https://github.com/j4mie/hotmetal/>

hawski• 2 years ago

I can only say that I like fast_html and how simple it is. Very easy source to read and understand. It is based on generators internally.

I started using it for some documentation generation purposes. I'm not a web-dev.

aitchnyu2• 2 years ago

Wish GVR updates pyxl4(used by Mixt) and includes it into Python. It could be used for html, xml, yaml and its config language friends and declarative UIs like React. IMO if and for are better implemented in Jinja or JSX better than Python's ternary and list comprehension. Comparison at bottom.

Most of the frameworks here are calling side effects upon shared objects in the guise of `with` statements. Not to mention the extreme verbosity.

<if cond>
<long code 1>
<else>
<long code 2>
[longcode1() if cond else longcode2()]
<for i in ls>
<long code 1>
[longcode() for i in ls]

maxpert• 2 years ago

On similar lines, I've put together a library for myself that is framework agnostic:

https://github.com/maxpert/htmxido

Hopefully will be useful for people as well

ta988• 2 years ago

Can it be used to make static html?

mixmastamyk6• 2 years ago

Looks like ludic.html could.

unclebucknasty4• 2 years ago

Call me traumatized by the Web framework wars, but this seems like the path back to some of the issues that HTMX is trying to solve.

There is a strong developer instinct to "simplify through complexity". That is, to stuff down so much functionality and saw off all edges in an effort to hide complexity. "Let's expose a simple interface that will allow devs to do everything they want to do."

It starts with good intentions and initially provides some improvements in productivity. But it continues to evolve to the point of untenability: the complexity starts to ooze out and become a thing to manage, but with abstractions that make it more difficult; the framework becomes so opinionated that it begins to stifle; performance issues are introduced as we drift farther from the metal.

We continue to try to solve with newer constructs and abstractions, frequently introducing more issues. Eventually, it comes full circle ("hey, let's add SSR!"), and we're again looking for another solution. One that's simpler.

Then the cycle starts over again.

I think there is an inflection point with all of these things, wherein whatever perceived productivity gains are had by pushing more functionality into the framework quietly begins the path to regression. I don't exactly know where it is, but I would bet that it happens far sooner than we suspect.

So, I kind of like HTMX where it is, in spirit and implementation. Whatever additional productivity I can pick up via helper functionality and reuse is something I believe there is value in devs doing on their own, for their individual use cases and preferences. There's a tendency to think "well why reinvent the wheel? Let's make it a framework". I've come to believe that impulse is somewhat pernicious in its effect. Ultimately we're going to have to write our own code at some point. I'd suggest that we're better off doing it much sooner than frameworks encourage.

m_rpn• 2 years ago

Spot on, in my opinion we need at least a 10 years ban on the creation of any new web framework of any kind. Hope that AI coders will make frameworks obsolete, but maybe i'm crazy :-).

onecandreamz• 2 years ago

I wish someone would sponsor development of HTMX support via NginX or Apache2 modules. That way you don't have to go to python, php, java, etc... to get HTMX support. You would just enable the module, configure a few things and let it rip. Kinda like server side includes in Apache2.

stefanha• 2 years ago

Can you explain what you mean? htmx is client-side, so you don't need server-side includes. Just host HTML snippets on your web server and let htmx stitch them together in the browser. I'm not sure where you want the web server to play a role?

sureonesec• 2 years ago

HTMX is a client side JS file which parses the dom and then interprets it in a certain way and then provides some functionality. However the choice whether someone creates the initial html in a SPA context or a full page refresh is up to the developer. Maybe I want to generate the HTML snippets HTMX processes from the server side in PHP/Python and then have HTMX run it's magic on page load.

DrFalkyn3• 2 years ago

There are some elements of HTMX that are done server side, such as HX-Trigger

polyrand2• 2 years ago

I think these kinds of tools are awesome, although I normally prefer not having to rely on getting the functions/classes/types defined for me, I usually find it easier to just use lxml[0] to build HTML components from composable Python functions[1], similar to React.

[0] https://lxml.de/lxmlhtml.html#creating-html-with-the-e-facto...

[1] https://ricardoanderegg.com/posts/python-build-html-componen...

mixmastamyk6• 2 years ago

Why does it use async? I thought that was for scrapers and the like, not producing a primary web page or app.

.js callbacks later to fill stuff in, sure.

mjf988• 2 years ago

cool!

Whats wrong with just writing HTML f-strings and using flask tho?

pelme2• 2 years ago

This looks great! Building HTML on the server with static types and htmx is a very productive combination!

At work, we have been exploring alternatives to templates in our Django project and just released it as a lib to help with this:

https://htpy.dev/

aitchnyu2• 2 years ago

Whoa, a creative (in a good way IMO) use of __ getitem__() dunder method to depict attributes and children. No action at a distance using with blocks, static and strongly typing instead of stringly typing, no need to extend Python, composable from fragments.

Have you tried to "bring up front" if and for? For example

if_(cond, tag1(), tag2()) # without eagerly execute both branches

Instead of

tag1() if cond else tag2()

And

for_(cars, lambda car: car_details(car)) # while still being type checked

Instead of

[car_details(car) for car in cars]

pelme2• 2 years ago

Python's inline if's still read backwards to me and it would be nice to switch the order. But we are kind of stuck with it. I also don't want to introduce new control structures in this lib. Just want to use whatever Python has to offer in terms of control structures/syntax to make it more approachable.

htpy supports generators/callables as children (https://htpy.dev/streaming/). As long as you are careful to wrap things in generators/lambdas everything is lazy. Implementing if_ and for_ is straightforward and anyone can build constructs like that to use. But will not be lazy unless all arguments are wrapped in lambdas:

  from htpy import div, li, ul

  def if_(cond, a, b):
      return a() if cond() else b()

  def for_(items, func):
      return (func(item) for item in items())

  print(div[if_(lambda: True, lambda: "True!", lambda: "False!")])
  print(div[if_(lambda: False, lambda: "True!", lambda: "False!")])
  print(ul[for_(lambda: ["a", "b", "c"], lambda x: li[x])])

output:

  <div>True!</div>
  <div>False!</div>
  <ul><li>a</li><li>b</li><li>c</li></ul>

yashasolutions• 2 years ago

Nice! A bit verbose for my taste but writing for the web in python is a space where all innovation is welcome.

I personally like hyperdiv [1] approach (even if no use of Htmx) which is very intuitive for python code.

[1] https://github.com/hyperdiv/hyperdiv

pietz• 2 years ago

I've been diving into Python/FastAPI + HTMX development and while I love both separately, I felt like the combined tooling could be a lot better. Jinja2 doesn't really satisfy me. At least not in this tech stack.

I'm still a bit sceptical if this is the solution but I'm definitely going to look into it as it fixes a few things I'm annoyed with.

Thanks!

mplemay97• 2 years ago

Agreed. As someone who primarily develops in python, I recently tried this exact same stack as a way of avoiding the inevitable (i.e. learning typescript). After dealing with the complexity of interleaving templates, overly complex information flows, etc. I copied some of the html from the jinja templates I created into a svelte project, added openapi-typescript to glue it all together, and moved on. While I am by no means a fan of typescript and modern web development, it is much easier build things given the extensive ecosystem. And if nothing else, the “frontend” is separate all of the core logic, which makes things easier to understand.

Ludic: New framework for Python with seamless Htmx support

Related Stories