Briefcase: Convert a Python project into a standalone native application

https://briefcase.readthedocs.io/en/stable/

by generichuman • 3 years ago

182 62 3 years ago

briefcase.readthedocs.io

nu11ptr• 3 years ago

How does this compare to PyInstaller or other packagers? What underlying tech does it use?

WillAdams2• 3 years ago

The big difference is it uses "Toga"

https://beeware.org/project/projects/libraries/toga/

which uses native widgets and manages basic functionality such as where the "Quit" menu entry is automatically.

Looking into this now --- it looks perfect for my needs, just hoping that there's a nice graphical tool for laying out a GUI (if it has a graphical programming mode like to Blockly or Ryven, then it is perfect for me).

fock• 3 years ago

so essentially what Kivy did already a bit ago.

skinner927• 3 years ago

It appears PyInstaller is much more portable. Briefcase seems to make an .msi installer on Windows and an AppImage for Linux. Both of those require installation to use. PyInstaller can create a single executable that you can just run (yea sure it extracts itself to a temp dir, but the optics are very different).

mvdwoord• 3 years ago

Although I have never used any of their stuff, I had the pleasure of watching Russell from BeeWare at Pycon Colombia some years ago. It consistently presents itsself as an ecosystem I would like to work with, and if anything the naming of their projects is very nice.

https://www.youtube.com/watch?v=TY4vb6V-gvc

velosol• 3 years ago

Anyone happened to use Toga with video streams? I've got some gstreamer stuff with Qt and it doesn't need a rewrite yet but it's a glimmer on the horizon.

mike_hearn7• 3 years ago

Interesting timing! We're just in the middle of adding Python support to Hydraulic Conveyor, which is a similar tool [1]. There's a github issue [2] and mailing list that'll get notified when it's done. Disclosure: it's commercial but free for open source projects.

There are many of these open source packagers and they all share very similar problems:

1. They don't let you do software updates, even though software updates are practically mandatory for any real project. Electron is a stand-out here because it does address this, but their update engines are unmaintained for years and have some major unfixed problems (causes a lot of issues with Windows networks, for example).

2. Even in the very rare cases that they do, they don't let you force updates on launch even though many apps need something like this to keep up with protocol changes. It's one of the reasons people like web apps.

3. They don't help you with signing. For example they don't simplify key management, they don't support cloud signing (essential since May because Microsoft now insist on HSMs for all keys, not just EV keys), or they don't do notarization, or they don't generate CSRs for you.

4. They require the use of CI to cross-build even when apps are written in portable frameworks that don't require compilation. This is because they are just thin wrappers around the native tooling.

5. They're invariably language specific even though there's no good reason to be because 80% of the work is the same regardless of what language or framework you use.

6. They make MSI files for Windows even though MSI is deprecated.

It's possible to bite the bullet, chew glass for a while and solve all these problems, which is what we did for Electron/JVM/Flutter/native apps. You can reimplement all the native tooling so users can cross-build (i.e. make Mac packages from Linux/Windows, Windows packages from Mac/Linux etc), which enables releasing from developer laptops or cheap Linux CI workers. You can support software update by integrating Sparkle on macOS, apt on Debian/Ubuntu and by using MSIX on Windows (and by then working around all the bugs in Windows to make it work well). You can generate download pages that work out the user's OS and gives them the right download, and instructions for how to install self-signed apps if the developer isn't code signing with a recognized certificate. You can abstract platform neutral things and expose platform specific things. Then you can write a parallel incremental build system so doing all the work is as fast as possible, and write lots of code to detect all the myriad mistakes people make and give good error messages or auto-fix them. Then you can make it support GitHub Releases. Then you can document it all.

But that big pile of glass isn't particularly tasty, which is why open source projects don't do it and we ask commercial users to pay for it.

Briefcase looks nice but it also seems to have all the problems listed above. I think once we add Python support Conveyor will be quite useful for the Python community, especially if we can find a workaround for pip not support cross-building of venvs. It would be great if you could just whip up a quick Python script, run one command and your installed clients start automatically updating, your download page updates, and the whole thing is no harder than releasing a static markdown-rendered website.

[1] https://hydraulic.dev/

[2] https://github.com/hydraulic-software/conveyor/issues/73

formerly_proven2• 3 years ago

essential since May because Microsoft now insist on HSMs for all keys

(Except their own authentication keys, of course)

mike_hearn7• 3 years ago

I'd be amazed if Microsoft held any important keys outside of HSMs. I think the incident you're referring to is where they appear to have been hacked by China (maybe the Chinese govt) and they blamed a "validation error in Microsoft code". HSMs help a bit but if the code that accesses them is insecure then you can still sign bad things. The HSM just means you don't have to do a key rotation to recover.

On that topic, one feature Conveyor has that open source packaging tools usually don't is that it can protect signing keys in your macOS keychain, in some cases. To utilize that the packaging tool itself must be signed, which of course it is here, and this is useful because the OS will protect both the private key and the process that's working with it from malware even if it's not explicitly sandboxed. The keys themselves are protected using the security chip. So it's a very high level of security that's sort of equivalent to having a dedicated user account for signing on Linux, but without needing to set anything up.

Unfortunately not every credential can be stored there yet, only the "root key" from which other keys are derived. Login credentials for Apple's notarization servers and cloud signing services like eSigner/KeyLocker don't get protected in the same way yet. But that'll come.

zie• 3 years ago

How do I get notified when you have python working? I want on that list.

mike_hearn7• 3 years ago

Follow the GitHub issue or enter your email address in "Subscribe to the blog" on https://hydraulic.dev/

ptx2• 3 years ago

MSI is deprecated

Is it? The WiX Toolset still seems to be actively developed and MSIX doesn't seem to support all use cases. Has MSI been officially deprecated despite that?

Also, MSIX's mandatory signing makes it too expensive and cumbersome for a hobby project, so it would be unfortunate if that became the only option.

mike_hearn7• 3 years ago

It was marked as such at one point on their website, but now I searched for that again I can't find it anymore. Maybe they undeprecated it, hard to keep track of what MS are doing. It's not been developed for many years though. All the new features seem to go into MSIX.

MSIX files can be self-signed if you install the cert first. Conveyor has an EXE wrapper that can do that for you. So you can distribute apps with it without buying signing certificates.

anonymouse0083• 3 years ago

I'm going to ask a 'self-taught' question, so my terms may be incorrect.

Has anyone solved the compiled libraries / dynamic libraries / linking problems for multiplatform apps? Everything works just fine with these python>native app tools until you hit some esoteric library that's not pure python.

willcipriano2• 3 years ago

Solved as in you don't have to do anything and it just works or solved as in there is a way to fix the problem but it requires some effort?

anonymouse0083• 3 years ago

Ha! Yeah I want the magic wand.

I have deep wounds from pre-chatGPT times of trying to manually figure out libraries. As in otooling each library, error, etc then recompiling into various places until things 'worked.' I'm appreciative of the experience, though now I think with LLMs the problem could be solved with some simple queries.

So I guess you're right, it requires effort, but I was a 10x engineer in the wrong direction and never want to experience that again ;)

KeplerBoy• 3 years ago

esoteric as in a whole lot of essential libraries?

stuaxo• 3 years ago

This needs solving.

It makes it really hard to distribute certain python libraries.

schemescape• 3 years ago

For Node, I’ve seen packages just bundle a copy of the shared library for each supported platform. I would assume you could do the same for Python.

aragilar• 3 years ago

That's what wheels do.

BiteCode_dev2• 3 years ago

Nuitka deals pretty well with those in general: https://nuitka.net/

Jack5500• 3 years ago

Until you require pytorch ;)

pjc50• 3 years ago

Python remains a mess for these use cases because it does not have packages.

It has a very specific filesystem layout that it calls a package, but this is not a first class citizen; it has a packaging format called "wheel" that is now used for distribution, and that can contain native libraries. That currently mostly handles the distribution of compiled libraries for multiple platforms, but only within the traditional venv-style setup. Being able to ship interpreter+venv+launcher would go a long way towards providing a native-ish experience.

A step beyond that would be being able to distribute a "virtual filesystem" containing a whole tree of python compiled as .pyc, appended to a native executable containing the interpreter and linked against the required native libraries already.

piyh• 3 years ago

Venvs not being able to be moved to other directories is such a PITA.

simonw2• 3 years ago

I'm a huge fan of https://github.com/indygreg/python-build-standalone which provides Python builds that CAN be moved around and work independently of any other Python installation.

I used that for my own Python+Electron app, which I wrote about here: https://til.simonwillison.net/electron/python-inside-electro...

oefrha• 3 years ago

PyInstaller bundles native dependencies just fine. Well, for Windows and macOS at least, never packaged as “native app” for Linux.

Frost1x• 3 years ago

Eh it depends, some python packages do things like pull data samples from my experience and it can be quite a mess detangling the package to give PyInstaller everything it needs to bundle stuff up. It also takes a lot of additional configuration for a lot of packages.

As much as I hate how containers are often misused, this is one of the reasons people obsess over them from my experience. The ability to bundle all your crap regardless of how many layers of package management idiosyncrasies your application ultimately woul have to deal with pretty much transparently.

simonw2• 3 years ago

Python wheels really can help here, but only if the downstream packages you depend on have published wheels for the various different platforms you need to support.

https://cibuildwheel.readthedocs.io/ is a great tool for assisting with that - many projects run that in GitHub Actions as part of their release process.

There's just one catch, in my experience: since GitHub don't offer Apple Silicon machines as part of GitHub Actions, it can be hard to build wheels there that work on those platforms. Maybe you can cross-compile them? I've not figured that out yet.

eyegor2• 3 years ago

I vaguely remember nuitka or pyinstaller being able to handle this for me, but I used vms or qemu to build them "natively" for each target platform. Never found a way to solve cross compiling.

yellowapple2• 3 years ago

If you’re on Linux, you should use the system Python provided by your operating system. You will be able to complete most of this tutorial using a non-system Python, but you won’t be able to package your application for distribution to others.

Why not?

packetlost• 3 years ago

Probably because it doesn't actually package the Python interpreter into the executable and relies on a hashbang (ie. `#!/bin/python`) to actually run the bundled source files. Pex [0] does the same thing.

It's a half solution at best, but if you control the infrastructure it does dramatically simplify things like Dockerfiles and other deployment stuff. Not super useful for general distribution IMO.

[0] https://pex.readthedocs.io/en/v2.1.140/

totetsu• 3 years ago

I ran into ssh cypher issues when I tried to share a packed python project. Worked on my machine but not my colleagues

hackan• 3 years ago

It's a cool trick, I used it to pack a tool long time ago: https://github.com/HacKanCuBa/passphrase-py/blob/219d6374338...

Basically, u zip the package, then add the shebang on the zip binary. Quite cool and simple! No need for external tools :p

yellowapple2• 3 years ago

That's a bummer. An app being self-contained is kinda the point of an AppImage IMO, and depending on an external Python (if one even exists) throws a wrench in that.

synergy20• 3 years ago

is this mainly for packaging python GUI programs cross-platform? how is it different from pyinstaller with tkinter?

40four• 3 years ago

I’ve had a lot of experience with another similar library, Kivy (kivy.org). These types of libraries, can be quite effective for the right use case. I shipped a couple mobile apps on both Google and Apple, so there’s that. I would check this out, but I do remember having to work through a lot of issues to get the mobile build just right. I wonder if a less mature project like this would have similar struggles?

gcanyon3• 3 years ago

This isn't intended as a dig at you, but in what world should an "Install Kivy" page take ~10 screens of text to display? If my goal is to build a simple Python app I can give to my friends and say, "here, I made this, give it a try," then that install page is screaming at me not to use Kivy to do that.

https://kivy.org/doc/stable/gettingstarted/installation.html

gcanyon3• 3 years ago

I'll add: the "getting started" collection of pages where I found that "install kivy" monstrosity then goes on to an "A first app" page, which...doesn't walk you through building your first app. Instead, it links to a page in a different collection of pages. And that page describes how to build a Pong app (too complex to be a "build my first app" -- there's a reason to start with "Hello World")...except it doesn't, it says that it assumes you already know how to create a Kivy application, and refers you to another page if you don't. And that page does "Hello World, but poorly.

Tommstein• 3 years ago

I too have used Kivy, using Buildozer for Android builds. One day I did something trivial like change the color of a button. The otherwise untouched project suddenly refused to build for random obscure Buildozer-related reasons I quickly lost the patience to figure out. Imagine if it was a security vulnerability I was trying to quickly patch. What's more, I think even rebuilding with the untouched source also failed now in the same way. And that's how I lost interest in pursuing Kivy further.

eternityforest2• 3 years ago

Yep. Kivy was a big part of my journey to usually wanting nothing to do with any niche or obscure tech.

It took me a long time to learn that no matter how awesome the concept seems, if it's not extremely popular (aside from very simple things) there's probably an issue somewhere.

At this point I would rather just make web apps for most everything, and I wish they'd just merge android and ChromeOS so we had an easy way to make simple cross platform stuff, while keeping the Android APIs for more powerful stuff.

doctorpangloss• 3 years ago

How do I turn a setuptools setup.py entrypoint into a standalone .exe for Windows users?

janalsncm• 3 years ago

This seems perfect for one of my projects.

Briefcase: Convert a Python project into a standalone native application

Related Stories