I'm going to ask a 'self-taught' question, so my terms may be incorrect.
Has anyone solved the compiled libraries / dynamic libraries / linking problems for multiplatform apps? Everything works just fine with these python>native app tools until you hit some esoteric library that's not pure python.
Python remains a mess for these use cases because it does not have packages.
It has a very specific filesystem layout that it calls a package, but this is not a first class citizen; it has a packaging format called "wheel" that is now used for distribution, and that can contain native libraries. That currently mostly handles the distribution of compiled libraries for multiple platforms, but only within the traditional venv-style setup. Being able to ship interpreter+venv+launcher would go a long way towards providing a native-ish experience.
A step beyond that would be being able to distribute a "virtual filesystem" containing a whole tree of python compiled as .pyc, appended to a native executable containing the interpreter and linked against the required native libraries already.
They are though, at least on Windows, I’ve moved them before. The whole Lib/ tree was independent of the venv’s path, it’s only the “activate” script that hard-codes a full path.
Since venvs on Linux symlink to the system’s Python executable, they should also work after moving.
> They are though, at least on Windows, I’ve moved them before. The whole Lib/ tree was independent of the venv’s path, it’s only the “activate” script that hard-codes a full path.
So, if activate.ps1 (and the other scripts for other shells) were updated to dynamically identify their own location rather than hardcoding the path, they’d be fully portable? Seems to be a fairly simple update.
> Since venvs on Linux symlink to the system’s Python executable
Hopefully, the executable with which they were created, not the system executable except where they happen to be the same.
> So, if activate.ps1 (and the other scripts for other shells) were updated to dynamically identify their own location rather than hardcoding the path, they’d be fully portable? Seems to be a fairly simple update.
I don't think the maintainers have gotten a request to do that. Personally I use a more complicated launcher because I need to set things like environment variables in my deployments, but a venv is almost an out-of-the-box relocatable Python environment. It's when you need to protect/compile your code that things get messy.
>> Since venvs on Linux symlink to the system’s Python executable
>Hopefully, the executable with which they were created, not the system executable except where they happen to be the same.
When you call /usr/bin/pythonx.y -m venv ./myenv, ./myenv/bin/python will link to /usr/bin/pythonx.y. Since there aren't many reasons to use a Python version not installed by the package manager, I call these the "system's Python executables", but sure, it can also point to /opt if you installed Python there.
They are in the sense that anything one would want to do with a venv, they can do with trivial effort. You don't need the activate script, you can just set PATH, PYTHONHOME and/or PYTHONPATH and be done with it.
Python wheels really can help here, but only if the downstream packages you depend on have published wheels for the various different platforms you need to support.
https://cibuildwheel.readthedocs.io/ is a great tool for assisting with that - many projects run that in GitHub Actions as part of their release process.
There's just one catch, in my experience: since GitHub don't offer Apple Silicon machines as part of GitHub Actions, it can be hard to build wheels there that work on those platforms. Maybe you can cross-compile them? I've not figured that out yet.
I have deep wounds from pre-chatGPT times of trying to manually figure out libraries. As in otooling each library, error, etc then recompiling into various places until things 'worked.' I'm appreciative of the experience, though now I think with LLMs the problem could be solved with some simple queries.
So I guess you're right, it requires effort, but I was a 10x engineer in the wrong direction and never want to experience that again ;)
What works for me is figuring out the platforms I need to support and creating a build pipeline that can handle those requirements. If you join onto a project I don't want you playing around with libraries on your local machine. If you are I consider that a failure of the build script/makefile/docker compose file/etc. First it's lost productivity, second you might get it wrong and have misleading results, causing more lost productivity.
A lot of places do this nonsense "spend hours configuring your machine" method for projects. The only reason I can think of doing that is "job security" and who wants to keep a job with horrible tooling?
That's definitely working smarter. I'm coming around to understanding how all the tooling works - the lost weeks learning the basics happen less often now.
In a way, that time was a chapter in 'job security.' Ran out of money, had to let everyone go, and picked up the pieces to fight another day (self-taught under threat of survival). By the grace of God I'm a reformed 'idea-guy' and am now a builder.
What kind of madman tries to deploy pytorch? If it's a product, convert the finished model to onnx. Otherwise it's probably for a dev and you can tell them to have fun because installing all the gpu libs sucks no matter what. In theory mambaforge should make most of these scenarios less painful (mostly portable faster version of anaconda), but you'll still need the right cuda runtime, rocm, etc.
What makes this stuff so hard? I could probably find a Python tutorial to render a spinning cube in like 5 minutes, but as soon as you want to do ML you spend an hour just getting libraries to work?
Is there ever going to be standardization, the way a WebGL shader just works everywhere?
Eh it depends, some python packages do things like pull data samples from my experience and it can be quite a mess detangling the package to give PyInstaller everything it needs to bundle stuff up. It also takes a lot of additional configuration for a lot of packages.
As much as I hate how containers are often misused, this is one of the reasons people obsess over them from my experience. The ability to bundle all your crap regardless of how many layers of package management idiosyncrasies your application ultimately woul have to deal with pretty much transparently.
I vaguely remember nuitka or pyinstaller being able to handle this for me, but I used vms or qemu to build them "natively" for each target platform. Never found a way to solve cross compiling.
which uses native widgets and manages basic functionality such as where the "Quit" menu entry is automatically.
Looking into this now --- it looks perfect for my needs, just hoping that there's a nice graphical tool for laying out a GUI (if it has a graphical programming mode like to Blockly or Ryven, then it is perfect for me).
I haven't used Kivy in quite a while, so maybe this has changed, but as far as I know -- no, Kivy hasn't done something like this. In Kivy, everything looks like Kivy, regardless of what platform you're on, and you need to explicitly lay everything out, including whichever changes you want for a particular platform. Whereas with Toga, the UI is changed to match the user's expectations for that platform, automatically.
One example of this, as parent mentioned, is where the "quit" button is located. Different platforms put this in different places. Toga will automatically put it in the right one, depending on which platform you're on. Buttons will look like normal buttons on a Mac, but also like normal buttons on Windows -- depending on which platform you're on.
Hi! Do you know whether Kivy or Toga/BeeWare lets you access Android API? For example, I want my app to listen to BatteryManager's broadcasts [0]. Is it possible using any of these frameworks? (Or any other Python-centric framework?) I read that Toga supports background tasks, but it is not clear to me if I could use them to listen to Android broadcasts.
It appears PyInstaller is much more portable. Briefcase seems to make an .msi installer on Windows and an AppImage for Linux. Both of those require installation to use. PyInstaller can create a single executable that you can just run (yea sure it extracts itself to a temp dir, but the optics are very different).
> If you’re on Linux, you should use the system Python provided by your operating system. You will be able to complete most of this tutorial using a non-system Python, but you won’t be able to package your application for distribution to others.
Probably because it doesn't actually package the Python interpreter into the executable and relies on a hashbang (ie. `#!/bin/python`) to actually run the bundled source files. Pex [0] does the same thing.
It's a half solution at best, but if you control the infrastructure it does dramatically simplify things like Dockerfiles and other deployment stuff. Not super useful for general distribution IMO.
That's a bummer. An app being self-contained is kinda the point of an AppImage IMO, and depending on an external Python (if one even exists) throws a wrench in that.
I’ve had a lot of experience with another similar library, Kivy (kivy.org). These types of libraries, can be quite effective for the right use case. I shipped a couple mobile apps on both Google and Apple, so there’s that. I would check this out, but I do remember having to work through a lot of issues to get the mobile build just right. I wonder if a less mature project like this would have similar struggles?
I too have used Kivy, using Buildozer for Android builds. One day I did something trivial like change the color of a button. The otherwise untouched project suddenly refused to build for random obscure Buildozer-related reasons I quickly lost the patience to figure out. Imagine if it was a security vulnerability I was trying to quickly patch. What's more, I think even rebuilding with the untouched source also failed now in the same way. And that's how I lost interest in pursuing Kivy further.
Yep. Kivy was a big part of my journey to usually wanting nothing to do with any niche or obscure tech.
It took me a long time to learn that no matter how awesome the concept seems, if it's not extremely popular (aside from very simple things) there's probably an issue somewhere.
At this point I would rather just make web apps for most everything, and I wish they'd just merge android and ChromeOS so we had an easy way to make simple cross platform stuff, while keeping the Android APIs for more powerful stuff.
This isn't intended as a dig at you, but in what world should an "Install Kivy" page take ~10 screens of text to display? If my goal is to build a simple Python app I can give to my friends and say, "here, I made this, give it a try," then that install page is screaming at me not to use Kivy to do that.
I'll add: the "getting started" collection of pages where I found that "install kivy" monstrosity then goes on to an "A first app" page, which...doesn't walk you through building your first app. Instead, it links to a page in a different collection of pages. And that page describes how to build a Pong app (too complex to be a "build my first app" -- there's a reason to start with "Hello World")...except it doesn't, it says that it assumes you already know how to create a Kivy application, and refers you to another page if you don't. And that page does "Hello World, but poorly.
Okay, one more and I'll shut up. The app is unsigned (on MacOS) so you can't double-click or command-O it -- you therefore have to right-click and select Open, and then acknowledge the dire warning about running unsigned code. I don't see any mention of this on the over-long install page, so anyone who doesn't know how to run unsigned code on a Mac is out of the game immediately.
And after all that, the app doesn't run. So yeah.
I put off learning python for years because of this ridiculousness. Finally I decided to just bite the bullet, because it just isn't getting any better.
Although I have never used any of their stuff, I had the pleasure of watching Russell from BeeWare at Pycon Colombia some years ago. It consistently presents itsself as an ecosystem I would like to work with, and if anything the naming of their projects is very nice.
Anyone happened to use Toga with video streams? I've got some gstreamer stuff with Qt and it doesn't need a rewrite yet but it's a glimmer on the horizon.
Interesting timing! We're just in the middle of adding Python support to Hydraulic Conveyor, which is a similar tool [1]. There's a github issue [2] and mailing list that'll get notified when it's done. Disclosure: it's commercial but free for open source projects.
There are many of these open source packagers and they all share very similar problems:
1. They don't let you do software updates, even though software updates are practically mandatory for any real project. Electron is a stand-out here because it does address this, but their update engines are unmaintained for years and have some major unfixed problems (causes a lot of issues with Windows networks, for example).
2. Even in the very rare cases that they do, they don't let you force updates on launch even though many apps need something like this to keep up with protocol changes. It's one of the reasons people like web apps.
3. They don't help you with signing. For example they don't simplify key management, they don't support cloud signing (essential since May because Microsoft now insist on HSMs for all keys, not just EV keys), or they don't do notarization, or they don't generate CSRs for you.
4. They require the use of CI to cross-build even when apps are written in portable frameworks that don't require compilation. This is because they are just thin wrappers around the native tooling.
5. They're invariably language specific even though there's no good reason to be because 80% of the work is the same regardless of what language or framework you use.
6. They make MSI files for Windows even though MSI is deprecated.
It's possible to bite the bullet, chew glass for a while and solve all these problems, which is what we did for Electron/JVM/Flutter/native apps. You can reimplement all the native tooling so users can cross-build (i.e. make Mac packages from Linux/Windows, Windows packages from Mac/Linux etc), which enables releasing from developer laptops or cheap Linux CI workers. You can support software update by integrating Sparkle on macOS, apt on Debian/Ubuntu and by using MSIX on Windows (and by then working around all the bugs in Windows to make it work well). You can generate download pages that work out the user's OS and gives them the right download, and instructions for how to install self-signed apps if the developer isn't code signing with a recognized certificate. You can abstract platform neutral things and expose platform specific things. Then you can write a parallel incremental build system so doing all the work is as fast as possible, and write lots of code to detect all the myriad mistakes people make and give good error messages or auto-fix them. Then you can make it support GitHub Releases. Then you can document it all.
But that big pile of glass isn't particularly tasty, which is why open source projects don't do it and we ask commercial users to pay for it.
Briefcase looks nice but it also seems to have all the problems listed above. I think once we add Python support Conveyor will be quite useful for the Python community, especially if we can find a workaround for pip not support cross-building of venvs. It would be great if you could just whip up a quick Python script, run one command and your installed clients start automatically updating, your download page updates, and the whole thing is no harder than releasing a static markdown-rendered website.
Is it? The WiX Toolset still seems to be actively developed and MSIX doesn't seem to support all use cases. Has MSI been officially deprecated despite that?
Also, MSIX's mandatory signing makes it too expensive and cumbersome for a hobby project, so it would be unfortunate if that became the only option.
It was marked as such at one point on their website, but now I searched for that again I can't find it anymore. Maybe they undeprecated it, hard to keep track of what MS are doing. It's not been developed for many years though. All the new features seem to go into MSIX.
MSIX files can be self-signed if you install the cert first. Conveyor has an EXE wrapper that can do that for you. So you can distribute apps with it without buying signing certificates.
But by installing a self-signed certificate, that certificate is trusted for all apps, not only for mine – right? This seems like a broken permission model.
I only want permission to modify my own app, not every app of the system. If users have to trust me (and all other developers of self-signed applications) with permission to modify all apps, that makes the whole signing rigamarole kind of pointless.
But if I do want to try this approach, do I have to install the certificate in Trusted Root Certification Authorities, or should it be in Trusted Publishers, or Trusted People? The first one would grant me even more privileges that I don't need, but what about the other two?
If you publish through the MS Store, it only costs $19 to verify an individual (one off, not a subscription) and they sign it for you. So that's pretty cheap, well within the range of a hobby project.
If you don't have some kind of signed identity then the concept of "your own" app doesn't exist, so indeed in that model you get permission to do what you want more or less. That's why it requires administrator elevation, whereas real signed packages don't. Of course this model is identical to the normal pre-MSIX model where any app installer can fiddle with any other app's files.
You can put it in Trusted People, I think. I'd have to check the code to see where we put it.
BTW, dead thread, I should note that the certs Conveyor generates aren't CA certificates. They have your chosen identity name in them, so you can't use it to sign for other packages.
I'd be amazed if Microsoft held any important keys outside of HSMs. I think the incident you're referring to is where they appear to have been hacked by China (maybe the Chinese govt) and they blamed a "validation error in Microsoft code". HSMs help a bit but if the code that accesses them is insecure then you can still sign bad things. The HSM just means you don't have to do a key rotation to recover.
On that topic, one feature Conveyor has that open source packaging tools usually don't is that it can protect signing keys in your macOS keychain, in some cases. To utilize that the packaging tool itself must be signed, which of course it is here, and this is useful because the OS will protect both the private key and the process that's working with it from malware even if it's not explicitly sandboxed. The keys themselves are protected using the security chip. So it's a very high level of security that's sort of equivalent to having a dedicated user account for signing on Linux, but without needing to set anything up.
Unfortunately not every credential can be stored there yet, only the "root key" from which other keys are derived. Login credentials for Apple's notarization servers and cloud signing services like eSigner/KeyLocker don't get protected in the same way yet. But that'll come.
> I'd be amazed if Microsoft held any important keys outside of HSMs. I think the incident you're referring to is where they appear to have been hacked by China (maybe the Chinese govt) and they blamed a "validation error in Microsoft code". HSMs help a bit but if the code that accesses them is insecure then you can still sign bad things. The HSM just means you don't have to do a key rotation to recover.
One of the few things that seem unambigiously worded in Microsoft's disclosure on this topic is that the hackers did actually have a copy of the key itself:
> was forging Azure AD tokens using an acquired Microsoft account (MSA) consumer signing key
> Storm-0558 acquired an inactive MSA consumer signing key and used it to forge authentication tokens for Azure AD enterprise and MSA consumer to access OWA and Outlook.com. All MSA keys active prior to the incident – including the actor-acquired MSA signing key – have been invalidated.
> The method by which the actor acquired the key is a matter of ongoing investigation.
Which sure sounds like something that wouldn't just happen if these lived in a HSM. (Presumably Microsoft data centers have access controls...?)
My guess is that because these keys see a high volume of operations and rolling out new keys is probably annoying because tons of different pieces validate against them that Microsoft chose to not use HSMs for them for cost reasons, because you can just use all compute power in a server to perform operations with the key (much lower $/sign/s) and if you need to scale up you can simply copy the private key to more instances.
Has anyone solved the compiled libraries / dynamic libraries / linking problems for multiplatform apps? Everything works just fine with these python>native app tools until you hit some esoteric library that's not pure python.