Python bindings

[DanielPharos]

QuArK 7 will use Python, and therefore needs a connection to the Python library. There's various ways of linking, and this is what I've come up with:

There will be no static linking (inclusion of the library file into the executable), because that would cause the executable to become larger, while Python is often already installed on many systems.

QuArK's executable will NOT have the OS load the Python lib when the executable is started; this would cause a pretty weird and non-friendly error message if the library mismatches. So the executable will load the needed function calls (after checking the lib's version) during start-up. (Technical note: This means all calls to Python in the executable will have to go through some extra hoops, but it should be pretty painless.)

However, all plugins will be loaded after all of that, so they will be able to use the not-at-runtime dynamic linking (what's the official term for this anyway?), since the Python lib is already loaded and its version verified. We need to be sure however no copy of Python gets loaded this way. Does anybody know a way to do that?

Any comments?

[xmm0]

However, all plugins will be loaded after all of that, so they will be able to use the not-at-runtime dynamic linking (what's the official term for this anyway?), since the Python lib is already loaded and its version verified. We need to be sure however no copy of Python gets loaded this way. Does anybody know a way to do that?

I'm not sure what you mean.
Are you saying that you want to implicitly link to the Python run-time and make sure that it doesn't link to the wrong version (to avoid "DLL Hell")? I know of several ways to do this on Windows but I'm no Unix guru  . It shouldn't be a problem on Windows since the python dynamic libs, in the System32 folder, are named python24.dll, python25.dll, python26.dll and so on.

Or are you saying that you want to link up the other way? With LoadLibrary() and GetProcAddress() and all that fun stuff
I apologize for my confusion

[DanielPharos]

On *nix, this is usually done by the "dlopen" call etc. (I had to google it too, but it works fine on even Mac OS X).

Nope, even though the filename looks nice on Windows, the different sub-versions of Python are also not compatible (at least not on the API-level), so you cannot rely on the filename.

Yep, the main executable will do the LoadLibrary and GetProcAddress stuff, verify the version, and then allow the plugins to load. The plugins will use the (what's the name) load-on-startup linking.

[xmm0]

Nope, even though the filename looks nice on Windows, the different sub-versions of Python are also not compatible (at least not on the API-level), so you cannot rely on the filename.

Really? This will be difficult then.
Since Python should already be installed on the user's system, should we include the version we will be using in the installer/package just in case?
What version were you planning on using anyway?

[DanielPharos]

QuArK 6 has a folder containing all needed Python files (the 'lib' folder) and has the Python dll in its 'dlls' folder. I think this is the approach that we should take on Windows systems to ensure we get the right Python version, with all the needed files. On *nix, the package manager should take care of getting the right version of Python, so we don't need to distribute any Python-stuff with Q7 there.

(Determining the version of the Python DLL is possible, though not nice: There's a function that returns the Python version number as part of a string, and with a little bit of parsing, one can get the version number from that. This is what I've done with QuArK 6, and it works fine (for at least Python 2.2-2.6).

Currently, I'm using Python 2.4.4 (the latest binary release of the 2.4 branch), as this is what QuArK 6 uses so this is the one I know. However, I plan to switch to Python 3 ASAP, since there's a few features in Python 3 that would really come in handy (plus, it's supposed to be 'much' faster). This WILL break Python 2- compatibility though...

[xmm0]

QuArK 6 has a folder containing all needed Python files (the 'lib' folder) and has the Python dll in its 'dlls' folder. I think this is the approach that we should take on Windows systems to ensure we get the right Python version, with all the needed files. On *nix, the package manager should take care of getting the right version of Python, so we don't need to distribute any Python-stuff with Q7 there.

Sounds good to me

Will all of the 2.x scripts have to be rewritten to be compatible with the new Python 3.x?

[DanielPharos]

(Sorry for the late reply; I lost track of this thread.)

Yeah, everything will have to be rewritten anyway, because even though the difference between Python 2.x and 3.x isn't THAT large, the entire QuArK-stuff will change, so I don't think it's going to be possible to fix most scripts with only minor adjustments. Of course, there are some exceptions, and a lot of code will be similar enough that a careful copy-paste-adjust strategy might work just fine.

[xmm0]

Awesome! I've been trying to bring my python 3 skills up to par for this project

[DanielPharos]

Some issues with Boost have gotten me thinking... This loading Python dynamically is just a pain in the you-know-what. I now think both Boost and Python should be *statically* linked at run-time. If either is corrupt, the OS will display a cryptic error message, but that's okay: If the install is corrupt, you're perfectly allowed to do that!

It'll vastly simplify the interface to Python (much less chance of bugs there), and makes all the Python tutorials etc. on the web work out-of-the-box. We can always switch back to dynamic loading (by just digging up the now-obsoleted code from the repository), but for now, I don't want to waste time doing this unless there's a good reason.

For Windows, we'll have to either include a Python into the main QuArK folder (recommended), or rely on a system-install (bad idea without a package manager). On *nix, we'll rely entirely on the package manager to deal with dependancies.

(Note: The main reason for this: without Boost, no filesystem interface, except for C++'s main one, which means certain parts of the code (logging) can NOT be unified into the QuArK filesystem. Boost cannot be statically linked (as it is now), because then plugins won't be able to access the Boost functions (BAD!).)

I'm not really happy with this, but sometimes choices need to be made...

[xmm0]

Sounds good to me Dan. Whatever needs to be done to make it work

[DanielPharos]

Just an update on this topic: Python 3 introduced Unicode support, but they borked up the implementation so bad (apparently), they're reworking it in Python 3.3. So we'll be doing 3.3+. Also, there's an (known) irritating bug in one of the main packages of Python, causing Python 3.2.x installs in custom locations to fail to start-up on *nix machines. I just hope they fix this bug before releasing 3.3...

[DanielPharos]

I haven't tried Python 3.3 yet (it's in Beta 1). I suppost I should get off my @$$ and check it out, just to see if the aforementioned problems are now fixed.