Major milestone achieved: JyNI builds successfully on Windows, DemoExtension is workable

After fixing the remaining symbol errors MSVC successfully created JyNI.dll for the very first time – what a historical moment!
Unfortunately the euphoria didn't last long, because when loading DemoExtension, it directly crashed.
After some investigation I found out that JyNI.dll was successfully loaded by the JVM and I also verified that code in the dll was actually accessed and properly run. Even the initial call into DemoExtension.pyd worked. However, the call to Py_InitModule4 crashed:
It seemed like symbols from JyNI.dll were not exported for access by DemoExtension.pyd. Strangely, the error message wasn't like one would expect for this. It rather looked like DemoExtension.pyd was calling into CPython's python27.dll, i.e. the one installed on the system.

The definite proof of concept

After some reading in MSDN docs I learned that the names of dll dependencies are hard coded at compile time, which means that DemoExtension.pyd will insist on loading python27.dll, no matter what.
So I tried to rename JyNI.dll to python27.dll but that only crashed the JVM.
After a sleepless night I reviewed everything and found that in makefile.win I was linking python27.lib into JyNI.dll. I had created the makefile based on DemoExtension's makefile, where linking to python27.lib had been perfectly fine.
For obvious (?) reasons, JyNI.dll should of course not link python27.
With that fixed I directly was rewarded with 30 fresh unresolved external symbol errors, which took some more hours to fix. Finally JyNI.dll built fine without linking python27. I rechecked behavior when using it with its original name JyNI.dll, but got the same results as before – this was expected.
Then, – finally – after renaming it to python27.dll I saw for the very first time the successful output of JyNIDemo.py using DemoExtension.pyd – in a Windows cmd prompt. This was the definite proof of concept that JyNI could be ported to Windows.
I didn't yet test ctypes, NumPy, etc., but in theory, everything might be working already – keep fingers crossed!
Will I eventually find time to do already a bit of buffer protocol support during this GSoC...?

It must be named python27.dll, no way around (?)

With that proof of concept in line I tried to figure out if it was possible to stick to the name JyNI.dll.
I read MSDN docs and stack overflow for several hours. Dll redirection, side by side assemblies, private assemblies, manifests. Some of it sounded promising, but none really did the trick (I tried a lot, maybe I always got it wrong somehow...).
So I finally came to the conclusion that just naming it python27.dll was the best way. Why do I actually bother?
Well, because having JyNI compile to python27.dll might yield confusion. How to ensure that the JVM will load the right python27.dll and not e.g. the system one? If people look into files installed by JyNI it might look like it would bundle CPython core. It is just best practice that dlls have correct and non-misleading names.
I considered that JyNI could rename JyNI.dll temporarily to python27.dll during loading, but that would cause trouble after crashes or with multiple JVMs running. So I finally decided to have JyNI.dll as a folder that must contain the actual JyNI.dll under the name python27.dll:
JyNI.dll\python27.dll.
I hope this makes sufficiently clear that the contained python27.dll is not actually python27.dll. Also, the JyNI.dll folder serves as a definite marker for JyNI's Java code to identify the right dll file. The folder JyNI.dll can be placed anywhere according to JVM and JyNI library search rules. Just that once JyNI.dll is found, it is required to be a folder containing python27.dll, which will then be loaded. This behavior is now implemented in the initializer of JyNI.java.

Makefile adventures

Finally I spent some time to improve the Windows makefile “makefile.win”. Assuming java and javac are properly configured on $path, it now finds JAVA_HOME automatically based on “where javac”. Note that this actually refers to JDK-home, which is required for JNI include folders.
While this sounds trivial, it took me a while to get gnumake to issue the command in the right way – naively using $(shell where javac) just didn't work. Neither did $(shell cmd where javac), $(shell cmd /C where javac), $(shell cmd //C where javac) or $(cmd where javac). Finally, a preceding “SHELL = cmd” did the trick. With that it can properly guess JAVA_HOME (in sense of JDK home) via $(subst \bin\javac.exe,, "$(shell where javac)").
In the same manner it can generate the user specific path to the default installation location of Visual C++ for Python:
USER = $(shell echo %username%)
VC_Python = "C:\Users\$(USER)\AppData\Local\Programs\Common\Microsoft\Visual C++ for Python\9.0"

What's next?

The next steps will be to check ctypes and NumPy for workability and eventually repair the necessary bits in JyNI.
One specific thing on this front that keeps me thinking is how to resolve the location of e.g. _ctypes.pyd. On Linux and OSX this was easy, because the system-installed python resides in a well defined location. On Windows it could be anywhere and not even tricks like python -c "import os, sys; print os.path.dirname(sys.executable)" are of much help for JyNI 2 as nowadays the plain python command usually links to Python 3.

So far I see these options:

1) Simply bundle the most important pyd files with JyNI, e.g. if they are part of std lib.
2) Bother the user to provide a proper pythonpath or some config file.
3) Query the Windows registry

Comments

Popular posts from this blog

Final report: ctypes and NumPy work with JyNI on Windows

The motto of my last few days: "error LNK2019: unresolved external symbol"

Lessons learned with Visual Studio, C89 and the preprocessor