Final report: ctypes and NumPy work with JyNI on Windows

After the last report, where I managed to run DemoExtension with JyNI on Windows, I investigated workability of ctypes.

Linking msvcrt

The first thing I discovered was that it didn't work:

Exception in thread "main" java.lang.UnsatisfiedLinkError: D:\workspace\linux\JyNI\build\JyNI.dll\python27.dll: Can't find dependent libraries

And further – that DemoExtension also was not working when built using setup.py rather than by my makefile. That was a drawback, but remember that DemoExtension was working fine with CPython, even if built with the makefile.
After carefully studying the difference between the build commands I found out that setup.py would add the /MD flag. Beneath several other options, e.g. /nologo /Ox /W3 /GS- /DNDEBUG, /MD was the one that made the difference. It tells MSVC to link msvcrt.dll, precisely speaking msvcr90.dll for Python 2.7.
I guess the DemoExtension build without explicit /MD worked, because it still linked libcmt.lib. However, after adding /MD to the makefile, I had to assure that the Python 2.7 compliant msvcrt90.dll would be loaded at runtime. Nowadays, the only viable way to force a specific version of a dll is to register a so called side by side assembly (SxS). This had to be done via a manifest file and I was able to extract the CPython 2.7 manifest file from a Visual Studio build: By setting some option in MS Visual Studio it can be forced to save the manifest file rather than merging it as a resource into python.exe. Deriving a proper manifest from CPython's original felt safer than writing my own based on MSDN docs. Using the manifest tool I now merge the manifest into JyNI.dll/python27.dll.
And – it worked! ctypes now presented a completely fresh, new, different error message:

ImportError: DLL load failed: The specified procedure could not be found.

Missing symbols and dumpbin

Here I was bitten by some differences between POSIX' and Windows' dynamic loading mechanisms.
From Linux I was used to be told explicitly about undefined symbols at runtime. At load time Linux' ld wouldn't check existence of symbols at all. In contrast to that, the Windows dll loader requires all symbols to be defined at load time and otherwise fails with a generic error message, not naming the missing symbol.

So I learned about the dumpbin tool that can list all sort of information of a given .exe or .dll file. Especially it can list its dependencies, imported and exported symbols. So I called

dumpbin /EXPORTS build\JyNI.dll\python27.dll

to see what symbols were currently provided by JyNI.dll and

dumpbin /IMPORTS "C:\Program Files\Python\Python2.7.13\DLLs\_ctypes.pyd"

to get a list of required symbols (full paths were simplified in this example).
I quickly hacked a Python script to perform a set exclusion of these lists and found the missing symbol for _ctypes.pyd:
After adding PyObject_GetBuffer, _ctypes.pyd finally was properly loaded and ctypes confronted me with yet another error.

PyWin_DLLhModule in sys.dllhandle

I quickly figured out that ctypes looks for a handle to python27.dll (PyWin_DLLhModule) at sys.dllhandle. So I added code to JyNI initialization that would also initialize this field of the sys module, along with sys.winver (PyWin_DLLVersionString).

However, so far I always got the empty string for sys.winver. I looked at the VERSION statement, the VERSIONINFO resource and tried to use the /V option on MSVC, but none of these did the trick.
It is loaded via
LoadString(hInst, 1000, dllVersionBuffer, sizeof(dllVersionBuffer));
where 1000 is described as a magic number ("1000 is a magic number I picked out of the air.  Could do with a #define, I spose...")
So I guess there must be some command or property that also uses this magic number to register the version string. So far I couldn't properly spot this in CPython's build process. However, things work fine without a proper version string, so this really does not have high priority. Maybe the easiest solution would be just to set it to "2.7" in code, which is supposed value for Python 2.7.

ctypes is workable

With all these adjustments, ctypes is finally workable with JyNI on Windows. Based on msvcrt rather than libc6, the whole ctypes JyNI demo is working smoothly now.

NumPy is workable

For NumPy I had to add some missing symbols again. Via the dumpbin based procedure described above it was straight forward to identify the right symbols.

For multiarray.pyd:
PyMemoryView_FromObject, PyFile_AsFile, PyObject_AsFileDescriptor and PyOS_InterruptOccurred.

For mtrand.pyd:
PyEval_EvalCodeEx and PyEval_EvalFrameEx.

With all of them fixed, NumPy loads fine and all stuff I tested on Linux and OSX so far works equally well on Windows.

Getting CPython location from the registry

Via the _winreg module I am now able to get the installed CPython 2.7 folder from the Windows registry. I wrote the utility python_home_winreg.py to automatically set PYTHON_HOME in makefile.win.
The next step is to exploit this info at runtime too. This would allow to find the Python\DLLs folder automatically. The problem here is that Jython does not support _winreg. I could bundle it with JyNI, but _winreg is not compiled to a .pyd file. I suppose it is included in python27.dll, so I guess I would have to do it the same way for JyNI. This will probably work somehow, but requires some more bits to figure out. Maybe the easier way would be to generate a config file at compile time. That would not be as portable between different installations and setups though.

What's next?

GSoC is mostly over now and all proposed goals were achieved. The only things left to do is some cleanup, improvement of setup (e.g. getting CPython location from Windows registry also at runtime), completing the WINE based build procedure makefile.wine and adding a Windows HowTo to the readme.
Finally it will be time for a JyNI alpha 5 release – featuring Windows support.

I will try to get as much as possible of this finalization work done within the – lets say – next three days. After that I will file my final evaluation and submit the work. Whatever was left out until then will be done directly after GSoC finalization. Stay tuned for JyNI's first Windows release!

Comments

Popular posts from this blog

The first two weeks of GSoC, or: Proud to release JyNI 2.7-alpha.4, featuring NumPy support.