support for the BufferProtocol

[ghost]

PyByteArray would be the next step towards PyByteArray, BufferProtocol, SciPy and Pandas support. It should be implemented in context of overall support for the BufferProtocol

[ghost]

@Stewori I want to work on that, please give hints and guide to get started.

[Stewori]

Are you familiar with the buffer protocol in CPython and how Jython implements it?
You should study the classes in org.python.core.buffer and classes in org.python.core that implement PyBuffer. Most important are IIRC BaseBytes and PyByteArray, but you should at least skim through other extenders (there are plenty, 12 or so) to see how they use the buffer protocol.
Then we'll have to make a plan how to bridge these notions. I already put a lot of thought into this, but that has been a while. Take a look at https://github.com/Stewori/JyNI/blob/master/JyNI-C/src/JySync.c to get a feeling how JyNI converts the various builtin types between Jython and CPython. Buffer protocol implementers can maybe handled in common by setting up the bridge for buffer API. Are you familiar how to expose data from Java via JNI to C? Make yourself familiar with exposing primitive arrays and direct nio.ByteBuffer. Let me conclude for now with an email where I once outlined to @jeff5 the goal of JyNI buffer protocol support:

Dear Jeff,

the true goal I want to achieve is a clean BufferProtocol support in JyNI. That means two things in my opinion.

1. If a CPython extension features types that support BufferProtocol and these are passed to Jython via JyNI, they shall appear as Jython PyObjects that support Jython's BufferProtocol. The exact memory-data that the CPython object exposes shall be exposed in Java by the Jython object. Optimally this should work for reading and writing and as direct as possible (for efficiency, minimizing memory requirements and -most important- to guarantee sync between Java-view and native view on the data). Using direct ByteBuffers this could be achieved, at least for JVMs that support these. On other JVMs, I would use byte[] as a fallback and try to obtain native access on the memory. However in this case, JNI does not guarantee to provide access to the actual memory of the array. It might only offer a copy, which would hold the risk of loosing sync. I see some techniques how to avoid that somehow, but it won't be very efficient nor elegant, nor absolutely save.

2. If a Jython object implements Jython's BufferProtocol in Java and JyNI is used to pass a PyObject from Jython down to a CPython extension, the native variant of the object shall support CPython's BufferProtocol. The object shall expose the corresponding memory from the JVM to the extension via this protocol. This shall work for reading and writing and as direct as possible for same reasons as above. I admit, this will be hard for Jython objects where a user implements PyBuffer in his own fashion. However, I think at least Jython's built-in types could and should support this. It can also be done with direct ByteBuffers.
   I know that direct ByteBuffers also have their disadvantages. They prohibit the JVM from doing memory optimization. Additionally the GC does not consider their memory when it determines what to delete or when to run (it would still clear the memory, if it collects a buffer). So direct ByteBuffers should only be used when really needed. I would propose a configuration-parameter for Jython that tells BaseBytes and BaseBuffer to use direct ByteBuffers. If this parameter is turned off, they could use ByteBuffer.wrap to fall back to the current implementation.

The doc of Jython's BufferProtocol could tell potential other implementers also to look at this parameter and store their data as direct or ordinary ByteBuffer accordingly. However, I would build fallbacks into JyNI to deal as good as possible with situations where implementers don't stick to this.

Both scenarios would mean that the backend in BaseBytes and BaseBuffer had to be of a type that unifies byte[] and ByteBuffer. One variant would be to use ByteBuffer as type and ByteBuffer.wrap for byte[] case. Another variant would be to have the storage of type Object. The PyBuffer implementation would know whether it is byte[] or ByteBuffer, or maybe even String and work accordingly (this would involve lots of explicit type casts though). However PyBuffer.pointer should not provide a byte[] and pretend it to be the actual backend in either variant.

I think the solution from PyString, i.e. SimpleStringBuffer only works well for read-only scenarios, since it would lead to asynchronity between byte[] and String backends if the user uses the obtained byte[] for writing (assuming a mutable String variant like StringBuffer). Additionally, such an approach potentially doubles the memory requirements, which might be significant in some situations.

I mentioned my thoughts about support for (>2^31 bytes)-arrays, because -afaIk- a CPython extension might expose such long data via BufferProtocol and I was wondering how to deal with this. Then I mentioned it as a maybe misleading example. I could imagine to provide kind of LongPyBuffer that enhanches PyBuffer by long-index methods and allows to obtain ordinary int-sized slices to portions of the data. But these are future thoughts - for a first step I would just document it as a JyNI-limitation to support BufferProtocol only for (>2^31 bytes)-buffers.

Again - I would be fine with postponing this decision if the public API was adjusted to be open for more variants than byte[]. I would also offer to help with implementing the adjustments and -if accepted- the refining of Jython's BufferProtocol.
I hope this explains my intentions a bit better.

Cheers

Stefan

[Stewori]

This thread from Jython mailing list is also an important read on this topic:
http://python.6.x6.nabble.com/Buffer-protocol-direct-vs-JVM-ByteBuffers-td5190804.html

[jeff5]

The most recent changes to the buffer interface improved support for direct buffers and deprecated the type that attempts to simulate a C pointer for this specific use. The aim was to make at least the first objective possible.

When passing a Jython object with the buffer protocol to a CPython extension, I would hope it were possible to create the right kind of façade to access the the byte array in store. Taking an interface freezes the object (in some sense), until a release, but that may not pin it in memory, which would have to be JyNI's responsibility for the filetime of the buffer view.

This is an extension of the lock JyNI must hold on the objects it handles for the lifetime of a native call. It is surely also necessary in order to simulate the GIL on which C extensions are will rely. It seems like this locking would work unless the CPython extension is badly-behaved enough to hold C pointers to implementation, which would be an error in CPython too. Oh, and call-backs are an interesting problem.

I'm missing what it is you can't do when Jython is implemented the most natural way.

[Stewori]

Taking an interface freezes the object (in some sense), until a release, but that may not pin it in memory, which would have to be JyNI's responsibility for the filetime of the buffer view.

Also CPython requires a release call to PyBuffer_Release, which we would use to trigger e.g. JNI's Release[PrimitiveType]ArrayElements() (see https://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/functions.html#wp17314) However, I suspect in CPython it is less critical than in the JVM and extensions are maybe sloppier or tend to allocate buffer access for longer periods, maybe the whole runtime, but that's just a believe. So, a direct nio.ByteBuffer might be the healthier solution.

I'm missing what it is you can't do when Jython is implemented the most natural way.

I suppose you refer to arrays. If a C-extension defines its own types etc it might expose internal, self-managed memory via buffer protocol. Since this memory would not be JVM-allocated I see no way how to make it accessible for Jython via arrays (i.e. without copying). Best solution in this case would be a direct nio.ByteBuffer.

You took care to add nio.ByteBuffer support to Jython, so that should be the way to go.

It is surely also necessary in order to simulate the GIL on which C extensions are will rely.

I hope that JyNI's curent GIL simulation is sufficient. Anyway, a C-extension could release the GIL and still keep buffer access open (or is that invalid behavior?). So buffer allocators need to be counted. Multithreading must be considered.

Oh, and call-backs are an interesting problem.

What callbacks do you mean?

[jeff5]

Ok, we're aligned I think on getting and releasing a PyBuffer, and JyNI's responsibility to pin and un-pin the JVM array.

If a C extension returns its own type that bears the (CPython) buffer interface then JyNI has to provide a Jython object (that is a PyObject) with the same API including Jython buffer API. I'm not making a rule here, just observing that it has to be so for a complete implementation. I believe that in deprecating Pointer I made it mandatory to be able to get a ByteBuffer, and optional to get a byte[], and that uses of the buffer interface in the code base always get that, never byte[].

You keep writing direct ByteBuffer, but that has a special meaning (see Javadoc for ByteBuffer) where it says "the contents of direct buffers may reside outside of the normal garbage-collected heap". It won't always be a direct buffer.

I'm not sure about GIL simulation, but you've thought about it at length, I guess. I think you shouldn't, perhaps can't, take a global interpreter lock on Jython. But in the presence of real concurrency, are not assumptions made by CPython extensions likely to fail? I think it would be ok as long as you pin all objects you touch, and un-pin on return. In contrast, it is fine to take a buffer and return still holding it (like the constructor of a memoryview).

The case is hard enough where the thread that entered calls back into a Python (compiled to Java), e.g. if you sub-class ndarray. Un-pin or not at that point? The case where you register a function with an extension, then call it sometime later from a C runtime thread makes my head hurt. We've had repeated trouble with this in Jython and I suspect it is because we are trying to follow CPython, which is itself victim of a thinking error. Not sure, though. And it may be academic for now.

[Stewori]

Yes, direct is used intentionally. It refers to memory hosted outside the JVM. I suspect JVM might not allow to obtain an array view on direct ByteBuffers. However, direct ByteBuffers are IMO the only way to expose external, possibly C-extension-managed memory on Java side. In the other direction, JyNI would request a direct ByteBuffr view on a PyBuffer. If that's not possible, it would use byte array as a fallback.

JyNI does not take a global interpreter lock. Its GIL only affects native code. It allows multiple threads on Java side but only one thread on native side. The native thread keeps the GIL when it calls back into JVM, but the GIL is a reentrant lock. This usually works fine because usually only a single thread is concerned in native work as extensions are designed for single threaded CPython. Or they release the GIL; JyNI supports the corresponding macros. See details in https://arxiv.org/abs/1607.00825.

[Stewori]

@ysz I think a good initial step would be to implement some test cases. Add some functions concerning buffer protocol operations to the DemoExtension. Add a test_JyNI_buffer_protocol.py to https://github.com/Stewori/JyNI/tree/master/JyNI-Demo/src, exercising some toy examples.