[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[jira] [Created] (ARROW-3175) Arrow Java: Upgrade to official FlatBuffers release (Flatbuffers incompatibility)

Alex Black created ARROW-3175:

             Summary: Arrow Java: Upgrade to official FlatBuffers release (Flatbuffers incompatibility)
                 Key: ARROW-3175
             Project: Apache Arrow
          Issue Type: Bug
          Components: Java
    Affects Versions: 0.10.0
            Reporter: Alex Black

Arrow Java currently uses an unofficial flatbuffers dependency - com.vlkan:flatbuffers:

The likely motivation here is that previously, no Java flatbuffers implementation was available on maven central.
 > Unfortunately, FlatBuffers project does not publish any artifacts to the Maven Central Repository

However, this is no longer the case:

The flatbuffers version used in Arrow java is a nearly 3-year-old snapshot, not even a version of an official release: []

The main problem is that this version of flatbuffers is not compatible with the official releases of flatbuffers.
 For example, we use the official flatbuffers releases in ND4J and Deeplearning4j: []

Running Arrow with an official flatbuffers library on the classpath results in issues such as:
 at org.apache.arrow.vector.types.pojo.Field.getField(
 at org.apache.arrow.vector.types.pojo.Schema.getSchema(
 at org.apache.arrow.vector.ipc.message.MessageSerializer.serialize(
 at org.apache.arrow.vector.ipc.ArrowWriter.ensureStarted(
 at org.apache.arrow.vector.ipc.ArrowWriter.start(
 at FlatBuffersDependencyIssue.test(

Simply excluding the com.vlkan:flatbuffers dependency in lieu of an official flatbuffers release is not a solution (same exception as above) and we aren't prepared to downgrade all of our projects to use the flatbuffers version that Arrow currently requires.
 Consequently, this is a major issue that prevents us using Arrow in our libraries.

I have prepared a simple repository to reproduce this issue, if required: []

Is there a reason for using this particular version of flatbuffers, and if not, can Arrow java use an official release of flatbuffers instead?

This message was sent by Atlassian JIRA