git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Weird symbol resolution error with llvm JIT in gandiva jni


Apparently you're using System.load() to load the compiled helpers
functions.  It seems it may load the library using RTLD_LOCAL, which
means it won't be seen from other dynamic libraries loaded in the same
process.

According to this SO entry, the solution would be to use
NativeLibrary.getInstance() instead:
https://stackoverflow.com/questions/5425034/java-load-shared-librariees-with-dependencies

Regards

Antoine.


Le 17/12/2018 à 16:48, Ravindra Pindikura a écrit :
> 
> 
>> On Dec 17, 2018, at 6:39 PM, Antoine Pitrou <antoine@xxxxxxxxxx> wrote:
>>
>>
>> Is LoadLibraryPermanently(nullptr) called after the libgandiva_jni.so is
>> loaded?  Perhaps you can try LoadLibraryPermanently("libgandiva_jni.so")
>> explicitly?
> 
> We are already doing this inside gandiva.
> 
> https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61 <https://github.com/apache/arrow/blob/master/cpp/src/gandiva/engine.cc#L61>
> 
>>
>>
>> Le 17/12/2018 à 13:50, Ravindra Pindikura a écrit :
>>> Antoine,
>>>
>>> I set ARROW_GANDIVA_STATIC_LIBSTDCPP and modified the jni/CMakeLists.txt to export all symbols (i.e removed the version-script parameter in LINK_FLAGS).
>>>
>>> $ nm ./debug/libgandiva_jni.so|grep  _ZNSt8ios_base4InitC1Ev
>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>
>>> $ nm --defined-only ./debug/libgandiva_jni.so|grep  _ZNSt8ios_base4InitC1Ev
>>> 0000000001df34f0 T _ZNSt8ios_base4InitC1Ev
>>>
>>> I still get the same “could not be resolved!” Error when I run the java test. It appears like when the library is loaded via jni, some symbols are getting hidden from the JIT.
>>>
>>> Of the four sub-cases of the test :
>>>
>>> 1. a cpp executable (shared/static), the decimal functions invoked from CPP fns
>>> 2. a cpp executable (shared/static), the decimal functions invoked from JIT code
>>> 3. a java executable, the decimal functions invoked from CPP Fns
>>> 4. a java executable, the decimal functions invoked from JIT code
>>>
>>> 1, 2 & 3 work good. Only 4 causes the symbol resolution issue.
>>>
>>> Thanks & regards,
>>> Ravindra.
>>>
>>>
>>>> On Dec 16, 2018, at 6:34 PM, Antoine Pitrou <antoine@xxxxxxxxxx> wrote:
>>>>
>>>>
>>>> In other words, see if:
>>>> nm --defined-only libgandiva.a | grep _ZNSt8ios_base4InitC1Ev
>>>>
>>>> outputs anything.
>>>>
>>>> Regards
>>>>
>>>> Antoine.
>>>>
>>>>
>>>> Le 16/12/2018 à 14:03, Antoine Pitrou a écrit :
>>>>>
>>>>> Does the static gandiva lib bundle libstdc++? Otherwise you'll probably
>>>>> need to load it explicitly in the Java process.
>>>>>
>>>>> Regards
>>>>>
>>>>> Antoine.
>>>>>
>>>>>
>>>>> Le 16/12/2018 à 04:53, Ravindra Pindikura a écrit :
>>>>>> I think this happens if any of the cpp files that are used for IR generation (clang —emit-llvm) includes <iostream>. For decimal support, I’m generating IR code from arrow/util/decimal.cc <http://decimal.cc/> which includes <iostream> (many times in the header file chain).
>>>>>>
>>>>>>> On Dec 16, 2018, at 9:14 AM, Ravindra Pindikura <ravindra@xxxxxxxxxx> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I’ve been working on adding decimal support in gandiva - I’m hitting a symbol resolution error with std::ios_base::Init with the java tests on ubuntu. Any ideas on what I may be doing wrong ?
>>>>>>>
>>>>>>> LLVM ERROR: Program used external function '_ZNSt8ios_base4InitC1Ev' which could not be resolved!
>>>>>>>
>>>>>>>
>>>>>>> I’ve the same test (add two decimals) in three forms : 
>>>>>>>
>>>>>>> 1. Cpp test executable with shared gandiva lib
>>>>>>>
>>>>>>> Works on both ubuntu and Mac
>>>>>>>
>>>>>>> 2. Cpp test executable with static gandiva lib
>>>>>>>
>>>>>>> Works on both ubuntu and Mac
>>>>>>>
>>>>>>> 3. Java test (internally uses jni over static gandiva lib)
>>>>>>>
>>>>>>> Works on Mac, but fails on ubuntu !!
>>>>>>>
>>>>>>> Ubuntu failure : https://travis-ci.org/apache/arrow/jobs/468562791
>>>>>>>
>>>>>>> I tried switching to ORC JIT (from MC JIT) but that didn’t help. In the JIT, the symbol is not found in the process address space for ubuntu+java. For all the other cases, the symbol is found.
>>>>>>>
>>>>>>> https://github.com/apache/arrow/pull/2942/files#diff-5ef29a5cabe384645dbf7a9c704aceeaR36
>>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Ravindra.
>>>>>>
>>>>>>
>>>
> 
>