git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [2/2] calcite git commit: [CALCITE-2347] running ElasticSearch in embedded mode for unit tests of ES adapter (Andrei Sereda)


Your changes seem to be generally beneficial and are much appreciated. But they do not fix the main problem, which is the missing package on JDK 9, 10, 11. (I believe it has always worked on JDK 8.) For example, here is output from the run against your branch on JDK 10 Ubuntu:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on project calcite: Error generating maven-javadoc-plugin:3.0.1:test-aggregate report: 
[ERROR] Exit code: 1 - /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29: error: package org.elasticsearch.node.internal does not exist
[ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer;
[ERROR]                                       ^
[ERROR] 

Let’s work on fixing the main problem first, then we’ll consider your PR.

We had issues before (there is a JIRA case, don’t recall the number) where ES2 and ES5 could not share the same class path. I think this is another manifestation of that — maybe javadoc tries to generate for all modules simultaneously, and therefore when ES5’s dependencies have been loaded into javadoc, ES2’s dependencies cannot be seen.

The solution may be to disable javadoc for the ES2 adapter. Or perhaps just for that one java file.

Julian


> On Jun 25, 2018, at 8:10 AM, Volodymyr Vysotskyi <volodymyr@xxxxxxxxxx> wrote:
> 
> Thanks for the changes, I have checked and it passed on my env.
> 
> I think it should be also checked with other JDK versions besides 8.
> 
> Kind regards,
> Volodymyr Vysotskyi
> 
> 
> пн, 25 черв. 2018 о 07:05 Andrei Sereda <andrei@xxxxxxxxx> пише:
> 
>> Can you please check the branch in my fork ?
>> 
>> fork: https://github.com/asereda-gs/calcite
>> branch:  javadoc-fixes
>> 
>> I've tried to remove most of the warnings and javadoc, javadoc-test and
>> site all work for me.
>> 
>> ~/I/c/elasticsearch5 ❯❯❯ mvn --version
>>                               javadoc-fixes ✭
>> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
>> 2018-06-17T14:33:14-04:00)
>> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
>> Java version: 1.8.0_151, vendor: Oracle Corporation, runtime:
>> /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre
>> Default locale: en_US, platform encoding: UTF-8
>> OS name: "mac os x", version: "10.13.2", arch: "x86_64", family: "mac"
>> 
>> 
>> 
>> On Sun, Jun 24, 2018 at 12:40 PM Volodymyr Vysotskyi <volodymyr@xxxxxxxxxx
>>> 
>> wrote:
>> 
>>> I have also checked "mvn javadoc:javadoc javadoc:test-javadoc" on my
>>> machine (Ubuntu, JDK 8, Maven 3.3.9) and it fails on elasticsearch2
>> module
>>> (didn't check this problem for the next modules).
>>> 
>>> "mvn clean -DskipTests site" fails for MongoDB module.
>>> 
>>> Kind regards,
>>> Volodymyr Vysotskyi
>>> 
>>> 
>>> сб, 23 черв. 2018 о 03:12 Julian Hyde <jhyde@xxxxxxxxxx> пише:
>>> 
>>>> Same failure for me on JDK 8/macOS. The most likely other variable is
>>>> maven: I have 3.5.2 on macOS, 3.5.3 on ubuntu.
>>>> 
>>>> I’m mystified why no one else is seeing javadoc failures on macOS.
>>>> 
>>>> Julian
>>>> 
>>>> 
>>>>> On Jun 22, 2018, at 4:35 PM, Julian Hyde <jhyde@xxxxxxxxxx> wrote:
>>>>> 
>>>>> On JDK 8/ubuntu, “mvn clean -DskipTests site” gives the following
>>>> failure:
>>>>> 
>>>>> [ERROR] Failed to execute goal
>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on
>>>> project calcite-mongodb: Error generating
>>>> maven-javadoc-plugin:3.0.1:test-javadoc report:
>>>>> [ERROR] Exit code: 1 -
>>>> 
>>> 
>> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:22:
>>>> error: cannot find symbol
>>>>> [ERROR] import org.apache.calcite.test.CalciteAssert;
>>>>> [ERROR]                               ^
>>>>> [ERROR]   symbol:   class CalciteAssert
>>>>> [ERROR]   location: package org.apache.calcite.test
>>>>> [ERROR]
>>>> 
>>> 
>> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:137:
>>>> error: package CalciteAssert does not exist
>>>>> [ERROR]   private CalciteAssert.AssertThat assertModel(String model)
>> {
>>>>> [ERROR]                        ^
>>>>> [ERROR]
>>>> 
>>> 
>> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:145:
>>>> error: package CalciteAssert does not exist
>>>>> [ERROR]   private CalciteAssert.AssertThat assertModel(URL url) {
>>>>> [ERROR]                        ^
>>>>> [ERROR]
>>>> 
>>> 
>> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/test/MongoAssertions.java:43:
>>>> error: reference not found
>>>>> [ERROR]   /** Similar to {@link CalciteAssert#checkResultUnordered},
>>> but
>>>> filters strings
>>>>> [ERROR]                         ^
>>>>> 
>>>>> 
>>>>> 
>>>>>> On Jun 22, 2018, at 3:46 PM, Andrei Sereda <andrei@xxxxxxxxx>
>> wrote:
>>>>>> 
>>>>>> Hmm. Both master and c12cb4b0de work for me. Perhaps the issue is
>> JDK
>>> 8
>>>> vs
>>>>>> 9 ?
>>>>>> 
>>>>>> # JDK8 (oracle) /  macOS
>>>>>> $ mvn clean -DskipTests site
>>>>>> 
>>>>>> On Fri, Jun 22, 2018 at 5:56 PM Julian Hyde <jhyde@xxxxxxxxxx>
>> wrote:
>>>>>> 
>>>>>>> I just tried it again. The following fails for me (jdk9, ubuntu):
>>>>>>> 
>>>>>>> $ git checkout c12cb4b0de
>>>>>>> $ mvn clean -DskipTests site
>>>>>>> 
>>>>>>> Constructing Javadoc information...
>>>>>>> 1 error
>>>>>>> [INFO]
>>>>>>> 
>>>> 
>> ------------------------------------------------------------------------
>>>>>>> [INFO] Reactor Summary:
>>>>>>> [INFO]
>>>>>>> [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE
>>>> [02:29
>>>>>>> min]
>>>>>>> [INFO] Calcite Linq4j ..................................... SKIPPED
>>>>>>> [INFO] Calcite Core ....................................... SKIPPED
>>>>>>> [INFO] Calcite Cassandra .................................. SKIPPED
>>>>>>> [INFO] Calcite Druid ...................................... SKIPPED
>>>>>>> [INFO] Calcite Elasticsearch .............................. SKIPPED
>>>>>>> [INFO] Calcite Elasticsearch5 ............................. SKIPPED
>>>>>>> [INFO] Calcite Examples ................................... SKIPPED
>>>>>>> [INFO] Calcite Example CSV ................................ SKIPPED
>>>>>>> [INFO] Calcite Example Function ........................... SKIPPED
>>>>>>> [INFO] Calcite File ....................................... SKIPPED
>>>>>>> [INFO] Calcite Geode ...................................... SKIPPED
>>>>>>> [INFO] Calcite MongoDB .................................... SKIPPED
>>>>>>> [INFO] Calcite Pig ........................................ SKIPPED
>>>>>>> [INFO] Calcite Piglet ..................................... SKIPPED
>>>>>>> [INFO] Calcite Plus ....................................... SKIPPED
>>>>>>> [INFO] Calcite Server ..................................... SKIPPED
>>>>>>> [INFO] Calcite Spark ...................................... SKIPPED
>>>>>>> [INFO] Calcite Splunk ..................................... SKIPPED
>>>>>>> [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED
>>>>>>> [INFO]
>>>>>>> 
>>>> 
>> ------------------------------------------------------------------------
>>>>>>> [INFO] BUILD FAILURE
>>>>>>> [INFO]
>>>>>>> 
>>>> 
>> ------------------------------------------------------------------------
>>>>>>> [INFO] Total time: 02:29 min
>>>>>>> [INFO] Finished at: 2018-06-22T14:53:13-07:00
>>>>>>> [INFO]
>>>>>>> 
>>>> 
>> ------------------------------------------------------------------------
>>>>>>> [ERROR] Failed to execute goal
>>>>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site)
>> on
>>>>>>> project calcite: Error generating
>>>> maven-javadoc-plugin:3.0.1:test-aggregate
>>>>>>> report:
>>>>>>> [ERROR] Exit code: 1 -
>>>>>>> 
>>>> 
>>> 
>> /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
>>>>>>> error: package org.elasticsearch.node.internal does not exist
>>>>>>> [ERROR] import
>>>> org.elasticsearch.node.internal.InternalSettingsPreparer;
>>>>>>> [ERROR]                                       ^
>>>>>>> [ERROR]
>>>>>>> [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options
>>>> @packages
>>>>>>> @argfile
>>>>>>> [ERROR]
>>>>>>> [ERROR] Refer to the generated Javadoc files in
>>>>>>> '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir.
>>>>>>> [ERROR] -> [Help 1]
>>>>>>> [ERROR]
>>>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven
>> with
>>>> the
>>>>>>> -e switch.
>>>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug
>>> logging.
>>>>>>> [ERROR]
>>>>>>> [ERROR] For more information about the errors and possible
>> solutions,
>>>>>>> please read the following articles:
>>>>>>> [ERROR] [Help 1]
>>>>>>> 
>>>> 
>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
>>>>>>> 
>>>>>>> 
>>>>>>>> On Jun 22, 2018, at 2:41 PM, Michael Mior <mmior@xxxxxxxxxx>
>> wrote:
>>>>>>>> 
>>>>>>>> Odd, mvn clean site still works fine for me.
>>>>>>>> --
>>>>>>>> Michael Mior
>>>>>>>> mmior@xxxxxxxxxx
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Le ven. 22 juin 2018 à 17:12, Julian Hyde <jhyde@xxxxxxxxxx> a
>>> écrit
>>>> :
>>>>>>>> 
>>>>>>>>> Looks like this change broke “mvn site” (perhaps also “mvn
>>>>>>>>> javadoc:test-javadoc”).
>>>>>>>>> 
>>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site
>> (default-site)
>>> on
>>>>>>>>> project calcite: Error generating
>>>>>>> maven-javadoc-plugin:3.0.1:test-aggregate
>>>>>>>>> report:
>>>>>>>>> [ERROR] Exit code: 1 -
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
>>>>>>>>> error: package org.elasticsearch.node.internal does not exist
>>>>>>>>> [ERROR] import
>>>> org.elasticsearch.node.internal.InternalSettingsPreparer;
>>>>>>>>> [ERROR]                                       ^
>>>>>>>>> [ERROR]
>>>>>>>>> 
>>>>>>>>>> On Jun 21, 2018, at 3:39 AM, mmior@xxxxxxxxxx wrote:
>>>>>>>>>> 
>>>>>>>>>> [CALCITE-2347] running ElasticSearch in embedded mode for unit
>>>> tests of
>>>>>>>>> ES adapter (Andrei Sereda)
>>>>>>>>>> 
>>>>>>>>>> After discussion on dev-list Integration tests (for ES) have
>> been
>>>>>>>>> removed. They're now
>>>>>>>>>> superseded by unit tests (which execute queries against a real
>>>> elastic
>>>>>>>>> instance)
>>>>>>>>>> 
>>>>>>>>>> Added local file (zips-mini.json) which contains a small subset
>> of
>>>>>>>>> original zips.json
>>>>>>>>>> (allows to bootstrap tests faster)
>>>>>>>>>> 
>>>>>>>>>> Created separate ES JUnit rule which can be re-used across
>>> different
>>>>>>>>> tests.
>>>>>>>>>> 
>>>>>>>>>> Both v2 and v5 of ES adapters are supported.
>>>>>>>>>> 
>>>>>>>>>> Close apache/calcite#716
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo
>>>>>>>>>> Commit:
>>>> http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0
>>>>>>>>>> Tree:
>>> http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0
>>>>>>>>>> Diff:
>>> http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0
>>>>>>>>>> 
>>>>>>>>>> Branch: refs/heads/master
>>>>>>>>>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d
>>>>>>>>>> Parents: 37944bb
>>>>>>>>>> Author: Andrei Sereda <andrei@xxxxxxxxxx>
>>>>>>>>>> Authored: Thu May 31 18:19:10 2018 -0400
>>>>>>>>>> Committer: Michael Mior <mmior@xxxxxxxxxxxx>
>>>>>>>>>> Committed: Thu Jun 21 06:38:50 2018 -0400
>>>>>>>>>> 
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> .../AbstractElasticsearchTable.java             |  12 +
>>>>>>>>>> .../elasticsearch/ElasticsearchProject.java     |  61 ++-
>>>>>>>>>> elasticsearch2/pom.xml                          |   6 +
>>>>>>>>>> .../Elasticsearch2Enumerator.java               |  12 +-
>>>>>>>>>> .../elasticsearch2/Elasticsearch2Schema.java    |  16 +-
>>>>>>>>>> .../elasticsearch2/Elasticsearch2Table.java     |   9 +-
>>>>>>>>>> .../ElasticSearch2AdapterTest.java              | 395
>>>>>>> ++++++++++++++++++
>>>>>>>>>> .../elasticsearch2/EmbeddedElasticNode.java     | 147 +++++++
>>>>>>>>>> .../elasticsearch2/EmbeddedElasticRule.java     |  97 +++++
>>>>>>>>>> .../org/apache/calcite/test/ElasticChecker.java |  49 +++
>>>>>>>>>> .../calcite/test/Elasticsearch2AdapterIT.java   | 270
>>> -------------
>>>>>>>>>> .../resources/elasticsearch-zips-model.json     |  50 ---
>>>>>>>>>> .../src/test/resources/zips-mini.json           | 149 +++++++
>>>>>>>>>> elasticsearch5/pom.xml                          |  31 ++
>>>>>>>>>> .../elasticsearch5/Elasticsearch5Schema.java    |  17 +-
>>>>>>>>>> .../elasticsearch5/Elasticsearch5Table.java     |  11 +-
>>>>>>>>>> .../ElasticSearch5AdapterTest.java              | 399
>>>>>>> +++++++++++++++++++
>>>>>>>>>> .../elasticsearch5/EmbeddedElasticNode.java     | 153 +++++++
>>>>>>>>>> .../elasticsearch5/EmbeddedElasticRule.java     |  98 +++++
>>>>>>>>>> .../org/apache/calcite/test/ElasticChecker.java |  49 +++
>>>>>>>>>> .../calcite/test/Elasticsearch5AdapterIT.java   | 270
>>> -------------
>>>>>>>>>> .../resources/elasticsearch-zips-model.json     |  50 ---
>>>>>>>>>> elasticsearch5/src/test/resources/log4j2.xml    |  16 +
>>>>>>>>>> .../src/test/resources/zips-mini.json           | 149 +++++++
>>>>>>>>>> pom.xml                                         |  20 +-
>>>>>>>>>> 25 files changed, 1866 insertions(+), 670 deletions(-)
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> diff --git
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>>>>>>> index 0980469..8cc5933 100644
>>>>>>>>>> ---
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>>>>>>> +++
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
>>>>>>>>>> @@ -75,6 +75,18 @@ public abstract class
>>> AbstractElasticsearchTable
>>>>>>>>> extends AbstractQueryableTable
>>>>>>>>>>     relOptTable, this, null);
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> +  /**
>>>>>>>>>> +   * In ES 5.x scripted fields start with {@code
>>>> params._source.foo}
>>>>>>>>> while in ES2.x
>>>>>>>>>> +   * {@code _source.foo}. Helper method to build correct query
>>>> based
>>>>>>> on
>>>>>>>>> runtime version of elastic.
>>>>>>>>>> +   *
>>>>>>>>>> +   * @see <a href="
>>>>>>>>> https://github.com/elastic/elasticsearch/issues/20068";>_source
>>>>>>>>> variable</a>
>>>>>>>>>> +   * @see <a href="
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html
>>>>>>> ">Scripted
>>>>>>>>> Fields</a>
>>>>>>>>>> +   */
>>>>>>>>>> +  protected String scriptedFieldPrefix() {
>>>>>>>>>> +    // this is default pattern starting 5.x
>>>>>>>>>> +    return "params._source";
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> /** Executes a "find" operation on the underlying type.
>>>>>>>>>> *
>>>>>>>>>> * <p>For example,
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> diff --git
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>>>>>>> index b42abd7..961c8b0 100644
>>>>>>>>>> ---
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>>>>>>> +++
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
>>>>>>>>>> @@ -27,11 +27,15 @@ import
>>>>>>>>> org.apache.calcite.rel.metadata.RelMetadataQuery;
>>>>>>>>>> import org.apache.calcite.rel.type.RelDataType;
>>>>>>>>>> import org.apache.calcite.rex.RexNode;
>>>>>>>>>> import org.apache.calcite.util.Pair;
>>>>>>>>>> -import org.apache.calcite.util.Util;
>>>>>>>>>> +
>>>>>>>>>> +import com.google.common.base.Function;
>>>>>>>>>> +import com.google.common.collect.Lists;
>>>>>>>>>> 
>>>>>>>>>> import java.util.ArrayList;
>>>>>>>>>> import java.util.List;
>>>>>>>>>> 
>>>>>>>>>> +import javax.annotation.Nullable;
>>>>>>>>>> +
>>>>>>>>>> /**
>>>>>>>>>> * Implementation of {@link org.apache.calcite.rel.core.Project}
>>>>>>>>>> * relational expression in Elasticsearch.
>>>>>>>>>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends
>>>> Project
>>>>>>>>> implements ElasticsearchRel {
>>>>>>>>>> implementor.visitChild(0, getInput());
>>>>>>>>>> 
>>>>>>>>>> final List<String> inFields =
>>>>>>>>>> -
>>>>>>>>> 
>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType());
>>>>>>>>>> +
>>>>>>>>> 
>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType());
>>>>>>>>>> final ElasticsearchRules.RexToElasticsearchTranslator
>> translator
>>> =
>>>>>>>>>> -        new ElasticsearchRules.RexToElasticsearchTranslator(
>>>>>>>>>> -            (JavaTypeFactory) getCluster().getTypeFactory(),
>>>>>>> inFields);
>>>>>>>>>> +            new
>> ElasticsearchRules.RexToElasticsearchTranslator(
>>>>>>>>>> +                    (JavaTypeFactory)
>>>> getCluster().getTypeFactory(),
>>>>>>>>> inFields);
>>>>>>>>>> 
>>>>>>>>>> -    final List<String> findItems = new ArrayList<>();
>>>>>>>>>> -    final List<String> scriptFieldItems = new ArrayList<>();
>>>>>>>>>> +    final List<String> fields = new ArrayList<>();
>>>>>>>>>> +    final List<String> scriptFields = new ArrayList<>();
>>>>>>>>>> for (Pair<RexNode, String> pair: getNamedProjects()) {
>>>>>>>>>>   final String name = pair.right;
>>>>>>>>>>   final String expr = pair.left.accept(translator);
>>>>>>>>>> 
>>>>>>>>>>   if (expr.equals("\"" + name + "\"")) {
>>>>>>>>>> -        findItems.add(ElasticsearchRules.quote(name));
>>>>>>>>>> +        fields.add(name);
>>>>>>>>>>   } else if (expr.matches("\"literal\":.+")) {
>>>>>>>>>> -        scriptFieldItems.add(ElasticsearchRules.quote(name)
>>>>>>>>>> -            + ":{\"script\": "
>>>>>>>>>> -            + expr.split(":")[1] + "}");
>>>>>>>>>> +        scriptFields.add(ElasticsearchRules.quote(name)
>>>>>>>>>> +                + ":{\"script\": "
>>>>>>>>>> +                + expr.split(":")[1] + "}");
>>>>>>>>>>   } else {
>>>>>>>>>> -        scriptFieldItems.add(ElasticsearchRules.quote(name)
>>>>>>>>>> -            + ":{\"script\":\"params._source."
>>>>>>>>>> -            + expr.replaceAll("\"", "") + "\"}");
>>>>>>>>>> +        scriptFields.add(ElasticsearchRules.quote(name)
>>>>>>>>>> +                + ":{\"script\":"
>>>>>>>>>> +                // _source (ES2) vs params._source (ES5)
>>>>>>>>>> +                + "\"" +
>>>>>>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "."
>>>>>>>>>> +                + expr.replaceAll("\"", "") + "\"}");
>>>>>>>>>> +      }
>>>>>>>>>> +    }
>>>>>>>>>> +
>>>>>>>>>> +    StringBuilder query = new StringBuilder();
>>>>>>>>>> +    if (scriptFields.isEmpty()) {
>>>>>>>>>> +      List<String> newList = Lists.transform(fields, new
>>>>>>>>> Function<String, String>() {
>>>>>>>>>> +        @Nullable
>>>>>>>>>> +        @Override public String apply(@Nullable String input) {
>>>>>>>>>> +          return ElasticsearchRules.quote(input);
>>>>>>>>>> +        }
>>>>>>>>>> +      });
>>>>>>>>>> +
>>>>>>>>>> +      final String findString = String.join(", ", newList);
>>>>>>>>>> +      query.append("\"_source\" :
>>>> [").append(findString).append("]");
>>>>>>>>>> +    } else {
>>>>>>>>>> +      // if scripted fields are present, ES ignores _source
>>>> attribute
>>>>>>>>>> +      for (String field: fields) {
>>>>>>>>>> +        scriptFields.add(ElasticsearchRules.quote(field) +
>>>>>>>>> ":{\"script\": "
>>>>>>>>>> +                // _source (ES2) vs params._source (ES5)
>>>>>>>>>> +                + "\"" +
>>>>>>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "."
>>>>>>>>>> +                + field + "\"}");
>>>>>>>>>>   }
>>>>>>>>>> +      query.append("\"script_fields\": {" + String.join(", ",
>>>>>>>>> scriptFields) + "}");
>>>>>>>>>> }
>>>>>>>>>> -    final String findString = Util.toString(findItems, "", ",
>> ",
>>>> "");
>>>>>>>>>> -    final String scriptFieldString = "\"script_fields\": {"
>>>>>>>>>> -        + Util.toString(scriptFieldItems, "", ", ", "") + "}";
>>>>>>>>>> -    final String fieldString = "\"_source\" : [" + findString +
>>> "]"
>>>>>>>>>> -        + ", " + scriptFieldString;
>>>>>>>>>> 
>>>>>>>>>> for (String opfield : implementor.list) {
>>>>>>>>>>   if (opfield.startsWith("\"_source\"")) {
>>>>>>>>>>     implementor.list.remove(opfield);
>>>>>>>>>>   }
>>>>>>>>>> }
>>>>>>>>>> -    implementor.add(fieldString);
>>>>>>>>>> +    implementor.add(query.toString());
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml
>>>>>>>>>> index f24622c..6fbee03 100644
>>>>>>>>>> --- a/elasticsearch2/pom.xml
>>>>>>>>>> +++ b/elasticsearch2/pom.xml
>>>>>>>>>> @@ -73,6 +73,12 @@ limitations under the License.
>>>>>>>>>>   <version>${elasticsearch-java-driver.version}</version>
>>>>>>>>>> </dependency>
>>>>>>>>>> <dependency>
>>>>>>>>>> +      <!-- Lang groovy dependency is needed for testing with
>>>> embedded
>>>>>>>>> ES (scripted fields like loc[0]) -->
>>>>>>>>>> +      <groupId>org.elasticsearch.module</groupId>
>>>>>>>>>> +      <artifactId>lang-groovy</artifactId>
>>>>>>>>>> +      <scope>test</scope>
>>>>>>>>>> +    </dependency>
>>>>>>>>>> +    <dependency>
>>>>>>>>>>   <groupId>com.carrotsearch</groupId>
>>>>>>>>>>   <artifactId>hppc</artifactId>
>>>>>>>>>>   <version>${hppc.version}</version>
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> diff --git
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>>>>>>> index 84370ab..c3d2ac0 100644
>>>>>>>>>> ---
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>>>>>>> +++
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
>>>>>>>>>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit;
>>>>>>>>>> import java.util.Date;
>>>>>>>>>> import java.util.Iterator;
>>>>>>>>>> import java.util.List;
>>>>>>>>>> +import java.util.Locale;
>>>>>>>>>> import java.util.Map;
>>>>>>>>>> 
>>>>>>>>>> /**
>>>>>>>>>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator
>>>> implements
>>>>>>>>> Enumerator<Object> {
>>>>>>>>>> private static Function1<SearchHit, Object[]> listGetter(
>>>>>>>>>>   final List<Map.Entry<String, Class>> fields) {
>>>>>>>>>> return new Function1<SearchHit, Object[]>() {
>>>>>>>>>> -      public Object[] apply(SearchHit searchHitFields) {
>>>>>>>>>> +      public Object[] apply(SearchHit hit) {
>>>>>>>>>>     Object[] objects = new Object[fields.size()];
>>>>>>>>>>     for (int i = 0; i < fields.size(); i++) {
>>>>>>>>>>       final Map.Entry<String, Class> field = fields.get(i);
>>>>>>>>>>       final String name = field.getKey();
>>>>>>>>>> -          if (searchHitFields.fields().isEmpty()) {
>>>>>>>>>> -            objects[i] =
>>>>>>> convert(searchHitFields.getSource().get(name),
>>>>>>>>> field.getValue());
>>>>>>>>>> +          if (hit.fields().isEmpty()) {
>>>>>>>>>> +            objects[i] = convert(hit.getSource().get(name),
>>>>>>>>> field.getValue());
>>>>>>>>>> +          } else if (hit.fields().containsKey(name)) {
>>>>>>>>>> +            objects[i] = convert(hit.field(name).getValue(),
>>>>>>>>> field.getValue());
>>>>>>>>>>       } else {
>>>>>>>>>> -            objects[i] =
>>>>>>>>> convert(searchHitFields.field(name).getValue(),
>> field.getValue());
>>>>>>>>>> +            throw new IllegalStateException(
>>>>>>>>>> +                    String.format(Locale.getDefault(), "No
>> result
>>>> for
>>>>>>>>> %s", field));
>>>>>>>>>>       }
>>>>>>>>>>     }
>>>>>>>>>>     return objects;
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> diff --git
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>>>>>>> index 668402b..46e3fc5 100644
>>>>>>>>>> ---
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>>>>>>> +++
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
>>>>>>>>>> @@ -22,6 +22,9 @@ import
>>>> org.apache.calcite.schema.impl.AbstractSchema;
>>>>>>>>>> 
>>>>>>>>>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
>>>>>>>>>> 
>>>>>>>>>> +import com.google.common.annotations.VisibleForTesting;
>>>>>>>>>> +import com.google.common.base.Preconditions;
>>>>>>>>>> +
>>>>>>>>>> import com.google.common.collect.ImmutableList;
>>>>>>>>>> import com.google.common.collect.ImmutableMap;
>>>>>>>>>> 
>>>>>>>>>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends
>>>>>>>>> AbstractSchema
>>>>>>>>>> }
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> +  /**
>>>>>>>>>> +   * Allows schema to be instantiated from existing elastic
>>> search
>>>>>>>>> client.
>>>>>>>>>> +   * This constructor is used in tests.
>>>>>>>>>> +   */
>>>>>>>>>> +  @VisibleForTesting
>>>>>>>>>> +  Elasticsearch2Schema(Client client, String index) {
>>>>>>>>>> +    this.client = Preconditions.checkNotNull(client, "client");
>>>>>>>>>> +    this.index = Preconditions.checkNotNull(index, "index");
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> @Override protected Map<String, Table> getTableMap() {
>>>>>>>>>> final ImmutableMap.Builder<String, Table> builder =
>>>>>>>>> ImmutableMap.builder();
>>>>>>>>>> 
>>>>>>>>>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends
>>>>>>>>> AbstractSchema
>>>>>>>>>> 
>>>>>>>>>> final List<DiscoveryNode> nodes =
>>>>>>>>> ImmutableList.copyOf(transportClient.connectedNodes());
>>>>>>>>>> if (nodes.isEmpty()) {
>>>>>>>>>> -      throw new RuntimeException("Cannot connect to any
>>>> elasticsearch
>>>>>>>>> nodes");
>>>>>>>>>> +      throw new IllegalStateException("Cannot connect to any
>>>>>>>>> elasticsearch node: "
>>>>>>>>>> +              + transportNodes);
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> client = transportClient;
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> diff --git
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>>>>>>> index 636aa5f..2928835 100644
>>>>>>>>>> ---
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>>>>>>> +++
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
>>>>>>>>>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends
>>>>>>>>> AbstractElasticsearchTable {
>>>>>>>>>> this.client = client;
>>>>>>>>>> }
>>>>>>>>>> 
>>>>>>>>>> +  /**
>>>>>>>>>> +   * ES version 2.x. To access document attributes ES2 uses
>>> {@code
>>>>>>>>> _source.foo} syntax.
>>>>>>>>>> +   */
>>>>>>>>>> +  @Override protected String scriptedFieldPrefix() {
>>>>>>>>>> +    return "_source";
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> @Override protected Enumerable<Object> find(String index,
>>>> List<String>
>>>>>>>>> ops,
>>>>>>>>>> -      List<Map.Entry<String, Class>> fields) {
>>>>>>>>>> +
>>>> List<Map.Entry<String,
>>>>>>>>> Class>> fields) {
>>>>>>>>>> final String dbName = index;
>>>>>>>>>> 
>>>>>>>>>> final String queryString = "{" + Util.toString(ops, "", ", ",
>>> "") +
>>>>>>>>> "}";
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
>>>>>>>>>> 
>>>> ----------------------------------------------------------------------
>>>>>>>>>> diff --git
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
>>>>>>>>>> new file mode 100644
>>>>>>>>>> index 0000000..287e094
>>>>>>>>>> --- /dev/null
>>>>>>>>>> +++
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
>>>>>>>>>> @@ -0,0 +1,395 @@
>>>>>>>>>> +/*
>>>>>>>>>> + * Licensed to the Apache Software Foundation (ASF) under one
>> or
>>>> more
>>>>>>>>>> + * contributor license agreements.  See the NOTICE file
>>> distributed
>>>>>>> with
>>>>>>>>>> + * this work for additional information regarding copyright
>>>> ownership.
>>>>>>>>>> + * The ASF licenses this file to you under the Apache License,
>>>> Version
>>>>>>>>> 2.0
>>>>>>>>>> + * (the "License"); you may not use this file except in
>>> compliance
>>>>>>> with
>>>>>>>>>> + * the License.  You may obtain a copy of the License at
>>>>>>>>>> + *
>>>>>>>>>> + * http://www.apache.org/licenses/LICENSE-2.0
>>>>>>>>>> + *
>>>>>>>>>> + * Unless required by applicable law or agreed to in writing,
>>>> software
>>>>>>>>>> + * distributed under the License is distributed on an "AS IS"
>>>> BASIS,
>>>>>>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
>>> or
>>>>>>>>> implied.
>>>>>>>>>> + * See the License for the specific language governing
>>> permissions
>>>> and
>>>>>>>>>> + * limitations under the License.
>>>>>>>>>> + */
>>>>>>>>>> +package org.apache.calcite.adapter.elasticsearch2;
>>>>>>>>>> +
>>>>>>>>>> +import org.apache.calcite.jdbc.CalciteConnection;
>>>>>>>>>> +import org.apache.calcite.schema.SchemaPlus;
>>>>>>>>>> +import org.apache.calcite.schema.impl.ViewTable;
>>>>>>>>>> +import org.apache.calcite.schema.impl.ViewTableMacro;
>>>>>>>>>> +import org.apache.calcite.test.CalciteAssert;
>>>>>>>>>> +import org.apache.calcite.test.ElasticChecker;
>>>>>>>>>> +
>>>>>>>>>> +import com.google.common.io.LineProcessor;
>>>>>>>>>> +import com.google.common.io.Resources;
>>>>>>>>>> +
>>>>>>>>>> +import org.elasticsearch.action.bulk.BulkItemResponse;
>>>>>>>>>> +import org.elasticsearch.action.bulk.BulkRequestBuilder;
>>>>>>>>>> +import org.elasticsearch.action.bulk.BulkResponse;
>>>>>>>>>> +import org.elasticsearch.common.xcontent.XContentBuilder;
>>>>>>>>>> +import org.elasticsearch.common.xcontent.XContentFactory;
>>>>>>>>>> +
>>>>>>>>>> +import org.junit.BeforeClass;
>>>>>>>>>> +import org.junit.ClassRule;
>>>>>>>>>> +import org.junit.Test;
>>>>>>>>>> +
>>>>>>>>>> +import java.io.IOException;
>>>>>>>>>> +import java.nio.charset.StandardCharsets;
>>>>>>>>>> +import java.sql.Connection;
>>>>>>>>>> +import java.sql.DriverManager;
>>>>>>>>>> +import java.sql.SQLException;
>>>>>>>>>> +import java.util.Arrays;
>>>>>>>>>> +import java.util.Collections;
>>>>>>>>>> +import java.util.Locale;
>>>>>>>>>> +
>>>>>>>>>> +/**
>>>>>>>>>> + * Set of tests for ES adapter. Uses real instance via {@link
>>>>>>>>> EmbeddedElasticRule}. Document
>>>>>>>>>> + * source is local {@code zips-mini.json} file (located in the
>>>>>>>>> classpath).
>>>>>>>>>> + */
>>>>>>>>>> +public class ElasticSearch2AdapterTest {
>>>>>>>>>> +
>>>>>>>>>> +  @ClassRule //init once for all tests
>>>>>>>>>> +  public static final EmbeddedElasticRule NODE =
>>>>>>>>> EmbeddedElasticRule.create();
>>>>>>>>>> +
>>>>>>>>>> +  private static final String ZIPS = "zips";
>>>>>>>>>> +
>>>>>>>>>> +  /**
>>>>>>>>>> +   * Used to create {@code zips} index and insert some data
>>>>>>>>>> +   */
>>>>>>>>>> +  @BeforeClass
>>>>>>>>>> +  public static void setupInstance() throws Exception {
>>>>>>>>>> +    // define mapping so fields are searchable (term query)
>>>>>>>>>> +    XContentBuilder mapping =
>>>>>>>>> XContentFactory.jsonBuilder().startObject()
>>>>>>>>>> +            .startObject("properties")
>>>>>>>>>> +            .startObject("city").field("type", "string")
>>>>>>>>>> +                  .field("index", "not_analyzed").endObject()
>>>>>>>>>> +            .startObject("state").field("type", "string")
>>>>>>>>>> +                  .field("index", "not_analyzed").endObject()
>>>>>>>>>> +            .startObject("pop").field("type",
>> "long").endObject()
>>>>>>>>>> +            .endObject()
>>>>>>>>>> +            .endObject();
>>>>>>>>>> +
>>>>>>>>>> +    // create index
>>>>>>>>>> +    NODE.client().admin().indices()
>>>>>>>>>> +            .prepareCreate(ZIPS)
>>>>>>>>>> +            .addMapping(ZIPS, mapping)
>>>>>>>>>> +            .get();
>>>>>>>>>> +
>>>>>>>>>> +    BulkRequestBuilder bulk =
>>>>>>>>> NODE.client().prepareBulk().setRefresh(true);
>>>>>>>>>> +
>>>>>>>>>> +    // load records from file
>>>>>>>>>> +
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"),
>>>>>>>>>> +            StandardCharsets.UTF_8, new LineProcessor<Void>() {
>>>>>>>>>> +              @Override public boolean processLine(String line)
>>>> throws
>>>>>>>>> IOException {
>>>>>>>>>> +                line = line.replaceAll("_id", "id"); // _id is
>> a
>>>>>>>>> reserved attribute in ES
>>>>>>>>>> +                bulk.add(NODE.client().prepareIndex(ZIPS,
>>>>>>>>> ZIPS).setSource(line));
>>>>>>>>>> +                return true;
>>>>>>>>>> +              }
>>>>>>>>>> +
>>>>>>>>>> +              @Override public Void getResult() {
>>>>>>>>>> +                return null;
>>>>>>>>>> +              }
>>>>>>>>>> +            });
>>>>>>>>>> +
>>>>>>>>>> +    if (bulk.numberOfActions() == 0) {
>>>>>>>>>> +      throw new IllegalStateException("No records to be
>>> indexed");
>>>>>>>>>> +    }
>>>>>>>>>> +
>>>>>>>>>> +    BulkResponse response = bulk.execute().get();
>>>>>>>>>> +
>>>>>>>>>> +    if (response.hasFailures()) {
>>>>>>>>>> +      throw new IllegalStateException(
>>>>>>>>>> +              String.format(Locale.getDefault(), "Failed to
>>>> populate
>>>>>>>>> %s:\n%s", NODE.httpAddress(),
>>>>>>>>>> +
>>>>>>>>> 
>>> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed)
>>>>>>>>>> +
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>")));
>>>>>>>>>> +    }
>>>>>>>>>> +
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  private CalciteAssert.ConnectionFactory
>> newConnectionFactory()
>>> {
>>>>>>>>>> +    return new CalciteAssert.ConnectionFactory() {
>>>>>>>>>> +      @Override public Connection createConnection() throws
>>>>>>>>> SQLException {
>>>>>>>>>> +        final Connection connection =
>>>>>>>>> DriverManager.getConnection("jdbc:calcite:");
>>>>>>>>>> +        final SchemaPlus root =
>>>>>>>>> connection.unwrap(CalciteConnection.class).getRootSchema();
>>>>>>>>>> +
>>>>>>>>>> +        root.add("elastic", new
>>> Elasticsearch2Schema(NODE.client(),
>>>>>>>>> ZIPS));
>>>>>>>>>> +
>>>>>>>>>> +        // add calcite view programmatically
>>>>>>>>>> +        final String viewSql = "select cast(_MAP['city'] AS
>>>>>>>>> varchar(20)) AS \"city\", "
>>>>>>>>>> +               + " cast(_MAP['loc'][0] AS float) AS
>>>> \"longitude\",\n"
>>>>>>>>>> +               + " cast(_MAP['loc'][1] AS float) AS
>>>> \"latitude\",\n"
>>>>>>>>>> +              + " cast(_MAP['pop'] AS integer) AS \"pop\", "
>>>>>>>>>> +              +  " cast(_MAP['state'] AS varchar(2)) AS
>>> \"state\",
>>>> "
>>>>>>>>>> +              +  " cast(_MAP['id'] AS varchar(5)) AS \"id\" "
>>>>>>>>>> +              +  "from \"elastic\".\"zips\"";
>>>>>>>>>> +
>>>>>>>>>> +        ViewTableMacro macro = ViewTable.viewMacro(root,
>> viewSql,
>>>>>>>>>> +                Collections.singletonList("elastic"),
>>>>>>>>> Arrays.asList("elastic", "view"), false);
>>>>>>>>>> +        root.add("ZIPS", macro);
>>>>>>>>>> +
>>>>>>>>>> +        return connection;
>>>>>>>>>> +      }
>>>>>>>>>> +    };
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  private CalciteAssert.AssertThat calciteAssert() {
>>>>>>>>>> +    return CalciteAssert.that()
>>>>>>>>>> +            .with(newConnectionFactory());
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  /**
>>>>>>>>>> +   * Tests using calcite view
>>>>>>>>>> +   */
>>>>>>>>>> +  @Test
>>>>>>>>>> +  public void view() throws Exception {
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +          .query("select * from zips where \"city\" =
>>> 'BROOKLYN'")
>>>>>>>>>> +          .returns("city=BROOKLYN; longitude=-73.956985;
>>>>>>>>> latitude=40.646694; "
>>>>>>>>>> +                  + "pop=111396; state=NY; id=11226\n")
>>>>>>>>>> +          .returnsCount(1);
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test
>>>>>>>>>> +  public void emptyResult() {
>>>>>>>>>> +    CalciteAssert.that()
>>>>>>>>>> +            .with(newConnectionFactory())
>>>>>>>>>> +            .query("select * from zips limit 0")
>>>>>>>>>> +            .returnsCount(0);
>>>>>>>>>> +
>>>>>>>>>> +    CalciteAssert.that()
>>>>>>>>>> +            .with(newConnectionFactory())
>>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where
>>>>>>>>> _MAP['Foo'] = '_MISSING_'")
>>>>>>>>>> +            .returnsCount(0);
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test
>>>>>>>>>> +  public void basic() throws Exception {
>>>>>>>>>> +    CalciteAssert.that()
>>>>>>>>>> +            .with(newConnectionFactory())
>>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where
>>>>>>>>> _MAP['city'] = 'BROOKLYN'")
>>>>>>>>>> +            .returnsCount(1);
>>>>>>>>>> +
>>>>>>>>>> +    CalciteAssert.that()
>>>>>>>>>> +            .with(newConnectionFactory())
>>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where"
>>>>>>>>>> +                    + " _MAP['city'] in ('BROOKLYN',
>>>> 'WASHINGTON')")
>>>>>>>>>> +            .returnsCount(2);
>>>>>>>>>> +
>>>>>>>>>> +    // lower-case
>>>>>>>>>> +    CalciteAssert.that()
>>>>>>>>>> +            .with(newConnectionFactory())
>>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where "
>>>>>>>>>> +                    + "_MAP['city'] in ('brooklyn', 'Brooklyn',
>>>>>>>>> 'BROOK') ")
>>>>>>>>>> +            .returnsCount(0);
>>>>>>>>>> +
>>>>>>>>>> +    // missing field
>>>>>>>>>> +    CalciteAssert.that()
>>>>>>>>>> +            .with(newConnectionFactory())
>>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where
>>>>>>>>> _MAP['CITY'] = 'BROOKLYN'")
>>>>>>>>>> +            .returnsCount(0);
>>>>>>>>>> +
>>>>>>>>>> +    // limit works
>>>>>>>>>> +    CalciteAssert.that()
>>>>>>>>>> +            .with(newConnectionFactory())
>>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" limit
>> 42")
>>>>>>>>>> +            .returnsCount(42);
>>>>>>>>>> +
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testSort() {
>>>>>>>>>> +    final String explain =
>>>> "PLAN=ElasticsearchToEnumerableConverter\n"
>>>>>>>>>> +            + "  ElasticsearchSort(sort0=[$4], dir0=[ASC])\n"
>>>>>>>>>> +            + "    ElasticsearchProject(city=[CAST(ITEM($0,
>>>>>>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>>>>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0,
>>> 'loc'),
>>>>>>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT],
>>>>>>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0,
>>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>>>>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0,
>> 'id')):VARCHAR(5)
>>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\"])\n"
>>>>>>>>>> +            + "      ElasticsearchTableScan(table=[[elastic,
>>>> zips]])";
>>>>>>>>>> +
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query("select * from zips order by \"state\"")
>>>>>>>>>> +            .returnsCount(10)
>>>>>>>>>> +            .explainContains(explain);
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testSortLimit() {
>>>>>>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
>>>>>>>>>> +            + "order by \"state\", \"id\" offset 2 rows fetch
>>> next
>>>> 3
>>>>>>>>> rows only";
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query(sql)
>>>>>>>>>> +            .returnsUnordered("state=AK; id=99801",
>>>>>>>>>> +                    "state=AL; id=35215",
>>>>>>>>>> +                    "state=AL; id=35401")
>>>>>>>>>> +            .queryContains(
>>>>>>>>>> +                    ElasticChecker.elasticsearchChecker(
>>>>>>>>>> +                            "\"_source\" : [\"state\",
>> \"id\"]",
>>>>>>>>>> +                            "\"sort\": [ {\"state\": \"asc\"},
>>>>>>> {\"id\":
>>>>>>>>> \"asc\"}]",
>>>>>>>>>> +                            "\"from\": 2",
>>>>>>>>>> +                            "\"size\": 3"));
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testOffsetLimit() {
>>>>>>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
>>>>>>>>>> +            + "offset 2 fetch next 3 rows only";
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query(sql)
>>>>>>>>>> +            .runs()
>>>>>>>>>> +            .queryContains(
>>>>>>>>>> +                    ElasticChecker.elasticsearchChecker(
>>>>>>>>>> +                            "\"from\": 2",
>>>>>>>>>> +                            "\"size\": 3",
>>>>>>>>>> +                            "\"_source\" : [\"state\",
>>> \"id\"]"));
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testLimit() {
>>>>>>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
>>>>>>>>>> +            + "fetch next 3 rows only";
>>>>>>>>>> +
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query(sql)
>>>>>>>>>> +            .runs()
>>>>>>>>>> +            .queryContains(
>>>>>>>>>> +                    ElasticChecker.elasticsearchChecker(
>>>>>>>>>> +                            "\"size\": 3",
>>>>>>>>>> +                            "\"_source\" : [\"state\",
>>> \"id\"]"));
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testFilterSort() {
>>>>>>>>>> +    final String sql = "select * from zips\n"
>>>>>>>>>> +            + "where \"state\" = 'CA' and \"id\" >= '70000'\n"
>>>>>>>>>> +            + "order by \"state\", \"id\"";
>>>>>>>>>> +    final String explain =
>>>> "PLAN=ElasticsearchToEnumerableConverter\n"
>>>>>>>>>> +            + "  ElasticsearchSort(sort0=[$4], sort1=[$5],
>>>> dir0=[ASC],
>>>>>>>>> dir1=[ASC])\n"
>>>>>>>>>> +            + "    ElasticsearchProject(city=[CAST(ITEM($0,
>>>>>>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>>>>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0,
>>> 'loc'),
>>>>>>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT],
>>>>>>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0,
>>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>>>>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0,
>> 'id')):VARCHAR(5)
>>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
>>>> \"ISO-8859-1$en_US$primary\"])\n"
>>>>>>>>>> +            + "
>>>>>>> ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0,
>>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
>>>>>>>>> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0,
>>>> 'id')):VARCHAR(5)
>>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
>> \"ISO-8859-1$en_US$primary\",
>>>>>>>>> '70000'))])\n"
>>>>>>>>>> +            + "        ElasticsearchTableScan(table=[[elastic,
>>>>>>> zips]])";
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query(sql)
>>>>>>>>>> +            .returnsOrdered("city=LOS ANGELES;
>>>> longitude=-118.258189;
>>>>>>>>> latitude=34.007856; "
>>>>>>>>>> +                            + "pop=96074; state=CA; id=90011",
>>>>>>>>>> +                    "city=BELL GARDENS; longitude=-118.17205;
>>>>>>>>> latitude=33.969177; "
>>>>>>>>>> +                            + "pop=99568; state=CA; id=90201",
>>>>>>>>>> +                    "city=NORWALK; longitude=-118.081767;
>>>>>>>>> latitude=33.90564; "
>>>>>>>>>> +                            + "pop=94188; state=CA; id=90650")
>>>>>>>>>> +            .queryContains(
>>>>>>>>>> +
>>> ElasticChecker.elasticsearchChecker("\"query\"
>>>> : "
>>>>>>>>>> +                                    +
>>>>>>>>> "{\"constant_score\":{\"filter\":{\"bool\":"
>>>>>>>>>> +                                    +
>>>>>>>>> "{\"must\":[{\"term\":{\"state\":\"CA\"}},"
>>>>>>>>>> +                                    +
>>>>>>>>> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}",
>>>>>>>>>> +                            "\"script_fields\":
>>>>>>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
>>>>>>>>>> +                                    +
>>>>>>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, "
>>>>>>>>>> +                                    + "\"city\":{\"script\":
>>>>>>>>> \"_source.city\"}, "
>>>>>>>>>> +                                    + "\"pop\":{\"script\":
>>>>>>>>> \"_source.pop\"}, "
>>>>>>>>>> +                                    + "\"state\":{\"script\":
>>>>>>>>> \"_source.state\"}, "
>>>>>>>>>> +                                    + "\"id\":{\"script\": \"_
>>>>>>> source.id
>>>>>>>>> \"}}",
>>>>>>>>>> +                            "\"sort\": [ {\"state\": \"asc\"},
>>>>>>> {\"id\":
>>>>>>>>> \"asc\"}]"))
>>>>>>>>>> +            .explainContains(explain);
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testFilterSortDesc() {
>>>>>>>>>> +    final String sql = "select * from zips\n"
>>>>>>>>>> +            + "where \"pop\" BETWEEN 95000 AND 100000\n"
>>>>>>>>>> +            + "order by \"state\" desc, \"pop\"";
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query(sql)
>>>>>>>>>> +            .limit(4)
>>>>>>>>>> +            .returnsOrdered(
>>>>>>>>>> +             "city=LOS ANGELES; longitude=-118.258189;
>>>>>>>>> latitude=34.007856; pop=96074; state=CA; id=90011",
>>>>>>>>>> +             "city=BELL GARDENS; longitude=-118.17205;
>>>>>>>>> latitude=33.969177; pop=99568; state=CA; id=90201");
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testFilterRedundant() {
>>>>>>>>>> +    final String sql = "select * from zips\n"
>>>>>>>>>> +            + "where \"state\" > 'CA' and \"state\" < 'AZ' and
>>>>>>>>> \"state\" = 'OK'";
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query(sql)
>>>>>>>>>> +            .runs()
>>>>>>>>>> +            .queryContains(
>>>>>>>>>> +                    ElasticChecker.elasticsearchChecker(""
>>>>>>>>>> +                                    + "\"query\" :
>>>>>>>>> {\"constant_score\":{\"filter\":{\"bool\":"
>>>>>>>>>> +                                    +
>>>>>>>>> "{\"must\":[{\"term\":{\"state\":\"OK\"}}]}}}}",
>>>>>>>>>> +                            "\"script_fields\":
>>>>>>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
>>>>>>>>>> +                                    +
>>>>>>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, "
>>>>>>>>>> +                                    +   "\"city\":{\"script\":
>>>>>>>>> \"_source.city\"}, "
>>>>>>>>>> +                                    +   "\"pop\":{\"script\":
>>>>>>>>> \"_source.pop\"}, \"state\":{\"script\": \"_source.state\"}, "
>>>>>>>>>> +                                    +
>>>> "\"id\":{\"script\":
>>>>>>>>> \"_source.id\"}}"
>>>>>>>>>> +                    ));
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testInPlan() {
>>>>>>>>>> +    final String[] searches = {
>>>>>>>>>> +        "\"query\" :
>>>>>>>>> {\"constant_score\":{\"filter\":{\"bool\":{\"should\":"
>>>>>>>>>> +                 +
>>>>>>>>> 
>>>>>>> 
>>>> 
>>> 
>> "[{\"bool\":{\"must\":[{\"term\":{\"pop\":96074}}]}},{\"bool\":{\"must\":[{\"term\":"
>>>>>>>>>> +                 + "{\"pop\":99568}}]}}]}}}}",
>>>>>>>>>> +        "\"script_fields\":
>>>>>>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
>>>>>>>>>> +                 +
>>> "\"latitude\":{\"script\":\"_source.loc[1]\"},
>>>> "
>>>>>>>>>> +                 +  "\"city\":{\"script\": \"_source.city\"}, "
>>>>>>>>>> +                 +  "\"pop\":{\"script\": \"_source.pop\"}, "
>>>>>>>>>> +                 +  "\"state\":{\"script\":
>> \"_source.state\"}, "
>>>>>>>>>> +                 +  "\"id\":{\"script\": \"_source.id\"}}"
>>>>>>>>>> +    };
>>>>>>>>>> +
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query("select * from zips where \"pop\" in (96074,
>>>>>>> 99568)")
>>>>>>>>>> +            .returnsUnordered(
>>>>>>>>>> +                    "city=BELL GARDENS; longitude=-118.17205;
>>>>>>>>> latitude=33.969177; pop=99568; state=CA; id=90201",
>>>>>>>>>> +                    "city=LOS ANGELES; longitude=-118.258189;
>>>>>>>>> latitude=34.007856; pop=96074; state=CA; id=90011"
>>>>>>>>>> +                    )
>>>>>>>>>> +
>>>>>>>>> .queryContains(ElasticChecker.elasticsearchChecker(searches));
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testZips() {
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +         .query("select \"state\", \"city\" from zips")
>>>>>>>>>> +         .returnsCount(10);
>>>>>>>>>> +  }
>>>>>>>>>> +
>>>>>>>>>> +  @Test public void testProject() {
>>>>>>>>>> +    final String sql = "select \"state\", \"city\", 0 as
>>>> \"zero\"\n"
>>>>>>>>>> +            + "from zips\n"
>>>>>>>>>> +            + "order by \"state\", \"city\"";
>>>>>>>>>> +
>>>>>>>>>> +    calciteAssert()
>>>>>>>>>> +            .query(sql)
>>>>>>>>>> +            .limit(2)
>>>>>>>>>> +            .returnsUnordered("state=AK; city=ANCHORAGE;
>> zero=0",
>>>>>>>>>> +                    "state=AK; city=FAIRBANKS; zero=0")
>>>>>>>>>> +            .queryContains(
>>>>>>>>>> +
>>>>>>>>> ElasticChecker.elasticsearchChecker("\"script_fields\": "
>>>>>>>>>> +                                    + "{\"zero\":{\"script\":
>>>> \"0\"},
>>>>>>> "
>>>>>>>>>> +                                    + "\"state\":{\"script\":
>>>>>>>>> \"_source.state\"}, "
>>>>>>>>>> +                                    + "\"city\":{\"script\":
>>>>>>>>> \"_source.city\"}}",
>>>>>>>>>> +                            "\"sort