git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [2/2] calcite git commit: [CALCITE-2347] running ElasticSearch in embedded mode for unit tests of ES adapter (Andrei Sereda)


I'm working on CALCITE-2376 which unifies ES2 and ES5 adapters. I'll make
sure it is passing both javadoc / site tasks in maven.

Would you consider merging this partial javadoc fix (PR 741
<https://github.com/apache/calcite/pull/741>) as precursor for CALCITE-2376 ?
Former PR has also fixes javadoc issues for mongo adapter.

On a separate note, do you think it makes sense to add javadoc generation
to travis build ?

On Mon, Jun 25, 2018 at 1:52 PM Julian Hyde <jhyde@xxxxxxxxxx> wrote:

> Your changes seem to be generally beneficial and are much appreciated. But
> they do not fix the main problem, which is the missing package on JDK 9,
> 10, 11. (I believe it has always worked on JDK 8.) For example, here is
> output from the run against your branch on JDK 10 Ubuntu:
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on
> project calcite: Error generating maven-javadoc-plugin:3.0.1:test-aggregate
> report:
> [ERROR] Exit code: 1 -
> /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
> error: package org.elasticsearch.node.internal does not exist
> [ERROR] import org.elasticsearch.node.internal.InternalSettingsPreparer;
> [ERROR]                                       ^
> [ERROR]
>
> Let’s work on fixing the main problem first, then we’ll consider your PR.
>
> We had issues before (there is a JIRA case, don’t recall the number) where
> ES2 and ES5 could not share the same class path. I think this is another
> manifestation of that — maybe javadoc tries to generate for all modules
> simultaneously, and therefore when ES5’s dependencies have been loaded into
> javadoc, ES2’s dependencies cannot be seen.
>
> The solution may be to disable javadoc for the ES2 adapter. Or perhaps
> just for that one java file.
>
> Julian
>
>
> > On Jun 25, 2018, at 8:10 AM, Volodymyr Vysotskyi <volodymyr@xxxxxxxxxx>
> wrote:
> >
> > Thanks for the changes, I have checked and it passed on my env.
> >
> > I think it should be also checked with other JDK versions besides 8.
> >
> > Kind regards,
> > Volodymyr Vysotskyi
> >
> >
> > пн, 25 черв. 2018 о 07:05 Andrei Sereda <andrei@xxxxxxxxx> пише:
> >
> >> Can you please check the branch in my fork ?
> >>
> >> fork: https://github.com/asereda-gs/calcite
> >> branch:  javadoc-fixes
> >>
> >> I've tried to remove most of the warnings and javadoc, javadoc-test and
> >> site all work for me.
> >>
> >> ~/I/c/elasticsearch5 ❯❯❯ mvn --version
> >>                               javadoc-fixes ✭
> >> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
> >> 2018-06-17T14:33:14-04:00)
> >> Maven home: /usr/local/Cellar/maven/3.5.4/libexec
> >> Java version: 1.8.0_151, vendor: Oracle Corporation, runtime:
> >> /Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home/jre
> >> Default locale: en_US, platform encoding: UTF-8
> >> OS name: "mac os x", version: "10.13.2", arch: "x86_64", family: "mac"
> >>
> >>
> >>
> >> On Sun, Jun 24, 2018 at 12:40 PM Volodymyr Vysotskyi <
> volodymyr@xxxxxxxxxx
> >>>
> >> wrote:
> >>
> >>> I have also checked "mvn javadoc:javadoc javadoc:test-javadoc" on my
> >>> machine (Ubuntu, JDK 8, Maven 3.3.9) and it fails on elasticsearch2
> >> module
> >>> (didn't check this problem for the next modules).
> >>>
> >>> "mvn clean -DskipTests site" fails for MongoDB module.
> >>>
> >>> Kind regards,
> >>> Volodymyr Vysotskyi
> >>>
> >>>
> >>> сб, 23 черв. 2018 о 03:12 Julian Hyde <jhyde@xxxxxxxxxx> пише:
> >>>
> >>>> Same failure for me on JDK 8/macOS. The most likely other variable is
> >>>> maven: I have 3.5.2 on macOS, 3.5.3 on ubuntu.
> >>>>
> >>>> I’m mystified why no one else is seeing javadoc failures on macOS.
> >>>>
> >>>> Julian
> >>>>
> >>>>
> >>>>> On Jun 22, 2018, at 4:35 PM, Julian Hyde <jhyde@xxxxxxxxxx> wrote:
> >>>>>
> >>>>> On JDK 8/ubuntu, “mvn clean -DskipTests site” gives the following
> >>>> failure:
> >>>>>
> >>>>> [ERROR] Failed to execute goal
> >>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on
> >>>> project calcite-mongodb: Error generating
> >>>> maven-javadoc-plugin:3.0.1:test-javadoc report:
> >>>>> [ERROR] Exit code: 1 -
> >>>>
> >>>
> >>
> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:22:
> >>>> error: cannot find symbol
> >>>>> [ERROR] import org.apache.calcite.test.CalciteAssert;
> >>>>> [ERROR]                               ^
> >>>>> [ERROR]   symbol:   class CalciteAssert
> >>>>> [ERROR]   location: package org.apache.calcite.test
> >>>>> [ERROR]
> >>>>
> >>>
> >>
> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:137:
> >>>> error: package CalciteAssert does not exist
> >>>>> [ERROR]   private CalciteAssert.AssertThat assertModel(String model)
> >> {
> >>>>> [ERROR]                        ^
> >>>>> [ERROR]
> >>>>
> >>>
> >>
> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:145:
> >>>> error: package CalciteAssert does not exist
> >>>>> [ERROR]   private CalciteAssert.AssertThat assertModel(URL url) {
> >>>>> [ERROR]                        ^
> >>>>> [ERROR]
> >>>>
> >>>
> >>
> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/test/MongoAssertions.java:43:
> >>>> error: reference not found
> >>>>> [ERROR]   /** Similar to {@link CalciteAssert#checkResultUnordered},
> >>> but
> >>>> filters strings
> >>>>> [ERROR]                         ^
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Jun 22, 2018, at 3:46 PM, Andrei Sereda <andrei@xxxxxxxxx>
> >> wrote:
> >>>>>>
> >>>>>> Hmm. Both master and c12cb4b0de work for me. Perhaps the issue is
> >> JDK
> >>> 8
> >>>> vs
> >>>>>> 9 ?
> >>>>>>
> >>>>>> # JDK8 (oracle) /  macOS
> >>>>>> $ mvn clean -DskipTests site
> >>>>>>
> >>>>>> On Fri, Jun 22, 2018 at 5:56 PM Julian Hyde <jhyde@xxxxxxxxxx>
> >> wrote:
> >>>>>>
> >>>>>>> I just tried it again. The following fails for me (jdk9, ubuntu):
> >>>>>>>
> >>>>>>> $ git checkout c12cb4b0de
> >>>>>>> $ mvn clean -DskipTests site
> >>>>>>>
> >>>>>>> Constructing Javadoc information...
> >>>>>>> 1 error
> >>>>>>> [INFO]
> >>>>>>>
> >>>>
> >> ------------------------------------------------------------------------
> >>>>>>> [INFO] Reactor Summary:
> >>>>>>> [INFO]
> >>>>>>> [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE
> >>>> [02:29
> >>>>>>> min]
> >>>>>>> [INFO] Calcite Linq4j ..................................... SKIPPED
> >>>>>>> [INFO] Calcite Core ....................................... SKIPPED
> >>>>>>> [INFO] Calcite Cassandra .................................. SKIPPED
> >>>>>>> [INFO] Calcite Druid ...................................... SKIPPED
> >>>>>>> [INFO] Calcite Elasticsearch .............................. SKIPPED
> >>>>>>> [INFO] Calcite Elasticsearch5 ............................. SKIPPED
> >>>>>>> [INFO] Calcite Examples ................................... SKIPPED
> >>>>>>> [INFO] Calcite Example CSV ................................ SKIPPED
> >>>>>>> [INFO] Calcite Example Function ........................... SKIPPED
> >>>>>>> [INFO] Calcite File ....................................... SKIPPED
> >>>>>>> [INFO] Calcite Geode ...................................... SKIPPED
> >>>>>>> [INFO] Calcite MongoDB .................................... SKIPPED
> >>>>>>> [INFO] Calcite Pig ........................................ SKIPPED
> >>>>>>> [INFO] Calcite Piglet ..................................... SKIPPED
> >>>>>>> [INFO] Calcite Plus ....................................... SKIPPED
> >>>>>>> [INFO] Calcite Server ..................................... SKIPPED
> >>>>>>> [INFO] Calcite Spark ...................................... SKIPPED
> >>>>>>> [INFO] Calcite Splunk ..................................... SKIPPED
> >>>>>>> [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED
> >>>>>>> [INFO]
> >>>>>>>
> >>>>
> >> ------------------------------------------------------------------------
> >>>>>>> [INFO] BUILD FAILURE
> >>>>>>> [INFO]
> >>>>>>>
> >>>>
> >> ------------------------------------------------------------------------
> >>>>>>> [INFO] Total time: 02:29 min
> >>>>>>> [INFO] Finished at: 2018-06-22T14:53:13-07:00
> >>>>>>> [INFO]
> >>>>>>>
> >>>>
> >> ------------------------------------------------------------------------
> >>>>>>> [ERROR] Failed to execute goal
> >>>>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site)
> >> on
> >>>>>>> project calcite: Error generating
> >>>> maven-javadoc-plugin:3.0.1:test-aggregate
> >>>>>>> report:
> >>>>>>> [ERROR] Exit code: 1 -
> >>>>>>>
> >>>>
> >>>
> >>
> /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
> >>>>>>> error: package org.elasticsearch.node.internal does not exist
> >>>>>>> [ERROR] import
> >>>> org.elasticsearch.node.internal.InternalSettingsPreparer;
> >>>>>>> [ERROR]                                       ^
> >>>>>>> [ERROR]
> >>>>>>> [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options
> >>>> @packages
> >>>>>>> @argfile
> >>>>>>> [ERROR]
> >>>>>>> [ERROR] Refer to the generated Javadoc files in
> >>>>>>> '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir.
> >>>>>>> [ERROR] -> [Help 1]
> >>>>>>> [ERROR]
> >>>>>>> [ERROR] To see the full stack trace of the errors, re-run Maven
> >> with
> >>>> the
> >>>>>>> -e switch.
> >>>>>>> [ERROR] Re-run Maven using the -X switch to enable full debug
> >>> logging.
> >>>>>>> [ERROR]
> >>>>>>> [ERROR] For more information about the errors and possible
> >> solutions,
> >>>>>>> please read the following articles:
> >>>>>>> [ERROR] [Help 1]
> >>>>>>>
> >>>>
> >> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> >>>>>>>
> >>>>>>>
> >>>>>>>> On Jun 22, 2018, at 2:41 PM, Michael Mior <mmior@xxxxxxxxxx>
> >> wrote:
> >>>>>>>>
> >>>>>>>> Odd, mvn clean site still works fine for me.
> >>>>>>>> --
> >>>>>>>> Michael Mior
> >>>>>>>> mmior@xxxxxxxxxx
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Le ven. 22 juin 2018 à 17:12, Julian Hyde <jhyde@xxxxxxxxxx> a
> >>> écrit
> >>>> :
> >>>>>>>>
> >>>>>>>>> Looks like this change broke “mvn site” (perhaps also “mvn
> >>>>>>>>> javadoc:test-javadoc”).
> >>>>>>>>>
> >>>>>>>>> [ERROR] Failed to execute goal
> >>>>>>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site
> >> (default-site)
> >>> on
> >>>>>>>>> project calcite: Error generating
> >>>>>>> maven-javadoc-plugin:3.0.1:test-aggregate
> >>>>>>>>> report:
> >>>>>>>>> [ERROR] Exit code: 1 -
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
> >>>>>>>>> error: package org.elasticsearch.node.internal does not exist
> >>>>>>>>> [ERROR] import
> >>>> org.elasticsearch.node.internal.InternalSettingsPreparer;
> >>>>>>>>> [ERROR]                                       ^
> >>>>>>>>> [ERROR]
> >>>>>>>>>
> >>>>>>>>>> On Jun 21, 2018, at 3:39 AM, mmior@xxxxxxxxxx wrote:
> >>>>>>>>>>
> >>>>>>>>>> [CALCITE-2347] running ElasticSearch in embedded mode for unit
> >>>> tests of
> >>>>>>>>> ES adapter (Andrei Sereda)
> >>>>>>>>>>
> >>>>>>>>>> After discussion on dev-list Integration tests (for ES) have
> >> been
> >>>>>>>>> removed. They're now
> >>>>>>>>>> superseded by unit tests (which execute queries against a real
> >>>> elastic
> >>>>>>>>> instance)
> >>>>>>>>>>
> >>>>>>>>>> Added local file (zips-mini.json) which contains a small subset
> >> of
> >>>>>>>>> original zips.json
> >>>>>>>>>> (allows to bootstrap tests faster)
> >>>>>>>>>>
> >>>>>>>>>> Created separate ES JUnit rule which can be re-used across
> >>> different
> >>>>>>>>> tests.
> >>>>>>>>>>
> >>>>>>>>>> Both v2 and v5 of ES adapters are supported.
> >>>>>>>>>>
> >>>>>>>>>> Close apache/calcite#716
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo
> >>>>>>>>>> Commit:
> >>>> http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0
> >>>>>>>>>> Tree:
> >>> http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0
> >>>>>>>>>> Diff:
> >>> http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0
> >>>>>>>>>>
> >>>>>>>>>> Branch: refs/heads/master
> >>>>>>>>>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d
> >>>>>>>>>> Parents: 37944bb
> >>>>>>>>>> Author: Andrei Sereda <andrei@xxxxxxxxxx>
> >>>>>>>>>> Authored: Thu May 31 18:19:10 2018 -0400
> >>>>>>>>>> Committer: Michael Mior <mmior@xxxxxxxxxxxx>
> >>>>>>>>>> Committed: Thu Jun 21 06:38:50 2018 -0400
> >>>>>>>>>>
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>> .../AbstractElasticsearchTable.java             |  12 +
> >>>>>>>>>> .../elasticsearch/ElasticsearchProject.java     |  61 ++-
> >>>>>>>>>> elasticsearch2/pom.xml                          |   6 +
> >>>>>>>>>> .../Elasticsearch2Enumerator.java               |  12 +-
> >>>>>>>>>> .../elasticsearch2/Elasticsearch2Schema.java    |  16 +-
> >>>>>>>>>> .../elasticsearch2/Elasticsearch2Table.java     |   9 +-
> >>>>>>>>>> .../ElasticSearch2AdapterTest.java              | 395
> >>>>>>> ++++++++++++++++++
> >>>>>>>>>> .../elasticsearch2/EmbeddedElasticNode.java     | 147 +++++++
> >>>>>>>>>> .../elasticsearch2/EmbeddedElasticRule.java     |  97 +++++
> >>>>>>>>>> .../org/apache/calcite/test/ElasticChecker.java |  49 +++
> >>>>>>>>>> .../calcite/test/Elasticsearch2AdapterIT.java   | 270
> >>> -------------
> >>>>>>>>>> .../resources/elasticsearch-zips-model.json     |  50 ---
> >>>>>>>>>> .../src/test/resources/zips-mini.json           | 149 +++++++
> >>>>>>>>>> elasticsearch5/pom.xml                          |  31 ++
> >>>>>>>>>> .../elasticsearch5/Elasticsearch5Schema.java    |  17 +-
> >>>>>>>>>> .../elasticsearch5/Elasticsearch5Table.java     |  11 +-
> >>>>>>>>>> .../ElasticSearch5AdapterTest.java              | 399
> >>>>>>> +++++++++++++++++++
> >>>>>>>>>> .../elasticsearch5/EmbeddedElasticNode.java     | 153 +++++++
> >>>>>>>>>> .../elasticsearch5/EmbeddedElasticRule.java     |  98 +++++
> >>>>>>>>>> .../org/apache/calcite/test/ElasticChecker.java |  49 +++
> >>>>>>>>>> .../calcite/test/Elasticsearch5AdapterIT.java   | 270
> >>> -------------
> >>>>>>>>>> .../resources/elasticsearch-zips-model.json     |  50 ---
> >>>>>>>>>> elasticsearch5/src/test/resources/log4j2.xml    |  16 +
> >>>>>>>>>> .../src/test/resources/zips-mini.json           | 149 +++++++
> >>>>>>>>>> pom.xml                                         |  20 +-
> >>>>>>>>>> 25 files changed, 1866 insertions(+), 670 deletions(-)
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>> diff --git
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>>>>>> index 0980469..8cc5933 100644
> >>>>>>>>>> ---
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>>>>>> +++
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>>>>>> @@ -75,6 +75,18 @@ public abstract class
> >>> AbstractElasticsearchTable
> >>>>>>>>> extends AbstractQueryableTable
> >>>>>>>>>>     relOptTable, this, null);
> >>>>>>>>>> }
> >>>>>>>>>>
> >>>>>>>>>> +  /**
> >>>>>>>>>> +   * In ES 5.x scripted fields start with {@code
> >>>> params._source.foo}
> >>>>>>>>> while in ES2.x
> >>>>>>>>>> +   * {@code _source.foo}. Helper method to build correct query
> >>>> based
> >>>>>>> on
> >>>>>>>>> runtime version of elastic.
> >>>>>>>>>> +   *
> >>>>>>>>>> +   * @see <a href="
> >>>>>>>>> https://github.com/elastic/elasticsearch/issues/20068";>_source
> >>>>>>>>> variable</a>
> >>>>>>>>>> +   * @see <a href="
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html
> >>>>>>> ">Scripted
> >>>>>>>>> Fields</a>
> >>>>>>>>>> +   */
> >>>>>>>>>> +  protected String scriptedFieldPrefix() {
> >>>>>>>>>> +    // this is default pattern starting 5.x
> >>>>>>>>>> +    return "params._source";
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> /** Executes a "find" operation on the underlying type.
> >>>>>>>>>> *
> >>>>>>>>>> * <p>For example,
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>> diff --git
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>>>>>> index b42abd7..961c8b0 100644
> >>>>>>>>>> ---
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>>>>>> +++
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>>>>>> @@ -27,11 +27,15 @@ import
> >>>>>>>>> org.apache.calcite.rel.metadata.RelMetadataQuery;
> >>>>>>>>>> import org.apache.calcite.rel.type.RelDataType;
> >>>>>>>>>> import org.apache.calcite.rex.RexNode;
> >>>>>>>>>> import org.apache.calcite.util.Pair;
> >>>>>>>>>> -import org.apache.calcite.util.Util;
> >>>>>>>>>> +
> >>>>>>>>>> +import com.google.common.base.Function;
> >>>>>>>>>> +import com.google.common.collect.Lists;
> >>>>>>>>>>
> >>>>>>>>>> import java.util.ArrayList;
> >>>>>>>>>> import java.util.List;
> >>>>>>>>>>
> >>>>>>>>>> +import javax.annotation.Nullable;
> >>>>>>>>>> +
> >>>>>>>>>> /**
> >>>>>>>>>> * Implementation of {@link org.apache.calcite.rel.core.Project}
> >>>>>>>>>> * relational expression in Elasticsearch.
> >>>>>>>>>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends
> >>>> Project
> >>>>>>>>> implements ElasticsearchRel {
> >>>>>>>>>> implementor.visitChild(0, getInput());
> >>>>>>>>>>
> >>>>>>>>>> final List<String> inFields =
> >>>>>>>>>> -
> >>>>>>>>>
> >>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType());
> >>>>>>>>>> +
> >>>>>>>>>
> >>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType());
> >>>>>>>>>> final ElasticsearchRules.RexToElasticsearchTranslator
> >> translator
> >>> =
> >>>>>>>>>> -        new ElasticsearchRules.RexToElasticsearchTranslator(
> >>>>>>>>>> -            (JavaTypeFactory) getCluster().getTypeFactory(),
> >>>>>>> inFields);
> >>>>>>>>>> +            new
> >> ElasticsearchRules.RexToElasticsearchTranslator(
> >>>>>>>>>> +                    (JavaTypeFactory)
> >>>> getCluster().getTypeFactory(),
> >>>>>>>>> inFields);
> >>>>>>>>>>
> >>>>>>>>>> -    final List<String> findItems = new ArrayList<>();
> >>>>>>>>>> -    final List<String> scriptFieldItems = new ArrayList<>();
> >>>>>>>>>> +    final List<String> fields = new ArrayList<>();
> >>>>>>>>>> +    final List<String> scriptFields = new ArrayList<>();
> >>>>>>>>>> for (Pair<RexNode, String> pair: getNamedProjects()) {
> >>>>>>>>>>   final String name = pair.right;
> >>>>>>>>>>   final String expr = pair.left.accept(translator);
> >>>>>>>>>>
> >>>>>>>>>>   if (expr.equals("\"" + name + "\"")) {
> >>>>>>>>>> -        findItems.add(ElasticsearchRules.quote(name));
> >>>>>>>>>> +        fields.add(name);
> >>>>>>>>>>   } else if (expr.matches("\"literal\":.+")) {
> >>>>>>>>>> -        scriptFieldItems.add(ElasticsearchRules.quote(name)
> >>>>>>>>>> -            + ":{\"script\": "
> >>>>>>>>>> -            + expr.split(":")[1] + "}");
> >>>>>>>>>> +        scriptFields.add(ElasticsearchRules.quote(name)
> >>>>>>>>>> +                + ":{\"script\": "
> >>>>>>>>>> +                + expr.split(":")[1] + "}");
> >>>>>>>>>>   } else {
> >>>>>>>>>> -        scriptFieldItems.add(ElasticsearchRules.quote(name)
> >>>>>>>>>> -            + ":{\"script\":\"params._source."
> >>>>>>>>>> -            + expr.replaceAll("\"", "") + "\"}");
> >>>>>>>>>> +        scriptFields.add(ElasticsearchRules.quote(name)
> >>>>>>>>>> +                + ":{\"script\":"
> >>>>>>>>>> +                // _source (ES2) vs params._source (ES5)
> >>>>>>>>>> +                + "\"" +
> >>>>>>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "."
> >>>>>>>>>> +                + expr.replaceAll("\"", "") + "\"}");
> >>>>>>>>>> +      }
> >>>>>>>>>> +    }
> >>>>>>>>>> +
> >>>>>>>>>> +    StringBuilder query = new StringBuilder();
> >>>>>>>>>> +    if (scriptFields.isEmpty()) {
> >>>>>>>>>> +      List<String> newList = Lists.transform(fields, new
> >>>>>>>>> Function<String, String>() {
> >>>>>>>>>> +        @Nullable
> >>>>>>>>>> +        @Override public String apply(@Nullable String input) {
> >>>>>>>>>> +          return ElasticsearchRules.quote(input);
> >>>>>>>>>> +        }
> >>>>>>>>>> +      });
> >>>>>>>>>> +
> >>>>>>>>>> +      final String findString = String.join(", ", newList);
> >>>>>>>>>> +      query.append("\"_source\" :
> >>>> [").append(findString).append("]");
> >>>>>>>>>> +    } else {
> >>>>>>>>>> +      // if scripted fields are present, ES ignores _source
> >>>> attribute
> >>>>>>>>>> +      for (String field: fields) {
> >>>>>>>>>> +        scriptFields.add(ElasticsearchRules.quote(field) +
> >>>>>>>>> ":{\"script\": "
> >>>>>>>>>> +                // _source (ES2) vs params._source (ES5)
> >>>>>>>>>> +                + "\"" +
> >>>>>>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "."
> >>>>>>>>>> +                + field + "\"}");
> >>>>>>>>>>   }
> >>>>>>>>>> +      query.append("\"script_fields\": {" + String.join(", ",
> >>>>>>>>> scriptFields) + "}");
> >>>>>>>>>> }
> >>>>>>>>>> -    final String findString = Util.toString(findItems, "", ",
> >> ",
> >>>> "");
> >>>>>>>>>> -    final String scriptFieldString = "\"script_fields\": {"
> >>>>>>>>>> -        + Util.toString(scriptFieldItems, "", ", ", "") + "}";
> >>>>>>>>>> -    final String fieldString = "\"_source\" : [" + findString +
> >>> "]"
> >>>>>>>>>> -        + ", " + scriptFieldString;
> >>>>>>>>>>
> >>>>>>>>>> for (String opfield : implementor.list) {
> >>>>>>>>>>   if (opfield.startsWith("\"_source\"")) {
> >>>>>>>>>>     implementor.list.remove(opfield);
> >>>>>>>>>>   }
> >>>>>>>>>> }
> >>>>>>>>>> -    implementor.add(fieldString);
> >>>>>>>>>> +    implementor.add(query.toString());
> >>>>>>>>>> }
> >>>>>>>>>> }
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml
> >>>>>>>>>> index f24622c..6fbee03 100644
> >>>>>>>>>> --- a/elasticsearch2/pom.xml
> >>>>>>>>>> +++ b/elasticsearch2/pom.xml
> >>>>>>>>>> @@ -73,6 +73,12 @@ limitations under the License.
> >>>>>>>>>>   <version>${elasticsearch-java-driver.version}</version>
> >>>>>>>>>> </dependency>
> >>>>>>>>>> <dependency>
> >>>>>>>>>> +      <!-- Lang groovy dependency is needed for testing with
> >>>> embedded
> >>>>>>>>> ES (scripted fields like loc[0]) -->
> >>>>>>>>>> +      <groupId>org.elasticsearch.module</groupId>
> >>>>>>>>>> +      <artifactId>lang-groovy</artifactId>
> >>>>>>>>>> +      <scope>test</scope>
> >>>>>>>>>> +    </dependency>
> >>>>>>>>>> +    <dependency>
> >>>>>>>>>>   <groupId>com.carrotsearch</groupId>
> >>>>>>>>>>   <artifactId>hppc</artifactId>
> >>>>>>>>>>   <version>${hppc.version}</version>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>> diff --git
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>>>>>> index 84370ab..c3d2ac0 100644
> >>>>>>>>>> ---
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>>>>>> +++
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>>>>>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit;
> >>>>>>>>>> import java.util.Date;
> >>>>>>>>>> import java.util.Iterator;
> >>>>>>>>>> import java.util.List;
> >>>>>>>>>> +import java.util.Locale;
> >>>>>>>>>> import java.util.Map;
> >>>>>>>>>>
> >>>>>>>>>> /**
> >>>>>>>>>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator
> >>>> implements
> >>>>>>>>> Enumerator<Object> {
> >>>>>>>>>> private static Function1<SearchHit, Object[]> listGetter(
> >>>>>>>>>>   final List<Map.Entry<String, Class>> fields) {
> >>>>>>>>>> return new Function1<SearchHit, Object[]>() {
> >>>>>>>>>> -      public Object[] apply(SearchHit searchHitFields) {
> >>>>>>>>>> +      public Object[] apply(SearchHit hit) {
> >>>>>>>>>>     Object[] objects = new Object[fields.size()];
> >>>>>>>>>>     for (int i = 0; i < fields.size(); i++) {
> >>>>>>>>>>       final Map.Entry<String, Class> field = fields.get(i);
> >>>>>>>>>>       final String name = field.getKey();
> >>>>>>>>>> -          if (searchHitFields.fields().isEmpty()) {
> >>>>>>>>>> -            objects[i] =
> >>>>>>> convert(searchHitFields.getSource().get(name),
> >>>>>>>>> field.getValue());
> >>>>>>>>>> +          if (hit.fields().isEmpty()) {
> >>>>>>>>>> +            objects[i] = convert(hit.getSource().get(name),
> >>>>>>>>> field.getValue());
> >>>>>>>>>> +          } else if (hit.fields().containsKey(name)) {
> >>>>>>>>>> +            objects[i] = convert(hit.field(name).getValue(),
> >>>>>>>>> field.getValue());
> >>>>>>>>>>       } else {
> >>>>>>>>>> -            objects[i] =
> >>>>>>>>> convert(searchHitFields.field(name).getValue(),
> >> field.getValue());
> >>>>>>>>>> +            throw new IllegalStateException(
> >>>>>>>>>> +                    String.format(Locale.getDefault(), "No
> >> result
> >>>> for
> >>>>>>>>> %s", field));
> >>>>>>>>>>       }
> >>>>>>>>>>     }
> >>>>>>>>>>     return objects;
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>> diff --git
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>>>>>> index 668402b..46e3fc5 100644
> >>>>>>>>>> ---
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>>>>>> +++
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>>>>>> @@ -22,6 +22,9 @@ import
> >>>> org.apache.calcite.schema.impl.AbstractSchema;
> >>>>>>>>>>
> >>>>>>>>>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
> >>>>>>>>>>
> >>>>>>>>>> +import com.google.common.annotations.VisibleForTesting;
> >>>>>>>>>> +import com.google.common.base.Preconditions;
> >>>>>>>>>> +
> >>>>>>>>>> import com.google.common.collect.ImmutableList;
> >>>>>>>>>> import com.google.common.collect.ImmutableMap;
> >>>>>>>>>>
> >>>>>>>>>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends
> >>>>>>>>> AbstractSchema
> >>>>>>>>>> }
> >>>>>>>>>> }
> >>>>>>>>>>
> >>>>>>>>>> +  /**
> >>>>>>>>>> +   * Allows schema to be instantiated from existing elastic
> >>> search
> >>>>>>>>> client.
> >>>>>>>>>> +   * This constructor is used in tests.
> >>>>>>>>>> +   */
> >>>>>>>>>> +  @VisibleForTesting
> >>>>>>>>>> +  Elasticsearch2Schema(Client client, String index) {
> >>>>>>>>>> +    this.client = Preconditions.checkNotNull(client, "client");
> >>>>>>>>>> +    this.index = Preconditions.checkNotNull(index, "index");
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> @Override protected Map<String, Table> getTableMap() {
> >>>>>>>>>> final ImmutableMap.Builder<String, Table> builder =
> >>>>>>>>> ImmutableMap.builder();
> >>>>>>>>>>
> >>>>>>>>>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends
> >>>>>>>>> AbstractSchema
> >>>>>>>>>>
> >>>>>>>>>> final List<DiscoveryNode> nodes =
> >>>>>>>>> ImmutableList.copyOf(transportClient.connectedNodes());
> >>>>>>>>>> if (nodes.isEmpty()) {
> >>>>>>>>>> -      throw new RuntimeException("Cannot connect to any
> >>>> elasticsearch
> >>>>>>>>> nodes");
> >>>>>>>>>> +      throw new IllegalStateException("Cannot connect to any
> >>>>>>>>> elasticsearch node: "
> >>>>>>>>>> +              + transportNodes);
> >>>>>>>>>> }
> >>>>>>>>>>
> >>>>>>>>>> client = transportClient;
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>> diff --git
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>>>>>> index 636aa5f..2928835 100644
> >>>>>>>>>> ---
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>>>>>> +++
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>>>>>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends
> >>>>>>>>> AbstractElasticsearchTable {
> >>>>>>>>>> this.client = client;
> >>>>>>>>>> }
> >>>>>>>>>>
> >>>>>>>>>> +  /**
> >>>>>>>>>> +   * ES version 2.x. To access document attributes ES2 uses
> >>> {@code
> >>>>>>>>> _source.foo} syntax.
> >>>>>>>>>> +   */
> >>>>>>>>>> +  @Override protected String scriptedFieldPrefix() {
> >>>>>>>>>> +    return "_source";
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> @Override protected Enumerable<Object> find(String index,
> >>>> List<String>
> >>>>>>>>> ops,
> >>>>>>>>>> -      List<Map.Entry<String, Class>> fields) {
> >>>>>>>>>> +
> >>>> List<Map.Entry<String,
> >>>>>>>>> Class>> fields) {
> >>>>>>>>>> final String dbName = index;
> >>>>>>>>>>
> >>>>>>>>>> final String queryString = "{" + Util.toString(ops, "", ", ",
> >>> "") +
> >>>>>>>>> "}";
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
> >>>>>>>>>>
> >>>> ----------------------------------------------------------------------
> >>>>>>>>>> diff --git
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
> >>>>>>>>>> new file mode 100644
> >>>>>>>>>> index 0000000..287e094
> >>>>>>>>>> --- /dev/null
> >>>>>>>>>> +++
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
> >>>>>>>>>> @@ -0,0 +1,395 @@
> >>>>>>>>>> +/*
> >>>>>>>>>> + * Licensed to the Apache Software Foundation (ASF) under one
> >> or
> >>>> more
> >>>>>>>>>> + * contributor license agreements.  See the NOTICE file
> >>> distributed
> >>>>>>> with
> >>>>>>>>>> + * this work for additional information regarding copyright
> >>>> ownership.
> >>>>>>>>>> + * The ASF licenses this file to you under the Apache License,
> >>>> Version
> >>>>>>>>> 2.0
> >>>>>>>>>> + * (the "License"); you may not use this file except in
> >>> compliance
> >>>>>>> with
> >>>>>>>>>> + * the License.  You may obtain a copy of the License at
> >>>>>>>>>> + *
> >>>>>>>>>> + * http://www.apache.org/licenses/LICENSE-2.0
> >>>>>>>>>> + *
> >>>>>>>>>> + * Unless required by applicable law or agreed to in writing,
> >>>> software
> >>>>>>>>>> + * distributed under the License is distributed on an "AS IS"
> >>>> BASIS,
> >>>>>>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
> >>> or
> >>>>>>>>> implied.
> >>>>>>>>>> + * See the License for the specific language governing
> >>> permissions
> >>>> and
> >>>>>>>>>> + * limitations under the License.
> >>>>>>>>>> + */
> >>>>>>>>>> +package org.apache.calcite.adapter.elasticsearch2;
> >>>>>>>>>> +
> >>>>>>>>>> +import org.apache.calcite.jdbc.CalciteConnection;
> >>>>>>>>>> +import org.apache.calcite.schema.SchemaPlus;
> >>>>>>>>>> +import org.apache.calcite.schema.impl.ViewTable;
> >>>>>>>>>> +import org.apache.calcite.schema.impl.ViewTableMacro;
> >>>>>>>>>> +import org.apache.calcite.test.CalciteAssert;
> >>>>>>>>>> +import org.apache.calcite.test.ElasticChecker;
> >>>>>>>>>> +
> >>>>>>>>>> +import com.google.common.io.LineProcessor;
> >>>>>>>>>> +import com.google.common.io.Resources;
> >>>>>>>>>> +
> >>>>>>>>>> +import org.elasticsearch.action.bulk.BulkItemResponse;
> >>>>>>>>>> +import org.elasticsearch.action.bulk.BulkRequestBuilder;
> >>>>>>>>>> +import org.elasticsearch.action.bulk.BulkResponse;
> >>>>>>>>>> +import org.elasticsearch.common.xcontent.XContentBuilder;
> >>>>>>>>>> +import org.elasticsearch.common.xcontent.XContentFactory;
> >>>>>>>>>> +
> >>>>>>>>>> +import org.junit.BeforeClass;
> >>>>>>>>>> +import org.junit.ClassRule;
> >>>>>>>>>> +import org.junit.Test;
> >>>>>>>>>> +
> >>>>>>>>>> +import java.io.IOException;
> >>>>>>>>>> +import java.nio.charset.StandardCharsets;
> >>>>>>>>>> +import java.sql.Connection;
> >>>>>>>>>> +import java.sql.DriverManager;
> >>>>>>>>>> +import java.sql.SQLException;
> >>>>>>>>>> +import java.util.Arrays;
> >>>>>>>>>> +import java.util.Collections;
> >>>>>>>>>> +import java.util.Locale;
> >>>>>>>>>> +
> >>>>>>>>>> +/**
> >>>>>>>>>> + * Set of tests for ES adapter. Uses real instance via {@link
> >>>>>>>>> EmbeddedElasticRule}. Document
> >>>>>>>>>> + * source is local {@code zips-mini.json} file (located in the
> >>>>>>>>> classpath).
> >>>>>>>>>> + */
> >>>>>>>>>> +public class ElasticSearch2AdapterTest {
> >>>>>>>>>> +
> >>>>>>>>>> +  @ClassRule //init once for all tests
> >>>>>>>>>> +  public static final EmbeddedElasticRule NODE =
> >>>>>>>>> EmbeddedElasticRule.create();
> >>>>>>>>>> +
> >>>>>>>>>> +  private static final String ZIPS = "zips";
> >>>>>>>>>> +
> >>>>>>>>>> +  /**
> >>>>>>>>>> +   * Used to create {@code zips} index and insert some data
> >>>>>>>>>> +   */
> >>>>>>>>>> +  @BeforeClass
> >>>>>>>>>> +  public static void setupInstance() throws Exception {
> >>>>>>>>>> +    // define mapping so fields are searchable (term query)
> >>>>>>>>>> +    XContentBuilder mapping =
> >>>>>>>>> XContentFactory.jsonBuilder().startObject()
> >>>>>>>>>> +            .startObject("properties")
> >>>>>>>>>> +            .startObject("city").field("type", "string")
> >>>>>>>>>> +                  .field("index", "not_analyzed").endObject()
> >>>>>>>>>> +            .startObject("state").field("type", "string")
> >>>>>>>>>> +                  .field("index", "not_analyzed").endObject()
> >>>>>>>>>> +            .startObject("pop").field("type",
> >> "long").endObject()
> >>>>>>>>>> +            .endObject()
> >>>>>>>>>> +            .endObject();
> >>>>>>>>>> +
> >>>>>>>>>> +    // create index
> >>>>>>>>>> +    NODE.client().admin().indices()
> >>>>>>>>>> +            .prepareCreate(ZIPS)
> >>>>>>>>>> +            .addMapping(ZIPS, mapping)
> >>>>>>>>>> +            .get();
> >>>>>>>>>> +
> >>>>>>>>>> +    BulkRequestBuilder bulk =
> >>>>>>>>> NODE.client().prepareBulk().setRefresh(true);
> >>>>>>>>>> +
> >>>>>>>>>> +    // load records from file
> >>>>>>>>>> +
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"),
> >>>>>>>>>> +            StandardCharsets.UTF_8, new LineProcessor<Void>() {
> >>>>>>>>>> +              @Override public boolean processLine(String line)
> >>>> throws
> >>>>>>>>> IOException {
> >>>>>>>>>> +                line = line.replaceAll("_id", "id"); // _id is
> >> a
> >>>>>>>>> reserved attribute in ES
> >>>>>>>>>> +                bulk.add(NODE.client().prepareIndex(ZIPS,
> >>>>>>>>> ZIPS).setSource(line));
> >>>>>>>>>> +                return true;
> >>>>>>>>>> +              }
> >>>>>>>>>> +
> >>>>>>>>>> +              @Override public Void getResult() {
> >>>>>>>>>> +                return null;
> >>>>>>>>>> +              }
> >>>>>>>>>> +            });
> >>>>>>>>>> +
> >>>>>>>>>> +    if (bulk.numberOfActions() == 0) {
> >>>>>>>>>> +      throw new IllegalStateException("No records to be
> >>> indexed");
> >>>>>>>>>> +    }
> >>>>>>>>>> +
> >>>>>>>>>> +    BulkResponse response = bulk.execute().get();
> >>>>>>>>>> +
> >>>>>>>>>> +    if (response.hasFailures()) {
> >>>>>>>>>> +      throw new IllegalStateException(
> >>>>>>>>>> +              String.format(Locale.getDefault(), "Failed to
> >>>> populate
> >>>>>>>>> %s:\n%s", NODE.httpAddress(),
> >>>>>>>>>> +
> >>>>>>>>>
> >>> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed)
> >>>>>>>>>> +
> >>>>>>>>>
> >>>>>>>
> >>>>
> >>>
> >>
> .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>")));
> >>>>>>>>>> +    }
> >>>>>>>>>> +
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  private CalciteAssert.ConnectionFactory
> >> newConnectionFactory()
> >>> {
> >>>>>>>>>> +    return new CalciteAssert.ConnectionFactory() {
> >>>>>>>>>> +      @Override public Connection createConnection() throws
> >>>>>>>>> SQLException {
> >>>>>>>>>> +        final Connection connection =
> >>>>>>>>> DriverManager.getConnection("jdbc:calcite:");
> >>>>>>>>>> +        final SchemaPlus root =
> >>>>>>>>> connection.unwrap(CalciteConnection.class).getRootSchema();
> >>>>>>>>>> +
> >>>>>>>>>> +        root.add("elastic", new
> >>> Elasticsearch2Schema(NODE.client(),
> >>>>>>>>> ZIPS));
> >>>>>>>>>> +
> >>>>>>>>>> +        // add calcite view programmatically
> >>>>>>>>>> +        final String viewSql = "select cast(_MAP['city'] AS
> >>>>>>>>> varchar(20)) AS \"city\", "
> >>>>>>>>>> +               + " cast(_MAP['loc'][0] AS float) AS
> >>>> \"longitude\",\n"
> >>>>>>>>>> +               + " cast(_MAP['loc'][1] AS float) AS
> >>>> \"latitude\",\n"
> >>>>>>>>>> +              + " cast(_MAP['pop'] AS integer) AS \"pop\", "
> >>>>>>>>>> +              +  " cast(_MAP['state'] AS varchar(2)) AS
> >>> \"state\",
> >>>> "
> >>>>>>>>>> +              +  " cast(_MAP['id'] AS varchar(5)) AS \"id\" "
> >>>>>>>>>> +              +  "from \"elastic\".\"zips\"";
> >>>>>>>>>> +
> >>>>>>>>>> +        ViewTableMacro macro = ViewTable.viewMacro(root,
> >> viewSql,
> >>>>>>>>>> +                Collections.singletonList("elastic"),
> >>>>>>>>> Arrays.asList("elastic", "view"), false);
> >>>>>>>>>> +        root.add("ZIPS", macro);
> >>>>>>>>>> +
> >>>>>>>>>> +        return connection;
> >>>>>>>>>> +      }
> >>>>>>>>>> +    };
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  private CalciteAssert.AssertThat calciteAssert() {
> >>>>>>>>>> +    return CalciteAssert.that()
> >>>>>>>>>> +            .with(newConnectionFactory());
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  /**
> >>>>>>>>>> +   * Tests using calcite view
> >>>>>>>>>> +   */
> >>>>>>>>>> +  @Test
> >>>>>>>>>> +  public void view() throws Exception {
> >>>>>>>>>> +    calciteAssert()
> >>>>>>>>>> +          .query("select * from zips where \"city\" =
> >>> 'BROOKLYN'")
> >>>>>>>>>> +          .returns("city=BROOKLYN; longitude=-73.956985;
> >>>>>>>>> latitude=40.646694; "
> >>>>>>>>>> +                  + "pop=111396; state=NY; id=11226\n")
> >>>>>>>>>> +          .returnsCount(1);
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  @Test
> >>>>>>>>>> +  public void emptyResult() {
> >>>>>>>>>> +    CalciteAssert.that()
> >>>>>>>>>> +            .with(newConnectionFactory())
> >>>>>>>>>> +            .query("select * from zips limit 0")
> >>>>>>>>>> +            .returnsCount(0);
> >>>>>>>>>> +
> >>>>>>>>>> +    CalciteAssert.that()
> >>>>>>>>>> +            .with(newConnectionFactory())
> >>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where
> >>>>>>>>> _MAP['Foo'] = '_MISSING_'")
> >>>>>>>>>> +            .returnsCount(0);
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  @Test
> >>>>>>>>>> +  public void basic() throws Exception {
> >>>>>>>>>> +    CalciteAssert.that()
> >>>>>>>>>> +            .with(newConnectionFactory())
> >>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where
> >>>>>>>>> _MAP['city'] = 'BROOKLYN'")
> >>>>>>>>>> +            .returnsCount(1);
> >>>>>>>>>> +
> >>>>>>>>>> +    CalciteAssert.that()
> >>>>>>>>>> +            .with(newConnectionFactory())
> >>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where"
> >>>>>>>>>> +                    + " _MAP['city'] in ('BROOKLYN',
> >>>> 'WASHINGTON')")
> >>>>>>>>>> +            .returnsCount(2);
> >>>>>>>>>> +
> >>>>>>>>>> +    // lower-case
> >>>>>>>>>> +    CalciteAssert.that()
> >>>>>>>>>> +            .with(newConnectionFactory())
> >>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where "
> >>>>>>>>>> +                    + "_MAP['city'] in ('brooklyn', 'Brooklyn',
> >>>>>>>>> 'BROOK') ")
> >>>>>>>>>> +            .returnsCount(0);
> >>>>>>>>>> +
> >>>>>>>>>> +    // missing field
> >>>>>>>>>> +    CalciteAssert.that()
> >>>>>>>>>> +            .with(newConnectionFactory())
> >>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" where
> >>>>>>>>> _MAP['CITY'] = 'BROOKLYN'")
> >>>>>>>>>> +            .returnsCount(0);
> >>>>>>>>>> +
> >>>>>>>>>> +    // limit works
> >>>>>>>>>> +    CalciteAssert.that()
> >>>>>>>>>> +            .with(newConnectionFactory())
> >>>>>>>>>> +            .query("select * from \"elastic\".\"zips\" limit
> >> 42")
> >>>>>>>>>> +            .returnsCount(42);
> >>>>>>>>>> +
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  @Test public void testSort() {
> >>>>>>>>>> +    final String explain =
> >>>> "PLAN=ElasticsearchToEnumerableConverter\n"
> >>>>>>>>>> +            + "  ElasticsearchSort(sort0=[$4], dir0=[ASC])\n"
> >>>>>>>>>> +            + "    ElasticsearchProject(city=[CAST(ITEM($0,
> >>>>>>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>>>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0,
> >>> 'loc'),
> >>>>>>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT],
> >>>>>>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0,
> >>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>>>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0,
> >> 'id')):VARCHAR(5)
> >>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>> \"ISO-8859-1$en_US$primary\"])\n"
> >>>>>>>>>> +            + "      ElasticsearchTableScan(table=[[elastic,
> >>>> zips]])";
> >>>>>>>>>> +
> >>>>>>>>>> +    calciteAssert()
> >>>>>>>>>> +            .query("select * from zips order by \"state\"")
> >>>>>>>>>> +            .returnsCount(10)
> >>>>>>>>>> +            .explainContains(explain);
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  @Test public void testSortLimit() {
> >>>>>>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
> >>>>>>>>>> +            + "order by \"state\", \"id\" offset 2 rows fetch
> >>> next
> >>>> 3
> >>>>>>>>> rows only";
> >>>>>>>>>> +    calciteAssert()
> >>>>>>>>>> +            .query(sql)
> >>>>>>>>>> +            .returnsUnordered("state=AK; id=99801",
> >>>>>>>>>> +                    "state=AL; id=35215",
> >>>>>>>>>> +                    "state=AL; id=35401")
> >>>>>>>>>> +            .queryContains(
> >>>>>>>>>> +                    ElasticChecker.elasticsearchChecker(
> >>>>>>>>>> +                            "\"_source\" : [\"state\",
> >> \"id\"]",
> >>>>>>>>>> +                            "\"sort\": [ {\"state\": \"asc\"},
> >>>>>>> {\"id\":
> >>>>>>>>> \"asc\"}]",
> >>>>>>>>>> +                            "\"from\": 2",
> >>>>>>>>>> +                            "\"size\": 3"));
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +
> >>>>>>>>>> +
> >>>>>>>>>> +  @Test public void testOffsetLimit() {
> >>>>>>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
> >>>>>>>>>> +            + "offset 2 fetch next 3 rows only";
> >>>>>>>>>> +    calciteAssert()
> >>>>>>>>>> +            .query(sql)
> >>>>>>>>>> +            .runs()
> >>>>>>>>>> +            .queryContains(
> >>>>>>>>>> +                    ElasticChecker.elasticsearchChecker(
> >>>>>>>>>> +                            "\"from\": 2",
> >>>>>>>>>> +                            "\"size\": 3",
> >>>>>>>>>> +                            "\"_source\" : [\"state\",
> >>> \"id\"]"));
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  @Test public void testLimit() {
> >>>>>>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
> >>>>>>>>>> +            + "fetch next 3 rows only";
> >>>>>>>>>> +
> >>>>>>>>>> +    calciteAssert()
> >>>>>>>>>> +            .query(sql)
> >>>>>>>>>> +            .runs()
> >>>>>>>>>> +            .queryContains(
> >>>>>>>>>> +                    ElasticChecker.elasticsearchChecker(
> >>>>>>>>>> +                            "\"size\": 3",
> >>>>>>>>>> +                            "\"_source\" : [\"state\",
> >>> \"id\"]"));
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  @Test public void testFilterSort() {
> >>>>>>>>>> +    final String sql = "select * from zips\n"
> >>>>>>>>>> +            + "where \"state\" = 'CA' and \"id\" >= '70000'\n"
> >>>>>>>>>> +            + "order by \"state\", \"id\"";
> >>>>>>>>>> +    final String explain =
> >>>> "PLAN=ElasticsearchToEnumerableConverter\n"
> >>>>>>>>>> +            + "  ElasticsearchSort(sort0=[$4], sort1=[$5],
> >>>> dir0=[ASC],
> >>>>>>>>> dir1=[ASC])\n"
> >>>>>>>>>> +            + "    ElasticsearchProject(city=[CAST(ITEM($0,
> >>>>>>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>>>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0,
> >>> 'loc'),
> >>>>>>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT],
> >>>>>>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0,
> >>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>>>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0,
> >> 'id')):VARCHAR(5)
> >>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>> \"ISO-8859-1$en_US$primary\"])\n"
> >>>>>>>>>> +            + "
> >>>>>>> ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0,
> >>>>>>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>>>>>> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0,
> >>>> 'id')):VARCHAR(5)
> >>>>>>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
> >> \"ISO-8859-1$en_US$primary\",
> >>>>>>>>> '70000'))])\n"
> >>>>>>>>>> +            + "        ElasticsearchTableScan(table=[[elastic,
> >>>>>>> zips]])";
> >>>>>>>>>> +    calciteAssert()
> >>>>>>>>>> +            .query(sql)
> >>>>>>>>>> +            .returnsOrdered("city=LOS ANGELES;
> >>>> longitude=-118.258189;
> >>>>>>>>> latitude=34.007856; "
> >>>>>>>>>> +                            + "pop=96074; state=CA; id=90011",
> >>>>>>>>>> +                    "city=BELL GARDENS; longitude=-118.17205;
> >>>>>>>>> latitude=33.969177; "
> >>>>>>>>>> +                            + "pop=99568; state=CA; id=90201",
> >>>>>>>>>> +                    "city=NORWALK; longitude=-118.081767;
> >>>>>>>>> latitude=33.90564; "
> >>>>>>>>>> +                            + "pop=94188; state=CA; id=90650")
> >>>>>>>>>> +            .queryContains(
> >>>>>>>>>> +
> >>> ElasticChecker.elasticsearchChecker("\"query\"
> >>>> : "
> >>>>>>>>>> +                                    +
> >>>>>>>>> "{\"constant_score\":{\"filter\":{\"bool\":"
> >>>>>>>>>> +                                    +
> >>>>>>>>> "{\"must\":[{\"term\":{\"state\":\"CA\"}},"
> >>>>>>>>>> +                                    +
> >>>>>>>>> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}",
> >>>>>>>>>> +                            "\"script_fields\":
> >>>>>>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
> >>>>>>>>>> +                                    +
> >>>>>>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, "
> >>>>>>>>>> +                                    + "\"city\":{\"script\":
> >>>>>>>>> \"_source.city\"}, "
> >>>>>>>>>> +                                    + "\"pop\":{\"script\":
> >>>>>>>>> \"_source.pop\"}, "
> >>>>>>>>>> +                                    + "\"state\":{\"script\":
> >>>>>>>>> \"_source.state\"}, "
> >>>>>>>>>> +                                    + "\"id\":{\"script\": \"_
> >>>>>>> source.id
> >>>>>>>>> \"}}",
> >>>>>>>>>> +                            "\"sort\": [ {\"state\": \"asc\"},
> >>>>>>> {\"id\":
> >>>>>>>>> \"asc\"}]"))
> >>>>>>>>>> +            .explainContains(explain);
> >>>>>>>>>> +  }
> >>>>>>>>>> +
> >>>>>>>>>> +  @Test public void testFilterSortDesc() {
> >>>>>>>>>> +    final String sql = "select * from zips\n"
> >>>>>>>>>> +            + "where \"pop\" BETWEEN 95000 AND 100000\n"
> >>>>>>>>>> +            + "order by \"state\" desc, \"pop\"";
> >>>>>>>>>> +    calciteAssert()
> >>>>>>>>>> +            .query(sql)
> >>>>>>>>>> +            .limit(4)
> >>>>>>>>>> +            .returnsOrdered(
> >>>>