git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [2/2] calcite git commit: [CALCITE-2347] running ElasticSearch in embedded mode for unit tests of ES adapter (Andrei Sereda)


I have also checked "mvn javadoc:javadoc javadoc:test-javadoc" on my
machine (Ubuntu, JDK 8, Maven 3.3.9) and it fails on elasticsearch2 module
(didn't check this problem for the next modules).

"mvn clean -DskipTests site" fails for MongoDB module.

Kind regards,
Volodymyr Vysotskyi


сб, 23 черв. 2018 о 03:12 Julian Hyde <jhyde@xxxxxxxxxx> пише:

> Same failure for me on JDK 8/macOS. The most likely other variable is
> maven: I have 3.5.2 on macOS, 3.5.3 on ubuntu.
>
> I’m mystified why no one else is seeing javadoc failures on macOS.
>
> Julian
>
>
> > On Jun 22, 2018, at 4:35 PM, Julian Hyde <jhyde@xxxxxxxxxx> wrote:
> >
> > On JDK 8/ubuntu, “mvn clean -DskipTests site” gives the following
> failure:
> >
> > [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on
> project calcite-mongodb: Error generating
> maven-javadoc-plugin:3.0.1:test-javadoc report:
> > [ERROR] Exit code: 1 -
> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:22:
> error: cannot find symbol
> > [ERROR] import org.apache.calcite.test.CalciteAssert;
> > [ERROR]                               ^
> > [ERROR]   symbol:   class CalciteAssert
> > [ERROR]   location: package org.apache.calcite.test
> > [ERROR]
> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:137:
> error: package CalciteAssert does not exist
> > [ERROR]   private CalciteAssert.AssertThat assertModel(String model) {
> > [ERROR]                        ^
> > [ERROR]
> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/adapter/mongodb/MongoAdapterTest.java:145:
> error: package CalciteAssert does not exist
> > [ERROR]   private CalciteAssert.AssertThat assertModel(URL url) {
> > [ERROR]                        ^
> > [ERROR]
> /home/jhyde/open1/calcite.4/mongodb/src/test/java/org/apache/calcite/test/MongoAssertions.java:43:
> error: reference not found
> > [ERROR]   /** Similar to {@link CalciteAssert#checkResultUnordered}, but
> filters strings
> > [ERROR]                         ^
> >
> >
> >
> >> On Jun 22, 2018, at 3:46 PM, Andrei Sereda <andrei@xxxxxxxxx> wrote:
> >>
> >> Hmm. Both master and c12cb4b0de work for me. Perhaps the issue is JDK 8
> vs
> >> 9 ?
> >>
> >> # JDK8 (oracle) /  macOS
> >> $ mvn clean -DskipTests site
> >>
> >> On Fri, Jun 22, 2018 at 5:56 PM Julian Hyde <jhyde@xxxxxxxxxx> wrote:
> >>
> >>> I just tried it again. The following fails for me (jdk9, ubuntu):
> >>>
> >>> $ git checkout c12cb4b0de
> >>> $ mvn clean -DskipTests site
> >>>
> >>> Constructing Javadoc information...
> >>> 1 error
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>> [INFO] Reactor Summary:
> >>> [INFO]
> >>> [INFO] Calcite 1.17.0-SNAPSHOT ............................ FAILURE
> [02:29
> >>> min]
> >>> [INFO] Calcite Linq4j ..................................... SKIPPED
> >>> [INFO] Calcite Core ....................................... SKIPPED
> >>> [INFO] Calcite Cassandra .................................. SKIPPED
> >>> [INFO] Calcite Druid ...................................... SKIPPED
> >>> [INFO] Calcite Elasticsearch .............................. SKIPPED
> >>> [INFO] Calcite Elasticsearch5 ............................. SKIPPED
> >>> [INFO] Calcite Examples ................................... SKIPPED
> >>> [INFO] Calcite Example CSV ................................ SKIPPED
> >>> [INFO] Calcite Example Function ........................... SKIPPED
> >>> [INFO] Calcite File ....................................... SKIPPED
> >>> [INFO] Calcite Geode ...................................... SKIPPED
> >>> [INFO] Calcite MongoDB .................................... SKIPPED
> >>> [INFO] Calcite Pig ........................................ SKIPPED
> >>> [INFO] Calcite Piglet ..................................... SKIPPED
> >>> [INFO] Calcite Plus ....................................... SKIPPED
> >>> [INFO] Calcite Server ..................................... SKIPPED
> >>> [INFO] Calcite Spark ...................................... SKIPPED
> >>> [INFO] Calcite Splunk ..................................... SKIPPED
> >>> [INFO] Calcite Ubenchmark 1.17.0-SNAPSHOT ................. SKIPPED
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>> [INFO] BUILD FAILURE
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>> [INFO] Total time: 02:29 min
> >>> [INFO] Finished at: 2018-06-22T14:53:13-07:00
> >>> [INFO]
> >>>
> ------------------------------------------------------------------------
> >>> [ERROR] Failed to execute goal
> >>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on
> >>> project calcite: Error generating
> maven-javadoc-plugin:3.0.1:test-aggregate
> >>> report:
> >>> [ERROR] Exit code: 1 -
> >>>
> /home/jhyde/open1/calcite.4/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
> >>> error: package org.elasticsearch.node.internal does not exist
> >>> [ERROR] import
> org.elasticsearch.node.internal.InternalSettingsPreparer;
> >>> [ERROR]                                       ^
> >>> [ERROR]
> >>> [ERROR] Command line was: /usr/lib/jvm/jdk9/bin/javadoc @options
> @packages
> >>> @argfile
> >>> [ERROR]
> >>> [ERROR] Refer to the generated Javadoc files in
> >>> '/home/jhyde/open1/calcite.4/target/site/testapidocs' dir.
> >>> [ERROR] -> [Help 1]
> >>> [ERROR]
> >>> [ERROR] To see the full stack trace of the errors, re-run Maven with
> the
> >>> -e switch.
> >>> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> >>> [ERROR]
> >>> [ERROR] For more information about the errors and possible solutions,
> >>> please read the following articles:
> >>> [ERROR] [Help 1]
> >>>
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> >>>
> >>>
> >>>> On Jun 22, 2018, at 2:41 PM, Michael Mior <mmior@xxxxxxxxxx> wrote:
> >>>>
> >>>> Odd, mvn clean site still works fine for me.
> >>>> --
> >>>> Michael Mior
> >>>> mmior@xxxxxxxxxx
> >>>>
> >>>>
> >>>>
> >>>> Le ven. 22 juin 2018 à 17:12, Julian Hyde <jhyde@xxxxxxxxxx> a écrit
> :
> >>>>
> >>>>> Looks like this change broke “mvn site” (perhaps also “mvn
> >>>>> javadoc:test-javadoc”).
> >>>>>
> >>>>> [ERROR] Failed to execute goal
> >>>>> org.apache.maven.plugins:maven-site-plugin:3.7:site (default-site) on
> >>>>> project calcite: Error generating
> >>> maven-javadoc-plugin:3.0.1:test-aggregate
> >>>>> report:
> >>>>> [ERROR] Exit code: 1 -
> >>>>>
> >>>
> /home/jhyde/regress/calcite/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java:29:
> >>>>> error: package org.elasticsearch.node.internal does not exist
> >>>>> [ERROR] import
> org.elasticsearch.node.internal.InternalSettingsPreparer;
> >>>>> [ERROR]                                       ^
> >>>>> [ERROR]
> >>>>>
> >>>>>> On Jun 21, 2018, at 3:39 AM, mmior@xxxxxxxxxx wrote:
> >>>>>>
> >>>>>> [CALCITE-2347] running ElasticSearch in embedded mode for unit
> tests of
> >>>>> ES adapter (Andrei Sereda)
> >>>>>>
> >>>>>> After discussion on dev-list Integration tests (for ES) have been
> >>>>> removed. They're now
> >>>>>> superseded by unit tests (which execute queries against a real
> elastic
> >>>>> instance)
> >>>>>>
> >>>>>> Added local file (zips-mini.json) which contains a small subset of
> >>>>> original zips.json
> >>>>>> (allows to bootstrap tests faster)
> >>>>>>
> >>>>>> Created separate ES JUnit rule which can be re-used across different
> >>>>> tests.
> >>>>>>
> >>>>>> Both v2 and v5 of ES adapters are supported.
> >>>>>>
> >>>>>> Close apache/calcite#716
> >>>>>>
> >>>>>>
> >>>>>> Project: http://git-wip-us.apache.org/repos/asf/calcite/repo
> >>>>>> Commit:
> http://git-wip-us.apache.org/repos/asf/calcite/commit/c12cb4b0
> >>>>>> Tree: http://git-wip-us.apache.org/repos/asf/calcite/tree/c12cb4b0
> >>>>>> Diff: http://git-wip-us.apache.org/repos/asf/calcite/diff/c12cb4b0
> >>>>>>
> >>>>>> Branch: refs/heads/master
> >>>>>> Commit: c12cb4b0de1baa3f7cbb9952ee350fdd1701662d
> >>>>>> Parents: 37944bb
> >>>>>> Author: Andrei Sereda <andrei@xxxxxxxxxx>
> >>>>>> Authored: Thu May 31 18:19:10 2018 -0400
> >>>>>> Committer: Michael Mior <mmior@xxxxxxxxxxxx>
> >>>>>> Committed: Thu Jun 21 06:38:50 2018 -0400
> >>>>>>
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> .../AbstractElasticsearchTable.java             |  12 +
> >>>>>> .../elasticsearch/ElasticsearchProject.java     |  61 ++-
> >>>>>> elasticsearch2/pom.xml                          |   6 +
> >>>>>> .../Elasticsearch2Enumerator.java               |  12 +-
> >>>>>> .../elasticsearch2/Elasticsearch2Schema.java    |  16 +-
> >>>>>> .../elasticsearch2/Elasticsearch2Table.java     |   9 +-
> >>>>>> .../ElasticSearch2AdapterTest.java              | 395
> >>> ++++++++++++++++++
> >>>>>> .../elasticsearch2/EmbeddedElasticNode.java     | 147 +++++++
> >>>>>> .../elasticsearch2/EmbeddedElasticRule.java     |  97 +++++
> >>>>>> .../org/apache/calcite/test/ElasticChecker.java |  49 +++
> >>>>>> .../calcite/test/Elasticsearch2AdapterIT.java   | 270 -------------
> >>>>>> .../resources/elasticsearch-zips-model.json     |  50 ---
> >>>>>> .../src/test/resources/zips-mini.json           | 149 +++++++
> >>>>>> elasticsearch5/pom.xml                          |  31 ++
> >>>>>> .../elasticsearch5/Elasticsearch5Schema.java    |  17 +-
> >>>>>> .../elasticsearch5/Elasticsearch5Table.java     |  11 +-
> >>>>>> .../ElasticSearch5AdapterTest.java              | 399
> >>> +++++++++++++++++++
> >>>>>> .../elasticsearch5/EmbeddedElasticNode.java     | 153 +++++++
> >>>>>> .../elasticsearch5/EmbeddedElasticRule.java     |  98 +++++
> >>>>>> .../org/apache/calcite/test/ElasticChecker.java |  49 +++
> >>>>>> .../calcite/test/Elasticsearch5AdapterIT.java   | 270 -------------
> >>>>>> .../resources/elasticsearch-zips-model.json     |  50 ---
> >>>>>> elasticsearch5/src/test/resources/log4j2.xml    |  16 +
> >>>>>> .../src/test/resources/zips-mini.json           | 149 +++++++
> >>>>>> pom.xml                                         |  20 +-
> >>>>>> 25 files changed, 1866 insertions(+), 670 deletions(-)
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> diff --git
> >>>>>
> >>>
> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>
> >>>
> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>> index 0980469..8cc5933 100644
> >>>>>> ---
> >>>>>
> >>>
> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>> +++
> >>>>>
> >>>
> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/AbstractElasticsearchTable.java
> >>>>>> @@ -75,6 +75,18 @@ public abstract class AbstractElasticsearchTable
> >>>>> extends AbstractQueryableTable
> >>>>>>      relOptTable, this, null);
> >>>>>> }
> >>>>>>
> >>>>>> +  /**
> >>>>>> +   * In ES 5.x scripted fields start with {@code
> params._source.foo}
> >>>>> while in ES2.x
> >>>>>> +   * {@code _source.foo}. Helper method to build correct query
> based
> >>> on
> >>>>> runtime version of elastic.
> >>>>>> +   *
> >>>>>> +   * @see <a href="
> >>>>> https://github.com/elastic/elasticsearch/issues/20068";>_source
> >>>>> variable</a>
> >>>>>> +   * @see <a href="
> >>>>>
> >>>
> https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-fields.html
> >>> ">Scripted
> >>>>> Fields</a>
> >>>>>> +   */
> >>>>>> +  protected String scriptedFieldPrefix() {
> >>>>>> +    // this is default pattern starting 5.x
> >>>>>> +    return "params._source";
> >>>>>> +  }
> >>>>>> +
> >>>>>> /** Executes a "find" operation on the underlying type.
> >>>>>> *
> >>>>>> * <p>For example,
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> diff --git
> >>>>>
> >>>
> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>
> >>>
> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>> index b42abd7..961c8b0 100644
> >>>>>> ---
> >>>>>
> >>>
> a/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>> +++
> >>>>>
> >>>
> b/core/src/main/java/org/apache/calcite/adapter/elasticsearch/ElasticsearchProject.java
> >>>>>> @@ -27,11 +27,15 @@ import
> >>>>> org.apache.calcite.rel.metadata.RelMetadataQuery;
> >>>>>> import org.apache.calcite.rel.type.RelDataType;
> >>>>>> import org.apache.calcite.rex.RexNode;
> >>>>>> import org.apache.calcite.util.Pair;
> >>>>>> -import org.apache.calcite.util.Util;
> >>>>>> +
> >>>>>> +import com.google.common.base.Function;
> >>>>>> +import com.google.common.collect.Lists;
> >>>>>>
> >>>>>> import java.util.ArrayList;
> >>>>>> import java.util.List;
> >>>>>>
> >>>>>> +import javax.annotation.Nullable;
> >>>>>> +
> >>>>>> /**
> >>>>>> * Implementation of {@link org.apache.calcite.rel.core.Project}
> >>>>>> * relational expression in Elasticsearch.
> >>>>>> @@ -57,41 +61,60 @@ public class ElasticsearchProject extends
> Project
> >>>>> implements ElasticsearchRel {
> >>>>>>  implementor.visitChild(0, getInput());
> >>>>>>
> >>>>>>  final List<String> inFields =
> >>>>>> -
> >>>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType());
> >>>>>> +
> >>>>> ElasticsearchRules.elasticsearchFieldNames(getInput().getRowType());
> >>>>>>  final ElasticsearchRules.RexToElasticsearchTranslator translator =
> >>>>>> -        new ElasticsearchRules.RexToElasticsearchTranslator(
> >>>>>> -            (JavaTypeFactory) getCluster().getTypeFactory(),
> >>> inFields);
> >>>>>> +            new ElasticsearchRules.RexToElasticsearchTranslator(
> >>>>>> +                    (JavaTypeFactory)
> getCluster().getTypeFactory(),
> >>>>> inFields);
> >>>>>>
> >>>>>> -    final List<String> findItems = new ArrayList<>();
> >>>>>> -    final List<String> scriptFieldItems = new ArrayList<>();
> >>>>>> +    final List<String> fields = new ArrayList<>();
> >>>>>> +    final List<String> scriptFields = new ArrayList<>();
> >>>>>>  for (Pair<RexNode, String> pair: getNamedProjects()) {
> >>>>>>    final String name = pair.right;
> >>>>>>    final String expr = pair.left.accept(translator);
> >>>>>>
> >>>>>>    if (expr.equals("\"" + name + "\"")) {
> >>>>>> -        findItems.add(ElasticsearchRules.quote(name));
> >>>>>> +        fields.add(name);
> >>>>>>    } else if (expr.matches("\"literal\":.+")) {
> >>>>>> -        scriptFieldItems.add(ElasticsearchRules.quote(name)
> >>>>>> -            + ":{\"script\": "
> >>>>>> -            + expr.split(":")[1] + "}");
> >>>>>> +        scriptFields.add(ElasticsearchRules.quote(name)
> >>>>>> +                + ":{\"script\": "
> >>>>>> +                + expr.split(":")[1] + "}");
> >>>>>>    } else {
> >>>>>> -        scriptFieldItems.add(ElasticsearchRules.quote(name)
> >>>>>> -            + ":{\"script\":\"params._source."
> >>>>>> -            + expr.replaceAll("\"", "") + "\"}");
> >>>>>> +        scriptFields.add(ElasticsearchRules.quote(name)
> >>>>>> +                + ":{\"script\":"
> >>>>>> +                // _source (ES2) vs params._source (ES5)
> >>>>>> +                + "\"" +
> >>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "."
> >>>>>> +                + expr.replaceAll("\"", "") + "\"}");
> >>>>>> +      }
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    StringBuilder query = new StringBuilder();
> >>>>>> +    if (scriptFields.isEmpty()) {
> >>>>>> +      List<String> newList = Lists.transform(fields, new
> >>>>> Function<String, String>() {
> >>>>>> +        @Nullable
> >>>>>> +        @Override public String apply(@Nullable String input) {
> >>>>>> +          return ElasticsearchRules.quote(input);
> >>>>>> +        }
> >>>>>> +      });
> >>>>>> +
> >>>>>> +      final String findString = String.join(", ", newList);
> >>>>>> +      query.append("\"_source\" :
> [").append(findString).append("]");
> >>>>>> +    } else {
> >>>>>> +      // if scripted fields are present, ES ignores _source
> attribute
> >>>>>> +      for (String field: fields) {
> >>>>>> +        scriptFields.add(ElasticsearchRules.quote(field) +
> >>>>> ":{\"script\": "
> >>>>>> +                // _source (ES2) vs params._source (ES5)
> >>>>>> +                + "\"" +
> >>>>> implementor.elasticsearchTable.scriptedFieldPrefix() + "."
> >>>>>> +                + field + "\"}");
> >>>>>>    }
> >>>>>> +      query.append("\"script_fields\": {" + String.join(", ",
> >>>>> scriptFields) + "}");
> >>>>>>  }
> >>>>>> -    final String findString = Util.toString(findItems, "", ", ",
> "");
> >>>>>> -    final String scriptFieldString = "\"script_fields\": {"
> >>>>>> -        + Util.toString(scriptFieldItems, "", ", ", "") + "}";
> >>>>>> -    final String fieldString = "\"_source\" : [" + findString + "]"
> >>>>>> -        + ", " + scriptFieldString;
> >>>>>>
> >>>>>>  for (String opfield : implementor.list) {
> >>>>>>    if (opfield.startsWith("\"_source\"")) {
> >>>>>>      implementor.list.remove(opfield);
> >>>>>>    }
> >>>>>>  }
> >>>>>> -    implementor.add(fieldString);
> >>>>>> +    implementor.add(query.toString());
> >>>>>> }
> >>>>>> }
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/pom.xml
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> diff --git a/elasticsearch2/pom.xml b/elasticsearch2/pom.xml
> >>>>>> index f24622c..6fbee03 100644
> >>>>>> --- a/elasticsearch2/pom.xml
> >>>>>> +++ b/elasticsearch2/pom.xml
> >>>>>> @@ -73,6 +73,12 @@ limitations under the License.
> >>>>>>    <version>${elasticsearch-java-driver.version}</version>
> >>>>>>  </dependency>
> >>>>>>  <dependency>
> >>>>>> +      <!-- Lang groovy dependency is needed for testing with
> embedded
> >>>>> ES (scripted fields like loc[0]) -->
> >>>>>> +      <groupId>org.elasticsearch.module</groupId>
> >>>>>> +      <artifactId>lang-groovy</artifactId>
> >>>>>> +      <scope>test</scope>
> >>>>>> +    </dependency>
> >>>>>> +    <dependency>
> >>>>>>    <groupId>com.carrotsearch</groupId>
> >>>>>>    <artifactId>hppc</artifactId>
> >>>>>>    <version>${hppc.version}</version>
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> diff --git
> >>>>>
> >>>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>
> >>>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>> index 84370ab..c3d2ac0 100644
> >>>>>> ---
> >>>>>
> >>>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>> +++
> >>>>>
> >>>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Enumerator.java
> >>>>>> @@ -26,6 +26,7 @@ import org.elasticsearch.search.SearchHit;
> >>>>>> import java.util.Date;
> >>>>>> import java.util.Iterator;
> >>>>>> import java.util.List;
> >>>>>> +import java.util.Locale;
> >>>>>> import java.util.Map;
> >>>>>>
> >>>>>> /**
> >>>>>> @@ -101,15 +102,18 @@ public class Elasticsearch2Enumerator
> implements
> >>>>> Enumerator<Object> {
> >>>>>> private static Function1<SearchHit, Object[]> listGetter(
> >>>>>>    final List<Map.Entry<String, Class>> fields) {
> >>>>>>  return new Function1<SearchHit, Object[]>() {
> >>>>>> -      public Object[] apply(SearchHit searchHitFields) {
> >>>>>> +      public Object[] apply(SearchHit hit) {
> >>>>>>      Object[] objects = new Object[fields.size()];
> >>>>>>      for (int i = 0; i < fields.size(); i++) {
> >>>>>>        final Map.Entry<String, Class> field = fields.get(i);
> >>>>>>        final String name = field.getKey();
> >>>>>> -          if (searchHitFields.fields().isEmpty()) {
> >>>>>> -            objects[i] =
> >>> convert(searchHitFields.getSource().get(name),
> >>>>> field.getValue());
> >>>>>> +          if (hit.fields().isEmpty()) {
> >>>>>> +            objects[i] = convert(hit.getSource().get(name),
> >>>>> field.getValue());
> >>>>>> +          } else if (hit.fields().containsKey(name)) {
> >>>>>> +            objects[i] = convert(hit.field(name).getValue(),
> >>>>> field.getValue());
> >>>>>>        } else {
> >>>>>> -            objects[i] =
> >>>>> convert(searchHitFields.field(name).getValue(), field.getValue());
> >>>>>> +            throw new IllegalStateException(
> >>>>>> +                    String.format(Locale.getDefault(), "No result
> for
> >>>>> %s", field));
> >>>>>>        }
> >>>>>>      }
> >>>>>>      return objects;
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> diff --git
> >>>>>
> >>>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>
> >>>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>> index 668402b..46e3fc5 100644
> >>>>>> ---
> >>>>>
> >>>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>> +++
> >>>>>
> >>>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Schema.java
> >>>>>> @@ -22,6 +22,9 @@ import
> org.apache.calcite.schema.impl.AbstractSchema;
> >>>>>>
> >>>>>> import com.carrotsearch.hppc.cursors.ObjectObjectCursor;
> >>>>>>
> >>>>>> +import com.google.common.annotations.VisibleForTesting;
> >>>>>> +import com.google.common.base.Preconditions;
> >>>>>> +
> >>>>>> import com.google.common.collect.ImmutableList;
> >>>>>> import com.google.common.collect.ImmutableMap;
> >>>>>>
> >>>>>> @@ -86,6 +89,16 @@ public class Elasticsearch2Schema extends
> >>>>> AbstractSchema
> >>>>>>  }
> >>>>>> }
> >>>>>>
> >>>>>> +  /**
> >>>>>> +   * Allows schema to be instantiated from existing elastic search
> >>>>> client.
> >>>>>> +   * This constructor is used in tests.
> >>>>>> +   */
> >>>>>> +  @VisibleForTesting
> >>>>>> +  Elasticsearch2Schema(Client client, String index) {
> >>>>>> +    this.client = Preconditions.checkNotNull(client, "client");
> >>>>>> +    this.index = Preconditions.checkNotNull(index, "index");
> >>>>>> +  }
> >>>>>> +
> >>>>>> @Override protected Map<String, Table> getTableMap() {
> >>>>>>  final ImmutableMap.Builder<String, Table> builder =
> >>>>> ImmutableMap.builder();
> >>>>>>
> >>>>>> @@ -120,7 +133,8 @@ public class Elasticsearch2Schema extends
> >>>>> AbstractSchema
> >>>>>>
> >>>>>>  final List<DiscoveryNode> nodes =
> >>>>> ImmutableList.copyOf(transportClient.connectedNodes());
> >>>>>>  if (nodes.isEmpty()) {
> >>>>>> -      throw new RuntimeException("Cannot connect to any
> elasticsearch
> >>>>> nodes");
> >>>>>> +      throw new IllegalStateException("Cannot connect to any
> >>>>> elasticsearch node: "
> >>>>>> +              + transportNodes);
> >>>>>>  }
> >>>>>>
> >>>>>>  client = transportClient;
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> diff --git
> >>>>>
> >>>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>
> >>>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>> index 636aa5f..2928835 100644
> >>>>>> ---
> >>>>>
> >>>
> a/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>> +++
> >>>>>
> >>>
> b/elasticsearch2/src/main/java/org/apache/calcite/adapter/elasticsearch2/Elasticsearch2Table.java
> >>>>>> @@ -45,8 +45,15 @@ public class Elasticsearch2Table extends
> >>>>> AbstractElasticsearchTable {
> >>>>>>  this.client = client;
> >>>>>> }
> >>>>>>
> >>>>>> +  /**
> >>>>>> +   * ES version 2.x. To access document attributes ES2 uses {@code
> >>>>> _source.foo} syntax.
> >>>>>> +   */
> >>>>>> +  @Override protected String scriptedFieldPrefix() {
> >>>>>> +    return "_source";
> >>>>>> +  }
> >>>>>> +
> >>>>>> @Override protected Enumerable<Object> find(String index,
> List<String>
> >>>>> ops,
> >>>>>> -      List<Map.Entry<String, Class>> fields) {
> >>>>>> +
> List<Map.Entry<String,
> >>>>> Class>> fields) {
> >>>>>>  final String dbName = index;
> >>>>>>
> >>>>>>  final String queryString = "{" + Util.toString(ops, "", ", ", "") +
> >>>>> "}";
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> diff --git
> >>>>>
> >>>
> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
> >>>>>
> >>>
> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
> >>>>>> new file mode 100644
> >>>>>> index 0000000..287e094
> >>>>>> --- /dev/null
> >>>>>> +++
> >>>>>
> >>>
> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/ElasticSearch2AdapterTest.java
> >>>>>> @@ -0,0 +1,395 @@
> >>>>>> +/*
> >>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or
> more
> >>>>>> + * contributor license agreements.  See the NOTICE file distributed
> >>> with
> >>>>>> + * this work for additional information regarding copyright
> ownership.
> >>>>>> + * The ASF licenses this file to you under the Apache License,
> Version
> >>>>> 2.0
> >>>>>> + * (the "License"); you may not use this file except in compliance
> >>> with
> >>>>>> + * the License.  You may obtain a copy of the License at
> >>>>>> + *
> >>>>>> + * http://www.apache.org/licenses/LICENSE-2.0
> >>>>>> + *
> >>>>>> + * Unless required by applicable law or agreed to in writing,
> software
> >>>>>> + * distributed under the License is distributed on an "AS IS"
> BASIS,
> >>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> >>>>> implied.
> >>>>>> + * See the License for the specific language governing permissions
> and
> >>>>>> + * limitations under the License.
> >>>>>> + */
> >>>>>> +package org.apache.calcite.adapter.elasticsearch2;
> >>>>>> +
> >>>>>> +import org.apache.calcite.jdbc.CalciteConnection;
> >>>>>> +import org.apache.calcite.schema.SchemaPlus;
> >>>>>> +import org.apache.calcite.schema.impl.ViewTable;
> >>>>>> +import org.apache.calcite.schema.impl.ViewTableMacro;
> >>>>>> +import org.apache.calcite.test.CalciteAssert;
> >>>>>> +import org.apache.calcite.test.ElasticChecker;
> >>>>>> +
> >>>>>> +import com.google.common.io.LineProcessor;
> >>>>>> +import com.google.common.io.Resources;
> >>>>>> +
> >>>>>> +import org.elasticsearch.action.bulk.BulkItemResponse;
> >>>>>> +import org.elasticsearch.action.bulk.BulkRequestBuilder;
> >>>>>> +import org.elasticsearch.action.bulk.BulkResponse;
> >>>>>> +import org.elasticsearch.common.xcontent.XContentBuilder;
> >>>>>> +import org.elasticsearch.common.xcontent.XContentFactory;
> >>>>>> +
> >>>>>> +import org.junit.BeforeClass;
> >>>>>> +import org.junit.ClassRule;
> >>>>>> +import org.junit.Test;
> >>>>>> +
> >>>>>> +import java.io.IOException;
> >>>>>> +import java.nio.charset.StandardCharsets;
> >>>>>> +import java.sql.Connection;
> >>>>>> +import java.sql.DriverManager;
> >>>>>> +import java.sql.SQLException;
> >>>>>> +import java.util.Arrays;
> >>>>>> +import java.util.Collections;
> >>>>>> +import java.util.Locale;
> >>>>>> +
> >>>>>> +/**
> >>>>>> + * Set of tests for ES adapter. Uses real instance via {@link
> >>>>> EmbeddedElasticRule}. Document
> >>>>>> + * source is local {@code zips-mini.json} file (located in the
> >>>>> classpath).
> >>>>>> + */
> >>>>>> +public class ElasticSearch2AdapterTest {
> >>>>>> +
> >>>>>> +  @ClassRule //init once for all tests
> >>>>>> +  public static final EmbeddedElasticRule NODE =
> >>>>> EmbeddedElasticRule.create();
> >>>>>> +
> >>>>>> +  private static final String ZIPS = "zips";
> >>>>>> +
> >>>>>> +  /**
> >>>>>> +   * Used to create {@code zips} index and insert some data
> >>>>>> +   */
> >>>>>> +  @BeforeClass
> >>>>>> +  public static void setupInstance() throws Exception {
> >>>>>> +    // define mapping so fields are searchable (term query)
> >>>>>> +    XContentBuilder mapping =
> >>>>> XContentFactory.jsonBuilder().startObject()
> >>>>>> +            .startObject("properties")
> >>>>>> +            .startObject("city").field("type", "string")
> >>>>>> +                  .field("index", "not_analyzed").endObject()
> >>>>>> +            .startObject("state").field("type", "string")
> >>>>>> +                  .field("index", "not_analyzed").endObject()
> >>>>>> +            .startObject("pop").field("type", "long").endObject()
> >>>>>> +            .endObject()
> >>>>>> +            .endObject();
> >>>>>> +
> >>>>>> +    // create index
> >>>>>> +    NODE.client().admin().indices()
> >>>>>> +            .prepareCreate(ZIPS)
> >>>>>> +            .addMapping(ZIPS, mapping)
> >>>>>> +            .get();
> >>>>>> +
> >>>>>> +    BulkRequestBuilder bulk =
> >>>>> NODE.client().prepareBulk().setRefresh(true);
> >>>>>> +
> >>>>>> +    // load records from file
> >>>>>> +
> >>>>>
> >>>
> Resources.readLines(ElasticSearch2AdapterTest.class.getResource("/zips-mini.json"),
> >>>>>> +            StandardCharsets.UTF_8, new LineProcessor<Void>() {
> >>>>>> +              @Override public boolean processLine(String line)
> throws
> >>>>> IOException {
> >>>>>> +                line = line.replaceAll("_id", "id"); // _id is a
> >>>>> reserved attribute in ES
> >>>>>> +                bulk.add(NODE.client().prepareIndex(ZIPS,
> >>>>> ZIPS).setSource(line));
> >>>>>> +                return true;
> >>>>>> +              }
> >>>>>> +
> >>>>>> +              @Override public Void getResult() {
> >>>>>> +                return null;
> >>>>>> +              }
> >>>>>> +            });
> >>>>>> +
> >>>>>> +    if (bulk.numberOfActions() == 0) {
> >>>>>> +      throw new IllegalStateException("No records to be indexed");
> >>>>>> +    }
> >>>>>> +
> >>>>>> +    BulkResponse response = bulk.execute().get();
> >>>>>> +
> >>>>>> +    if (response.hasFailures()) {
> >>>>>> +      throw new IllegalStateException(
> >>>>>> +              String.format(Locale.getDefault(), "Failed to
> populate
> >>>>> %s:\n%s", NODE.httpAddress(),
> >>>>>> +
> >>>>> Arrays.stream(response.getItems()).filter(BulkItemResponse::isFailed)
> >>>>>> +
> >>>>>
> >>>
> .map(BulkItemResponse::getFailureMessage).findFirst().orElse("<unknown>")));
> >>>>>> +    }
> >>>>>> +
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  private CalciteAssert.ConnectionFactory newConnectionFactory() {
> >>>>>> +    return new CalciteAssert.ConnectionFactory() {
> >>>>>> +      @Override public Connection createConnection() throws
> >>>>> SQLException {
> >>>>>> +        final Connection connection =
> >>>>> DriverManager.getConnection("jdbc:calcite:");
> >>>>>> +        final SchemaPlus root =
> >>>>> connection.unwrap(CalciteConnection.class).getRootSchema();
> >>>>>> +
> >>>>>> +        root.add("elastic", new Elasticsearch2Schema(NODE.client(),
> >>>>> ZIPS));
> >>>>>> +
> >>>>>> +        // add calcite view programmatically
> >>>>>> +        final String viewSql = "select cast(_MAP['city'] AS
> >>>>> varchar(20)) AS \"city\", "
> >>>>>> +               + " cast(_MAP['loc'][0] AS float) AS
> \"longitude\",\n"
> >>>>>> +               + " cast(_MAP['loc'][1] AS float) AS
> \"latitude\",\n"
> >>>>>> +              + " cast(_MAP['pop'] AS integer) AS \"pop\", "
> >>>>>> +              +  " cast(_MAP['state'] AS varchar(2)) AS \"state\",
> "
> >>>>>> +              +  " cast(_MAP['id'] AS varchar(5)) AS \"id\" "
> >>>>>> +              +  "from \"elastic\".\"zips\"";
> >>>>>> +
> >>>>>> +        ViewTableMacro macro = ViewTable.viewMacro(root, viewSql,
> >>>>>> +                Collections.singletonList("elastic"),
> >>>>> Arrays.asList("elastic", "view"), false);
> >>>>>> +        root.add("ZIPS", macro);
> >>>>>> +
> >>>>>> +        return connection;
> >>>>>> +      }
> >>>>>> +    };
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  private CalciteAssert.AssertThat calciteAssert() {
> >>>>>> +    return CalciteAssert.that()
> >>>>>> +            .with(newConnectionFactory());
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  /**
> >>>>>> +   * Tests using calcite view
> >>>>>> +   */
> >>>>>> +  @Test
> >>>>>> +  public void view() throws Exception {
> >>>>>> +    calciteAssert()
> >>>>>> +          .query("select * from zips where \"city\" = 'BROOKLYN'")
> >>>>>> +          .returns("city=BROOKLYN; longitude=-73.956985;
> >>>>> latitude=40.646694; "
> >>>>>> +                  + "pop=111396; state=NY; id=11226\n")
> >>>>>> +          .returnsCount(1);
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test
> >>>>>> +  public void emptyResult() {
> >>>>>> +    CalciteAssert.that()
> >>>>>> +            .with(newConnectionFactory())
> >>>>>> +            .query("select * from zips limit 0")
> >>>>>> +            .returnsCount(0);
> >>>>>> +
> >>>>>> +    CalciteAssert.that()
> >>>>>> +            .with(newConnectionFactory())
> >>>>>> +            .query("select * from \"elastic\".\"zips\" where
> >>>>> _MAP['Foo'] = '_MISSING_'")
> >>>>>> +            .returnsCount(0);
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test
> >>>>>> +  public void basic() throws Exception {
> >>>>>> +    CalciteAssert.that()
> >>>>>> +            .with(newConnectionFactory())
> >>>>>> +            .query("select * from \"elastic\".\"zips\" where
> >>>>> _MAP['city'] = 'BROOKLYN'")
> >>>>>> +            .returnsCount(1);
> >>>>>> +
> >>>>>> +    CalciteAssert.that()
> >>>>>> +            .with(newConnectionFactory())
> >>>>>> +            .query("select * from \"elastic\".\"zips\" where"
> >>>>>> +                    + " _MAP['city'] in ('BROOKLYN',
> 'WASHINGTON')")
> >>>>>> +            .returnsCount(2);
> >>>>>> +
> >>>>>> +    // lower-case
> >>>>>> +    CalciteAssert.that()
> >>>>>> +            .with(newConnectionFactory())
> >>>>>> +            .query("select * from \"elastic\".\"zips\" where "
> >>>>>> +                    + "_MAP['city'] in ('brooklyn', 'Brooklyn',
> >>>>> 'BROOK') ")
> >>>>>> +            .returnsCount(0);
> >>>>>> +
> >>>>>> +    // missing field
> >>>>>> +    CalciteAssert.that()
> >>>>>> +            .with(newConnectionFactory())
> >>>>>> +            .query("select * from \"elastic\".\"zips\" where
> >>>>> _MAP['CITY'] = 'BROOKLYN'")
> >>>>>> +            .returnsCount(0);
> >>>>>> +
> >>>>>> +    // limit works
> >>>>>> +    CalciteAssert.that()
> >>>>>> +            .with(newConnectionFactory())
> >>>>>> +            .query("select * from \"elastic\".\"zips\" limit 42")
> >>>>>> +            .returnsCount(42);
> >>>>>> +
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testSort() {
> >>>>>> +    final String explain =
> "PLAN=ElasticsearchToEnumerableConverter\n"
> >>>>>> +            + "  ElasticsearchSort(sort0=[$4], dir0=[ASC])\n"
> >>>>>> +            + "    ElasticsearchProject(city=[CAST(ITEM($0,
> >>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'),
> >>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT],
> >>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0,
> >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5)
> >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
> \"ISO-8859-1$en_US$primary\"])\n"
> >>>>>> +            + "      ElasticsearchTableScan(table=[[elastic,
> zips]])";
> >>>>>> +
> >>>>>> +    calciteAssert()
> >>>>>> +            .query("select * from zips order by \"state\"")
> >>>>>> +            .returnsCount(10)
> >>>>>> +            .explainContains(explain);
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testSortLimit() {
> >>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
> >>>>>> +            + "order by \"state\", \"id\" offset 2 rows fetch next
> 3
> >>>>> rows only";
> >>>>>> +    calciteAssert()
> >>>>>> +            .query(sql)
> >>>>>> +            .returnsUnordered("state=AK; id=99801",
> >>>>>> +                    "state=AL; id=35215",
> >>>>>> +                    "state=AL; id=35401")
> >>>>>> +            .queryContains(
> >>>>>> +                    ElasticChecker.elasticsearchChecker(
> >>>>>> +                            "\"_source\" : [\"state\", \"id\"]",
> >>>>>> +                            "\"sort\": [ {\"state\": \"asc\"},
> >>> {\"id\":
> >>>>> \"asc\"}]",
> >>>>>> +                            "\"from\": 2",
> >>>>>> +                            "\"size\": 3"));
> >>>>>> +  }
> >>>>>> +
> >>>>>> +
> >>>>>> +
> >>>>>> +  @Test public void testOffsetLimit() {
> >>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
> >>>>>> +            + "offset 2 fetch next 3 rows only";
> >>>>>> +    calciteAssert()
> >>>>>> +            .query(sql)
> >>>>>> +            .runs()
> >>>>>> +            .queryContains(
> >>>>>> +                    ElasticChecker.elasticsearchChecker(
> >>>>>> +                            "\"from\": 2",
> >>>>>> +                            "\"size\": 3",
> >>>>>> +                            "\"_source\" : [\"state\", \"id\"]"));
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testLimit() {
> >>>>>> +    final String sql = "select \"state\", \"id\" from zips\n"
> >>>>>> +            + "fetch next 3 rows only";
> >>>>>> +
> >>>>>> +    calciteAssert()
> >>>>>> +            .query(sql)
> >>>>>> +            .runs()
> >>>>>> +            .queryContains(
> >>>>>> +                    ElasticChecker.elasticsearchChecker(
> >>>>>> +                            "\"size\": 3",
> >>>>>> +                            "\"_source\" : [\"state\", \"id\"]"));
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testFilterSort() {
> >>>>>> +    final String sql = "select * from zips\n"
> >>>>>> +            + "where \"state\" = 'CA' and \"id\" >= '70000'\n"
> >>>>>> +            + "order by \"state\", \"id\"";
> >>>>>> +    final String explain =
> "PLAN=ElasticsearchToEnumerableConverter\n"
> >>>>>> +            + "  ElasticsearchSort(sort0=[$4], sort1=[$5],
> dir0=[ASC],
> >>>>> dir1=[ASC])\n"
> >>>>>> +            + "    ElasticsearchProject(city=[CAST(ITEM($0,
> >>>>> 'city')):VARCHAR(20) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>> \"ISO-8859-1$en_US$primary\"], longitude=[CAST(ITEM(ITEM($0, 'loc'),
> >>>>> 0)):FLOAT], latitude=[CAST(ITEM(ITEM($0, 'loc'), 1)):FLOAT],
> >>>>> pop=[CAST(ITEM($0, 'pop')):INTEGER], state=[CAST(ITEM($0,
> >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>> \"ISO-8859-1$en_US$primary\"], id=[CAST(ITEM($0, 'id')):VARCHAR(5)
> >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
> \"ISO-8859-1$en_US$primary\"])\n"
> >>>>>> +            + "
> >>> ElasticsearchFilter(condition=[AND(=(CAST(ITEM($0,
> >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>> \"ISO-8859-1$en_US$primary\", 'CA'), >=(CAST(ITEM($0,
> 'id')):VARCHAR(5)
> >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE \"ISO-8859-1$en_US$primary\",
> >>>>> '70000'))])\n"
> >>>>>> +            + "        ElasticsearchTableScan(table=[[elastic,
> >>> zips]])";
> >>>>>> +    calciteAssert()
> >>>>>> +            .query(sql)
> >>>>>> +            .returnsOrdered("city=LOS ANGELES;
> longitude=-118.258189;
> >>>>> latitude=34.007856; "
> >>>>>> +                            + "pop=96074; state=CA; id=90011",
> >>>>>> +                    "city=BELL GARDENS; longitude=-118.17205;
> >>>>> latitude=33.969177; "
> >>>>>> +                            + "pop=99568; state=CA; id=90201",
> >>>>>> +                    "city=NORWALK; longitude=-118.081767;
> >>>>> latitude=33.90564; "
> >>>>>> +                            + "pop=94188; state=CA; id=90650")
> >>>>>> +            .queryContains(
> >>>>>> +                    ElasticChecker.elasticsearchChecker("\"query\"
> : "
> >>>>>> +                                    +
> >>>>> "{\"constant_score\":{\"filter\":{\"bool\":"
> >>>>>> +                                    +
> >>>>> "{\"must\":[{\"term\":{\"state\":\"CA\"}},"
> >>>>>> +                                    +
> >>>>> "{\"range\":{\"id\":{\"gte\":\"70000\"}}}]}}}}",
> >>>>>> +                            "\"script_fields\":
> >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
> >>>>>> +                                    +
> >>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, "
> >>>>>> +                                    + "\"city\":{\"script\":
> >>>>> \"_source.city\"}, "
> >>>>>> +                                    + "\"pop\":{\"script\":
> >>>>> \"_source.pop\"}, "
> >>>>>> +                                    + "\"state\":{\"script\":
> >>>>> \"_source.state\"}, "
> >>>>>> +                                    + "\"id\":{\"script\": \"_
> >>> source.id
> >>>>> \"}}",
> >>>>>> +                            "\"sort\": [ {\"state\": \"asc\"},
> >>> {\"id\":
> >>>>> \"asc\"}]"))
> >>>>>> +            .explainContains(explain);
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testFilterSortDesc() {
> >>>>>> +    final String sql = "select * from zips\n"
> >>>>>> +            + "where \"pop\" BETWEEN 95000 AND 100000\n"
> >>>>>> +            + "order by \"state\" desc, \"pop\"";
> >>>>>> +    calciteAssert()
> >>>>>> +            .query(sql)
> >>>>>> +            .limit(4)
> >>>>>> +            .returnsOrdered(
> >>>>>> +             "city=LOS ANGELES; longitude=-118.258189;
> >>>>> latitude=34.007856; pop=96074; state=CA; id=90011",
> >>>>>> +             "city=BELL GARDENS; longitude=-118.17205;
> >>>>> latitude=33.969177; pop=99568; state=CA; id=90201");
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testFilterRedundant() {
> >>>>>> +    final String sql = "select * from zips\n"
> >>>>>> +            + "where \"state\" > 'CA' and \"state\" < 'AZ' and
> >>>>> \"state\" = 'OK'";
> >>>>>> +    calciteAssert()
> >>>>>> +            .query(sql)
> >>>>>> +            .runs()
> >>>>>> +            .queryContains(
> >>>>>> +                    ElasticChecker.elasticsearchChecker(""
> >>>>>> +                                    + "\"query\" :
> >>>>> {\"constant_score\":{\"filter\":{\"bool\":"
> >>>>>> +                                    +
> >>>>> "{\"must\":[{\"term\":{\"state\":\"OK\"}}]}}}}",
> >>>>>> +                            "\"script_fields\":
> >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
> >>>>>> +                                    +
> >>>>> "\"latitude\":{\"script\":\"_source.loc[1]\"}, "
> >>>>>> +                                    +   "\"city\":{\"script\":
> >>>>> \"_source.city\"}, "
> >>>>>> +                                    +   "\"pop\":{\"script\":
> >>>>> \"_source.pop\"}, \"state\":{\"script\": \"_source.state\"}, "
> >>>>>> +                                    +
> "\"id\":{\"script\":
> >>>>> \"_source.id\"}}"
> >>>>>> +                    ));
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testInPlan() {
> >>>>>> +    final String[] searches = {
> >>>>>> +        "\"query\" :
> >>>>> {\"constant_score\":{\"filter\":{\"bool\":{\"should\":"
> >>>>>> +                 +
> >>>>>
> >>>
> "[{\"bool\":{\"must\":[{\"term\":{\"pop\":96074}}]}},{\"bool\":{\"must\":[{\"term\":"
> >>>>>> +                 + "{\"pop\":99568}}]}}]}}}}",
> >>>>>> +        "\"script_fields\":
> >>>>> {\"longitude\":{\"script\":\"_source.loc[0]\"}, "
> >>>>>> +                 +  "\"latitude\":{\"script\":\"_source.loc[1]\"},
> "
> >>>>>> +                 +  "\"city\":{\"script\": \"_source.city\"}, "
> >>>>>> +                 +  "\"pop\":{\"script\": \"_source.pop\"}, "
> >>>>>> +                 +  "\"state\":{\"script\": \"_source.state\"}, "
> >>>>>> +                 +  "\"id\":{\"script\": \"_source.id\"}}"
> >>>>>> +    };
> >>>>>> +
> >>>>>> +    calciteAssert()
> >>>>>> +            .query("select * from zips where \"pop\" in (96074,
> >>> 99568)")
> >>>>>> +            .returnsUnordered(
> >>>>>> +                    "city=BELL GARDENS; longitude=-118.17205;
> >>>>> latitude=33.969177; pop=99568; state=CA; id=90201",
> >>>>>> +                    "city=LOS ANGELES; longitude=-118.258189;
> >>>>> latitude=34.007856; pop=96074; state=CA; id=90011"
> >>>>>> +                    )
> >>>>>> +
> >>>>> .queryContains(ElasticChecker.elasticsearchChecker(searches));
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testZips() {
> >>>>>> +    calciteAssert()
> >>>>>> +         .query("select \"state\", \"city\" from zips")
> >>>>>> +         .returnsCount(10);
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testProject() {
> >>>>>> +    final String sql = "select \"state\", \"city\", 0 as
> \"zero\"\n"
> >>>>>> +            + "from zips\n"
> >>>>>> +            + "order by \"state\", \"city\"";
> >>>>>> +
> >>>>>> +    calciteAssert()
> >>>>>> +            .query(sql)
> >>>>>> +            .limit(2)
> >>>>>> +            .returnsUnordered("state=AK; city=ANCHORAGE; zero=0",
> >>>>>> +                    "state=AK; city=FAIRBANKS; zero=0")
> >>>>>> +            .queryContains(
> >>>>>> +
> >>>>> ElasticChecker.elasticsearchChecker("\"script_fields\": "
> >>>>>> +                                    + "{\"zero\":{\"script\":
> \"0\"},
> >>> "
> >>>>>> +                                    + "\"state\":{\"script\":
> >>>>> \"_source.state\"}, "
> >>>>>> +                                    + "\"city\":{\"script\":
> >>>>> \"_source.city\"}}",
> >>>>>> +                            "\"sort\": [ {\"state\": \"asc\"},
> >>>>> {\"city\": \"asc\"}]"));
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testFilter() {
> >>>>>> +    final String explain =
> "PLAN=ElasticsearchToEnumerableConverter\n"
> >>>>>> +            + "  ElasticsearchProject(state=[CAST(ITEM($0,
> >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>> \"ISO-8859-1$en_US$primary\"], city=[CAST(ITEM($0,
> 'city')):VARCHAR(20)
> >>>>> CHARACTER SET \"ISO-8859-1\" COLLATE
> \"ISO-8859-1$en_US$primary\"])\n"
> >>>>>> +            + "    ElasticsearchFilter(condition=[=(CAST(ITEM($0,
> >>>>> 'state')):VARCHAR(2) CHARACTER SET \"ISO-8859-1\" COLLATE
> >>>>> \"ISO-8859-1$en_US$primary\", 'CA')])\n"
> >>>>>> +            + "      ElasticsearchTableScan(table=[[elastic,
> zips]])";
> >>>>>> +    calciteAssert()
> >>>>>> +            .query("select \"state\", \"city\" from zips where
> >>>>> \"state\" = 'CA'")
> >>>>>> +            .limit(3)
> >>>>>> +            .returnsUnordered("state=CA; city=BELL GARDENS",
> >>>>>> +                    "state=CA; city=LOS ANGELES",
> >>>>>> +                    "state=CA; city=NORWALK")
> >>>>>> +            .explainContains(explain);
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  @Test public void testFilterReversed() {
> >>>>>> +    calciteAssert()
> >>>>>> +          .query("select \"state\", \"city\" from zips where 'WI' <
> >>>>> \"state\" order by \"city\"")
> >>>>>> +          .limit(2)
> >>>>>> +          .returnsUnordered("state=WV; city=BECKLEY",
> >>>>>> +                    "state=WY; city=CHEYENNE");
> >>>>>> +    calciteAssert()
> >>>>>> +          .query("select \"state\", \"city\" from zips where
> \"state\"
> >>>>>> 'WI' order by \"city\"")
> >>>>>> +          .limit(2)
> >>>>>> +          .returnsUnordered("state=WV; city=BECKLEY",
> >>>>>> +                    "state=WY; city=CHEYENNE");
> >>>>>> +  }
> >>>>>> +
> >>>>>> +}
> >>>>>> +
> >>>>>> +// End ElasticSearch2AdapterTest.java
> >>>>>>
> >>>>>>
> >>>>>
> >>>
> http://git-wip-us.apache.org/repos/asf/calcite/blob/c12cb4b0/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java
> >>>>>>
> ----------------------------------------------------------------------
> >>>>>> diff --git
> >>>>>
> >>>
> a/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java
> >>>>>
> >>>
> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java
> >>>>>> new file mode 100644
> >>>>>> index 0000000..4474add
> >>>>>> --- /dev/null
> >>>>>> +++
> >>>>>
> >>>
> b/elasticsearch2/src/test/java/org/apache/calcite/adapter/elasticsearch2/EmbeddedElasticNode.java
> >>>>>> @@ -0,0 +1,147 @@
> >>>>>> +/*
> >>>>>> + * Licensed to the Apache Software Foundation (ASF) under one or
> more
> >>>>>> + * contributor license agreements.  See the NOTICE file distributed
> >>> with
> >>>>>> + * this work for additional information regarding copyright
> ownership.
> >>>>>> + * The ASF licenses this file to you under the Apache License,
> Version
> >>>>> 2.0
> >>>>>> + * (the "License"); you may not use this file except in compliance
> >>> with
> >>>>>> + * the License.  You may obtain a copy of the License at
> >>>>>> + *
> >>>>>> + * http://www.apache.org/licenses/LICENSE-2.0
> >>>>>> + *
> >>>>>> + * Unless required by applicable law or agreed to in writing,
> software
> >>>>>> + * distributed under the License is distributed on an "AS IS"
> BASIS,
> >>>>>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> >>>>> implied.
> >>>>>> + * See the License for the specific language governing permissions
> and
> >>>>>> + * limitations under the License.
> >>>>>> + */
> >>>>>> +package org.apache.calcite.adapter.elasticsearch2;
> >>>>>> +
> >>>>>> +import com.google.common.base.Preconditions;
> >>>>>> +import com.google.common.io.Files;
> >>>>>> +
> >>>>>> +import org.elasticsearch.Version;
> >>>>>> +import org.elasticsearch.action.admin.cluster.node.info.NodeInfo;
> >>>>>> +import org.elasticsearch.action.admin.cluster.node.info
> >>>>> .NodesInfoResponse;
> >>>>>> +import org.elasticsearch.client.Client;
> >>>>>> +import org.elasticsearch.common.settings.Settings;
> >>>>>> +import org.elasticsearch.common.transport.TransportAddress;
> >>>>>> +import org.elasticsearch.node.Node;
> >>>>>> +import org.elasticsearch.node.internal.InternalSettingsPreparer;
> >>>>>> +import org.elasticsearch.plugins.Plugin;
> >>>>>> +import org.elasticsearch.script.groovy.GroovyPlugin;
> >>>>>> +
> >>>>>> +import java.io.File;
> >>>>>> +import java.util.Arrays;
> >>>>>> +import java.util.Collection;
> >>>>>> +import java.util.Collections;
> >>>>>> +
> >>>>>> +/**
> >>>>>> + * Represents a single elastic search node which can run embedded
> in a
> >>>>> java application.
> >>>>>> + * Intended for unit and integration tests. Settings and plugins
> are
> >>>>> crafted for Calcite.
> >>>>>> + */
> >>>>>> +class EmbeddedElasticNode implements AutoCloseable {
> >>>>>> +
> >>>>>> +  private final LocalNode node;
> >>>>>> +  private volatile boolean  isStarted;
> >>>>>> +
> >>>>>> +  private EmbeddedElasticNode(LocalNode node) {
> >>>>>> +    this.node = Preconditions.checkNotNull(node, "node");
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  /**
> >>>>>> +   * Having separate class to expose (protected) constructor which
> >>>>> allows to install
> >>>>>> +   * different plugins. In our case it is {@code GroovyPlugin} for
> >>>>> scripted fields like
> >>>>>> +   * {@code loc[0]} or {@code loc[1]['foo']}.
> >>>>>> +   */
> >>>>>> +  private static class LocalNode extends Node {
> >>>>>> +    private LocalNode(Settings settings, Collection<Class<? extends
> >>>>> Plugin>> classpathPlugins) {
> >>>>>> +      super(InternalSettingsPreparer.prepareEnvironment(settings,
> >>> null),
> >>>>>> +              Version.CURRENT,
> >>>>>> +              classpathPlugins);
> >>>>>> +    }
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  /**
> >>>>>> +   * Creates an instance with existing settings
> >>>>>> +   */
> >>>>>> +  private static EmbeddedElasticNode create(Settings settings) {
> >>>>>> +    // ensure GroovyPlugin is installed or otherwise scripted
> fields
> >>>>> would not work
> >>>>>> +    LocalNode node = new LocalNode(settings,
> >>>>> Collections.singleton(GroovyPlugin.class));
> >>>>>> +    return new EmbeddedElasticNode(node);
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  /**
> >>>>>> +   * Creates elastic node as single member of a cluster. Node will
> not
> >>>>> be started
> >>>>>> +   * unless {@link #start()} is explicitly called.
> >>>>>> +   */
> >>>>>> +  public static EmbeddedElasticNode create() {
> >>>>>> +    File data = Files.createTempDir();
> >>>>>> +    data.deleteOnExit();
> >>>>>> +    File home = Files.createTempDir();
> >>>>>> +    home.deleteOnExit();
> >>>>>> +
> >>>>>> +    Settings settings = Settings.builder()
> >>>>>> +            .put("node.name", "fake-elastic")
> >>>>>> +            .put("path.home", home.getAbsolutePath())
> >>>>>> +            .put("path.data", data.getAbsolutePath())
> >>>>>> +            .put("script.inline", true)  // requires GroovyPlugin
> >>>>>> +            .put("script.indexed", true) // requires GroovyPlugin
> >>>>>> +
> .put("cluster.routing.allocation.disk.threshold_enabled",
> >>>>> false)
> >>>>>> +            .put("node.local", true)
> >>>>>> +            .put("node.data", true)
> >>>>>> +            .put("network.host", "localhost")
> >>>>>> +            .build();
> >>>>>> +
> >>>>>> +    return create(settings);
> >>>>>> +  }
> >>>>>> +
> >>>>>> +  /**
> >>>>>> +   * Starts current node
> >>>>>> +   */
> >>>>>> +  public void start() {
> >>>>>> +    Preconditions.checkState(!isStarted, "already started");
> >>>>>> +    node.start();
> >>>


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-calcite-development/msg03840.html on line 1411
Call Stack
#TimeMemoryFunctionLocation
10.0016450776{main}( ).../msg03840.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-calcite-development/msg03840.html on line 1411
Call Stack
#TimeMemoryFunctionLocation
10.0016450776{main}( ).../msg03840.html:0