git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table


> 1) What's the time horizon for the current adapter no longer working with these
changes to ES ?
Current adapter will be working for a while with existing setup. The
problem is nomenclature and ease of use.

Their new SQL concepts mapping
<https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html>
drops
the notion of ES type (which before was equivalent of RDBMS table) and uses
ES index as new table equivalent (before ES index was equal to database).
Most users use elastic this way (one type , one index) index == table.

Currently calcite requires schema per index. In RDBMS parlance database per
table (I'd like to change that).

> 2) Any guess how complicated it would be to maintain code paths for both
> behaviours? I know this is probably really challenging to estimate, but I
> really have no idea of the scope of these changes. Would it mean two
> different ES adapters?

One can have just a separate calcite schema implementations (same adapter /
module) :
1)  LegacySchema (old). Schema can have only one index (but multiple
types). Type == table in this case.
2)  NewSchema (new). Single schema can have multiple indexes (type is
dropped). Index == table in this case

> 3) Do we really need compatibility with the current version of the
adapter?
> IMO this depends on what versions of ES we would lose support for and how
> complex it would be for users of the current ES adapter to make updates
for
> any Calcite API changes.

The issue is not in adapter but how calcite schema exposes tables.  Should
it expose index as individual table (new), or ES type (old) ?

Andrei.

On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <mmior@xxxxxxxxxx> wrote:

> Unfortunately I know very little about ES so I'm not in a great position to
> asses the impact of these changes. I will say that that legacy
> compatibility is great, but maintaining two sets of logic is always a
> challenge. A few follow up questions:
>
> 1) What's the time horizon for the current adapter no longer working with
> these changes to ES?
>
> 2) Any guess how complicated it would be to maintain code paths for both
> behaviours? I know this is probably really challenging to estimate, but I
> really have no idea of the scope of these changes. Would it mean two
> different ES adapters?
>
> 3) Do we really need compatibility with the current version of the adapter?
> IMO this depends on what versions of ES we would lose support for and how
> complex it would be for users of the current ES adapter to make updates for
> any Calcite API changes.
>
> Thanks for your continued work on the ES adapter Andrei!
>
> --
> Michael Mior
> mmior@xxxxxxxxxx
>
>
>
> Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <andrei@xxxxxxxxx> a écrit :
>
> > Hello,
> >
> > Elastic announced
> > <
> >
> https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
> > >
> > that they will be deprecating mapping types in ES6 and indexes will be
> > single-typed only.
> >
> > Historical analogy <https://www.elastic.co/blog/index-vs-type> between
> > RDBMS and elastic was that index is equivalent to a database and type
> > corresponds to table in that database. In a couple of releases (ES6-8)
> this
> > shall not longer be true.
> >
> > Recent SQL addition
> > <https://www.elastic.co/blog/elasticsearch-6-3-0-released> to elastic
> > confirms
> > this trend
> > <
> >
> https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
> > >.
> > Index is equivalent to a table and there are no more ES types.
> >
> > I would like to propose to include this logic in Calcite ES adapter. IE,
> > expose each ES single-typed index as a separate table inside calcite
> > schema. This is in contrast to  current integration where schema can only
> > have a single index. Current approach forces you to create multiple
> schemas
> > to query single-typed indexes (on the same ES cluster).
> >
> > Legacy compatibility can always be controlled with configuration
> > parameters.
> >
> > Do you agree with such changes ? If yes, would you consider a PR ?
> >
> > Regards,
> > Andrei.
> >
>


( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-calcite-development/msg03910.html on line 179
Call Stack
#TimeMemoryFunctionLocation
10.0008368824{main}( ).../msg03910.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-calcite-development/msg03910.html on line 179
Call Stack
#TimeMemoryFunctionLocation
10.0008368824{main}( ).../msg03910.html:0