git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Elasticsearch Adapter. Removal of Mapping Types (by vendor). Index == Table


Is there an API to discover indexes? If there is, I'd suggest we allow a config option that to make the adapter discover the possible indexes. We'd still have to adapt the code a bit, but internally, the schema could just keep a cache of type name to index name map and be able to support both scenarios.


Mit freundlichen Grüßen,
------------------------------------------------------------------------
*Christian Beikov*
Am 29.06.2018 um 00:12 schrieb Andrei Sereda:
1) What's the time horizon for the current adapter no longer working with these
changes to ES ?
Current adapter will be working for a while with existing setup. The
problem is nomenclature and ease of use.

Their new SQL concepts mapping
<https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html>
drops
the notion of ES type (which before was equivalent of RDBMS table) and uses
ES index as new table equivalent (before ES index was equal to database).
Most users use elastic this way (one type , one index) index == table.

Currently calcite requires schema per index. In RDBMS parlance database per
table (I'd like to change that).

2) Any guess how complicated it would be to maintain code paths for both
behaviours? I know this is probably really challenging to estimate, but I
really have no idea of the scope of these changes. Would it mean two
different ES adapters?
One can have just a separate calcite schema implementations (same adapter /
module) :
1)  LegacySchema (old). Schema can have only one index (but multiple
types). Type == table in this case.
2)  NewSchema (new). Single schema can have multiple indexes (type is
dropped). Index == table in this case

3) Do we really need compatibility with the current version of the
adapter?
IMO this depends on what versions of ES we would lose support for and how
complex it would be for users of the current ES adapter to make updates
for
any Calcite API changes.
The issue is not in adapter but how calcite schema exposes tables.  Should
it expose index as individual table (new), or ES type (old) ?

Andrei.

On Thu, Jun 28, 2018 at 5:23 PM Michael Mior <mmior@xxxxxxxxxx> wrote:

Unfortunately I know very little about ES so I'm not in a great position to
asses the impact of these changes. I will say that that legacy
compatibility is great, but maintaining two sets of logic is always a
challenge. A few follow up questions:

1) What's the time horizon for the current adapter no longer working with
these changes to ES?

2) Any guess how complicated it would be to maintain code paths for both
behaviours? I know this is probably really challenging to estimate, but I
really have no idea of the scope of these changes. Would it mean two
different ES adapters?

3) Do we really need compatibility with the current version of the adapter?
IMO this depends on what versions of ES we would lose support for and how
complex it would be for users of the current ES adapter to make updates for
any Calcite API changes.

Thanks for your continued work on the ES adapter Andrei!

--
Michael Mior
mmior@xxxxxxxxxx



Le jeu. 28 juin 2018 à 12:57, Andrei Sereda <andrei@xxxxxxxxx> a écrit :

Hello,

Elastic announced
<

https://www.elastic.co/guide/en/elasticsearch/reference/master/removal-of-types.html
that they will be deprecating mapping types in ES6 and indexes will be
single-typed only.

Historical analogy <https://www.elastic.co/blog/index-vs-type> between
RDBMS and elastic was that index is equivalent to a database and type
corresponds to table in that database. In a couple of releases (ES6-8)
this
shall not longer be true.

Recent SQL addition
<https://www.elastic.co/blog/elasticsearch-6-3-0-released> to elastic
confirms
this trend
<

https://www.elastic.co/guide/en/elasticsearch/reference/current/_mapping_concepts_across_sql_and_elasticsearch.html
.
Index is equivalent to a table and there are no more ES types.

I would like to propose to include this logic in Calcite ES adapter. IE,
expose each ES single-typed index as a separate table inside calcite
schema. This is in contrast to  current integration where schema can only
have a single index. Current approach forces you to create multiple
schemas
to query single-typed indexes (on the same ES cluster).

Legacy compatibility can always be controlled with configuration
parameters.

Do you agree with such changes ? If yes, would you consider a PR ?

Regards,
Andrei.