git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Improvements to the Unified SQL Connector API


Hi Timo,

Thanks for putting together the proposal!
I really love the idea to combining solution for historic and recent data
and left some suggestions on that part.

Regarding the table type, e.g. for kafka streams, I agree with @hequn's
idea that it should be pretty much inferable from the SQL context.
I think there might be some questions need to be addressed when unifying
the definition, for example:
- Should a Kafka table used in "INSERT INTO" statement be used again in
"FROM" statement, and vise versa ?
- How to enforce checks in combo-table use cases ?
- Can user change the way a table is used (e.g. source/sink) in interactive
env such as sql-client ?

Thanks,
Rong

On Thu, Oct 4, 2018 at 7:31 AM Hequn Cheng <chenghequn@xxxxxxxxx> wrote:

> Hi,
>
> Thanks a lot for the proposal. I like the idea to unify table definitions.
> I think we can drop the table type since the type can be derived from the
> sql, i.e, a table be inserted can only be a sink table.
>
> I left some minor suggestions in the document, mainly include:
> - Maybe we also need to allow define properties for tables.
> - Support specify Computed Columns in a table
> - Support define keys for sources.
>
> Best, Hequn
>
>
> On Thu, Oct 4, 2018 at 4:09 PM Shuyi Chen <suez1224@xxxxxxxxx> wrote:
>
> > Thanks a lot for the proposal, Timo. I left a few comments. Also, it
> seems
> > the example in the doc does not have the table type (source, sink and
> both)
> > property anymore. Are you suggesting drop it? I think the table type
> > properties is still useful as it can restrict a certain connector to be
> > only source/sink, for example, we usually want a Kafka topic to be either
> > read-only or write-only, but not both.
> >
> > Shuyi
> >
> > On Mon, Oct 1, 2018 at 1:53 AM Timo Walther <twalthr@xxxxxxxxxx> wrote:
> >
> > > Hi everyone,
> > >
> > > as some of you might have noticed, in the last two releases we aimed to
> > > unify SQL connectors and make them more modular. The first connectors
> > > and formats have been implemented and are usable via the SQL Client and
> > > Java/Scala/SQL APIs.
> > >
> > > However, after writing more connectors/example programs and talking to
> > > users, there are still a couple of improvements that should be applied
> > > to unified SQL connector API.
> > >
> > > I wrote a design document [1] that discusses limitations that I have
> > > observed and consideres feedback that I have collected over the last
> > > months. I don't know whether we will implement all of these
> > > improvements, but it would be great to get feedback for a satisfactory
> > > API and for future priorization.
> > >
> > > The general goal should be to connect to external systems as convenient
> > > and type-safe as possible. Any feedback is highly appreciated.
> > >
> > > Thanks,
> > >
> > > Timo
> > >
> > > [1]
> > >
> > >
> >
> https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit?usp=sharing
> > >
> > >
> >
> > --
> > "So you have to trust that the dots will somehow connect in your future."
> >
>