git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Plans for next Tranquility release


> It still does make sense if you don't need delivery guarantees and you
don't want to run Kafka.

To add a data point, as some talk about Tranquility recently has had me
wondering if the project is going to be abandoned, this is our situation.
Druid is just an endpoint of a longer at-least-once pipeline.

We distribute to Druid through embedded Tranquility as one endpoint of a
data-distribution hub. I would rather not have to add Kafka to our stack
just to stream data into Druid.

The listed benefits of Kafka:
exactly-once (we already are at-least-once)
no external service (already have one running anyway, and adding kafka
would then be a new external service)
reading late data (we take care of this in the de-dup/data cleansing
process, a process which is necessary even with kafka)

I have to assume we're not the only people with an at-least-once system,
where the benefits of adding Kafka look to be low.

Dyana

On 8 May 2018 at 20:05, Gian Merlino <gian@xxxxxxxxxx> wrote:

> Hi Stig,
>
> Tranquility seems to be in the "if it ain't broke don't fix it" stage of
> its lifecycle. It still works great with the latest Druid releases.
> Although, nobody has seemed to be too interested in taking over its
> maintenance and adding new features. At Imply we do still support it for
> our users, and if it stopped working with a new Druid release then at this
> point we would be motivated to patch it.
>
> However, most of the community effort into streaming ingestion work in the
> past couple of years has gone into Kafka indexing (
> http://druid.io/docs/latest/development/extensions-core/
> kafka-ingestion.html).
> It has a number of benefits over Tranquility, namely that it is capable of
> exactly once ingestion, can read late data, and doesn't require external
> processes (although I can see how the latter one is flipped in the Storm
> case). It has also generally been much more active dev-wise than
> Tranquility.
>
> What I usually do is direct people to the Kafka indexing method and stress
> its benefits, and then if they still want to use Tranquillity (maybe they
> just really don't like Kafka) then they can go for it. It still does make
> sense if you don't need delivery guarantees and you don't want to run
> Kafka.
>
> On Tue, May 8, 2018 at 9:50 AM, Stig Rohde Døssing <stigdoessing@xxxxxxxxx
> >
> wrote:
>
> > Hi Druid devs,
> >
> > We're using Tranquility in the Storm project to allow users to write to
> > Druid from Storm (
> > https://github.com/apache/storm/tree/master/external/storm-druid), and
> > we've noticed that the library hasn't had a release for a few years.
> >
> > Looking at the Druid mailing list and Tranquility issue tracker, it's not
> > obvious to me whether Tranquility will be developed going forward (
> > https://lists.apache.org/thread.html/ea60ba5294de5eb39761c8fd1c9a88
> > 54d2ffe4a212394078e0e67f54@%3Cdev.druid.apache.org%3E,
> > https://github.com/druid-io/tranquility/pull/233), or whether it is
> > compatible with the latest Druid releases.
> >
> > Are there plans to release a new version of Tranquility any time in the
> > near future? Is Tranquility compatible with the newest Druid releases?
> Does
> > it make sense for Storm to continue to support integration via the
> > Tranquility library, or should we go another way (e.g. direct users to
> > ingest via Kafka instead)?
> >
> > Thanks
> >
>



-- 

Dyana Rose
Software Engineer


W: www.salecycle.com <http://www.salecycle.com/>
[image: Marketing Permissions Service]
<https://t.xink.io/Tracking/Index/9LwBAKNtAAAwphkA0>