[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Plans for next Tranquility release

> It still does make sense if you don't need delivery guarantees and you
don't want to run Kafka.

To add a data point, as some talk about Tranquility recently has had me
wondering if the project is going to be abandoned, this is our situation.
Druid is just an endpoint of a longer at-least-once pipeline.

We distribute to Druid through embedded Tranquility as one endpoint of a
data-distribution hub. I would rather not have to add Kafka to our stack
just to stream data into Druid.

The listed benefits of Kafka:
exactly-once (we already are at-least-once)
no external service (already have one running anyway, and adding kafka
would then be a new external service)
reading late data (we take care of this in the de-dup/data cleansing
process, a process which is necessary even with kafka)

I have to assume we're not the only people with an at-least-once system,
where the benefits of adding Kafka look to be low.


On 8 May 2018 at 20:05, Gian Merlino <gian@xxxxxxxxxx> wrote:

> Hi Stig,
> Tranquility seems to be in the "if it ain't broke don't fix it" stage of
> its lifecycle. It still works great with the latest Druid releases.
> Although, nobody has seemed to be too interested in taking over its
> maintenance and adding new features. At Imply we do still support it for
> our users, and if it stopped working with a new Druid release then at this
> point we would be motivated to patch it.
> However, most of the community effort into streaming ingestion work in the
> past couple of years has gone into Kafka indexing (
> kafka-ingestion.html).
> It has a number of benefits over Tranquility, namely that it is capable of
> exactly once ingestion, can read late data, and doesn't require external
> processes (although I can see how the latter one is flipped in the Storm
> case). It has also generally been much more active dev-wise than
> Tranquility.
> What I usually do is direct people to the Kafka indexing method and stress
> its benefits, and then if they still want to use Tranquillity (maybe they
> just really don't like Kafka) then they can go for it. It still does make
> sense if you don't need delivery guarantees and you don't want to run
> Kafka.
> On Tue, May 8, 2018 at 9:50 AM, Stig Rohde Døssing <stigdoessing@xxxxxxxxx
> >
> wrote:
> > Hi Druid devs,
> >
> > We're using Tranquility in the Storm project to allow users to write to
> > Druid from Storm (
> >, and
> > we've noticed that the library hasn't had a release for a few years.
> >
> > Looking at the Druid mailing list and Tranquility issue tracker, it's not
> > obvious to me whether Tranquility will be developed going forward (
> >
> >,
> >, or whether it is
> > compatible with the latest Druid releases.
> >
> > Are there plans to release a new version of Tranquility any time in the
> > near future? Is Tranquility compatible with the newest Druid releases?
> Does
> > it make sense for Storm to continue to support integration via the
> > Tranquility library, or should we go another way (e.g. direct users to
> > ingest via Kafka instead)?
> >
> > Thanks
> >


Dyana Rose
Software Engineer

W: <>
[image: Marketing Permissions Service]

( ! ) Warning: include(msgfooter.php): failed to open stream: No such file or directory in /var/www/git/apache-druid-developers/msg00158.html on line 158
Call Stack
10.0008368664{main}( ).../msg00158.html:0

( ! ) Warning: include(): Failed opening 'msgfooter.php' for inclusion (include_path='.:/var/www/git') in /var/www/git/apache-druid-developers/msg00158.html on line 158
Call Stack
10.0008368664{main}( ).../msg00158.html:0