Re: Plans for next Tranquility release
It is definitely true that there are some use cases where Tranquility makes
sense. It just hasn't attracted a dev community in the same way that Druid
itself has (Druid has a ton of active contributors and the contributor base
is growing, while Tranquillity has only had a few over its lifetime, and
those folks -- one of them is me -- are no longer actively working on it).
If anyone steps up as interested in maintaining it then the project will
spring to life once again. That could be you! If you are interested, please
dive in and we will totally support you.
Fwiw, Tranquility is not quite at-least-once; if you lose all your Druid
realtime tasks, then some data will be dropped. It's neither at-most-once
nor at-least-once since it can double up data but it can also drop data.
On Wed, May 9, 2018 at 1:33 AM, Dyana Rose <dyana.rose@xxxxxxxxxxxxx> wrote:
> > It still does make sense if you don't need delivery guarantees and you
> don't want to run Kafka.
> To add a data point, as some talk about Tranquility recently has had me
> wondering if the project is going to be abandoned, this is our situation.
> Druid is just an endpoint of a longer at-least-once pipeline.
> We distribute to Druid through embedded Tranquility as one endpoint of a
> data-distribution hub. I would rather not have to add Kafka to our stack
> just to stream data into Druid.
> The listed benefits of Kafka:
> exactly-once (we already are at-least-once)
> no external service (already have one running anyway, and adding kafka
> would then be a new external service)
> reading late data (we take care of this in the de-dup/data cleansing
> process, a process which is necessary even with kafka)
> I have to assume we're not the only people with an at-least-once system,
> where the benefits of adding Kafka look to be low.
> On 8 May 2018 at 20:05, Gian Merlino <gian@xxxxxxxxxx> wrote:
> > Hi Stig,
> > Tranquility seems to be in the "if it ain't broke don't fix it" stage of
> > its lifecycle. It still works great with the latest Druid releases.
> > Although, nobody has seemed to be too interested in taking over its
> > maintenance and adding new features. At Imply we do still support it for
> > our users, and if it stopped working with a new Druid release then at
> > point we would be motivated to patch it.
> > However, most of the community effort into streaming ingestion work in
> > past couple of years has gone into Kafka indexing (
> > http://druid.io/docs/latest/development/extensions-core/
> > kafka-ingestion.html).
> > It has a number of benefits over Tranquility, namely that it is capable
> > exactly once ingestion, can read late data, and doesn't require external
> > processes (although I can see how the latter one is flipped in the Storm
> > case). It has also generally been much more active dev-wise than
> > Tranquility.
> > What I usually do is direct people to the Kafka indexing method and
> > its benefits, and then if they still want to use Tranquillity (maybe they
> > just really don't like Kafka) then they can go for it. It still does make
> > sense if you don't need delivery guarantees and you don't want to run
> > Kafka.
> > On Tue, May 8, 2018 at 9:50 AM, Stig Rohde Døssing <
> > >
> > wrote:
> > > Hi Druid devs,
> > >
> > > We're using Tranquility in the Storm project to allow users to write to
> > > Druid from Storm (
> > > https://github.com/apache/storm/tree/master/external/storm-druid), and
> > > we've noticed that the library hasn't had a release for a few years.
> > >
> > > Looking at the Druid mailing list and Tranquility issue tracker, it's
> > > obvious to me whether Tranquility will be developed going forward (
> > > https://lists.apache.org/thread.html/ea60ba5294de5eb39761c8fd1c9a88
> > > 54d2ffe4a212394078e0e67f54@%3Cdev.druid.apache.org%3E,
> > > https://github.com/druid-io/tranquility/pull/233), or whether it is
> > > compatible with the latest Druid releases.
> > >
> > > Are there plans to release a new version of Tranquility any time in the
> > > near future? Is Tranquility compatible with the newest Druid releases?
> > Does
> > > it make sense for Storm to continue to support integration via the
> > > Tranquility library, or should we go another way (e.g. direct users to
> > > ingest via Kafka instead)?
> > >
> > > Thanks
> > >
> Dyana Rose
> Software Engineer
> W: www.salecycle.com <http://www.salecycle.com/>
> [image: Marketing Permissions Service]