git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: A "Kamel" crazy idea


++1


---
Luca Burgazzoli


On Tue, Jul 31, 2018 at 9:47 AM, Andrea Cosentino
<ancosen1985@xxxxxxxxx.invalid> wrote:
> This is a wonderful idea and I believe we can improve our community with a subproject like this.
>
> So +1 for me.
>
> --
> Andrea Cosentino
> ----------------------------------
> Apache Camel PMC Chair
> Apache Karaf Committer
> Apache Servicemix PMC Member
> Email: ancosen1985@xxxxxxxxx
> Twitter: @oscerd2
> Github: oscerd
>
>
>
>
>
>
> On Monday, July 30, 2018, 5:49:50 PM GMT+2, Nicola Ferraro <ni.ferraro@xxxxxxxxx> wrote:
>
>
>
>
>
> Hi Cameleers,
> it seems from the comments that this "Kamel" subproject is something we
> want to start and I think that also the main camel core will benefit from
> the new features it will bring.
>
> I would like to donate the current "Kamel" code to Apache Camel, in order
> to have a initial brick to start from.
>
> From your reactions, the name "Kamel" seems cool and I'd like to keep it.
> However if you talk to people about the "Kamel" project, they won't
> understand if you're talking about "Kamel with the K" or "Camel with the C".
> For this reasons I propose to keep "Kamel", but also use "Camel K" [ˈkæməl
> keɪ] as friendly name when talking about it. This is in line with the
> repository we should create for the subproject, that needs to start with
> "camel-" (it will be "apache/camel-k" in that case).
>
> These days I've been experimenting different ideas with Luca Burgazzoli. In
> particular, there have been some concerns here on the power of a
> declarative DSL (a new one or the existing XML one) and we've found a
> strategy that will allow us to use also the Java DSL in Kamel integrations.
> Without adding too many details here, at integration build time we can run
> the user code in a build container and inspect the produced routes to get
> metadata for the optimizer. This is one of the first things we should do
> next.
>
> This possibility of running code that produces integrations opened a lot of
> unexpected paths, that I've written down in the project roadmap (
> https://github.com/nicolaferraro/integration-operator) and summarized below.
>
> One nice feature (kudos to Luca) is that we can simplify life for Kamel
> users up to the point that they'll just need to write their routes on a
> Java file and run them with e.g.:
> "kamel run Routes.java"
>
> Where "kamel" is a binary we release within the project. Kubernetes custom
> resources will be used under the hood, but the "kamel" binary is a utility
> that will provide a user experience comparable to that of current
> serverless platforms (or even better).
>
> But there's not just that.
>
> Luca wrote this weekend a prototype for having a polyglot Camel (
> https://github.com/lburgazzoli/camel-routes-loader).
>
> With that, you will be able to e.g. write integrations in groovy and use
> the same Kamel engine:
> "kamel run routes.groovy"
>
> But also JavaScript:
> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>
> This is a prototype right now, but a similar approach based on GraalVM has
> a lot of potential, because users can not only use their preferred language
> to write routes and processors, but also bind them to functionalities
> available in their preferred libraries.
>
> How much this will be feasible depends on the adoption of GraalVM, but I'm
> seeing many frameworks adding metadata to make GraalVM work with
> reflection. We've also done some work in Camel, there are some Jiras for it
> and first tests made by Guillaume signal that it's something feasible, at
> least for the Camel core and a subset of components. GraalVM would be
> important also to reduce memory footprint and improve startup time, as
> already said.
>
> If the GraalVM approach works (it is working for other frameworks), instead
> of rewriting a subset of Camel in Go (as the original proposal mentions),
> we can just sanitize and recompile our existing codebase: this way we fully
> leverage the strength of Apache Camel.
>
> Another important thing in the roadmap is actually how we'll integrate with
> current FaaS platforms. We will evaluate together if it's better to have a
> tighter integration with some of them or to leverage knative for some
> aspects of Kamel...
>
> But, let's get started!
>
> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni.ferraro@xxxxxxxxx> ha scritto:
>
> It's clear to me that we need to add support for our existing XML DSL, that
> is powerful. But there are multiple reasons why I'd like to also "add" a
> limited yaml/json notation to "Kamel".
>
> The first one (and simplest) is that json/yaml is the primary encoding for
> all resources exchanged with the Kubernetes API server. I'm not saying that
> for this reason the Integration resource cannot contain a XML
> (/Java/Ballerina) section, but from a pure presentational point of view,
> having the possibility to write simple use cases (even a "hello world") in
> pure json/yaml is better than always requiring a mix of json and XML.
>
> The second one is simplicity. Writing a optimizer (the module that
> materializes the integration into running code, choosing a specific runtime
> and configuration) for a turing-complete language like Java is not easy:
> even if you manage to create a good parser, it may be able to do
> optimizations only if you write routes in a particular way, without complex
> constructs.
> XML is ok from this point of view. The thing is that both json/yaml and XML
> are just two different ways to serialize object trees, that can be then
> statically analyzed.
> The point is not XML vs json/yaml, it's more about tailoring a new
> minimalistic DSL to the emerging use cases vs proposing "only" our classic
> way of writing integrations. I think XML can be the "advanced" way. We can
> experiment optimizations easily with the new DSL, and enable them also on
> XML if it's worth.
>
> Scripting should be part of the spec, but I'd try to use programming
> languages only for processing/transformation, not for the route definition.
>
> Third one is performance. Apart from the fact that json parsers are in
> general said to be faster than XML parsers... Given the "fast startup"
> target that we want to reach, we may think e.g. to translate the new DSL
> into Java or Go code, then compiling it. This allows doing parsing at build
> time in order to avoid it on startup. This kind of improvements are much
> easier with a limited DSL but much more difficult with a existing
> fully-fledged DSL..
>
> Nicola
>
>
>
> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ullgren@xxxxxxxxx> wrote:
>
>> I also like the idea but with some comments.
>>
>> As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
>> Perhaps a more fluent DSL like the Camel Java DSL or perhaps something like
>> Ballerina language would be better suited ?
>> Also in my experience even simple integrations, that is simple real world
>> integration and not just hello world demos, requires
>> you to add one or more Java class or scripting in addition to the core
>> components. So for it to be useful there must be some way to add custom
>> code in some way for aggregation/enrichment strategies.
>>
>> If you go with a GO implementation I would also like some fallback for
>> using Java since I find it unlikely that the existing base of components
>> will be ported any time soon.
>>
>> As for the name as other I like the "Kamel"  name however a being a newborn
>> kotlin fan I know of one pre-existing project with that name
>> that is close enough to allow for potential confusion.
>> https://github.com/topicusoverheid/kamel
>>
>> Looking forward to see the result.
>>
>> // Pontus
>>
>>
>>
>>
>>
>>
>> On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hiram@xxxxxxxxxxxxxxxx> wrote:
>>
>> > Love the idea.
>> >
>> > Personally, I'd keep using the existing Camel XML DSL if possible.  You
>> can
>> > still embed it in the CRD.  The operator that deploys it could inspect it
>> > and figure whats the most optimized runtime that can support the DSL.
>> > Perhaps if it's only using the restricted set of camel components
>> supported
>> > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses that.
>> > Otherwise it falls back to using camel spring boot.
>> >
>> > For that to work I think we need the different runtime implementations to
>> > provide a way to ask them: 'hey do you support running this camel route?'
>> > Not a trivial thing to respond to, it might require a build step in there
>> > for traditional Camel runtimes.
>> >
>> >
>> > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> antonin@xxxxxxxxxxxxx>
>> > wrote:
>> >
>> > > Hi Nicola,
>> > >
>> > > I love the idea.
>> > >
>> > > I just wonder whether YAML/JSON is an expressive enough format in the
>> > long
>> > > term. But as you’ve mentioned, starting simple would enable
>> experimenting
>> > > some very interesting / promising optimisations. So it seems worth
>> taking
>> > > that path, instead of trying to embed a complex DSL or the existing XML
>> > DSL
>> > > into the CRD.
>> > >
>> > > Definitely +1
>> > >
>> > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni.ferraro@xxxxxxxxx>
>> wrote:
>> > > >
>> > > > Hi Cameleers,
>> > > > it's now passed some time since I started thinking about a new
>> project
>> > > that
>> > > > we can begin here at Apache Camel, and I'd like to have your opinion.
>> > > >
>> > > > We've already been targeting cloud-native applications with Camel,
>> > > > especially on top of Kubernetes, that is becoming "the standard"
>> cloud
>> > > > platform. But writing a Camel integration and running it on
>> Kubernetes
>> > > > requires some effort: choosing the base platform (spring-boot, karaf,
>> > > > simple main?), adding health checks (actuator?), packaging a docker
>> > image
>> > > > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
>> > > > publishing the image on a docker registry, then finally deploying the
>> > > > resources on a Kubernetes cluster.
>> > > >
>> > > > The resulting integration container is then far from being optimal
>> > from a
>> > > > resource consumption point of view: it is likely that a Camel
>> > Spring-Boot
>> > > > application will require at least 200MB of RAM and also some CPU
>> shares
>> > > > because of polling threads used by many components.
>> > > >
>> > > > In case people use a CI/CD pipeline, it will take also a long time to
>> > get
>> > > > from a code update to having a Kubernetes POD up and running.
>> > > > Apart from compilation and image push/pull time, also startup time is
>> > > often
>> > > > ~10 seconds for Camel + Spring-Boot in a container with standard
>> limits
>> > > on
>> > > > resources, making it difficult to propose this combination for
>> > > "serverless
>> > > > integration" (this term is becoming increasingly more popular).
>> > > >
>> > > > So, my proposal is to start to investigate a "more cloud-native"
>> > approach
>> > > > to integration: *making Camel integrations first-class citizens in
>> > > > Kubernetes, and making them super fast and lightweight.*
>> > > >
>> > > > We can base the project on Kubernetes Custom Resource Definitions
>> (CRD)
>> > > > <
>> > >
>> >
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> > > >,
>> > > > for example a Integration CRD and have a Kubernetes "operator"
>> > > > <https://coreos.com/operators/> taking care of:
>> > > > - Optimizing the integration that we want to run
>> > > > - Packaging in a container
>> > > > - Running it on Kubernetes
>> > > > - Managing its entire lifecycle
>> > > >
>> > > > A Kubernetes-native integration may look like:
>> > > >
>> > > > -------------------
>> > > > kind: "Integration"
>> > > > apiVersion: "camel.apache.org/v1alpha1"
>> > > > metadata:
>> > > > name: "example"
>> > > > spec:
>> > > > replicas: 1
>> > > > routes:
>> > > >  - id: timer
>> > > >    route:
>> > > >    - type: endpoint
>> > > >      uri: timer:tick
>> > > >    - type: endpoint
>> > > >      uri: log:info
>> > > > -------------------
>> > > >
>> > > > For those who are not familiar with Kubernetes resources, this kind
>> of
>> > > > YAML/JSON resource definitions are really common.
>> > > > The example route is embedded in the Kubernetes resource declaration
>> > and
>> > > > follows a basic "flow DSL". We may start from a basic one and evolve
>> it
>> > > as
>> > > > new requirements arrive from the community.
>> > > >
>> > > > I've made a very simple (but working) POC here:
>> > > > https://github.com/nicolaferraro/integration-operator.
>> > > >
>> > > > This idea of a "Cloud-Native Camel" on Kubernetes (project codename
>> can
>> > > be "
>> > > > *Kamel*", if you like it :D), will be an enabler for a lot of nice
>> > > features.
>> > > >
>> > > > For example, we can propose "Kamel" as "ideal" platform for
>> "serverless
>> > > > integration" (I see many people reinventing the wheel out there): the
>> > > > operator can reduce resource consumption of a single integration by
>> > > > optimizing the runtime and also pause/resume integrations when they
>> are
>> > > not
>> > > > used, that is the basic idea behind "serverless" (e.g. think to
>> > > > HTTP-triggered integrations, but not only).
>> > > > Focusing on serverless will bring more emphasis on push-based
>> > > notifications
>> > > > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
>> > > used in
>> > > > Camel components, that prefer a poll based approach being it simpler
>> to
>> > > use
>> > > > in classic deployments, but not so good in the cloud, where more
>> > > resources
>> > > > become higher direct costs for the users.
>> > > >
>> > > > The presence of the simplified DSL enables also experimenting on
>> > > "*reduced*
>> > > > subsets of Camel" implemented in languages other than Java, for
>> example
>> > > one
>> > > > language that has a reactive approach on thread scheduling and a
>> really
>> > > low
>> > > > memory footprint, like Go.
>> > > >
>> > > > But apart from this kind of experiments (that are valid IMO), the
>> > "Kamel"
>> > > > optimizer will have free room to choose the right platform for the
>> > > > integration that the user wants to run, including, in the future,
>> doing
>> > > AOT
>> > > > compilation using Graal/VM (less memory, faster startup) if the
>> > features
>> > > > (components) used in the integration are supporting it (maybe we can
>> > add
>> > > > AOT compilation in the roadmap for Camel 3).
>> > > > A silly optimization: integrations starting from "timer:..." may be
>> > > > scheduled directly with Kubernetes CronJobs, so they will consume
>> > > resources
>> > > > only when actually running.
>> > > >
>> > > > Being the final integrations lightweight and being the DSL
>> > > > language-independent, we may see a increased adoption of Camel also
>> as
>> > > > agile integration layer for not-only-java applications (both "cloud"
>> > and
>> > > > "serverless" applications).
>> > > >
>> > > > I'm the first one that would like to work on a project ilke this.
>> I've
>> > > > worked on many Kubernetes/Openshift based applications and frameworks
>> > in
>> > > > the past years, also on operators and CRDs, and I think this way of
>> > > > redesigning integrations has a lot of potential.
>> > > >
>> > > > Integrations will not be necessarily limited to the simplified DSL,
>> but
>> > > we
>> > > > can add extension points for scripting and even custom libraries
>> > > (although
>> > > > limiting the freedom of the optimizer).
>> > > >
>> > > > The most important thing: it may become a great project, since it's
>> > > driven
>> > > > by a great community.
>> > > >
>> > > > So, what do you think? Is it crazy enough?
>> > > >
>> > > > Nicola
>> > >
>> > >
>> >
>> > --
>> > Hiram Chirino
>> > Engineering | Red Hat, Inc.
>> > hchirino@xxxxxxxxxx | fusesource.com | redhat.com
>> > skype: hiramchirino | twitter: @hiramchirino
>> >
>>