git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: A "Kamel" crazy idea


On Thu, Aug 2, 2018 at 12:17 AM, Luca Burgazzoli <lburgazzoli@xxxxxxxxx> wrote:
> That’s very nice start, thx Guillaume
>

Yes this is really awesome to see this prototype in action.



> On Wed, 1 Aug 2018 at 20:15, Guillaume Nodet <gnodet@xxxxxxxxxx> wrote:
>
>> I've pushed a branch with graalvm experiments.
>>    https://github.com/apache/camel/tree/graalvm
>> In order to build the project, you need to use the GraalVM jdk and use the
>> following ~/.m2/toolchains.xml adapted to your path:
>>
>> <?xml version="1.0" encoding="UTF8"?>
>>
>> <toolchains>
>>
>>   <toolchain>
>>
>>     <type>jdk</type>
>>
>>     <provides>
>>
>>       <version>1.0.0-rc4</version>
>>
>>       <vendor>oracle</vendor>
>>
>>       <id>graalvm-ee-1.0.0-rc4</id>
>>
>>     </provides>
>>
>>     <configuration>
>>
>>
>>
>> <jdkHome>/Users/gnodet/Downloads/graalvm-ee-1.0.0-rc4/Contents/Home/</jdkHome>
>>
>>     </configuration>
>>
>>   </toolchain>
>>
>> </toolchains>
>>
>> Build and run:
>>  gnodet•camel*(*graalvm*)*» cd platforms/graalvm
>>  gnodet•camel/platforms/graalvm*(*graalvm*)*» mvn install
>> ...
>>  gnodet•camel/platforms/graalvm*(*graalvm*)*»
>> ./example/target/org.apache.camel.graalvm.main
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> 2.23.0-SNAPSHOT (CamelContext: camel-1) is starting
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - StreamCaching is
>> not in use. If using streams then its recommended to enable stream caching.
>> See more details at http://camel.apache.org/stream-caching.html
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Total 0 routes, of
>> which 0 are started
>>
>> [main] INFO org.apache.camel.graalvm.FastCamelContext - Apache Camel
>> 2.23.0-SNAPSHOT (CamelContext: camel-1) started in 0.006 seconds
>>
>>  Guillaume
>>
>> Le mar. 31 juil. 2018 à 09:44, Guillaume Nodet <gnodet@xxxxxxxxxx> a
>> écrit :
>>
>> >
>> >
>> > Le lun. 30 juil. 2018 à 17:49, Nicola Ferraro <ni.ferraro@xxxxxxxxx> a
>> > écrit :
>> >
>> >> Hi Cameleers,
>> >> it seems from the comments that this "Kamel" subproject is something we
>> >> want to start and I think that also the main camel core will benefit
>> from
>> >> the new features it will bring.
>> >>
>> >> I would like to donate the current "Kamel" code to Apache Camel, in
>> order
>> >> to have a initial brick to start from.
>> >>
>> >> From your reactions, the name "Kamel" seems cool and I'd like to keep
>> it.
>> >> However if you talk to people about the "Kamel" project, they won't
>> >> understand if you're talking about "Kamel with the K" or "Camel with the
>> >> C".
>> >> For this reasons I propose to keep "Kamel", but also use "Camel K"
>> [ˈkæməl
>> >> keɪ] as friendly name when talking about it. This is in line with the
>> >> repository we should create for the subproject, that needs to start with
>> >> "camel-" (it will be "apache/camel-k" in that case).
>> >>
>> >> These days I've been experimenting different ideas with Luca Burgazzoli.
>> >> In
>> >> particular, there have been some concerns here on the power of a
>> >> declarative DSL (a new one or the existing XML one) and we've found a
>> >> strategy that will allow us to use also the Java DSL in Kamel
>> >> integrations.
>> >> Without adding too many details here, at integration build time we can
>> run
>> >> the user code in a build container and inspect the produced routes to
>> get
>> >> metadata for the optimizer. This is one of the first things we should do
>> >> next.
>> >>
>> >> This possibility of running code that produces integrations opened a lot
>> >> of
>> >> unexpected paths, that I've written down in the project roadmap (
>> >> https://github.com/nicolaferraro/integration-operator) and summarized
>> >> below.
>> >>
>> >> One nice feature (kudos to Luca) is that we can simplify life for Kamel
>> >> users up to the point that they'll just need to write their routes on a
>> >> Java file and run them with e.g.:
>> >> "kamel run Routes.java"
>> >>
>> >> Where "kamel" is a binary we release within the project. Kubernetes
>> custom
>> >> resources will be used under the hood, but the "kamel" binary is a
>> utility
>> >> that will provide a user experience comparable to that of current
>> >> serverless platforms (or even better).
>> >>
>> >> But there's not just that.
>> >>
>> >> Luca wrote this weekend a prototype for having a polyglot Camel (
>> >> https://github.com/lburgazzoli/camel-routes-loader).
>> >>
>> >> With that, you will be able to e.g. write integrations in groovy and use
>> >> the same Kamel engine:
>> >> "kamel run routes.groovy"
>> >>
>> >> But also JavaScript:
>> >>
>> >>
>> https://github.com/lburgazzoli/camel-routes-loader/blob/ca986541f7c422ee02c21727cdfe4293d64a364e/src/test/resources/ext/camel/routes.js#L2
>> >>
>> >> This is a prototype right now, but a similar approach based on GraalVM
>> has
>> >> a lot of potential, because users can not only use their preferred
>> >> language
>> >> to write routes and processors, but also bind them to functionalities
>> >> available in their preferred libraries.
>> >>
>> >> How much this will be feasible depends on the adoption of GraalVM, but
>> I'm
>> >> seeing many frameworks adding metadata to make GraalVM work with
>> >> reflection. We've also done some work in Camel, there are some Jiras for
>> >> it
>> >> and first tests made by Guillaume signal that it's something feasible,
>> at
>> >> least for the Camel core and a subset of components. GraalVM would be
>> >> important also to reduce memory footprint and improve startup time, as
>> >> already said.
>> >>
>> >
>> > I'll continue experimenting and I'll report back.
>> > I did some initial experiments leveraging some modifications I did  to
>> > speed up the start up time [1] and on a single route [2].
>> > There may be some limitations down the road of course, but at least it
>> > shows that it's feasible. Fwiw, the experiments lead to a startup time of
>> > 14 ms compared to a few hundreds milliseconds (roughly 680 with [1]).
>> >
>> >
>> >>
>> >> If the GraalVM approach works (it is working for other frameworks),
>> >> instead
>> >> of rewriting a subset of Camel in Go (as the original proposal
>> mentions),
>> >> we can just sanitize and recompile our existing codebase: this way we
>> >> fully
>> >> leverage the strength of Apache Camel.
>> >>
>> >
>> > Right, if we can avoid rewriting Camel, that would be much better !
>> >
>> > [1] https://issues.apache.org/jira/browse/CAMEL-12688
>> > [2]
>> >
>> https://github.com/gnodet/openwhisk-runtime-camel/blob/master/camel-openwhisk-example/src/main/java/org/jboss/fuse/openwhisk/camel/example/SimpleCamelFunction.java
>> >
>> >
>> >>
>> >> Another important thing in the roadmap is actually how we'll integrate
>> >> with
>> >> current FaaS platforms. We will evaluate together if it's better to
>> have a
>> >> tighter integration with some of them or to leverage knative for some
>> >> aspects of Kamel...
>> >>
>> >> But, let's get started!
>> >>
>> >> Il 19 lug 2018 10:45, "Nicola Ferraro" <ni.ferraro@xxxxxxxxx> ha
>> scritto:
>> >>
>> >> It's clear to me that we need to add support for our existing XML DSL,
>> >> that
>> >> is powerful. But there are multiple reasons why I'd like to also "add" a
>> >> limited yaml/json notation to "Kamel".
>> >>
>> >> The first one (and simplest) is that json/yaml is the primary encoding
>> for
>> >> all resources exchanged with the Kubernetes API server. I'm not saying
>> >> that
>> >> for this reason the Integration resource cannot contain a XML
>> >> (/Java/Ballerina) section, but from a pure presentational point of view,
>> >> having the possibility to write simple use cases (even a "hello world")
>> in
>> >> pure json/yaml is better than always requiring a mix of json and XML.
>> >>
>> >> The second one is simplicity. Writing a optimizer (the module that
>> >> materializes the integration into running code, choosing a specific
>> >> runtime
>> >> and configuration) for a turing-complete language like Java is not easy:
>> >> even if you manage to create a good parser, it may be able to do
>> >> optimizations only if you write routes in a particular way, without
>> >> complex
>> >> constructs.
>> >> XML is ok from this point of view. The thing is that both json/yaml and
>> >> XML
>> >> are just two different ways to serialize object trees, that can be then
>> >> statically analyzed.
>> >> The point is not XML vs json/yaml, it's more about tailoring a new
>> >> minimalistic DSL to the emerging use cases vs proposing "only" our
>> classic
>> >> way of writing integrations. I think XML can be the "advanced" way. We
>> can
>> >> experiment optimizations easily with the new DSL, and enable them also
>> on
>> >> XML if it's worth.
>> >>
>> >> Scripting should be part of the spec, but I'd try to use programming
>> >> languages only for processing/transformation, not for the route
>> >> definition.
>> >>
>> >> Third one is performance. Apart from the fact that json parsers are in
>> >> general said to be faster than XML parsers... Given the "fast startup"
>> >> target that we want to reach, we may think e.g. to translate the new DSL
>> >> into Java or Go code, then compiling it. This allows doing parsing at
>> >> build
>> >> time in order to avoid it on startup. This kind of improvements are much
>> >> easier with a limited DSL but much more difficult with a existing
>> >> fully-fledged DSL..
>> >>
>> >> Nicola
>> >>
>> >>
>> >>
>> >> On Wed, Jul 18, 2018 at 11:51 PM Pontus Ullgren <ullgren@xxxxxxxxx>
>> >> wrote:
>> >>
>> >> > I also like the idea but with some comments.
>> >> >
>> >> > As Hiram Chirino I'm not sure YAML/JSON is the best language for this.
>> >> > Perhaps a more fluent DSL like the Camel Java DSL or perhaps something
>> >> like
>> >> > Ballerina language would be better suited ?
>> >> > Also in my experience even simple integrations, that is simple real
>> >> world
>> >> > integration and not just hello world demos, requires
>> >> > you to add one or more Java class or scripting in addition to the core
>> >> > components. So for it to be useful there must be some way to add
>> custom
>> >> > code in some way for aggregation/enrichment strategies.
>> >> >
>> >> > If you go with a GO implementation I would also like some fallback for
>> >> > using Java since I find it unlikely that the existing base of
>> components
>> >> > will be ported any time soon.
>> >> >
>> >> > As for the name as other I like the "Kamel"  name however a being a
>> >> newborn
>> >> > kotlin fan I know of one pre-existing project with that name
>> >> > that is close enough to allow for potential confusion.
>> >> > https://github.com/topicusoverheid/kamel
>> >> >
>> >> > Looking forward to see the result.
>> >> >
>> >> > // Pontus
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > On Mon, 16 Jul 2018 at 15:15 Hiram Chirino <hiram@xxxxxxxxxxxxxxxx>
>> >> wrote:
>> >> >
>> >> > > Love the idea.
>> >> > >
>> >> > > Personally, I'd keep using the existing Camel XML DSL if possible.
>> >> You
>> >> > can
>> >> > > still embed it in the CRD.  The operator that deploys it could
>> >> inspect it
>> >> > > and figure whats the most optimized runtime that can support the
>> DSL.
>> >> > > Perhaps if it's only using the restricted set of camel components
>> >> > supported
>> >> > > by camel-go (https://github.com/lburgazzoli/camel-go) then is uses
>> >> that.
>> >> > > Otherwise it falls back to using camel spring boot.
>> >> > >
>> >> > > For that to work I think we need the different runtime
>> >> implementations to
>> >> > > provide a way to ask them: 'hey do you support running this camel
>> >> route?'
>> >> > > Not a trivial thing to respond to, it might require a build step in
>> >> there
>> >> > > for traditional Camel runtimes.
>> >> > >
>> >> > >
>> >> > > On Fri, Jul 13, 2018 at 4:32 AM Antonin Stefanutti <
>> >> > antonin@xxxxxxxxxxxxx>
>> >> > > wrote:
>> >> > >
>> >> > > > Hi Nicola,
>> >> > > >
>> >> > > > I love the idea.
>> >> > > >
>> >> > > > I just wonder whether YAML/JSON is an expressive enough format in
>> >> the
>> >> > > long
>> >> > > > term. But as you’ve mentioned, starting simple would enable
>> >> > experimenting
>> >> > > > some very interesting / promising optimisations. So it seems worth
>> >> > taking
>> >> > > > that path, instead of trying to embed a complex DSL or the
>> existing
>> >> XML
>> >> > > DSL
>> >> > > > into the CRD.
>> >> > > >
>> >> > > > Definitely +1
>> >> > > >
>> >> > > > > On 13 Jul 2018, at 01:30, Nicola Ferraro <ni.ferraro@xxxxxxxxx>
>> >> > wrote:
>> >> > > > >
>> >> > > > > Hi Cameleers,
>> >> > > > > it's now passed some time since I started thinking about a new
>> >> > project
>> >> > > > that
>> >> > > > > we can begin here at Apache Camel, and I'd like to have your
>> >> opinion.
>> >> > > > >
>> >> > > > > We've already been targeting cloud-native applications with
>> Camel,
>> >> > > > > especially on top of Kubernetes, that is becoming "the standard"
>> >> > cloud
>> >> > > > > platform. But writing a Camel integration and running it on
>> >> > Kubernetes
>> >> > > > > requires some effort: choosing the base platform (spring-boot,
>> >> karaf,
>> >> > > > > simple main?), adding health checks (actuator?), packaging a
>> >> docker
>> >> > > image
>> >> > > > > and creating the Kubernetes resources (fabric8-maven-plugin,
>> >> helm?),
>> >> > > > > publishing the image on a docker registry, then finally
>> deploying
>> >> the
>> >> > > > > resources on a Kubernetes cluster.
>> >> > > > >
>> >> > > > > The resulting integration container is then far from being
>> optimal
>> >> > > from a
>> >> > > > > resource consumption point of view: it is likely that a Camel
>> >> > > Spring-Boot
>> >> > > > > application will require at least 200MB of RAM and also some CPU
>> >> > shares
>> >> > > > > because of polling threads used by many components.
>> >> > > > >
>> >> > > > > In case people use a CI/CD pipeline, it will take also a long
>> >> time to
>> >> > > get
>> >> > > > > from a code update to having a Kubernetes POD up and running.
>> >> > > > > Apart from compilation and image push/pull time, also startup
>> >> time is
>> >> > > > often
>> >> > > > > ~10 seconds for Camel + Spring-Boot in a container with standard
>> >> > limits
>> >> > > > on
>> >> > > > > resources, making it difficult to propose this combination for
>> >> > > > "serverless
>> >> > > > > integration" (this term is becoming increasingly more popular).
>> >> > > > >
>> >> > > > > So, my proposal is to start to investigate a "more cloud-native"
>> >> > > approach
>> >> > > > > to integration: *making Camel integrations first-class citizens
>> in
>> >> > > > > Kubernetes, and making them super fast and lightweight.*
>> >> > > > >
>> >> > > > > We can base the project on Kubernetes Custom Resource
>> Definitions
>> >> > (CRD)
>> >> > > > > <
>> >> > > >
>> >> > >
>> >> >
>> >>
>> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
>> >> > > > >,
>> >> > > > > for example a Integration CRD and have a Kubernetes "operator"
>> >> > > > > <https://coreos.com/operators/> taking care of:
>> >> > > > > - Optimizing the integration that we want to run
>> >> > > > > - Packaging in a container
>> >> > > > > - Running it on Kubernetes
>> >> > > > > - Managing its entire lifecycle
>> >> > > > >
>> >> > > > > A Kubernetes-native integration may look like:
>> >> > > > >
>> >> > > > > -------------------
>> >> > > > > kind: "Integration"
>> >> > > > > apiVersion: "camel.apache.org/v1alpha1"
>> >> > > > > metadata:
>> >> > > > > name: "example"
>> >> > > > > spec:
>> >> > > > > replicas: 1
>> >> > > > > routes:
>> >> > > > >  - id: timer
>> >> > > > >    route:
>> >> > > > >     - type: endpoint
>> >> > > > >       uri: timer:tick
>> >> > > > >     - type: endpoint
>> >> > > > >       uri: log:info
>> >> > > > > -------------------
>> >> > > > >
>> >> > > > > For those who are not familiar with Kubernetes resources, this
>> >> kind
>> >> > of
>> >> > > > > YAML/JSON resource definitions are really common.
>> >> > > > > The example route is embedded in the Kubernetes resource
>> >> declaration
>> >> > > and
>> >> > > > > follows a basic "flow DSL". We may start from a basic one and
>> >> evolve
>> >> > it
>> >> > > > as
>> >> > > > > new requirements arrive from the community.
>> >> > > > >
>> >> > > > > I've made a very simple (but working) POC here:
>> >> > > > > https://github.com/nicolaferraro/integration-operator.
>> >> > > > >
>> >> > > > > This idea of a "Cloud-Native Camel" on Kubernetes (project
>> >> codename
>> >> > can
>> >> > > > be "
>> >> > > > > *Kamel*", if you like it :D), will be an enabler for a lot of
>> nice
>> >> > > > features.
>> >> > > > >
>> >> > > > > For example, we can propose "Kamel" as "ideal" platform for
>> >> > "serverless
>> >> > > > > integration" (I see many people reinventing the wheel out
>> there):
>> >> the
>> >> > > > > operator can reduce resource consumption of a single integration
>> >> by
>> >> > > > > optimizing the runtime and also pause/resume integrations when
>> >> they
>> >> > are
>> >> > > > not
>> >> > > > > used, that is the basic idea behind "serverless" (e.g. think to
>> >> > > > > HTTP-triggered integrations, but not only).
>> >> > > > > Focusing on serverless will bring more emphasis on push-based
>> >> > > > notifications
>> >> > > > > (webhooks, cloud events <https://cloudevents.io/>), that are
>> >> rarely
>> >> > > > used in
>> >> > > > > Camel components, that prefer a poll based approach being it
>> >> simpler
>> >> > to
>> >> > > > use
>> >> > > > > in classic deployments, but not so good in the cloud, where more
>> >> > > > resources
>> >> > > > > become higher direct costs for the users.
>> >> > > > >
>> >> > > > > The presence of the simplified DSL enables also experimenting on
>> >> > > > "*reduced*
>> >> > > > > subsets of Camel" implemented in languages other than Java, for
>> >> > example
>> >> > > > one
>> >> > > > > language that has a reactive approach on thread scheduling and a
>> >> > really
>> >> > > > low
>> >> > > > > memory footprint, like Go.
>> >> > > > >
>> >> > > > > But apart from this kind of experiments (that are valid IMO),
>> the
>> >> > > "Kamel"
>> >> > > > > optimizer will have free room to choose the right platform for
>> the
>> >> > > > > integration that the user wants to run, including, in the
>> future,
>> >> > doing
>> >> > > > AOT
>> >> > > > > compilation using Graal/VM (less memory, faster startup) if the
>> >> > > features
>> >> > > > > (components) used in the integration are supporting it (maybe we
>> >> can
>> >> > > add
>> >> > > > > AOT compilation in the roadmap for Camel 3).
>> >> > > > > A silly optimization: integrations starting from "timer:..." may
>> >> be
>> >> > > > > scheduled directly with Kubernetes CronJobs, so they will
>> consume
>> >> > > > resources
>> >> > > > > only when actually running.
>> >> > > > >
>> >> > > > > Being the final integrations lightweight and being the DSL
>> >> > > > > language-independent, we may see a increased adoption of Camel
>> >> also
>> >> > as
>> >> > > > > agile integration layer for not-only-java applications (both
>> >> "cloud"
>> >> > > and
>> >> > > > > "serverless" applications).
>> >> > > > >
>> >> > > > > I'm the first one that would like to work on a project ilke
>> this.
>> >> > I've
>> >> > > > > worked on many Kubernetes/Openshift based applications and
>> >> frameworks
>> >> > > in
>> >> > > > > the past years, also on operators and CRDs, and I think this way
>> >> of
>> >> > > > > redesigning integrations has a lot of potential.
>> >> > > > >
>> >> > > > > Integrations will not be necessarily limited to the simplified
>> >> DSL,
>> >> > but
>> >> > > > we
>> >> > > > > can add extension points for scripting and even custom libraries
>> >> > > > (although
>> >> > > > > limiting the freedom of the optimizer).
>> >> > > > >
>> >> > > > > The most important thing: it may become a great project, since
>> >> it's
>> >> > > > driven
>> >> > > > > by a great community.
>> >> > > > >
>> >> > > > > So, what do you think? Is it crazy enough?
>> >> > > > >
>> >> > > > > Nicola
>> >> > > >
>> >> > > >
>> >> > >
>> >> > > --
>> >> > > Hiram Chirino
>> >> > > Engineering | Red Hat, Inc.
>> >> > > hchirino@xxxxxxxxxx | fusesource.com | redhat.com
>> >> > > skype: hiramchirino | twitter: @hiramchirino
>> >> > >
>> >> >
>> >>
>> >
>> >
>> > --
>> > ------------------------
>> > Guillaume Nodet
>> >
>> >
>>
>> --
>> ------------------------
>> Guillaume Nodet
>>
> --
> --
> Luca Burgazzoli



-- 
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2