git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: A "Kamel" crazy idea


Love it so ++1

Btw, I did start some (early) experiments around alternative canel runtime
in my spare time:

- https://github.com/lburgazzoli/camel-go
- https://github.com/lburgazzoli/camel-go-examples/tree/master/example-yaml

Happy to make it part of the kamel initiative if you find it interesting.

On Fri, 13 Jul 2018 at 08:14, Sascha Dirbach <
sascha.dirbach@xxxxxxxxxxxxxxxxxxxxxx> wrote:

> Hi Nicola,
>
> +1
>
> Great idea, do you already have a concept/idea/sketch on how to deal
> with complex configurations? i.e. SSL certs for outgoing calls, XSLT
> transformations, etc.
>
> When I think about it, you could probably use configmaps/secrets to
> mount these config in the container an then reference them in the route.
>
> Best regards,
>
> Sascha
>
> Am 13.07.2018 um 01:30 schrieb Nicola Ferraro:
> > Hi Cameleers,
> > it's now passed some time since I started thinking about a new project
> that
> > we can begin here at Apache Camel, and I'd like to have your opinion.
> >
> > We've already been targeting cloud-native applications with Camel,
> > especially on top of Kubernetes, that is becoming "the standard" cloud
> > platform. But writing a Camel integration and running it on Kubernetes
> > requires some effort: choosing the base platform (spring-boot, karaf,
> > simple main?), adding health checks (actuator?), packaging a docker image
> > and creating the Kubernetes resources (fabric8-maven-plugin, helm?),
> > publishing the image on a docker registry, then finally deploying the
> > resources on a Kubernetes cluster.
> >
> > The resulting integration container is then far from being optimal from a
> > resource consumption point of view: it is likely that a Camel Spring-Boot
> > application will require at least 200MB of RAM and also some CPU shares
> > because of polling threads used by many components.
> >
> > In case people use a CI/CD pipeline, it will take also a long time to get
> > from a code update to having a Kubernetes POD up and running.
> > Apart from compilation and image push/pull time, also startup time is
> often
> > ~10 seconds for Camel + Spring-Boot in a container with standard limits
> on
> > resources, making it difficult to propose this combination for
> "serverless
> > integration" (this term is becoming increasingly more popular).
> >
> > So, my proposal is to start to investigate a "more cloud-native" approach
> > to integration: *making Camel integrations first-class citizens in
> > Kubernetes, and making them super fast and lightweight.*
> >
> > We can base the project on Kubernetes Custom Resource Definitions (CRD)
> > <
> https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/
> >,
> > for example a Integration CRD and have a Kubernetes "operator"
> > <https://coreos.com/operators/> taking care of:
> > - Optimizing the integration that we want to run
> > - Packaging in a container
> > - Running it on Kubernetes
> > - Managing its entire lifecycle
> >
> > A Kubernetes-native integration may look like:
> >
> > -------------------
> > kind: "Integration"
> > apiVersion: "camel.apache.org/v1alpha1"
> > metadata:
> >  name: "example"
> > spec:
> >  replicas: 1
> >  routes:
> >   - id: timer
> >     route:
> >      - type: endpoint
> >        uri: timer:tick
> >      - type: endpoint
> >        uri: log:info
> > -------------------
> >
> > For those who are not familiar with Kubernetes resources, this kind of
> > YAML/JSON resource definitions are really common.
> > The example route is embedded in the Kubernetes resource declaration and
> > follows a basic "flow DSL". We may start from a basic one and evolve it
> as
> > new requirements arrive from the community.
> >
> > I've made a very simple (but working) POC here:
> > https://github.com/nicolaferraro/integration-operator.
> >
> > This idea of a "Cloud-Native Camel" on Kubernetes (project codename can
> be "
> > *Kamel*", if you like it :D), will be an enabler for a lot of nice
> features.
> >
> > For example, we can propose "Kamel" as "ideal" platform for "serverless
> > integration" (I see many people reinventing the wheel out there): the
> > operator can reduce resource consumption of a single integration by
> > optimizing the runtime and also pause/resume integrations when they are
> not
> > used, that is the basic idea behind "serverless" (e.g. think to
> > HTTP-triggered integrations, but not only).
> > Focusing on serverless will bring more emphasis on push-based
> notifications
> > (webhooks, cloud events <https://cloudevents.io/>), that are rarely
> used in
> > Camel components, that prefer a poll based approach being it simpler to
> use
> > in classic deployments, but not so good in the cloud, where more
> resources
> > become higher direct costs for the users.
> >
> > The presence of the simplified DSL enables also experimenting on
> "*reduced*
> > subsets of Camel" implemented in languages other than Java, for example
> one
> > language that has a reactive approach on thread scheduling and a really
> low
> > memory footprint, like Go.
> >
> > But apart from this kind of experiments (that are valid IMO), the "Kamel"
> > optimizer will have free room to choose the right platform for the
> > integration that the user wants to run, including, in the future, doing
> AOT
> > compilation using Graal/VM (less memory, faster startup) if the features
> > (components) used in the integration are supporting it (maybe we can add
> > AOT compilation in the roadmap for Camel 3).
> > A silly optimization: integrations starting from "timer:..." may be
> > scheduled directly with Kubernetes CronJobs, so they will consume
> resources
> > only when actually running.
> >
> > Being the final integrations lightweight and being the DSL
> > language-independent, we may see a increased adoption of Camel also as
> > agile integration layer for not-only-java applications (both "cloud" and
> > "serverless" applications).
> >
> > I'm the first one that would like to work on a project ilke this. I've
> > worked on many Kubernetes/Openshift based applications and frameworks in
> > the past years, also on operators and CRDs, and I think this way of
> > redesigning integrations has a lot of potential.
> >
> > Integrations will not be necessarily limited to the simplified DSL, but
> we
> > can add extension points for scripting and even custom libraries
> (although
> > limiting the freedom of the optimizer).
> >
> > The most important thing: it may become a great project, since it's
> driven
> > by a great community.
> >
> > So, what do you think? Is it crazy enough?
> >
> > Nicola
> >
>
> --
> Sascha Dirbach
>
> Inhaber
>
> endless webservices
> Marco Paetschke & Sascha Dirbach GbR
>
> Kirchweg 113
> 28201 Bremen
>
> Mobil: +49 (0)160-94182103
> Mail: sascha.dirbach@xxxxxxxxxxxxxxxxxxxxxx
> Web: www.endless-webservices.de
>
> USt-IdNr.: DE310969215
>
>
> --
--
Luca Burgazzoli