git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Apache CloudStack Generic Key-Value and Communication Plug-in


Interesting. Have you thought about packaging everything together? I mean,
incorporating the whole plugin in ACS. Then, ACS manage the ES via one new
system VM type (OSVM – Object store system virtual machine), and another
system VM could be used to run the standalone service, which could execute
not just the authorization, but also the actual delivery.  This would then,
offload the delivery of data out of the MS, and would make it feasible to
deliver long binary data without much problem. Then, the MS would only have
APIs to register and delete data. The registration would then return an URL
that the user could use to do the actual upload of the key/value to the ES
via the standalone server or directly to the OSVM.

I do like the plugin you designed, but I would like to see more of these
services to get upstream and to work out of the box. Cloud nowadays is made
of so much more services than just IaaS, and we should try to create a more
comprehensive solution.

On Fri, Sep 21, 2018 at 8:15 AM, Ivan Kudryavtsev <kudryavtsev_ia@xxxxxxxxx>
wrote:

> Hi, Rafael. Second option. ES is configured beforehand, then, when plugin
> is deployed, only endpoint is specified in ES, but actual VMs don't
> interact with ES directly, they interact with API server which is deployed
> in docker container. So, basically three components: ES, CS plugin
> (authorized access and storages management), standalone service (access if
> you know certain storage UUID and secret for it). So, ES is a single
> instance for all accounts. But every account have separate indices in ES to
> keep data secure, data isolation is maintained by API services. So, suppose
> answered well))
>
> пт, 21 сент. 2018 г., 18:04 Rafael Weingärtner <
> rafaelweingartner@xxxxxxxxx
> >:
>
> > That is interesting Ivan. You answer create another question.
> >
> > You say that the plugin uses an Elasticsearch instance and that a
> clustered
> > ES is supported as
> > well. Is the plugin going to manage (install this ES) or you (the
> operator)
> > will have to deploy and manage the ES and only configure/point the ES
> > instance to ACS?
> >
> >
> > On Thu, Sep 20, 2018 at 11:34 PM, Ivan Kudryavtsev <
> > kudryavtsev_ia@xxxxxxxxx
> > > wrote:
> >
> > > Hello, Daan, Rafael.
> > >
> > > Ok, let's discover some points.
> > >
> > > Rafael:
> > > All the stuff is managed in Elasticsearch, clustered ES is supported as
> > > well. As for now, value datatype is 'keyword' and doesn't fit well for
> > > blobs. But, actually, there are no practical limitations except we
> don't
> > > want press the CS server with streaming for blobs as it's design is not
> > > well designed for long lasting jobs like that.
> > >
> > > To adress the case we also plan to develop artifact storage which is
> > > intended to keep blobs with versioning and managed security. Probably
> > will
> > > use Ceph under the hood.
> > >
> > > Daan:
> > > Let's go with the cases. Actually, in the past I published the
> > announcement
> > > for log management plugin which intention is to provide access to VM
> logs
> > > from the API and other interfaces. Now we develop CLI utility for it.
> > That
> > > plugin helps to observe what logs are generate during VM deployment.
> > >
> > > This plugin (KV) helps to deploy complex topologies, like Docker Swarm,
> > > Kubernetes, MySQL Galera replicaset, DC/OS, monitor the state of the
> > > deployments and do trivial things:
> > >
> > > Observe current VM load, RAM availability, disk space. All these
> > parameters
> > > can be observed in historical perspective or last values.
> > >
> > > A quorum example. Say, you deploy two VMs programmatically and they
> have
> > to
> > > decide who is Docker master and who is worker, so you create shared
> > storage
> > > and both VMs place they UUID in 'order' key. They check the history for
> > > 'order' and find who is the first. Next, the first generates cluster
> > > initialization and places secret key to 'secret'. After, worker reads
> it
> > > and joins the cluster. Want to add a new VM to the cluster, just pass
> > that
> > > KV to it as well. Of cource, proper templates with bootstrap code are
> > > required.
> > >
> > > Actually, there are tons of very sophisticated deployments can be
> > > implemented in a very generic way with theese two plugins.
> > >
> > > Another piece of software to be developed is one-time accessor plugin
> > which
> > > is important for secure sharing of various credentials when multi
> tenant
> > > VMs are used.
> > >
> > > Have a good day.
> > >
> > >
> > > пт, 21 сент. 2018 г., 2:21 Rafael Weingärtner <
> > rafaelweingartner@xxxxxxxxx
> > > >:
> > >
> > > > That is very interesting Ivan!
> > > >
> > > > I have a few questions though:
> > > >
> > > >    - Do you have a limit on ”value” size? Can it be a binary of a
> file?
> > > >    - Where do you guys store these “keys” and ”values”?
> > > >
> > > >
> > > > On Wed, Sep 19, 2018 at 4:02 AM, Ivan Kudryavtsev <
> > > > kudryavtsev_ia@xxxxxxxxx>
> > > > wrote:
> > > >
> > > > > Hello, community.
> > > > >
> > > > > We glad to announce a new Open Source plug-in for Apache CloudStack
> > > which
> > > > > helps to maintain generic key-value storages. There are three types
> > are
> > > > > supported right now:
> > > > >
> > > > > - temporary KV with TTL (many per account);
> > > > > - persistent KV bound to VM (one per VM);
> > > > > - persistent KV bound to account (many per account).
> > > > >
> > > > > KV storages support a commit-log based approach where old values
> are
> > > not
> > > > > erased with new ones and it provides the access to the last
> > key/values
> > > > and
> > > > > to the history which helps to maintain things like quorums and
> other
> > > > > operations which require serialized access.
> > > > >
> > > > > The plugin uses Elasticsearch. I
> > > > >
> > > > > If you are interested to know more, contact our website for the
> full
> > > > > announcement and download details:
> > > > >
> > https://bitworks.software/en/2018-09-19-acs-vm-kv-storage-plugin.html
> > > > >
> > > > > All questions, improvements and considerations are welcome!
> > > > >
> > > > > --
> > > > > With best regards, Ivan Kudryavtsev
> > > > > Bitworks LLC
> > > > > Cell: +7-923-414-1515
> > > > > WWW: http://bitworks.software/ <http://bw-sw.com/>
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
>



-- 
Rafael Weingärtner