[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Cyborg][Ironic][Nova][Neutron][TripleO][Cinder] accelerators management

Hi Arkady and all,

    Good discussions and questions.

First, it is good to clarify what we mean by lifecycle management. It includes:

* Discovery: We need to get more than just the PCI IDs/addresses of devices. We would need their properties and features as well. This is especially the case for programmable devices, as the properties and changes can change over time, though the PCI ID may not.

* Scheduling: We would want to schedule the application that needs offload based on the properties/features discovered above.

* Programming/configuration and/or firmware update. More on this later.

* Health management: discover the health of a device, esp. if programming/configuration etc. fail.

* Inventory management: Track the fleet of accelerators based on their properties/features.

* Other aspects that I won't dwell on here.

In short, lifecycle management is more than just firmware update.

Secondly, regarding the difference between programming and firmware updates, some key questions are:

1. What does the device configuration do?

   A. Expose properties/features relevant to scheduling:

      Could be for a specific application or workload (e.g. apply a new AI algorithm)

      Or expose new/premium device features (e.g. enable more memory banks)

   B. Update general features not relevant to scheduling. E.g. fix a bug in BMC firmware.

2. When/how is the device configuration done?

   A. Dynamically: as instances are provisioned/retired, based on time of day, workload demand, etc. This would be part of OpenStack workflow.

   B. Statically: as part of the physical host configuration. This is typically done 'offline', perhaps in a maintenance window, often using external frameworks like Redfish/Ansible/Puppet/Chef/...

The combination 1A+2A is what I'd call programming, while 1B+2B is firmware update. I don't see a motivation for 1B+2A.

The case 1A+2B is interesting. It basically means that a programmable device is being treated like a fixed-function accelerator for a period of time before it gets reprogrammed offline. This model is being used in the industry today, esp. telcos. I am fine with calling this a 'firmware update' too.

There are some grey areas to consider. For example, many FPGA deployments are structured to have a 'shell', which is hardware logic that exposes some generic features like PCI and DMA, and a separate user/custom logic that is application/workload-specific. Would updating the shell qualify as 'programming' or a 'firmware update'? Today, it often falls under 2B, esp. if it requires server reboots. But it could conceivably come under 1A+2A as products and use cases evolve. IOW, what is called a firmware update today could become a programming update tomorrow.

Cyborg is designed for programming, i.e. 1A+2A. It can be used with Nova (to program devices as instances are provisioned/retired) or standalone (based on time of day, traffic patterns, etc.) Other cases (1A/1B + 2B) can be classified as firmware update and outside of Cyborg.


* Agree with Arkady that firmware updates should follow the server vendors' guidelines, and can/should be done as part of the server configuration.

* If the claim is that firmware updates, as defined above (i.e. 1A/1B + 2B), should be done by Ironic, I am fine with it.

* To reiterate, it is NOT enough to handle devices based only on their PCI IDs -- we should be able to use their features/properties for scheduling, inventory management, etc. This is extra true for programmable devices where features can change dynamically while PCI IDs potentially stay constant.

* Cyborg is designed for these devices and its stated role includes all other aspects of lifecycle management.

*  I see value in having Cyborg and Ironic work together, esp. for 1A+2B, where Ironic can do the 'firmware update' and Cyborg discovers the schedulable properties of the device.

> From: Arkady.Kanevsky at <Arkady.Kanevsky at>

> Sent: Friday, January 3, 2020 1:19 PM

> To: openstack-discuss at

> Subject: [Cyborg][Ironic][Nova][Neutron][TripleO][Cinder] accelerators management

> 1. Application user need to program a portion of the device ...


> 2. Administrator need to program the whole device for specific usage. That covers the scenario when device can only support single tenant or single use case.

Why does it have to be single-tenant or single use-case? For example, one could program an FPGA with an Open Vswitch implementation, which is shared by VMs from different tenants.

> That is done once during OpenStack deployment but may need reprogramming to configure device for different usage.

If the change exposes workload-specific or schedulable properties, this would not necessarily be a one-shot thing at deployment time.

> 3. Administrator need to setup device for its use, like burning specific FW on it. This is typically done as part of server life-cycle event.

With the definition of firmware update as above, I agree.

> The first 2 cases cover application life cycle of device usage.


> The last one covers device life cycle independently how it is used.

Here's where I beg to disagree. As I said, the term 'device lifecycle' is far broader than just firmware update.

> Managing life cycle of devices is Ironic responsibility,

Disagree here. To the best of my knowledge, Ironic handles devices based on PCI IDs. Cyborg is designed to go deeper for discovering device features/properties and utilize Placement for scheduling based on these.

> One cannot and should not manage lifecycle of server components independently.

If what you meant to say is: ' do not update device firmware independently of other server components', agreed.

> Managing server devices outside server management violates customer service agreements with server vendors and breaks server support agreements.


> Nova and Neutron are getting info about all devices and their capabilities from Ironic; that they use for scheduling

Hmm, this seems overly broad to me: not every deployment includes Ironic, and getting PCI IDs is not enough for scheduling and management.

> Finally we want Cyborg to be able to be used in standalone capacity, say for Kubernetes


> Thus, I propose that Cyborg cover use cases 1 & 2, and Ironic would cover use case 3

Use case 3 says "setup device for its use, like burning specific FW." With the definition of firmware above, I agree. Other aspects of lifecycle management, not covered by use cases 1 - 3, would come under Cyborg.

> Thus, move all device Life-cycle code from Cyborg to Ironic

To recap, there is more to device lifecycle than firmware update. I'd suggest the other aspects can remain in Cyborg.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>