Re: [tempest] what is a proper way to install a package into a vm instance spawned by tempest?
On Mon, Mar 23, 2020, at 10:51 AM, Roman Safronov wrote:
> Actually we need such test because we are testing Openstack as a whole
> product (including Neutron+openvswitch+OVN+Nova+libvirt+Octavia etc.)
> that's why I think neutron functional test would be not enough. We are
> creating tests covering scenarios that our customers tried to use and
> encountered issues.
> For example this is a bug reported downstream on an issue happened in
> this scenario: https://bugzilla.redhat.com/show_bug.cgi?id=1707241
> There were more reported issues on similar use case and we would like
> to catch such issues before the product is released.
Absolutely do what you need to test these use cases. I'm simply suggesting that the most reliable and effective way to do that may be to avoid virtualization because we don't have nested virt.
> Anyway, as I said this specific test runs stable in downstream CI on
> virtual multi node environments with nested virtualization. It usually
> runs with RHEL8 image but I also tried it with standard Ubuntu-18.04
> guest image used by upstream CI gates. The only issue is that keepalive
> package installation by 'apt install' for some reason does not work
> when running on upstream gates causing the test to be skipped. I just
> would like to understand if running 'apt install/yum install' inside
> VMs spawned by upstream tempest tests is not acceptable at all or I am
> missing something (maybe proxy settings?).
Our test nodes should have full access to the Internet. We typically advise against relying on Internet resources as they can be flaky. For this reason we do run package mirrors within each cloud region that hosts our test nodes.
My hunch is that the network topology set up by devstack and neutron has VMs on rfc 1918 subnets for floating IPs. These floating IPs are then not further NAT'd by the host's IP address so any outbound traffic originating from those IPs will have no route back to home.
> On Mon, Mar 23, 2020 at 5:36 PM Clark Boylan <cboylan at sapwetik.org> wrote:
> > On Sun, Mar 22, 2020, at 9:10 AM, Roman Safronov wrote:
> > > Hi,
> > >
> > > I wrote a tempest test
> > > <https://review.opendev.org/#/c/710975/https://review.opendev.org/#/c/710975/ <https://review.opendev.org/#/c/710975/https://review.opendev.org/%23/c/710975/>)> which requires keepalived to be running inside vm instance.
> > > The test tries to install keepalived package inside vm by running "apt
> > > install/yum install".
> > > However as I see in upstream gates logs this does not work while the
> > > same code works perfectly in our downstream CI using the same image.
> > >
> > > Do vm instances spawned on upstream gates during tests have a route to
> > > the internet?
> > > Is there an alternative way that I can use in order to install a
> > > package?
> > By default the tempest jobs use a cirros image. This is a very small, quick to boot linux without a package manager. If you need services to be running in the image typically you'll need to boot a more typical linux installation. Keep in mind that nested virt isn't on by default as it isn't available everywhere and has been flaky in the past. This makes these types of installations very small which may make reliable VRRP testing difficult.
> > Thinking out loud here, I'm not sure how much benefit there is to testing VRRP failures in this manner. Do we think that OpenStack sitting on top of libvirt and OVS will somehow produce different results with VRRP than using those tools as is? To put this another way: are we testing OpenStack or are we testing OVS and libvirt?
> > One option here may be to construct this as a Neutron functional test and run VRRP on Neutron networks without virtualization mixed in.
> > >
> > > Thanks in advance
> > >
> > > --
> > > ROMAN SAFRONOV