git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Openstack] live_migration only using 8 Mb speed


Matt,

I am going to override following in user_variable.yml file in that
case do i need to run ./bootstrap-ansible.sh script?

## Nova service
nova_git_repo: https://git.openstack.org/openstack/nova
nova_git_install_branch: a9c9285a5a68ab89a6543d143c364d90a01cd51c #
HEAD of "stable/queens" as of 06.08.2018
nova_git_project_group: nova_all



On Thu, Aug 23, 2018 at 7:18 AM, Satish Patel <satish.txt at gmail.com> wrote:
> I'm testing this in lab, no load yet
>
> Sent from my iPhone
>
>> On Aug 23, 2018, at 2:30 AM, Matthew Thode <prometheanfire at gentoo.org> wrote:
>>
>>> On 18-08-22 23:04:57, Satish Patel wrote:
>>> Mathew,
>>>
>>> I haven't applied any patch yet but i am noticing in cluster some host
>>> migrating VM super fast and some host migrating very slow. Is this
>>> known behavior?
>>>
>>> On Wed, Aug 22, 2018 at 11:28 AM, Matthew Thode
>>> <prometheanfire at gentoo.org> wrote:
>>>> On 18-08-22 10:58:48, Satish Patel wrote:
>>>>> Matthew,
>>>>>
>>>>> I have two option looks like, correct me if i am wrong.
>>>>>
>>>>> 1. I have two option, upgrade minor release from 17.0.7-6-g9187bb1 to
>>>>> 17.0.8-23-g0aff517  and upgrade full OSA
>>>>>
>>>>> 2. Just do override as you said "nova_git_install_branch:" in my
>>>>> /etc/openstack_deploy/user_variables.yml  file, and run playbooks.
>>>>>
>>>>>
>>>>> I think option [2] is safe to just touch specific component, also am i
>>>>> correct about override in /etc/openstack_deploy/user_variables.yml
>>>>> file?
>>>>>
>>>>> You mentioned "nova_git_install_branch:
>>>>> dee99b1ed03de4b6ded94f3cf6d2ea7214bca93b" but i believe it should be
>>>>> "a9c9285a5a68ab89a6543d143c364d90a01cd51c" am i correct?
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 22, 2018 at 10:46 AM, Matthew Thode
>>>>> <prometheanfire at gentoo.org> wrote:
>>>>>> On 18-08-22 10:33:11, Satish Patel wrote:
>>>>>>> Thanks Matthew,
>>>>>>>
>>>>>>> Can i put that sha in my OSA at
>>>>>>> playbooks/defaults/repo_packages/openstack_services.yml by hand and
>>>>>>> run playbooks [repo/nova] ?
>>>>>>>
>>>>>>> On Wed, Aug 22, 2018 at 10:24 AM, Matthew Thode
>>>>>>> <prometheanfire at gentoo.org> wrote:
>>>>>>>> On 18-08-22 08:35:09, Satish Patel wrote:
>>>>>>>>> Currently in stable/queens i am seeing this sha
>>>>>>>>> https://github.com/openstack/openstack-ansible/blob/stable/queens/ansible-role-requirements.yml#L112
>>>>>>>>>
>>>>>>>>> On Wed, Aug 22, 2018 at 2:02 AM, Matthew Thode
>>>>>>>>> <prometheanfire at gentoo.org> wrote:
>>>>>>>>>> On 18-08-22 01:57:17, Satish Patel wrote:
>>>>>>>>>>> What I need to upgrade, any specific component?
>>>>>>>>>>>
>>>>>>>>>>> I have deployed openstack-ansible
>>>>>>>>>>>
>>>>>>>>>>> Sent from my iPhone
>>>>>>>>>>>
>>>>>>>>>>>>> On Aug 22, 2018, at 1:06 AM, Matthew Thode <prometheanfire at gentoo.org> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 18-08-22 01:02:53, Satish Patel wrote:
>>>>>>>>>>>>> Matthew,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for reply, Look like i don't have this patch
>>>>>>>>>>>>> https://review.openstack.org/#/c/591761/
>>>>>>>>>>>>>
>>>>>>>>>>>>> So i have to patch following 3 file manually?
>>>>>>>>>>>>>
>>>>>>>>>>>>> nova/tests/unit/virt/libvirt/test_driver.py213
>>>>>>>>>>>>> nova/tests/unit/virt/test_virt_drivers.py2
>>>>>>>>>>>>> nova/virt/libvirt/driver.py
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 22, 2018 at 12:42 AM, Matthew Thode
>>>>>>>>>>>>> <prometheanfire at gentoo.org> wrote:
>>>>>>>>>>>>>> On 18-08-22 00:27:08, Satish Patel wrote:
>>>>>>>>>>>>>>> Folks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I am running openstack queens and hypervisor is kvm, my live migration
>>>>>>>>>>>>>>> working fine. but somehow it stuck to 8 Mb network speed and taking
>>>>>>>>>>>>>>> long time to migrate 1G instance.  I have 10Gbps network and i have
>>>>>>>>>>>>>>> tried to copy 10G file between two compute node and it did copy in 2
>>>>>>>>>>>>>>> minute, so i am not seeing any network issue also.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> it seem live_migration has some bandwidth limit, I have tried
>>>>>>>>>>>>>>> following option in nova.conf but it didn't work
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> live_migration_bandwidth = 500
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My nova.conf look like following:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> live_migration_uri =
>>>>>>>>>>>>>>> "qemu+ssh://nova@%s/system?no_verify=1&keyfile=/var/lib/nova/.ssh/id_rsa"
>>>>>>>>>>>>>>> live_migration_tunnelled = True
>>>>>>>>>>>>>>> live_migration_bandwidth = 500
>>>>>>>>>>>>>>> hw_disk_discard = unmap
>>>>>>>>>>>>>>> disk_cachemodes = network=writeback
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Do you have a this patch (and a couple of patches up to it)?
>>>>>>>>>>>>>> https://bugs.launchpad.net/nova/+bug/1786346
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I don't know if that would cleanly apply (there are other patches that
>>>>>>>>>>>> changed those functions within the last month and a half.  It'd be best
>>>>>>>>>>>> to upgrade and not do just one patch (which would be an untested
>>>>>>>>>>>> process).
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The sha for nova has not been updated yet (next update is 24-48 hours
>>>>>>>>>> away iirc), once that's done you can use the head of stable/queens from
>>>>>>>>>> OSA and run a inter-series upgrade (but the minimal thing to do would be
>>>>>>>>>> to run repo-build and os-nova plays).  I'm not sure when that sha bump
>>>>>>>>>> will be tagged in a full release if you would rather wait on that.
>>>>>>>>
>>>>>>>> it's this sha that needs updating.
>>>>>>>> https://github.com/openstack/openstack-ansible/blob/stable/queens/playbooks/defaults/repo_packages/openstack_services.yml#L173
>>>>>>>>
>>>>>>
>>>>>> I'm not sure how you are doing overrides, but set the following as an
>>>>>> override, then rerun the repo-build playbook (to rebuild the nova venv)
>>>>>> then rerun the nova playbook to install it.
>>>>>>
>>>>>> nova_git_install_branch: dee99b1ed03de4b6ded94f3cf6d2ea7214bca93b
>>>>>>
>>>>
>>>> The sha I gave was head of the queens branch of openstack/nova.  It's
>>>> also the commit in that branch that containst the fix.
>>>>
>>
>> I don't think that is known behavior, different memory pressure causing
>> the difference maybe?
>>
>> --
>> Matthew Thode (prometheanfire)