git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: SSVM's not starting, timeout for libvirt python script in agent.log


Hello All,

Maybe is not related to your case but in case of....as we don't use the same OS

I was facing the exact same error during a new creation of virtual router.
The only change i made was to update my qemu version on our KVM (centOS7 + CS 4.11) from 1.5 to 2.3
Before that VR creation was working fine.

Apperently some users got the same issue but with an old version of CS (4.9) 
http://dev.cloudstack.apache.narkive.com/qsKmnsa4/patchviasocket-seems-to-be-broken-with-qemu-2-3



-----Message d'origine-----
De : Christoffer Pedersen [mailto:vrod@xxxxxxx] 
Envoyé : mardi 12 juin 2018 19:17
À : users@xxxxxxxxxxxxxxxxxxxxx
Objet : Re: SSVM's not starting, timeout for libvirt python script in agent.log

Hi James,

I looked at the value and it's already on 50GB. I looked a little more and it seems like after restarting the cloudstack-management service I could no longer reach the console or storage system vm.... looking at the host, now the cloud0 interface is gone. I am using OVS, but according to a Shapeblue article, no extra config is needed for the cloud0 interface.



On Tue, Jun 12, 2018 at 6:17 PM, McClune, James < mcclunej@xxxxxxxxxxxxxxxxxxx> wrote:

> Hi Christoffer,
>
> Did you try setting the max.template.iso.size to a higher value (e.g.
> 50GB)?
>
> I think the default is set pretty low.
>
> Best,
> James
>
> On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <vrod@xxxxxxx>
> wrote:
>
> > Just want to update that I still get the ISO error:
> >
> > 2018-06-12 17:09:42,194 INFO  [c.c.t.HypervisorTemplateAdapter] 
> > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7) Image 
> > store doesn't have enough capacity. Skip downloading template to 
> > this image
> store
> > 1
> >
> > Thanks!
> >
> > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <vrod@xxxxxxx>
> > wrote:
> >
> >> Hi Dag,
> >>
> >> Yes, I altered the IP addresses as I do not fancy throwing them out 
> >> on the public net. If you think they have value to the 
> >> troubleshooting, I
> can
> >> send you the original logdata directly. I configured the networking 
> >> with OVS and as follows:
> >>
> >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also hosts 
> >> the system VMs I guess. In the Zone setup, I labelled the 
> >> Management
> network as
> >> MGMT0, I guess that's OK? It's a Internal network where other 
> >> servers
> are
> >> also connected. I can ping the VMs here, also over a S2S connection
> >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999). I 
> >> use real public IP-addresses from a scope and I am able to ping the
> addresses
> >> of both the CPVM and SSVM from my home.
> >>
> >> I have to update the situation though...
> >>
> >> Somehow at 13:02, I had the last error from the agent. Now, I do 
> >> not
> have
> >> any other errors and both VMs are now showing as "running" as well 
> >> as
> agent
> >> being "up" in the UI...  had these errors since yesterday and did 
> >> not change anything after 9:00 this morning. Maybe I'm impatient 
> >> but 4 hours seems a bit long for them to get to work. :)
> >>
> >> Console VM works though at least:
> >>
> >>
> >>
> >> My impression is that cloudstack needs a while to get hold of things...
> >> or am I just experiencing unusual things? I have a question though: 
> >> When adding an ISO this morning, I had an error about that there 
> >> was no space left (though the storage is 20TB). Was this because 
> >> the SSVM was not running at the time?
> >>
> >> Thank you!
> >> Chris
> >>
> >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
> Dag.Sonstebo@xxxxxxxxxxxxx
> >> > wrote:
> >>
> >>> Chris,
> >>>
> >>> Going off in a slightly different direction to previous answers. I 
> >>> suspect your problem is with networking - how have you configured this?
> >>> When you say you can ping the SSVM on the private interface which 
> >>> IP address do you use and where do you successfully ping from?
> >>>
> >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
> patchviasocket.py
> >>> -n
> >>> v-1-VM -p
> >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
> >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
> >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
> >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
> >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> >>> %dns2=8.8.4.4
> >>>
> >>> It could be you have edited the above IP addresses to mask your 
> >>> real addresses – if so ignore this.
> >>>
> >>> If not then the above points to:
> >>> - Management host is on 1.1.1.1
> >>> - Eth2 which for a console proxy is public traffic is on 
> >>> 9.9.9.9/24
> >>> - Eth0 which is the link local management interface is on
> >>> 169.254.3.159/16 (system generated)
> >>> - Eth1 is the main management interface on 6.6.6.6/24
> >>> - You have a gateway address of 1.2.3.4
> >>>
> >>> So in this case – the CPVM can not check in to the management host 
> >>> on
> >>> 1.1.1.1 -  It’s got no interface on that subnet and it also has a
> gateway
> >>> it’s not able to reach.
> >>>
> >>> Regards,
> >>> Dag Sonstebo
> >>> Cloud Architect
> >>> ShapeBlue
> >>>
> >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n.bouige@xxxxxxxx> wrote:
> >>>
> >>>     Hi Ivan,
> >>>
> >>>
> >>>     Are you talking about this global parameters :
> >>>
> >>>     router.aggregation.command.each.timeout
> >>>
> >>>
> >>>
> >>>     Best regards,
> >>>
> >>>     Nicolas Bouige
> >>>     DIMSI
> >>>     cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> >>>     4, avenue Laurent Cely
> >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> gmail&source=g>
> >>>     Tour d’Asnière – 92600 Asnière sur Seine
> >>>     T/ +33 (0)6 28 98 53 40
> >>>
> >>>
> >>>     ________________________________
> >>>     De : Ivan Kudryavtsev <kudryavtsev_ia@xxxxxxxxx>
> >>>     Envoyé : mardi 12 juin 2018 13:59:39
> >>>     À : users
> >>>     Objet : Re: SSVM's not starting, timeout for libvirt python 
> >>> script in agent.log
> >>>
> >>>     Increasing command timeouts in global parameters can work 
> >>> here. At least I
> >>>     met similar behaviour with VR.
> >>>
> >>>     вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <vrod@xxxxxxx>:
> >>>
> >>>     > Hi Nicolas,
> >>>     >
> >>>     > I did a apt show qemu and it gave me this version:
> >>>     >
> >>>     > Version: 1:2.5+dfsg-5ubuntu10.29
> >>>     >
> >>>     > So I guess tha would be version 2.5?
> >>>     >
> >>>
> >>> Dag.Sonstebo@xxxxxxxxxxxxx
> >>> www.shapeblue.com
> >>> 53 Chandos Place, Covent Garden, London 
> >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
> Garden,+London&entry=gmail&source=g>
> >>> WC2N 4HSUK
> >>> @shapeblue
> >>>
> >>>
> >>>
> >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige 
> >>> > <n.bouige@xxxxxxxx>
> >>> wrote:
> >>>     >
> >>>     > > Hello Christoffer,
> >>>     > >
> >>>     > >
> >>>     > > Could you tell us wich qemu version are you using ?
> >>>     > >
> >>>     > > Nicolas Bouige
> >>>     > > DIMSI
> >>>     > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
> >>>     > > 4, avenue Laurent Cely
> >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
> gmail&source=g>
> >>>     > > Tour d’Asnière – 92600 Asnière sur Seine
> >>>     > > T/ +33 (0)6 28 98 53 40
> >>>     > >
> >>>     > >
> >>>     > > ________________________________
> >>>     > > De : Christoffer Pedersen <vrod@xxxxxxx>
> >>>     > > Envoyé : mardi 12 juin 2018 12:30:48
> >>>     > > À : users@xxxxxxxxxxxxxxxxxxxxx
> >>>     > > Objet : SSVM's not starting, timeout for libvirt python 
> >>> script
> in
> >>>     > agent.log
> >>>     > >
> >>>     > > Hi all,
> >>>     > >
> >>>     > > I have an issue regarding the system VMs. After deploying 
> >>> an advanced
> >>>     > zone,
> >>>     > > the system VMs are trying to be created but gets stuck in 
> >>> a "Starting"
> >>>     > > state, however the Agent state is "Up". I have these logs 
> >>> in
> the
> >>>     > agent.log
> >>>     > > (sorry for the formatting)
> >>>     > >
> >>>     > > 2018-06-12 12:22:06,354 WARN  
> >>> [kvm.resource.LibvirtComputing Resource]
> >>>     > > (Script-8:null) (logid:) Interrupting script.
> >>>     > > 2018-06-12 12:22:06,355 WARN  
> >>> [kvm.resource.LibvirtComputing Resource]
> >>>     > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed out:
> >>>     > > 
> >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> >>> viasocket.py
> >>>     > > -n
> >>>     > > v-1-VM -p
> >>>     > > %template=domP%type=consoleproxy%host=1.1.1.1%
> >>>     > > port=8250%name=v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%
> >>>     > > disable_rp_filter=true%eth2ip=9.9.9.9%eth2mask=255.255.255.
> >>>     > > 0%gateway=9.9.9.1%eth0ip=169.254.3.159%eth0mask=255.255.0.
> >>>     > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
> >>>     > >
> >>>     > 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
> >>> %dns2=8.8.4.4
> >>>     > > .  Output is:
> >>>     > > 2018-06-12 12:22:06,355 ERROR 
> >>> [kvm.resource.LibvirtComputing Resource]
> >>>     > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd 
> >>> failed:timeout
> >>>     > > 2018-06-12 12:22:08,914 WARN  
> >>> [kvm.resource.LibvirtComputing Resource]
> >>>     > > (Script-4:null) (logid:) Interrupting script.
> >>>     > > 2018-06-12 12:22:08,915 WARN  
> >>> [kvm.resource.LibvirtComputing Resource]
> >>>     > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed out:
> >>>     > > 
> >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
> >>> viasocket.py
> >>>     > > -n
> >>>     > > s-2-VM -p
> >>>     > > %template=domP%type=secstorage%host=1.1.1.1%port=
> >>>     > > 
> >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
> >>> =org.apache.
> >>>     > > cloudstack.storage.resource.NfsSecondaryStorageResource%
> >>>     > > instance=SecStorage%sslcopy=false%role=templateProcessor%
> >>>     > > 
> >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
> >>> .1%public.
> >>>     > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
> >>>     > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%mgmtcidr=
> >>>     > > 9.9.9.0/24%localgw=1.2.3.4%private.network.device=eth1%
> >>>     > > internaldns1=1.2.3.4%dns1=8.8.8.8%dns2=8.8.4.4%nfsVersion=null
> >>>     > > .  Output is:
> >>>     > > 2018-06-12 12:22:08,915 ERROR 
> >>> [kvm.resource.LibvirtComputing Resource]
> >>>     > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd 
> >>> failed:timeout
> >>>     > >
> >>>     > > I have seen this error around but did not really find a
> solution
> >>> to it. I
> >>>     > > am not exactly sure whats "timing" out? I can ping both 
> >>> SSVM's on their
> >>>     > > private and public interface.
> >>>     > >
> >>>     > > I hope someone can help me out here. :)
> >>>     > >
> >>>     > > --
> >>>     > > Thanks,
> >>>     > > Chris pedersen
> >>>     > >
> >>>     >
> >>>     >
> >>>     >
> >>>     > --
> >>>     > Thanks,
> >>>     > Chris pedersen
> >>>     >
> >>>
> >>>
> >>>
> >>
> >>
> >> --
> >> Thanks,
> >> Chris pedersen
> >>
> >
> >
> >
> > --
> > Thanks,
> > Chris pedersen
> >
>



--
Thanks,
Chris pedersen