git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: SSVM's not starting, timeout for libvirt python script in agent.log


Hi Chris,


    - Network menu > select view pulldown box > VPN users
    =>>> You mean under "Select view:" (which defaults to Guest Network),
    right? There I am only able to see "Guest Network", "VPC" and "VPN Customer
    Gateway"

=>>> yes this is where it should be if you have enabled VPN on the source NAT. As above – which CloudStack version is this? I’ve tested this on 4.11.1RC2 and seems fine for both isolated networks and VPC networks.
    
    - I have just tested creating a new project as a normal user, change this
    on the pulldown project menu and clicked on the “network” menu item, no
    error in RC2 – is this where you found it?
    =>>> I just tried again with a user i created, it's an admin though. I
    still get the message.

=>>> Can you put a screen dump somewhere? (attachments don’t work on the mailing list)
    
    - Minutes yes, hours no. Going by your previous posts I would guess you
    still have some tuning up to do and possibly run on low spec hardware
    though?
    =>>> The 2 hosts I have run with 2x E5-2680's and 64GB of ram. The
    management server itself is a VMware hosted VM with 16gb of memory and
    4vCPU's (E5-2650). It could be the tuning I guess, I am currently at a loss
    here. The storage is a FreeNAS based NFS share but it's pretty fast.

=>>> This should be sufficient, and I would not expect it to take hours for a VPC to come up. Do keep in mind the first rollout of any template to a previously un-used primary storage pool will take longer as the template has to be copied from secondary storage to primary. Subsequent rollouts should be faster though. What to do is spin up a new VPC, ssh onto this once it’s up and tail the /var/log/cloud.log to determine how long it does processing, then watch how long it take for the VR to come online in ACS. Also check that your “host” global setting is correct.


Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 13/06/2018, 18:12, "Christoffer Pedersen" <vrod@xxxxxxx> wrote:

    Hi Dag,
    
    
    - Network menu > select view pulldown box > VPN users
    =>>> You mean under "Select view:" (which defaults to Guest Network),
    right? There I am only able to see "Guest Network", "VPC" and "VPN Customer
    Gateway"
    
    - I have just tested creating a new project as a normal user, change this
    on the pulldown project menu and clicked on the “network” menu item, no
    error in RC2 – is this where you found it?
    =>>> I just tried again with a user i created, it's an admin though. I
    still get the message.
    
    - This is by design, for VPCs you have to add additional IPs to configure
    services. The only service available on the primary VPC IP is VPN.
    =>>> Ok thanks, good to know!
    
    - Minutes yes, hours no. Going by your previous posts I would guess you
    still have some tuning up to do and possibly run on low spec hardware
    though?
    =>>> The 2 hosts I have run with 2x E5-2680's and 64GB of ram. The
    management server itself is a VMware hosted VM with 16gb of memory and
    4vCPU's (E5-2650). It could be the tuning I guess, I am currently at a loss
    here. The storage is a FreeNAS based NFS share but it's pretty fast.
    
    
    On Wed, Jun 13, 2018 at 5:54 PM, Dag Sonstebo <Dag.Sonstebo@xxxxxxxxxxxxx>
    wrote:
    
    > Hi Chris,
    >
    > - I have enabled remote VPN in a VPC but I cannot seem to add users
    > anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
    > users are now accessed by changing views at the networks tab". However, I
    > cannot find out where such view would be? The documentation doesn't state
    > anything about this.
    >
    > =>>> Network menu > select view pulldown box > VPN users
    >
    > - When I create a new project, choose the view from that project and go to
    > "Network", I always get this message: " Account and projectId can't be
    > specified together". What is it exactly and how can I get rid of it?
    >
    > =>>> I have just tested creating a new project as a normal user, change
    > this on the pulldown project menu and clicked on the “network” menu item,
    > no error in RC2 – is this where you found it?
    >
    > - I don't seem to be able to do port-forwarding with the first configured
    > IP for the VPC. However when adding another IP, I can do it on that
    > specific IP. Isn't it possible to do forwarding on the first assigned IP?
    >
    > =>>> This is by design, for VPCs you have to add additional IPs to
    > configure services. The only service available on the primary VPC IP is VPN.
    >
    > - Is it normal that it takes minutes, sometimes hours to create the VPC? It
    > really takes a while for the system to get going once I have created a VPC.
    >
    > =>>> Minutes yes, hours no. Going by your previous posts I would guess you
    > still have some tuning up to do and possibly run on low spec hardware
    > though?
    >
    >
    > Regards,
    > Dag Sonstebo
    > Cloud Architect
    > ShapeBlue
    >
    > On 13/06/2018, 16:25, "Christoffer Pedersen" <vrod@xxxxxxx> wrote:
    >
    >     The routers managed to start after a while. Looks like most things are
    > OK
    >     now, however I hope some of you could answer some questions for me.
    >
    >     - I have enabled remote VPN in a VPC but I cannot seem to add users
    >     anywhere. On the VPN tab where the PSK is displayed it says: "Note: VPN
    >     users are now accessed by changing views at the networks tab".
    > However, I
    >     cannot find out where such view would be? The documentation doesn't
    > state
    >     anything about this.
    >     - When I create a new project, choose the view from that project and
    > go to
    >     "Network", I always get this message: " Account and projectId can't be
    >     specified together". What is it exactly and how can I get rid of it?
    >     - I don't seem to be able to do port-forwarding with the first
    > configured
    >     IP for the VPC. However when adding another IP, I can do it on that
    >     specific IP. Isn't it possible to do forwarding on the first assigned
    > IP?
    >     - Is it normal that it takes minutes, sometimes hours to create the
    > VPC? It
    >     really takes a while for the system to get going once I have created a
    > VPC.
    >
    >     Perhaps this is for another email-thread, please let me know if so.
    >
    >     Apart from that it seems like the cloudstack instance is finally
    > working,
    >     first VM booted up!
    >
    >     Thank you!
    >     Chris
    >
    >
    > Dag.Sonstebo@xxxxxxxxxxxxx
    > www.shapeblue.com
    > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
    > @shapeblue
    >
    >
    >
    
Dag.Sonstebo@xxxxxxxxxxxxx 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

> On Wed, Jun 13, 2018 at 4:15 PM, Nicolas Bouige <n.bouige@xxxxxxxx> wrote:
    >
    >     > Hello All,
    >     >
    >     > Maybe is not related to your case but in case of....as we don't use
    > the
    >     > same OS
    >     >
    >     > I was facing the exact same error during a new creation of virtual
    > router.
    >     > The only change i made was to update my qemu version on our KVM
    > (centOS7 +
    >     > CS 4.11) from 1.5 to 2.3
    >     > Before that VR creation was working fine.
    >     >
    >     > Apperently some users got the same issue but with an old version of
    > CS
    >     > (4.9)
    >     > http://dev.cloudstack.apache.narkive.com/qsKmnsa4/
    >     > patchviasocket-seems-to-be-broken-with-qemu-2-3
    >     >
    >     >
    >     >
    >     > -----Message d'origine-----
    >     > De : Christoffer Pedersen [mailto:vrod@xxxxxxx]
    >     > Envoyé : mardi 12 juin 2018 19:17
    >     > À : users@xxxxxxxxxxxxxxxxxxxxx
    >     > Objet : Re: SSVM's not starting, timeout for libvirt python script in
    >     > agent.log
    >     >
    >     > Hi James,
    >     >
    >     > I looked at the value and it's already on 50GB. I looked a little
    > more and
    >     > it seems like after restarting the cloudstack-management service I
    > could no
    >     > longer reach the console or storage system vm.... looking at the
    > host, now
    >     > the cloud0 interface is gone. I am using OVS, but according to a
    > Shapeblue
    >     > article, no extra config is needed for the cloud0 interface.
    >     >
    >     >
    >     >
    >     > On Tue, Jun 12, 2018 at 6:17 PM, McClune, James <
    >     > mcclunej@xxxxxxxxxxxxxxxxxxx> wrote:
    >     >
    >     > > Hi Christoffer,
    >     > >
    >     > > Did you try setting the max.template.iso.size to a higher value
    > (e.g.
    >     > > 50GB)?
    >     > >
    >     > > I think the default is set pretty low.
    >     > >
    >     > > Best,
    >     > > James
    >     > >
    >     > > On Tue, Jun 12, 2018 at 11:10 AM, Christoffer Pedersen <
    > vrod@xxxxxxx>
    >     > > wrote:
    >     > >
    >     > > > Just want to update that I still get the ISO error:
    >     > > >
    >     > > > 2018-06-12 17:09:42,194 INFO  [c.c.t.HypervisorTemplateAdapter]
    >     > > > (qtp1096283470-12:ctx-962ec49c ctx-2c82c0e2) (logid:0c7df1d7)
    > Image
    >     > > > store doesn't have enough capacity. Skip downloading template to
    >     > > > this image
    >     > > store
    >     > > > 1
    >     > > >
    >     > > > Thanks!
    >     > > >
    >     > > > On Tue, Jun 12, 2018 at 5:07 PM, Christoffer Pedersen <
    > vrod@xxxxxxx>
    >     > > > wrote:
    >     > > >
    >     > > >> Hi Dag,
    >     > > >>
    >     > > >> Yes, I altered the IP addresses as I do not fancy throwing them
    > out
    >     > > >> on the public net. If you think they have value to the
    >     > > >> troubleshooting, I
    >     > > can
    >     > > >> send you the original logdata directly. I configured the
    > networking
    >     > > >> with OVS and as follows:
    >     > > >>
    >     > > >> cloudbr0 - MGMT0 (management interface, VLAN1000), this also
    > hosts
    >     > > >> the system VMs I guess. In the Zone setup, I labelled the
    >     > > >> Management
    >     > > network as
    >     > > >> MGMT0, I guess that's OK? It's a Internal network where other
    >     > > >> servers
    >     > > are
    >     > > >> also connected. I can ping the VMs here, also over a S2S
    > connection
    >     > > >> cloudbr1 - Public (VLAN 4000) and Guest network (VLAN 500-999).
    > I
    >     > > >> use real public IP-addresses from a scope and I am able to ping
    > the
    >     > > addresses
    >     > > >> of both the CPVM and SSVM from my home.
    >     > > >>
    >     > > >> I have to update the situation though...
    >     > > >>
    >     > > >> Somehow at 13:02, I had the last error from the agent. Now, I do
    >     > > >> not
    >     > > have
    >     > > >> any other errors and both VMs are now showing as "running" as
    > well
    >     > > >> as
    >     > > agent
    >     > > >> being "up" in the UI...  had these errors since yesterday and
    > did
    >     > > >> not change anything after 9:00 this morning. Maybe I'm impatient
    >     > > >> but 4 hours seems a bit long for them to get to work. :)
    >     > > >>
    >     > > >> Console VM works though at least:
    >     > > >>
    >     > > >>
    >     > > >>
    >     > > >> My impression is that cloudstack needs a while to get hold of
    >     > things...
    >     > > >> or am I just experiencing unusual things? I have a question
    > though:
    >     > > >> When adding an ISO this morning, I had an error about that there
    >     > > >> was no space left (though the storage is 20TB). Was this because
    >     > > >> the SSVM was not running at the time?
    >     > > >>
    >     > > >> Thank you!
    >     > > >> Chris
    >     > > >>
    >     > > >> On Tue, Jun 12, 2018 at 4:39 PM, Dag Sonstebo <
    >     > > Dag.Sonstebo@xxxxxxxxxxxxx
    >     > > >> > wrote:
    >     > > >>
    >     > > >>> Chris,
    >     > > >>>
    >     > > >>> Going off in a slightly different direction to previous
    > answers. I
    >     > > >>> suspect your problem is with networking - how have you
    > configured
    >     > this?
    >     > > >>> When you say you can ping the SSVM on the private interface
    > which
    >     > > >>> IP address do you use and where do you successfully ping from?
    >     > > >>>
    >     > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/
    >     > > patchviasocket.py
    >     > > >>> -n
    >     > > >>> v-1-VM -p
    >     > > >>> %template=domP%type=consoleproxy%host=1.1.1.1%port=8250%name
    >     > > >>> =v-1-VM%zone=1%pod=1%guid=Proxy.1%proxy_vm=1%disable_rp_filt
    >     > > >>> er=true%eth2ip=9.9.9.9%eth2mask=255.255.255.0%gateway=9.9.9.
    >     > > >>> 1%eth0ip=169.254.3.159%eth0mask=255.255.0.0%eth1ip=6.
    >     > > >>> 6.6.6%eth1mask=255.255.255.0%mgmtcidr=
    >     > > >>> 9.9.9.0/24%localgw=1.2.3.4%internaldns1=1.2.3.4%dns1=8.8.8.8
    >     > > >>> %dns2=8.8.4.4
    >     > > >>>
    >     > > >>> It could be you have edited the above IP addresses to mask your
    >     > > >>> real addresses – if so ignore this.
    >     > > >>>
    >     > > >>> If not then the above points to:
    >     > > >>> - Management host is on 1.1.1.1
    >     > > >>> - Eth2 which for a console proxy is public traffic is on
    >     > > >>> 9.9.9.9/24
    >     > > >>> - Eth0 which is the link local management interface is on
    >     > > >>> 169.254.3.159/16 (system generated)
    >     > > >>> - Eth1 is the main management interface on 6.6.6.6/24
    >     > > >>> - You have a gateway address of 1.2.3.4
    >     > > >>>
    >     > > >>> So in this case – the CPVM can not check in to the management
    > host
    >     > > >>> on
    >     > > >>> 1.1.1.1 -  It’s got no interface on that subnet and it also
    > has a
    >     > > gateway
    >     > > >>> it’s not able to reach.
    >     > > >>>
    >     > > >>> Regards,
    >     > > >>> Dag Sonstebo
    >     > > >>> Cloud Architect
    >     > > >>> ShapeBlue
    >     > > >>>
    >     > > >>> On 12/06/2018, 13:12, "Nicolas Bouige" <n.bouige@xxxxxxxx>
    > wrote:
    >     > > >>>
    >     > > >>>     Hi Ivan,
    >     > > >>>
    >     > > >>>
    >     > > >>>     Are you talking about this global parameters :
    >     > > >>>
    >     > > >>>     router.aggregation.command.each.timeout
    >     > > >>>
    >     > > >>>
    >     > > >>>
    >     > > >>>     Best regards,
    >     > > >>>
    >     > > >>>     Nicolas Bouige
    >     > > >>>     DIMSI
    >     > > >>>     cloud.dimsi.fr<http://www.cloud.dimsi.fr>
    >     > > >>>     4, avenue Laurent Cely
    >     > > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
    >     > > gmail&source=g>
    >     > > >>>     Tour d’Asnière – 92600 Asnière sur Seine
    >     > > >>>     T/ +33 (0)6 28 98 53 40
    >     > > >>>
    >     > > >>>
    >     > > >>>     ________________________________
    >     > > >>>     De : Ivan Kudryavtsev <kudryavtsev_ia@xxxxxxxxx>
    >     > > >>>     Envoyé : mardi 12 juin 2018 13:59:39
    >     > > >>>     À : users
    >     > > >>>     Objet : Re: SSVM's not starting, timeout for libvirt python
    >     > > >>> script in agent.log
    >     > > >>>
    >     > > >>>     Increasing command timeouts in global parameters can work
    >     > > >>> here. At least I
    >     > > >>>     met similar behaviour with VR.
    >     > > >>>
    >     > > >>>     вт, 12 июн. 2018 г., 14:39 Christoffer Pedersen <
    > vrod@xxxxxxx>:
    >     > > >>>
    >     > > >>>     > Hi Nicolas,
    >     > > >>>     >
    >     > > >>>     > I did a apt show qemu and it gave me this version:
    >     > > >>>     >
    >     > > >>>     > Version: 1:2.5+dfsg-5ubuntu10.29
    >     > > >>>     >
    >     > > >>>     > So I guess tha would be version 2.5?
    >     > > >>>     >
    >     > > >>>
    >     > > >>> Dag.Sonstebo@xxxxxxxxxxxxx
    >     > > >>> www.shapeblue.com
    >     > > >>> 53 Chandos Place, Covent Garden, London
    >     > > >>> <https://maps.google.com/?q=53+Chandos+Place,+Covent+
    >     > > Garden,+London&entry=gmail&source=g>
    >     > > >>> WC2N 4HSUK
    >     > > >>> @shapeblue
    >     > > >>>
    >     > > >>>
    >     > > >>>
    >     > > >>> > On Tue, Jun 12, 2018 at 1:04 PM, Nicolas Bouige
    >     > > >>> > <n.bouige@xxxxxxxx>
    >     > > >>> wrote:
    >     > > >>>     >
    >     > > >>>     > > Hello Christoffer,
    >     > > >>>     > >
    >     > > >>>     > >
    >     > > >>>     > > Could you tell us wich qemu version are you using ?
    >     > > >>>     > >
    >     > > >>>     > > Nicolas Bouige
    >     > > >>>     > > DIMSI
    >     > > >>>     > > cloud.dimsi.fr<http://www.cloud.dimsi.fr>
    >     > > >>>     > > 4, avenue Laurent Cely
    >     > > >>> <https://maps.google.com/?q=4,+avenue+Laurent+Cely&entry=
    >     > > gmail&source=g>
    >     > > >>>     > > Tour d’Asnière – 92600 Asnière sur Seine
    >     > > >>>     > > T/ +33 (0)6 28 98 53 40
    >     > > >>>     > >
    >     > > >>>     > >
    >     > > >>>     > > ________________________________
    >     > > >>>     > > De : Christoffer Pedersen <vrod@xxxxxxx>
    >     > > >>>     > > Envoyé : mardi 12 juin 2018 12:30:48
    >     > > >>>     > > À : users@xxxxxxxxxxxxxxxxxxxxx
    >     > > >>>     > > Objet : SSVM's not starting, timeout for libvirt python
    >     > > >>> script
    >     > > in
    >     > > >>>     > agent.log
    >     > > >>>     > >
    >     > > >>>     > > Hi all,
    >     > > >>>     > >
    >     > > >>>     > > I have an issue regarding the system VMs. After
    > deploying
    >     > > >>> an advanced
    >     > > >>>     > zone,
    >     > > >>>     > > the system VMs are trying to be created but gets stuck
    > in
    >     > > >>> a "Starting"
    >     > > >>>     > > state, however the Agent state is "Up". I have these
    > logs
    >     > > >>> in
    >     > > the
    >     > > >>>     > agent.log
    >     > > >>>     > > (sorry for the formatting)
    >     > > >>>     > >
    >     > > >>>     > > 2018-06-12 12:22:06,354 WARN
    >     > > >>> [kvm.resource.LibvirtComputing Resource]
    >     > > >>>     > > (Script-8:null) (logid:) Interrupting script.
    >     > > >>>     > > 2018-06-12 12:22:06,355 WARN
    >     > > >>> [kvm.resource.LibvirtComputing Resource]
    >     > > >>>     > > (agentRequest-Handler-4:null) (logid:ea9cb55a) Timed
    > out:
    >     > > >>>     > >
    >     > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
    >     > > >>> viasocket.py
    >     > > >>>     > > -n
    >     > > >>>     > > v-1-VM -p
    >     > > >>>     > > %template=domP%type=consoleproxy%host=1.1.1.1%
    >     > > >>>     > > port=8250%name=v-1-VM%zone=1%
    > pod=1%guid=Proxy.1%proxy_vm=1%
    >     > > >>>     > > disable_rp_filter=true%eth2ip=
    > 9.9.9.9%eth2mask=255.255.255.
    >     > > >>>     > > 0%gateway=9.9.9.1%eth0ip=169.
    > 254.3.159%eth0mask=255.255.0.
    >     > > >>>     > > 0%eth1ip=6.6.6.6%eth1mask=255.255.255.0%mgmtcidr=
    >     > > >>>     > >
    >     > > >>>     > 9.9.9.0/24%localgw=1.2.3.4%
    > internaldns1=1.2.3.4%dns1=8.8.8.8
    >     > > >>> %dns2=8.8.4.4
    >     > > >>>     > > .  Output is:
    >     > > >>>     > > 2018-06-12 12:22:06,355 ERROR
    >     > > >>> [kvm.resource.LibvirtComputing Resource]
    >     > > >>>     > > (agentRequest-Handler-4:null) (logid:ea9cb55a) passcmd
    >     > > >>> failed:timeout
    >     > > >>>     > > 2018-06-12 12:22:08,914 WARN
    >     > > >>> [kvm.resource.LibvirtComputing Resource]
    >     > > >>>     > > (Script-4:null) (logid:) Interrupting script.
    >     > > >>>     > > 2018-06-12 12:22:08,915 WARN
    >     > > >>> [kvm.resource.LibvirtComputing Resource]
    >     > > >>>     > > (agentRequest-Handler-5:null) (logid:8e44093e) Timed
    > out:
    >     > > >>>     > >
    >     > > >>> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patch
    >     > > >>> viasocket.py
    >     > > >>>     > > -n
    >     > > >>>     > > s-2-VM -p
    >     > > >>>     > > %template=domP%type=secstorage%host=1.1.1.1%port=
    >     > > >>>     > >
    >     > > >>> 8250%name=s-2-VM%zone=1%pod=1%guid=s-2-VM%workers=5%resource
    >     > > >>> =org.apache.
    >     > > >>>     > > cloudstack.storage.resource.
    > NfsSecondaryStorageResource%
    >     > > >>>     > > instance=SecStorage%sslcopy=
    > false%role=templateProcessor%
    >     > > >>>     > >
    >     > > >>> mtu=1500%eth2ip=7.7.7.7%eth2mask=255.255.255.0%gateway=9.9.9
    >     > > >>> .1%public.
    >     > > >>>     > > network.device=eth2%eth0ip=169.254.2.193%eth0mask=255.
    >     > > >>>     > > 255.0.0%eth1ip=10.120.0.61%eth1mask=255.255.255.0%
    > mgmtcidr=
    >     > > >>>     > > 9.9.9.0/24%localgw=1.2.3.4%
    > private.network.device=eth1%
    >     > > >>>     > > internaldns1=1.2.3.4%dns1=8.8.
    > 8.8%dns2=8.8.4.4%nfsVersion=
    >     > null
    >     > > >>>     > > .  Output is:
    >     > > >>>     > > 2018-06-12 12:22:08,915 ERROR
    >     > > >>> [kvm.resource.LibvirtComputing Resource]
    >     > > >>>     > > (agentRequest-Handler-5:null) (logid:8e44093e) passcmd
    >     > > >>> failed:timeout
    >     > > >>>     > >
    >     > > >>>     > > I have seen this error around but did not really find a
    >     > > solution
    >     > > >>> to it. I
    >     > > >>>     > > am not exactly sure whats "timing" out? I can ping both
    >     > > >>> SSVM's on their
    >     > > >>>     > > private and public interface.
    >     > > >>>     > >
    >     > > >>>     > > I hope someone can help me out here. :)
    >     > > >>>     > >
    >     > > >>>     > > --
    >     > > >>>     > > Thanks,
    >     > > >>>     > > Chris pedersen
    >     > > >>>     > >
    >     > > >>>     >
    >     > > >>>     >
    >     > > >>>     >
    >     > > >>>     > --
    >     > > >>>     > Thanks,
    >     > > >>>     > Chris pedersen
    >     > > >>>     >
    >     > > >>>
    >     > > >>>
    >     > > >>>
    >     > > >>
    >     > > >>
    >     > > >> --
    >     > > >> Thanks,
    >     > > >> Chris pedersen
    >     > > >>
    >     > > >
    >     > > >
    >     > > >
    >     > > > --
    >     > > > Thanks,
    >     > > > Chris pedersen
    >     > > >
    >     > >
    >     >
    >     >
    >     >
    >     > --
    >     > Thanks,
    >     > Chris pedersen
    >     >
    >
    >
    >
    >     --
    >     Thanks,
    >     Chris pedersen
    >
    >
    >
    
    
    -- 
    Thanks,
    Chris pedersen