git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: urgent: Unable to apply firewall rules on router


Did you check the logs in the affected router?

On Fri, Nov 9, 2018 at 9:28 AM Ugo Vasi <ugo.vasi@xxxxxxxxx.invalid> wrote:

> Hi Glenn,
> I tried to restart the manager but nothing changed. Note that this
> behavior only occurs on this router, the others work regularly.
> As soon as possible restart the router and see what happens.
>
> Thanks
>
> Il 08/11/18 19:36, Glenn Wagner ha scritto:
> > Hi Ugo,
> >
> > Have you tried to just restart the management service to clear any
> running tasks?
> > And then try add the rules again.
> >
> > Regards
> > Glenn Wagner
> >
> >
> > glenn.wagner@xxxxxxxxxxxxx
> > www.shapeblue.com
> > Winter Suite, 1st Floor, The Avenues, Drama Street, Somerset West, Cape
> Town  7129South Africa
> > @shapeblue
> >
> >
> >
> >
> > -----Original Message-----
> > From: Ugo Vasi <ugo.vasi@xxxxxxxxx.INVALID>
> > Sent: Thursday, 08 November 2018 5:33 PM
> > To: users@xxxxxxxxxxxxxxxxxxxxx; Andrija Panic <andrija.panic@xxxxxxxxx>
> > Subject: Re: urgent: Unable to apply firewall rules on router
> >
> > Hi Andrija,
> > from the checks you have suggested I do not show up long running jobs.
> >
> > There are no error messages in the agent logs. By migrating the router,
> the behavior has not changed.
> >
> > Doing further tests I found that the added rules become effective
> immediately but the interface takes about 25 minutes to show it as active.
> A couple of times gave error:
> >
> > 2018-11-08 16:22:28,588 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-17:ctx-36b7f3eb job-942) (logid:a107efdf) Complete async
> job-942, jobStatus: FAILED, resultCode: 530, result:
> >
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> > to create firewall rule"}
> >
> > When I delete a rule, it remains active until the status is updated and
> then disappears (about 20 minutes after).
> >
> > Il 07/11/18 18:38, Andrija Panic ha scritto:
> >> Hi Ugo,
> >>
> >> I have seen similar issues with i.e. starting a VM when there are
> >> other long running jobs - check if there are any ongoing long jobs
> >> already, that might be blocking the executioon of this job - i.e. long
> >> running snapshots, or other thing.
> >> I would also examine agent.log on the host where this VR is located -
> >> there might be some traces there...
> >>
> >> Try this SQL to list aysnc jobs:
> >>
> >> select aj.id,
> >>               case when aj.job_status=1 then 'completed' when
> >> aj.job_status=2 then 'progress' when aj.job_status=3 then 'error' end
> as status,
> >>               aj.created, aj.last_updated, aj.related,
> >> account.account_name, user.username, host.name as host, vm.name as
> instance, vmj.step, aj.job_cmd
> >>             from async_job aj
> >>             inner join vm_work_job vmj on aj.id = vmj.id
> >>             left join vm_instance vm on vmj.vm_instance_id=vm.id
> >>             left join user on aj.user_id=user.id
> >>             left join account on aj.account_id=account.id
> >>             left join host on vm.host_id=host.id
> >>
> >> Alternatively, try to live-migrate VR to another host, and try to add
> >> rule again.
> >>
> >> Cheers
> >> Andrija
> >>
> >>
> >> On Wed, 7 Nov 2018 at 17:59, Ugo Vasi <ugo.vasi@xxxxxxxxx.invalid>
> wrote:
> >>
> >>> Hi all,
> >>> I'm having a problem when I try to insert a firewall rule of an
> >>> address connected to a new VM of a Guest Isolated Network.
> >>>
> >>> After a while the job is removed as FAILED. I try to repeat the
> >>> operation but the problem remains. How can I unblock the situation?
> >>>
> >>> here it is the log of job-927:
> >>>
> >>> 2018-11-07 17:16:45,256 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0787853c) Add
> >>> job-927 into job monitoring
> >>> 2018-11-07 17:16:45,279 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Executing
> >>> AsyncJobVO {id:927, userId: 2, accountId: 2, instanceType:
> >>> FirewallRule,
> >>> instanceId: 289, cmd:
> >>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
> >>> ,
> >>> cmdInfo:
> >>> {"startport":"1","ipaddressid":"39e4cce4-6a6c-4f31-9f19-85a1bfc47705"
> >>> ,"httpmethod":"GET","ctxAccountId":"2","uuid":"8bccd152-ce2b-4917-986
> >>> 5-3563806cc457","cmdEventType":"FIREWALL.OPEN","cidrlist":"XX.XX.XX.X
> >>> X/29","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEve
> >>> ntId":"5163","id":"289","endport":"65535","ctxDetails":"{\"interface
> >>>
> >>> com.cloud.network.rules.FirewallRule\":\"8bccd152-ce2b-4917-9865-3563
> >>> 806cc457\",\"interface
> >>>
> >>> com.cloud.network.IpAddress\":\"39e4cce4-6a6c-4f31-9f19-85a1bfc47705\
> >>> "}","_":"1541607404902"},
> >>>
> >>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >>> result: null, initMsid: 220777304233416, completeMsid: null,
> >>> lastUpdated: null, lastPolled: null, created: null}
> >>> 2018-11-07 17:16:45,280 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (qtp1096283470-466:ctx-27e3330a ctx-7e984b1b) (logid:5ebca5bb) submit
> >>> async job-927, details: AsyncJobVO {id:927, userId: 2, accountId: 2,
> >>> instanceType: FirewallRule, instanceId: 289, cmd:
> >>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
> >>> ,
> >>> cmdInfo:
> >>> {"startport":"1","ipaddressid":"39e4cce4-6a6c-4f31-9f19-85a1bfc47705"
> >>> ,"httpmethod":"GET","ctxAccountId":"2","uuid":"8bccd152-ce2b-4917-986
> >>> 5-3563806cc457","cmdEventType":"FIREWALL.OPEN","cidrlist":"XX.XX.XX.X
> >>> X/29","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEve
> >>> ntId":"5163","id":"289","endport":"65535","ctxDetails":"{\"interface
> >>>
> >>> com.cloud.network.rules.FirewallRule\":\"8bccd152-ce2b-4917-9865-3563
> >>> 806cc457\",\"interface
> >>>
> >>> com.cloud.network.IpAddress\":\"39e4cce4-6a6c-4f31-9f19-85a1bfc47705\
> >>> "}","_":"1541607404902"},
> >>>
> >>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> >>> result: null, initMsid: 220777304233416, completeMsid: null,
> >>> lastUpdated: null, lastPolled: null, created: null}
> >>> 2018-11-07 17:16:45,330 DEBUG [o.a.c.n.t.BasicNetworkTopology]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) APPLYING FIREWALL RULES
> >>> 2018-11-07 17:16:45,330 DEBUG [o.a.c.n.t.BasicNetworkTopology]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) Applying firewall rules in network Ntwk[206|Guest|8]
> >>> 2018-11-07 17:16:45,345 DEBUG [c.c.a.t.Request]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) Seq 1-5860309015115866969: Sending  { Cmd , MgmtId:
> >>> 220777304233416,
> >>> via: 1(cshp121), Ver: v1, Flags: 100001,
> >>>
> >>>
> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":289,"srcIp":"193.239.54.35","protocol":"tcp","srcPortRange":[1,65535],"revoked":false,"alreadyAdded":false,"sourceCidrList":["XX.XX.XX.XX/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"
> >>> router.name":"r-12-VM","router.guest.ip":"10.11.12.1","router.ip":"16
> >>> 9.254.1.114","zone.network.type":"Advanced","firewall.egress.default"
> >>> :"false"},"wait":0}}]
> >>>
> >>> }
> >>> 2018-11-07 17:18:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-1960b382) (logid:bcb6ab77) Task (job-927) has been
> >>> pending for 107 seconds
> >>> 2018-11-07 17:19:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-c7b405f5) (logid:2eda05d8) Task (job-927) has been
> >>> pending for 167 seconds
> >>> 2018-11-07 17:20:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-9661b60b) (logid:432b6bd2) Task (job-927) has been
> >>> pending for 227 seconds
> >>> 2018-11-07 17:21:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-18fa2315) (logid:fa867749) Task (job-927) has been
> >>> pending for 287 seconds
> >>> 2018-11-07 17:22:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-ba0654c9) (logid:572f3a44) Task (job-927) has been
> >>> pending for 347 seconds
> >>> 2018-11-07 17:23:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-2acb9ef9) (logid:83a6be92) Task (job-927) has been
> >>> pending for 407 seconds
> >>> 2018-11-07 17:24:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-8658487d) (logid:8ad384ee) Task (job-927) has been
> >>> pending for 467 seconds
> >>> 2018-11-07 17:25:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-9b2a9bc2) (logid:6d4f5007) Task (job-927) has been
> >>> pending for 527 seconds
> >>> 2018-11-07 17:26:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-3522c7f8) (logid:c5609631) Task (job-927) has been
> >>> pending for 587 seconds
> >>> 2018-11-07 17:27:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-762be74d) (logid:2942dfbd) Task (job-927) has been
> >>> pending for 647 seconds
> >>> 2018-11-07 17:28:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-2ce78e8b) (logid:ae408435) Task (job-927) has been
> >>> pending for 707 seconds
> >>> 2018-11-07 17:29:31,232 DEBUG [c.c.a.t.Request]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) Seq 1-5860309015115866969: Received:  { Ans: ,
> >>> MgmtId: 220777304233416,
> >>> via: 1(cshp121), Ver: v1, Flags: 0, { GroupAnswer } }
> >>> 2018-11-07 17:29:31,235 WARN  [c.c.n.f.FirewallManagerImpl]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) Failed to apply firewall rules due to : Resource
> >>> [DataCenter:1] is
> >>> unreachable: Unable to apply firewall rules on router
> >>> 2018-11-07 17:29:31,300 DEBUG [o.a.c.n.t.BasicNetworkTopology]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) APPLYING FIREWALL RULES
> >>> 2018-11-07 17:29:31,301 DEBUG [o.a.c.n.t.BasicNetworkTopology]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) Applying firewall rules in network Ntwk[206|Guest|8]
> >>> 2018-11-07 17:29:31,314 DEBUG [c.c.a.t.Request]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) Seq 1-5860309015115867196: Sending  { Cmd , MgmtId:
> >>> 220777304233416,
> >>> via: 1(cshp121), Ver: v1, Flags: 100001,
> >>>
> >>>
> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":289,"srcIp":"193.239.54.35","protocol":"tcp","srcPortRange":[1,65535],"revoked":true,"alreadyAdded":false,"sourceCidrList":["XX.XX.XX.XX/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"
> >>> router.name":"r-12-VM","router.guest.ip":"10.11.12.1","router.ip":"16
> >>> 9.254.1.114","zone.network.type":"Advanced","firewall.egress.default"
> >>> :"false"},"wait":0}}]
> >>>
> >>> }
> >>> 2018-11-07 17:29:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-23b76d0d) (logid:57a65a25) Task (job-927) has been
> >>> pending for 767 seconds
> >>> 2018-11-07 17:30:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-f049b29a) (logid:7fbb726e) Task (job-927) has been
> >>> pending for 827 seconds
> >>> 2018-11-07 17:31:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-717decf8) (logid:88f19102) Task (job-927) has been
> >>> pending for 887 seconds
> >>> 2018-11-07 17:32:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-4768ae42) (logid:55f233fa) Task (job-927) has been
> >>> pending for 947 seconds
> >>> 2018-11-07 17:33:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-816fef7b) (logid:5d9db903) Task (job-927) has been
> >>> pending for 1007 seconds
> >>> 2018-11-07 17:34:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-b8559261) (logid:4dcb351e) Task (job-927) has been
> >>> pending for 1067 seconds
> >>> 2018-11-07 17:35:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-94e242a4) (logid:6388b17a) Task (job-927) has been
> >>> pending for 1127 seconds
> >>> 2018-11-07 17:36:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-79404740) (logid:0dcdd7aa) Task (job-927) has been
> >>> pending for 1187 seconds
> >>> 2018-11-07 17:37:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-5f60335c) (logid:2039a058) Task (job-927) has been
> >>> pending for 1247 seconds
> >>> 2018-11-07 17:38:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (Timer-1:ctx-ca5488fa) (logid:0c78bc1a) Task (job-927) has been
> >>> pending for 1307 seconds
> >>> 2018-11-07 17:39:31,688 DEBUG [c.c.a.t.Request]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5)
> >>> (logid:0e6c51f7) Seq 1-5860309015115867196: Received:  { Ans: ,
> >>> MgmtId: 220777304233416,
> >>> via: 1(cshp121), Ver: v1, Flags: 0, { GroupAnswer } }
> >>> 2018-11-07 17:39:31,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Complete
> >>> async job-927, jobStatus: FAILED, resultCode: 530, result:
> >>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList"
> >>> :[],"errorcode":530,"errortext":"Failed
> >>>
> >>> to create firewall rule"}
> >>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Publish
> >>> async
> >>> job-927 complete on message bus
> >>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Wake up
> >>> jobs related to job-927
> >>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Update db
> >>> status for job-927
> >>> 2018-11-07 17:39:31,739 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Wake up
> >>> jobs joined with job-927 and disjoin all subjobs created from job-
> >>> 927
> >>> 2018-11-07 17:39:31,743 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Done
> >>> executing
> >>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
> >>> for job-927
> >>> 2018-11-07 17:39:31,744 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> >>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Remove
> >>> job-927 from job monitoring
> >>>
> >>>
> >>>
> >>> Configuration:
> >>> ACS version 4.11.1.0
> >>> Hypervisor KVM
> >>> S.O. Ubuntu 16.04
> >>> --
> >>>
> >>> *Ugo Vasi* / System Administrator
> >>> ugo.vasi@xxxxxxxxx <mailto:ugo.vasi@xxxxxxxxx>
> >>>
> >>>
> >>>
> >>>
> >>> *Procne S.r.l.*
> >>> +39 0432 486 523
> >>> via Cotonificio, 45
> >>> 33010 Tavagnacco (UD)
> >>> www.procne.it <http://www.procne.it/>
> >>>
> >>>
> >>> Le informazioni contenute nella presente comunicazione ed i relativi
> >>> allegati possono essere riservate e sono, comunque, destinate
> >>> esclusivamente alle persone od alla Società sopraindicati. La
> >>> diffusione, distribuzione e/o copiatura del documento trasmesso da
> >>> parte di qualsiasi soggetto diverso dal destinatario è proibita sia
> >>> ai sensi dell'art. 616 c.p., che ai sensi del Decreto Legislativo n.
> >>> 196/2003 "Codice in materia di protezione dei dati personali". Se
> >>> avete ricevuto questo messaggio per errore, vi preghiamo di
> >>> distruggerlo e di informare immediatamente Procne S.r.l. scrivendo
> >>> all' indirizzo e-mail info@xxxxxxxxx <mailto:info@xxxxxxxxx>.
> >>>
> >>>
> >
>
>
> --
>
> *Ugo Vasi* / System Administrator
> ugo.vasi@xxxxxxxxx <mailto:ugo.vasi@xxxxxxxxx>
>
>
>
>
> *Procne S.r.l.*
> +39 0432 486 523
> via Cotonificio, 45
> 33010 Tavagnacco (UD)
> www.procne.it <http://www.procne.it/>
>
>
> Le informazioni contenute nella presente comunicazione ed i relativi
> allegati possono essere riservate e sono, comunque, destinate
> esclusivamente alle persone od alla Società sopraindicati. La
> diffusione, distribuzione e/o copiatura del documento trasmesso da parte
> di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi
> dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003
> "Codice in materia di protezione dei dati personali". Se avete ricevuto
> questo messaggio per errore, vi preghiamo di distruggerlo e di informare
> immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail
> info@xxxxxxxxx <mailto:info@xxxxxxxxx>.
>
>

-- 
Rafael Weingärtner