git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: urgent: Unable to apply firewall rules on router


Hi Ugo,

Have you tried to just restart the management service to clear any running tasks?
And then try add the rules again.

Regards
Glenn Wagner


glenn.wagner@xxxxxxxxxxxxx 
www.shapeblue.com
Winter Suite, 1st Floor, The Avenues, Drama Street, Somerset West, Cape Town  7129South Africa
@shapeblue
  
 


-----Original Message-----
From: Ugo Vasi <ugo.vasi@xxxxxxxxx.INVALID> 
Sent: Thursday, 08 November 2018 5:33 PM
To: users@xxxxxxxxxxxxxxxxxxxxx; Andrija Panic <andrija.panic@xxxxxxxxx>
Subject: Re: urgent: Unable to apply firewall rules on router

Hi Andrija,
from the checks you have suggested I do not show up long running jobs.

There are no error messages in the agent logs. By migrating the router, the behavior has not changed.

Doing further tests I found that the added rules become effective immediately but the interface takes about 25 minutes to show it as active. A couple of times gave error:

2018-11-08 16:22:28,588 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-17:ctx-36b7f3eb job-942) (logid:a107efdf) Complete async job-942, jobStatus: FAILED, resultCode: 530, result: 
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to create firewall rule"}

When I delete a rule, it remains active until the status is updated and then disappears (about 20 minutes after).

Il 07/11/18 18:38, Andrija Panic ha scritto:
> Hi Ugo,
>
> I have seen similar issues with i.e. starting a VM when there are 
> other long running jobs - check if there are any ongoing long jobs 
> already, that might be blocking the executioon of this job - i.e. long 
> running snapshots, or other thing.
> I would also examine agent.log on the host where this VR is located - 
> there might be some traces there...
>
> Try this SQL to list aysnc jobs:
>
> select aj.id,
>              case when aj.job_status=1 then 'completed' when 
> aj.job_status=2 then 'progress' when aj.job_status=3 then 'error' end as status,
>              aj.created, aj.last_updated, aj.related, 
> account.account_name, user.username, host.name as host, vm.name as instance, vmj.step, aj.job_cmd
>            from async_job aj
>            inner join vm_work_job vmj on aj.id = vmj.id
>            left join vm_instance vm on vmj.vm_instance_id=vm.id
>            left join user on aj.user_id=user.id
>            left join account on aj.account_id=account.id
>            left join host on vm.host_id=host.id
>
> Alternatively, try to live-migrate VR to another host, and try to add 
> rule again.
>
> Cheers
> Andrija
>
>
> On Wed, 7 Nov 2018 at 17:59, Ugo Vasi <ugo.vasi@xxxxxxxxx.invalid> wrote:
>
>> Hi all,
>> I'm having a problem when I try to insert a firewall rule of an 
>> address connected to a new VM of a Guest Isolated Network.
>>
>> After a while the job is removed as FAILED. I try to repeat the 
>> operation but the problem remains. How can I unblock the situation?
>>
>> here it is the log of job-927:
>>
>> 2018-11-07 17:16:45,256 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0787853c) Add 
>> job-927 into job monitoring
>> 2018-11-07 17:16:45,279 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Executing 
>> AsyncJobVO {id:927, userId: 2, accountId: 2, instanceType: 
>> FirewallRule,
>> instanceId: 289, cmd:
>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>> ,
>> cmdInfo:
>> {"startport":"1","ipaddressid":"39e4cce4-6a6c-4f31-9f19-85a1bfc47705"
>> ,"httpmethod":"GET","ctxAccountId":"2","uuid":"8bccd152-ce2b-4917-986
>> 5-3563806cc457","cmdEventType":"FIREWALL.OPEN","cidrlist":"XX.XX.XX.X
>> X/29","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEve
>> ntId":"5163","id":"289","endport":"65535","ctxDetails":"{\"interface
>>
>> com.cloud.network.rules.FirewallRule\":\"8bccd152-ce2b-4917-9865-3563
>> 806cc457\",\"interface
>>
>> com.cloud.network.IpAddress\":\"39e4cce4-6a6c-4f31-9f19-85a1bfc47705\
>> "}","_":"1541607404902"},
>>
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null}
>> 2018-11-07 17:16:45,280 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
>> (qtp1096283470-466:ctx-27e3330a ctx-7e984b1b) (logid:5ebca5bb) submit 
>> async job-927, details: AsyncJobVO {id:927, userId: 2, accountId: 2,
>> instanceType: FirewallRule, instanceId: 289, cmd:
>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>> ,
>> cmdInfo:
>> {"startport":"1","ipaddressid":"39e4cce4-6a6c-4f31-9f19-85a1bfc47705"
>> ,"httpmethod":"GET","ctxAccountId":"2","uuid":"8bccd152-ce2b-4917-986
>> 5-3563806cc457","cmdEventType":"FIREWALL.OPEN","cidrlist":"XX.XX.XX.X
>> X/29","protocol":"tcp","response":"json","ctxUserId":"2","ctxStartEve
>> ntId":"5163","id":"289","endport":"65535","ctxDetails":"{\"interface
>>
>> com.cloud.network.rules.FirewallRule\":\"8bccd152-ce2b-4917-9865-3563
>> 806cc457\",\"interface
>>
>> com.cloud.network.IpAddress\":\"39e4cce4-6a6c-4f31-9f19-85a1bfc47705\
>> "}","_":"1541607404902"},
>>
>> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
>> result: null, initMsid: 220777304233416, completeMsid: null,
>> lastUpdated: null, lastPolled: null, created: null}
>> 2018-11-07 17:16:45,330 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) APPLYING FIREWALL RULES
>> 2018-11-07 17:16:45,330 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) Applying firewall rules in network Ntwk[206|Guest|8]
>> 2018-11-07 17:16:45,345 DEBUG [c.c.a.t.Request]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) Seq 1-5860309015115866969: Sending  { Cmd , MgmtId: 
>> 220777304233416,
>> via: 1(cshp121), Ver: v1, Flags: 100001,
>>
>> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":289,"srcIp":"193.239.54.35","protocol":"tcp","srcPortRange":[1,65535],"revoked":false,"alreadyAdded":false,"sourceCidrList":["XX.XX.XX.XX/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"
>> router.name":"r-12-VM","router.guest.ip":"10.11.12.1","router.ip":"16
>> 9.254.1.114","zone.network.type":"Advanced","firewall.egress.default"
>> :"false"},"wait":0}}]
>>
>> }
>> 2018-11-07 17:18:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-1960b382) (logid:bcb6ab77) Task (job-927) has been 
>> pending for 107 seconds
>> 2018-11-07 17:19:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-c7b405f5) (logid:2eda05d8) Task (job-927) has been 
>> pending for 167 seconds
>> 2018-11-07 17:20:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-9661b60b) (logid:432b6bd2) Task (job-927) has been 
>> pending for 227 seconds
>> 2018-11-07 17:21:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-18fa2315) (logid:fa867749) Task (job-927) has been 
>> pending for 287 seconds
>> 2018-11-07 17:22:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-ba0654c9) (logid:572f3a44) Task (job-927) has been 
>> pending for 347 seconds
>> 2018-11-07 17:23:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-2acb9ef9) (logid:83a6be92) Task (job-927) has been 
>> pending for 407 seconds
>> 2018-11-07 17:24:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-8658487d) (logid:8ad384ee) Task (job-927) has been 
>> pending for 467 seconds
>> 2018-11-07 17:25:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-9b2a9bc2) (logid:6d4f5007) Task (job-927) has been 
>> pending for 527 seconds
>> 2018-11-07 17:26:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-3522c7f8) (logid:c5609631) Task (job-927) has been 
>> pending for 587 seconds
>> 2018-11-07 17:27:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-762be74d) (logid:2942dfbd) Task (job-927) has been 
>> pending for 647 seconds
>> 2018-11-07 17:28:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-2ce78e8b) (logid:ae408435) Task (job-927) has been 
>> pending for 707 seconds
>> 2018-11-07 17:29:31,232 DEBUG [c.c.a.t.Request]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) Seq 1-5860309015115866969: Received:  { Ans: , 
>> MgmtId: 220777304233416,
>> via: 1(cshp121), Ver: v1, Flags: 0, { GroupAnswer } }
>> 2018-11-07 17:29:31,235 WARN  [c.c.n.f.FirewallManagerImpl]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) Failed to apply firewall rules due to : Resource 
>> [DataCenter:1] is
>> unreachable: Unable to apply firewall rules on router
>> 2018-11-07 17:29:31,300 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) APPLYING FIREWALL RULES
>> 2018-11-07 17:29:31,301 DEBUG [o.a.c.n.t.BasicNetworkTopology]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) Applying firewall rules in network Ntwk[206|Guest|8]
>> 2018-11-07 17:29:31,314 DEBUG [c.c.a.t.Request]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) Seq 1-5860309015115867196: Sending  { Cmd , MgmtId: 
>> 220777304233416,
>> via: 1(cshp121), Ver: v1, Flags: 100001,
>>
>> [{"com.cloud.agent.api.routing.SetFirewallRulesCommand":{"rules":[{"id":289,"srcIp":"193.239.54.35","protocol":"tcp","srcPortRange":[1,65535],"revoked":true,"alreadyAdded":false,"sourceCidrList":["XX.XX.XX.XX/29"],"purpose":"Firewall","trafficType":"Ingress","defaultEgressPolicy":false}],"accessDetails":{"
>> router.name":"r-12-VM","router.guest.ip":"10.11.12.1","router.ip":"16
>> 9.254.1.114","zone.network.type":"Advanced","firewall.egress.default"
>> :"false"},"wait":0}}]
>>
>> }
>> 2018-11-07 17:29:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-23b76d0d) (logid:57a65a25) Task (job-927) has been 
>> pending for 767 seconds
>> 2018-11-07 17:30:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-f049b29a) (logid:7fbb726e) Task (job-927) has been 
>> pending for 827 seconds
>> 2018-11-07 17:31:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-717decf8) (logid:88f19102) Task (job-927) has been 
>> pending for 887 seconds
>> 2018-11-07 17:32:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-4768ae42) (logid:55f233fa) Task (job-927) has been 
>> pending for 947 seconds
>> 2018-11-07 17:33:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-816fef7b) (logid:5d9db903) Task (job-927) has been 
>> pending for 1007 seconds
>> 2018-11-07 17:34:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-b8559261) (logid:4dcb351e) Task (job-927) has been 
>> pending for 1067 seconds
>> 2018-11-07 17:35:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-94e242a4) (logid:6388b17a) Task (job-927) has been 
>> pending for 1127 seconds
>> 2018-11-07 17:36:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-79404740) (logid:0dcdd7aa) Task (job-927) has been 
>> pending for 1187 seconds
>> 2018-11-07 17:37:32,512 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-5f60335c) (logid:2039a058) Task (job-927) has been 
>> pending for 1247 seconds
>> 2018-11-07 17:38:32,511 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
>> (Timer-1:ctx-ca5488fa) (logid:0c78bc1a) Task (job-927) has been 
>> pending for 1307 seconds
>> 2018-11-07 17:39:31,688 DEBUG [c.c.a.t.Request]
>> (API-Job-Executor-3:ctx-75ed3861 job-927 ctx-2af633c5) 
>> (logid:0e6c51f7) Seq 1-5860309015115867196: Received:  { Ans: , 
>> MgmtId: 220777304233416,
>> via: 1(cshp121), Ver: v1, Flags: 0, { GroupAnswer } }
>> 2018-11-07 17:39:31,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Complete 
>> async job-927, jobStatus: FAILED, resultCode: 530, result:
>> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList"
>> :[],"errorcode":530,"errortext":"Failed
>>
>> to create firewall rule"}
>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Publish 
>> async
>> job-927 complete on message bus
>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Wake up 
>> jobs related to job-927
>> 2018-11-07 17:39:31,737 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Update db 
>> status for job-927
>> 2018-11-07 17:39:31,739 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Wake up 
>> jobs joined with job-927 and disjoin all subjobs created from job- 
>> 927
>> 2018-11-07 17:39:31,743 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Done 
>> executing 
>> org.apache.cloudstack.api.command.user.firewall.CreateFirewallRuleCmd
>> for job-927
>> 2018-11-07 17:39:31,744 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
>> (API-Job-Executor-3:ctx-75ed3861 job-927) (logid:0e6c51f7) Remove
>> job-927 from job monitoring
>>
>>
>>
>> Configuration:
>> ACS version 4.11.1.0
>> Hypervisor KVM
>> S.O. Ubuntu 16.04
>> --
>>
>> *Ugo Vasi* / System Administrator
>> ugo.vasi@xxxxxxxxx <mailto:ugo.vasi@xxxxxxxxx>
>>
>>
>>
>>
>> *Procne S.r.l.*
>> +39 0432 486 523
>> via Cotonificio, 45
>> 33010 Tavagnacco (UD)
>> www.procne.it <http://www.procne.it/>
>>
>>
>> Le informazioni contenute nella presente comunicazione ed i relativi 
>> allegati possono essere riservate e sono, comunque, destinate 
>> esclusivamente alle persone od alla Società sopraindicati. La 
>> diffusione, distribuzione e/o copiatura del documento trasmesso da 
>> parte di qualsiasi soggetto diverso dal destinatario è proibita sia 
>> ai sensi dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 
>> 196/2003 "Codice in materia di protezione dei dati personali". Se 
>> avete ricevuto questo messaggio per errore, vi preghiamo di 
>> distruggerlo e di informare immediatamente Procne S.r.l. scrivendo 
>> all' indirizzo e-mail info@xxxxxxxxx <mailto:info@xxxxxxxxx>.
>>
>>


-- 

*Ugo Vasi* / System Administrator
ugo.vasi@xxxxxxxxx <mailto:ugo.vasi@xxxxxxxxx>




*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>


Le informazioni contenute nella presente comunicazione ed i relativi allegati possono essere riservate e sono, comunque, destinate esclusivamente alle persone od alla Società sopraindicati. La diffusione, distribuzione e/o copiatura del documento trasmesso da parte di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003 "Codice in materia di protezione dei dati personali". Se avete ricevuto questo messaggio per errore, vi preghiamo di distruggerlo e di informare immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail info@xxxxxxxxx <mailto:info@xxxxxxxxx>.