git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

VM Snapshot not removed from primary SR


Hello

I have had weird issue since I updated CS from 4.9 to 4.10.
CS does not delete/remove temporary snapshot from primary SR and I have
had to removed it by manual time to time, before 30 snapshot limit has
reached.
+ doesn¹t CS not update secondary_storege count on snapshot_count table

It works fine on 4.9, but stop to work on 4.10 and I have wait possibility
to update 4.11.1.
Update to 4.11.1 has done now and looks like this not work still

I removed all snapshot and start over from clean table, snapshot policy
is: DAILY, keep 2, delta = 5
There are now 3 snapshot on primary SR and 3 on secondary SR

The Environment is Xen 7 + CS 4.11.1 + NFS storage

Is there some other parameter I should look and any other work around I
has to do to get it work ?

Any hits?

BR
-sami

VM snapshot Log for last round:

2018-09-13 06:02:15,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Executing
AsyncJobVO {id:28020, userId: 1, accountId: 7, instanceType: Snapshot,
instanceId: 2656, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd,
cmdInfo: 
{"policyid":"17","ctxUserId":"1","volumeid":"238","ctxStartEventId":"1","id
":"2656","ctxAccountId":"7"}, cmdVersion: 0, status: IN_PROGRESS,
processStatus: 0, resultCode: 0, result: null, initMsid: 19873467853209,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2018-09-13 06:02:15,991 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Access to Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] granted to
Acct[479c643e-9c84-41fb-9f0a-9bb999893a25-juha] by DomainChecker
2018-09-13 06:02:16,045 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Sync job-28021 execution on object VmWorkJobQueue.228
2018-09-13 06:02:17,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Executing AsyncJobVO {id:28021, userId: 1, accountId: 7, instanceType:
null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot,
cmdInfo: 
rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAZ
aAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3
RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvb
mc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm1X
b3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV
0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAAHAAAAAAAAAAEAAAAAAAAA5HQAFFZvbHVtZU
FwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyA
BBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAARc3EAfgAHAAAAAAAACmBzcQB-
AAcAAAAAAAAA7g, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 19873467853209, completeMsid: null,
lastUpdated: null, lastPolled: null, created: Thu Sep 13 06:02:16 EEST
2018}
2018-09-13 06:02:17,116 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 228, job
origin: 28020
2018-09-13 06:02:17,117 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Execute VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snapsho
tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,"v
mId":228,"handlerName":"VolumeApiServiceImpl"}
2018-09-13 06:02:17,197 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CreateObjectCommand
2018-09-13 06:02:17,197 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) We are returning the default host to execute commands
because the command is not of Copy type.
2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 128-4630826316843724436: Sending  { Cmd , MgmtId:
19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org
.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c796d
-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apache.
cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-
be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pa
th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/poo
l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","
isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-8f
a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountId"
:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hypervis
orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed4520
72ad","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"
uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFile
system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"Net
workFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f
-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","na
me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id":2
656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],"p
hysicalSize":0}},"wait":0}}] }
2018-09-13 06:02:17,199 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 128-4630826316843724436: Executing:  { Cmd , MgmtId:
19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org
.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"969c796d
-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","dataStore":{"org.apache.
cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-
be17c1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","pa
th":"/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/poo
l3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","
isManaged":false}},"name":"ROOT-228","size":21474836480,"path":"3d2a9716-8f
a1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7-228-VM","accountId"
:7,"format":"VHD","provisioningType":"THIN","id":238,"deviceId":0,"hypervis
orType":"XenServer"},"parentSnapshotPath":"c7b2824a-7402-4f66-b2e6-06ed4520
72ad","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"
uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFile
system","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"Net
workFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f
-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"vmName":"i-7-228-VM","na
me":"BGI-Maria_ROOT-228_20180913030215","hypervisorType":"XenServer","id":2
656,"quiescevm":false,"parents":["c7b2824a-7402-4f66-b2e6-06ed452072ad"],"p
hysicalSize":0}},"wait":0}}] }
2018-09-13 06:02:19,815 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
128-4630826316843724436: Response Received:
2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
(DirectAgent-206:ctx-370f4cd9) (logid:4c7a1c38) Seq
128-4630826316843724436: Processing:  { Ans: , MgmtId: 19873467853209,
via: 128(xenserver-15), Ver: v1, Flags: 10,
[{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.
apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-
b14c-4f57f4cda54b","id":0,"quiescevm":false,"physicalSize":0}},"result":tru
e,"wait":0}}] }
2018-09-13 06:02:19,815 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 128-4630826316843724436: Received:  { Ans: , MgmtId:
19873467853209, via: 128(xenserver-15), Ver: v1, Flags: 10, {
CreateObjectAnswer } }
2018-09-13 06:02:19,934 DEBUG [o.a.c.s.m.AncientDataMotionStrategy]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) copyAsync inspecting src type SNAPSHOT copyAsync
inspecting dest type SNAPSHOT
2018-09-13 06:02:20,017 ERROR [o.a.c.s.e.DefaultEndPointSelector]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) IR24 select BACKUPSNAPSHOT from primary to secondary 2656
dest=2656
2018-09-13 06:02:20,023 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) getCommandHostDelegation: class
org.apache.cloudstack.storage.command.CopyCommand
2018-09-13 06:02:20,023 DEBUG [c.c.h.XenServerGuru]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) We are returning the default host to execute commands
because the source and destination objects are not NFS type.
2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 130-7345089517264579763: Sending  { Cmd , MgmtId:
19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache
.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c-4
f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volum
eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSto
reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Net
workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"u
rl":"NetworkFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=
6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228"
,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId
":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningType"
:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshot
Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.cloud
stack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c
1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"
/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool3/pr
imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isMan
aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_201809130302
15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["c7
b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.apac
he.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238","volum
e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","data
Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568
d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","ho
st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFilesys
tem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-
bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":21474836480
,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7
-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,"d
eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/7/
238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.agent
.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8","_role":"Image"}
},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hypervi
sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/23
8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3d-
4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,"o
ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
2018-09-13 06:02:20,025 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 130-7345089517264579763: Executing:  { Cmd , MgmtId:
19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache
.cloudstack.storage.to.SnapshotObjectTO":{"path":"e91fb041-779d-4090-b14c-4
f57f4cda54b","volume":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volum
eType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataSto
reTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"Net
workFilesystem","host":"10.10.1.33","path":"/pool3/primary3","port":2049,"u
rl":"NetworkFilesystem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=
6568d30f-82b6-35a0-bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228"
,"size":21474836480,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId
":238,"vmName":"i-7-228-VM","accountId":7,"format":"VHD","provisioningType"
:"THIN","id":238,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshot
Path":"c7b2824a-7402-4f66-b2e6-06ed452072ad","dataStore":{"org.apache.cloud
stack.storage.to.PrimaryDataStoreTO":{"uuid":"6568d30f-82b6-35a0-bfe5-be17c
1398ed8","id":2,"poolType":"NetworkFilesystem","host":"10.10.1.33","path":"
/pool3/primary3","port":2049,"url":"NetworkFilesystem://10.10.1.33/pool3/pr
imary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-bfe5-be17c1398ed8","isMan
aged":false}},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_201809130302
15","hypervisorType":"XenServer","id":2656,"quiescevm":false,"parents":["c7
b2824a-7402-4f66-b2e6-06ed452072ad"],"physicalSize":0}},"destTO":{"org.apac
he.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238","volum
e":{"uuid":"969c796d-072e-4272-adc6-93a492c03ea3","volumeType":"ROOT","data
Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6568
d30f-82b6-35a0-bfe5-be17c1398ed8","id":2,"poolType":"NetworkFilesystem","ho
st":"10.10.1.33","path":"/pool3/primary3","port":2049,"url":"NetworkFilesys
tem://10.10.1.33/pool3/primary3/?ROLE=Primary&STOREUUID=6568d30f-82b6-35a0-
bfe5-be17c1398ed8","isManaged":false}},"name":"ROOT-228","size":21474836480
,"path":"3d2a9716-8fa1-497e-8753-941117a0ef50","volumeId":238,"vmName":"i-7
-228-VM","accountId":7,"format":"VHD","provisioningType":"THIN","id":238,"d
eviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/7/
238/0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","dataStore":{"com.cloud.agent
.api.to.NfsTO":{"_url":"nfs://10.10.1.34/pool8/secondary8","_role":"Image"}
},"vmName":"i-7-228-VM","name":"BGI-Maria_ROOT-228_20180913030215","hypervi
sorType":"XenServer","id":2656,"quiescevm":false,"parents":["snapshots/7/23
8/93a42896-0da1-4b82-8985-d28464f1ac4a.vhd","snapshots/7/238/0c8c6ee3-ce3d-
4241-8509-5193d2cff491.vhd"],"physicalSize":0}},"executeInSequence":true,"o
ptions":{"fullSnapshot":"false"},"options2":{},"wait":28800}}] }
2018-09-13 06:04:28,895 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Removing SR
2018-09-13 06:04:28,913 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
OpaqueRef:05380b03-4cb8-85a7-6a6a-a1595c188f60: Unplugging pbd
2018-09-13 06:04:30,881 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Host 10.10.1.81
OpaqueRef:f83331df-6634-63fd-6b95-44607a6069b8: Forgetting
2018-09-13 06:04:30,891 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Successfully destroyed
snapshot on volume: 3d2a9716-8fa1-497e-8753-941117a0ef50 execept this
current snapshot e91fb041-779d-4090-b14c-4f57f4cda54b
2018-09-13 06:04:30,891 INFO  [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot details:
SnapshotTO[datastore=null|volume=null|pathsnapshots/7/238/0560d536-c945-4a6
1-8c4c-dbaf29f29993.vhd]
2018-09-13 06:04:30,891 INFO  [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) New snapshot physical
utilization: 2342932992
2018-09-13 06:04:30,891 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
130-7345089517264579763: Response Received:
2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
130-7345089517264579763: Processing:  { Ans: , MgmtId: 19873467853209,
via: 130(xenserver-81), Ver: v1, Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.ap
ache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/238/0560d
536-c945-4a61-8c4c-dbaf29f29993.vhd","parentSnapshotPath":"snapshots/7/238/
0c8c6ee3-ce3d-4241-8509-5193d2cff491.vhd","id":0,"quiescevm":false,"physica
lSize":2342932992}},"result":true,"wait":0}}] }
2018-09-13 06:04:30,891 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-382:ctx-36b817c8) (logid:4c7a1c38) Seq
130-7345089517264579763: No more commands found
2018-09-13 06:04:30,891 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Seq 130-7345089517264579763: Received:  { Ans: , MgmtId:
19873467853209, via: 130(xenserver-81), Ver: v1, Flags: 110, {
CopyCmdAnswer } }
2018-09-13 06:04:30,957 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Updating resource Type = secondary_storage count for
Account = 7 Operation = decreasing Amount = 19131903488
2018-09-13 06:04:30,969 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Done executing VM work job:
com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":238,"policyId":17,"snapsho
tId":2656,"quiesceVm":false,"asyncBackup":false,"userId":1,"accountId":7,"v
mId":228,"handlerName":"VolumeApiServiceImpl"}
2018-09-13 06:04:30,969 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Complete async job-28021, jobStatus: SUCCEEDED,
resultCode: 0, result:
rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk5
1bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAKYA
2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Publish async job-28021 complete on message bus
2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Wake up jobs related to job-28021
2018-09-13 06:04:30,970 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Update db status for job-28021
2018-09-13 06:04:30,971 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021 ctx-e4ed14c2)
(logid:4c7a1c38) Wake up jobs joined with job-28021 and disjoin all
subjobs created from job- 28021
2018-09-13 06:04:30,983 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM
228, job origin: 28020
2018-09-13 06:04:30,983 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-28021
2018-09-13 06:04:30,984 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-53:ctx-3728caef job-28020/job-28021) (logid:4c7a1c38)
Remove job-28021 from job monitoring
2018-09-13 06:04:31,026 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Complete async job-28020, jobStatus: SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"3c8662b
2-c76f-4e3a-a928-f76dad820122","account":"juha","domainid":"d3c2420f-7503-4
560-a2da-d7e7149ce390","domain":"BGI","snapshottype":"DAILY","volumeid":"96
9c796d-072e-4272-adc6-93a492c03ea3","volumename":"ROOT-228","volumetype":"R
OOT","created":"2018-09-13T06:02:15+0300","name":"BGI-Maria_ROOT-228_201809
13030215","intervaltype":"DAILY","state":"BackedUp","physicalsize":23429329
92,"zoneid":"533c8f88-2637-4c0e-8210-e0a355b62d82","tags":[],"revertable":f
alse,"ostypeid":"da491b96-3748-11e5-a683-121326fde999","osdisplayname":"Cen
tOS 6.5 (64-bit)","virtualsize":21474836480}
2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Publish async job-28020 complete on message bus
2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Wake up jobs related to job-28020
2018-09-13 06:04:31,027 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Update db status for job-28020
2018-09-13 06:04:31,028 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020 ctx-5b8156a6) (logid:4c7a1c38)
Wake up jobs joined with job-28020 and disjoin all subjobs created from
job- 28020
2018-09-13 06:04:31,039 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Done
executing 
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for
job-28020
2018-09-13 06:04:31,039 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-83:ctx-cfb1cd95 job-28020) (logid:4c7a1c38) Remove
job-28020 from job monitoring