git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Data storage space unbalance issue


(I am sending the previous mail again because it seems that it has not been sent properly.)

HI experts,

I am running 2 datacenters each containing five nodes. (total 10 nodes, all 3.11.3)

My data is stored one at each data center. (REPLICATION = { 'class' : 'org.apache.cassandra.locator.NetworkTopologyStrategy''datacenter1''1''datacenter2''1 })

Most of my data have a short TTL(14days). The gc_grace_seconds value for all tables is also 600sec.

I expect the two data centers to use the same size but datacenter2 is using more size. It seems that the datas of datacenter2 is rarely deleted. While the disk usage for datacenter1 remains constant, the disk usage for datacenter2 continues to grow.

——————
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.61.58.228  925.48 GiB  256          21.5%             60d1bac8-b4d6-4e02-a05f-badee0bb36f5  rack1
UN  10.61.58.167  840 GiB    256          20.0%             a04fc77a-907f-490c-971c-4e1f964c7b14  rack1
UN  10.61.75.86   1.13 TiB   256          19.3%             618c101b-036d-42e7-bf9f-2bcbd429cbd1  rack1
UN  10.61.59.22   844.19 GiB  256          20.0%             d8a4a165-13f0-4f4a-9278-4024730b8116  rack1
UN  10.61.59.82   737.88 GiB  256          19.2%             054a4eb5-6d1c-46fa-b550-34da610da4e0  rack1
Datacenter: datacenter2
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.42.6.120   1.11 TiB   256          18.6%             69f15be0-e5a1-474e-87cf-b063e6854402  rack1
UN  10.42.5.207   1.17 TiB   256          20.0%             f78bdce5-cb01-47e0-90b9-fcc31568e49e  rack1
UN  10.42.6.47    1.01 TiB   256          20.1%             3ff93b47-2c15-4e1a-a4ea-2596f26b4281  rack1
UN  10.42.6.48    1007.67 GiB  256          20.4%             8cbbe76d-6496-403a-8b09-fe6812c9dea2  rack1
UN  10.42.5.208   1.29 TiB   256          20.9%             4aa96c6a-6083-417f-a58a-ec847bcbfc7e  rack1
------------------

A few days ago, one node of datacenter1 broke down and replaced it, and I worked on rebuild, repair, and cleanup.


What else can I do?

Thank you in advance.