git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Data storage space unbalance issue


It depends on the type of repair, but you'll want to make sure all the data is where it should be before running cleanup.  Somewhat related, if you're not running regular repairs already, you should be.  You can do it via cron, but I strongly suggest checking out Reaper.

On Wed, Nov 28, 2018, 8:05 PM Eunsu Kim <eunsu.bill23@xxxxxxxxx wrote:
Thank you for your response.

I will run repair from datacenter2 with your advice. Do I have to run repair on every node in datacenter2?

There is no snapshot when checked with nodetool listsnaphosts.

Thank you.

On 29 Nov 2018, at 4:31 AM, Elliott Sims <elliott@xxxxxxxxxxxxx> wrote:

I think you answered your own question, sort of.

When you expand a cluster, it copies the appropriate rows to the new node(s) but doesn't automatically remove them from the old nodes.  When you ran cleanup on datacenter1, it cleared out those old extra copies.  I would suggest running a repair first for safety on datacenter2, then a "nodetool cleanup" on those hosts. 

Also run "nodetool snapshot" to make sure you don't have any old snapshots sitting around taking up space.

On Wed, Nov 28, 2018 at 5:29 AM Eunsu Kim <eunsu.bill23@xxxxxxxxx> wrote:
(I am sending the previous mail again because it seems that it has not been sent properly.)

HI experts,

I am running 2 datacenters each containing five nodes. (total 10 nodes, all 3.11.3)

My data is stored one at each data center. (REPLICATION = { 'class' : 'org.apache.cassandra.locator.NetworkTopologyStrategy''datacenter1''1''datacenter2''1 })

Most of my data have a short TTL(14days). The gc_grace_seconds value for all tables is also 600sec.

I expect the two data centers to use the same size but datacenter2 is using more size. It seems that the datas of datacenter2 is rarely deleted. While the disk usage for datacenter1 remains constant, the disk usage for datacenter2 continues to grow.

——————
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.61.58.228  925.48 GiB  256          21.5%             60d1bac8-b4d6-4e02-a05f-badee0bb36f5  rack1
UN  10.61.58.167  840 GiB    256          20.0%             a04fc77a-907f-490c-971c-4e1f964c7b14  rack1
UN  10.61.75.86   1.13 TiB   256          19.3%             618c101b-036d-42e7-bf9f-2bcbd429cbd1  rack1
UN  10.61.59.22   844.19 GiB  256          20.0%             d8a4a165-13f0-4f4a-9278-4024730b8116  rack1
UN  10.61.59.82   737.88 GiB  256          19.2%             054a4eb5-6d1c-46fa-b550-34da610da4e0  rack1
Datacenter: datacenter2
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.42.6.120   1.11 TiB   256          18.6%             69f15be0-e5a1-474e-87cf-b063e6854402  rack1
UN  10.42.5.207   1.17 TiB   256          20.0%             f78bdce5-cb01-47e0-90b9-fcc31568e49e  rack1
UN  10.42.6.47    1.01 TiB   256          20.1%             3ff93b47-2c15-4e1a-a4ea-2596f26b4281  rack1
UN  10.42.6.48    1007.67 GiB  256          20.4%             8cbbe76d-6496-403a-8b09-fe6812c9dea2  rack1
UN  10.42.5.208   1.29 TiB   256          20.9%             4aa96c6a-6083-417f-a58a-ec847bcbfc7e  rack1
------------------

A few days ago, one node of datacenter1 broke down and replaced it, and I worked on rebuild, repair, and cleanup.


What else can I do?

Thank you in advance.