git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: RE: [EXTERNAL] Cluster is unbalanced


>If it was partition key issue, we would see similar number of partition keys across nodes. If we look closely number of keys across nodes vary a lot.

I'm not sure about that, is it possible you're writing more new partitions to some nodes even though each node owns the same number of tokens?

On Mon, Jun 18, 2018 at 6:07 PM, learner dba <cassandradba@xxxxxxxxx.invalid> wrote:
Hi Sean,

Are you using any rack aware topology? --> we are using gossip file
Are you using any rack aware topology? --> we are using gossip file
 What are your partition keys? --> Partition key is uniq
Is it possible that your partition keys do not divide up as cleanly as you would like across the cluster because the data is not evenly distributed (by partition key)?  --> No, we verified it.

If it was partition key issue, we would see similar number of partition keys across nodes. If we look closely number of keys across nodes vary a lot.


Number of partitions (estimate): 3142552
Number of partitions (estimate): 15625442
Number of partitions (estimate): 15244021
Number of partitions (estimate): 9592992
Number of partitions (estimate): 15839280





On Monday, June 18, 2018, 5:39:08 PM EDT, Durity, Sean R <SEAN_R_DURITY@xxxxxxxxxxxxx> wrote:


Are you using any rack aware topology? What are your partition keys? Is it possible that your partition keys do not divide up as cleanly as you would like across the cluster because the data is not evenly distributed (by partition key)?

 

 

Sean Durity

lord of the (C*) rings (Staff Systems Engineer – Cassandra)

MTC 2250

#cassandra - for the latest news and updates

 

From: learner dba <cassandradba@xxxxxxxxx.INVALID>
Sent: Monday, June 18, 2018 2:06 PM
To: User cassandra.apache.org <user@xxxxxxxxxxxxxxxxxxxx>
Subject: [EXTERNAL] Cluster is unbalanced

 

Hi,

 

Data volume varies a lot in our two DC cluster:

 Load       Tokens       Owns  

 20.01 GiB  256          ?     

 65.32 GiB  256          ?     

 60.09 GiB  256          ?     

 46.95 GiB  256          ?     

 50.73 GiB  256          ?     

kaiprodv2

=========

/Leaving/Joining/Moving

 Load       Tokens       Owns  

 25.19 GiB  256          ?     

 30.26 GiB  256          ?     

 9.82 GiB   256          ?     

 20.54 GiB  256          ?     

 9.7 GiB    256          ?     

 

I ran clearsnapshot, garbagecollect and cleanup, but it increased the size on heavier nodes instead of decreasing. Based on nodetool cfstats, I can see partition keys on each node varies a lot:

 

Number of partitions (estimate): 3142552

Number of partitions (estimate): 15625442

Number of partitions (estimate): 15244021

Number of partitions (estimate): 9592992

Number of partitions (estimate): 15839280

 

How can I diagnose this imbalance further?

 




--
Joshua Galbraith | Senior Software Engineer | New Relic
C: 907-209-1208 | jgalbraith@xxxxxxxxxxxx