git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Cluster configuration issue


Hi Francesco, it can't work! Milano and Firenze, oh boy, Calcio vs Calcio Storico X-D

Ok more seriously, "Updating topology ..." is not a problem. But you have low resources and system misconfiguration:

  - Small heap size: 3.867GiB
From the logs: "Unable to lock JVM memory (ENOMEM). This can result in part of the JVM being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK or run Cassandra as root."

 - System settings: Swap shoud be disabled, bad system limits, etc.
From the logs: "Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : true, nproc limit adequate? : false"

For system tuning see https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html


You said "Cassandra node did not startup". What is the problem exactly? The process is stuck or does it dies?
What do you see with "nodetool status" on nodes that are up and running? 

Btw cassandra-topology.properties is not required with GossipingPropertyFileSnitch (unless your are migratig from PropertyFileSnitch).


Best,

Romain


Le vendredi 9 novembre 2018 à 11:34:16 UTC+1, Francesco Messere <f.messere@xxxxxxxxxxxxxx> a écrit :


Hi to all,

I have a problem with distribuited cluster configuration.
This is a test environment 
Cassandra version is 3.11.3
2 site Milan and Florence
2 servers on each site

1 common "cluster-name" and 2 DC

First installation and startup goes ok all the nodes are present in the cluster.

The issue startup after a server reboot in FLORENCE DC

Cassandra node did not startup and in system.log last line written is

INFO  [ScheduledTasks:1] 2018-11-09 10:36:54,306 TokenMetadata.java:498 - Updating topology for all endpoints that have changed



The only way to correct the thing I found is to cleanup the node, remove from cluster and re-join it.

How can I solve it?


here are configuration files

less cassandra-topology.properties
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Cassandra Node IP=Data Center:Rack
192.168.204.175=DC-FIRENZE:RACK1
192.168.204.176=DC-FIRENZE:RACK1
192.168.71.210=DC-MILANO:RACK1
192.168.71.211=DC-MILANO:RACK1

# default for unknown nodes
default=DC-FIRENZE:r1

# Native IPv6 is supported, however you must escape the colon in the IPv6 Address
# Also be sure to comment out JVM_OPTS="$JVM_OPTS -Djava.net.preferIPv4Stack=true"
# in cassandra-env.sh
#fe80\:0\:0\:0\:202\:b3ff\:fe1e\:8329=DC1:RAC3


cassandra-rackdc.properties

# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# These properties are used with GossipingPropertyFileSnitch and will
# indicate the rack and dc for this node
dc=DC-FIRENZE
rack=RACK1

# Add a suffix to a datacenter name. Used by the Ec2Snitch and Ec2MultiRegionSnitch
# to append a string to the EC2 region name.
#dc_suffix=

# Uncomment the following line to make this snitch prefer the internal ip when possible, as the Ec2MultiRegionSnitch does.
# prefer_local=true


In attach the system.log file

Regards

Francesco Messere



---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: user-help@xxxxxxxxxxxxxxxxxxxx