git.net

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Check Cluster Health


Hi,
Ok.
Thanks a lot for answers.
Kind regards.


2018-07-11 16:13 GMT+01:00 Furkan Cifci <furkancifci@xxxxxxxxx>:
No, you dont need to install Prometheus on each node. Install Prometheus on one machine and configure it. For basic configuration use this documentation: https://www.robustperception.io/monitoring-cassandra-with-prometheus/
You need to use exporter on each node for collecting metrics.

Node exporter will give you system metrics like cpu,ram,disk I/O etc. https://github.com/prometheus/node_exporter
Jmx/Cassandra Exporter will give you  detailed Cassandra metrics like Client r-w latencies, node status, jvm metrics and such. https://github.com/prometheus/jmx_exporter




2018-07-06 18:01 GMT+03:00 Thouraya TH <thouraya87@xxxxxxxxx>:


2018-07-06 13:04 GMT+01:00 Simon Fontana Oscarsson <simon.fontana.oscarsson@ericsson.com>:
Running nodetool status is okay if you want the simplest solution.
But it generates a lot of output and creates a new JMX connection for every execution.
Cassandra uses JMX to expose metrics via mbeans.
Read this to get a first understanding: https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/ops_monitoring_c.html
Use Jconsole to explore the different metrics. Use documentation as reference: https://cassandra.apache.org/doc/latest/operating/metrics.html

As for your solution I recommend one of the following:
* Create a simple JMX client and add your beans. You can do some simple logging with Logback or log4j. You can get some help by googling.
* Use a monitoring system such as Prometheus. This is the best solution but most time consuming.
 
If i use  Prometheus , i have to install it on each node on my data center ? It will give me details about all nodes connections as does nodetool status?

--
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscarsson@ericsson.com
www.ericsson.com

On fre, 2018-07-06 at 11:18 +0100, Thouraya TH wrote:
> Hi,
> Thank you so much for answers.
>
> Please, can you explain more what's metric libraries ? and give me some examples ?
>
> Using nodetool status, to generate the history of my data center, i intend to proceed as follows:
>
> From a node A: 
>
> For i  1 ..24 hours  (every 2 minutes do)
>
> ./nodetool status >> file.txt
>
> End For
>
>
> is it a good idea?
>
> Thanks a lot.
> Kind regards. 
>
> 2018-07-05 1:30 GMT+01:00 Anthony Grasso <anthony.grasso@xxxxxxxxx>:
> > Hi,
> >
> > Yes, you can use nodetool status to inspect the health/status of the cluster. Using nodetool status <keyspace> will show the cluster health/status as well as the amount of data that each node has
> > for the specified <keyspace>.  Using nodetool status without the <keyspace> argument will only show the cluster health/status.
> >
> > Unless there is a special reason for using nodetool to capture history, you may want to consider using metric libraries to capture and push information about each node to a metric server. It is
> > much easier to view the data captured on the metric server as there are tools already made for this. Using metrics libraries will save you time creating and maintaining a parser for the nodetool
> > output. It also makes monitoring the health of cluster very easy.
> >
> > Regards,
> > Anthony
> >
> > On Sun, 1 Jul 2018 at 20:19, Thouraya TH <thouraya87@xxxxxxxxx> wrote:
> > > Hi,
> > > Thank you so much for answer.
> > > Please, is it possible to use this command ?
> > > nodetool status mykeyspace
> > >
> > > Datacenter: datacenter1
> > > =======================
> > > Status=Up/Down
> > > |/ State=Normal/Leaving/Joining/Moving
> > > --  Address    Load       Tokens  Owns    Host ID                               Rack
> > > UN  127.0.0.1  47.66 KB   1       33.3%   aaa1b7c1-6049-4a08-ad3e-3697a0e30e10  rack1
> > > UN  127.0.0.2  47.67 KB   1       33.3%   1848c369-4306-4874-afdf-5c1e95b8732e  rack1
> > > UN
> > > Thank you so much.
> > > Kind regards. 
> > >
> > > 2018-06-29 1:40 GMT+01:00 Rahul Singh <rahul.xavier.singh@xxxxxxxxx>:
> > > >
> > > >
> > > > When you run TPstats or Tablestats subcommands in nodetool you are actually accessing data inside Cassandra via JMX.
> > > >
> > > > You can start there at first.
> > > >
> > > > Rahul
> > > > On Jun 28, 2018, 10:55 AM -0500, Thouraya TH <thouraya87@xxxxxxxxx>, wrote:
> > > > > Hi,
> > > > >
> > > > > Please, how can check the health of my cluster / data center using cassandra ?
> > > > > In fact i'd like to generate a hitory of the state of each node. an history about the failure of my cluster ( 20% of failure in a day, 40% of failure in a day etc...)
> > > > >
> > > > > Thank you so much.
> > > > > Kind regards.
> > >