Yes Kurt, system log is flooded with hints sent and replayed messages.On Monday, June 18, 2018, 7:30:34 AM EDT, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:Not sure what to make of that. Are there any log messages regarding the file and replaying hints? Sounds like maybe it's corrupt (although not sure why it keeps getting rewritten).On 14 June 2018 at 13:19, Nitan Kainth <nitankainth@xxxxxxxxx> wrote:Kurt,Hint file matches UUID matches with another node in the cluster:
-rw-r--r--. 1 root root 6848246 May 13 23:37 1b694180-210a-4b75-8f2a- 748f4a5b6a3d-1526254645089-1. hints
/opt/cassandra/bin/nodetool status |grep 1b694180
UN x.x.x. 23.77 GiB 256 ? 1b694180-210a-4b75-8f2a- 748f4a5b6a3d RAC1On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:Does the UUID on the filename correspond with a UUID in nodetool status?Sounds to me like it could be something weird with an old node that no longer exists, although hints for old nodes are meant to be cleaned up.On 14 June 2018 at 01:54, Nitan Kainth <nitankainth@xxxxxxxxx> wrote:Kurt,No node is down for months. And yes, I am surprised to look at Unix timestamp on files.
On Jun 13, 2018, at 6:41 PM, kurt greaves <kurt@xxxxxxxxxxxxxxx> wrote:system.hints is not used in Cassandra 3. Can't explain the files though, are you referring to the files timestamp or the Unix timestamp in the file name? Is there a node that's been down for several months?On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <nitankainth@xxxxxxxxx> wrote:Hi,I observed a strange behavior about stored hints.Time stamp of hints file shows several months old. I deleted them and saw new hints files created with same old date. Why is that?Also, I see hints files on disk but if I query system.hints table, it shows 0 rows. Why system.hints is not populated?Version 3.11-1