✅ View the hints directory for the running node:
| Node (IP) | ls comand |
|---|---|
| 127.0.0.1 |
|
| 127.0.0.2 |
|
### {"execute": false}
total 8
-rw-r--r-- 1 ubuntu ubuntu 318 Apr 7 08:11 b6f8e8ca-b0ae-48f7-855e-98ebbe97d153-1586247113080-v1.hints
-rw-r--r-- 1 ubuntu ubuntu 447 Apr 7 08:11 c4eb8051-ea98-49bc-bb5f-96dbad73a1cd-1586247113076-v1.hints
You should see that the node is storing hints. You can also run ls multiple times to see the hints files grow larger over time. The first part of the file name should correspond with the Host ID of the node that hints are being saved for.
✅ Run the ls command on the other nodes’ hints directory and check that they are empty (meaning neither of the offline nodes are storing hints).
✅ Start cqlsh on the running node:
✅ Verify that the consistency level is set back to the default of ONE:
Solution
node1/bin/cqlsh
CONSISTENCY;✅ Execute the following SELECT statement to try the read back the new row:
SELECT * FROM killrvideo.videos_by_tag WHERE tag = 'cassandra';The query fails with NoHostAvailable: because the sole online node is not a replica node for the cassandra partition, i.e. it does not own the part of the token range that cassandra would hash to.
✅ Quit cqlsh and start node3:
QUIT
./node3/bin/cassandra
Wait for node3 to start.
✅ Run ls on the hints directory of the non-replica node like you did before:
### {"execute": false}
total 288
-rw-r--r-- 1 ubuntu ubuntu 293255 Apr 7 08:13 c4eb8051-ea98-49bc-bb5f-96dbad73a1cd-1586247113076-v1.hints
You will see that one of the set of hint files has disappeared. This is due to this node forwarding the hints to node3 after it has come back online.
✅ Start cqlsh again and connect to node3:
./node3/bin/cqlsh
✅ Switch to the killrvideo keyspace and SELECT the cassandra partition
Solution
USE killrvideo;
SELECT * FROM killrvideo.videos_by_tag WHERE tag = 'cassandra';The query succeeds and returns the new row.
✅ Quit cqlsh:
QUIT
✅ Start the remaining node:
✅ Run ls on the hints directory of the non-replica node like you did before:
You should see no more hints since they have been passed to the two replicas nodes.