I have a 6-node cluster (Windows Server 2008 R2), and I have OpsCenter running on a Windows 7 machine. I can see from the ring information in the OpsCenter interface that the cluster is up. However, OpsCenter says 0 of 6 agents connected. Also, the performance metrics charts say "No Data" (for all the metrics). I'd appreciate any help in resolving this. I've been encountering this problem since I've upgraded my cluster to the latest version of Datastax Cassandra Community Server, which has Murmur3 as the default partitioner. With the earlier version, where the default partitioner was RandomPartitioner, I could see the write/read throughput and write/read latency, but not the system parameters. I'd like to continue using the latest version.
Here's the opscenterd.conf on the OpsCenter Community Server machine:
[webserver]
port = 8888
interface = 0.0.0.0
# The following settings can be used to enable ssl support for the opscenter
# web application. Change these values to point to the ssl certificate and key
# that you wish to use for your OpsCenter install, as well as the port you would like
# to serve ssl traffic from.
#ssl_keyfile = /var/lib/opscenter/ssl/opscenter.key
#ssl_certfile = /var/lib/opscenter/ssl/opscenter.pem
#ssl_port = 8443
log_path=C:\Program Files (x86)\DataStax Community\logs\opscenterd-http.log
[logging]
# level may be TRACE, DEBUG, INFO, WARN, or ERROR
#level = INFO
log_path=C:\Program Files (x86)\DataStax Community\logs\opscenterd.log
[authentication]
# if this file does not exist, there will be no password protection. Use the
# set_passwd.py tool (included with OpsCenter) to set passwords. This property will
# default to /etc/opscenter/.passwd in packaged installations and ./passwds in
# tarball installations.
#passwd_file =
[agents]
use_ssl=0
incoming_interface=<IP of this machine>
Here's the address.yaml on the OpsCenter Community Server machine:
stomp_interface: <IP of this machine>
local_interface: <IP of this machine>
use_ssl: 0
Here's the address.yaml on the individual nodes in the cluster (the agents):
stomp_interface: <IP of the OpsCenter Community Server machine>
local_interface: <IP of the agent machine>
use_ssl: 0
Here's what http://<OpsCenter IP>:8888/Philip/nodes/all shows:
[{"load": null, "has_jna": false, "devices": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "task_progress": {}, "node_ip": "172.16.55.11", "network_interfaces": null, "ec2": {}, "node_version": {}, "dc": null, "node_name": null, "num_procs": null, "streaming": {}, "token": "-6148914691236517206", "data_held": null, "mode": "unknown", "rpc_ip": "172.16.55.11", "partitions": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "os": null, "rack": null, "last_seen": 0}, {"load": null, "has_jna": false, "devices": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "task_progress": {}, "node_ip": "172.16.55.13", "network_interfaces": null, "ec2": {}, "node_version": {}, "dc": null, "node_name": null, "num_procs": null, "streaming": {}, "token": "3074457345618258600", "data_held": null, "mode": "unknown", "rpc_ip": "172.16.55.13", "partitions": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "os": null, "rack": null, "last_seen": 0}, {"load": null, "has_jna": false, "devices": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "task_progress": {}, "node_ip": "172.16.55.14", "network_interfaces": null, "ec2": {}, "node_version": {}, "dc": null, "node_name": null, "num_procs": null, "streaming": {}, "token": "-3074457345618258604", "data_held": null, "mode": "unknown", "rpc_ip": "172.16.55.14", "partitions": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "os": null, "rack": null, "last_seen": 0}, {"load": null, "has_jna": false, "devices": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "task_progress": {}, "node_ip": "172.16.55.15", "network_interfaces": null, "ec2": {}, "node_version": {}, "dc": null, "node_name": null, "num_procs": null, "streaming": {}, "token": "6148914691236517202", "data_held": null, "mode": "unknown", "rpc_ip": "172.16.55.15", "partitions": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "os": null, "rack": null, "last_seen": 0}, {"load": null, "has_jna": false, "devices": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "task_progress": {}, "node_ip": "172.16.55.16", "network_interfaces": null, "ec2": {}, "node_version": {}, "dc": null, "node_name": null, "num_procs": null, "streaming": {}, "token": "-2", "data_held": null, "mode": "unknown", "rpc_ip": "172.16.55.16", "partitions": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "os": null, "rack": null, "last_seen": 0}, {"load": null, "has_jna": false, "devices": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "task_progress": {}, "node_ip": "172.16.55.2", "network_interfaces": null, "ec2": {}, "node_version": {}, "dc": null, "node_name": null, "num_procs": null, "streaming": {}, "token": "-9223372036854775808", "data_held": null, "mode": "unknown", "rpc_ip": "172.16.55.2", "partitions": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "os": null, "rack": null, "last_seen": 0}]
Here's a dump of opscenterd.log:
2013-06-03 23:46:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.13 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:46:46+0530 [local] WARN: HTTP request http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.16 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:46:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.16 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:46:46+0530 [local] WARN: HTTP request http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.14 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:46:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.14 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:46:46+0530 [local] WARN: HTTP request http://172.16.55.14:61621/cluster/datacenter?node_ip=172.16.55.11 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:46:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.14:61621/cluster/datacenter?node_ip=172.16.55.11 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:47:27+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was 3.0.0. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
2013-06-03 23:48:27+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was 3.0.0. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
2013-06-03 23:48:46+0530 [local] WARN: HTTP request http://172.16.55.14:61621/cluster/datacenter?node_ip=172.16.55.14 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.14:61621/cluster/datacenter?node_ip=172.16.55.14 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: HTTP request http://172.16.55.16:61621/cluster/datacenter?node_ip=172.16.55.11 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.16:61621/cluster/datacenter?node_ip=172.16.55.11 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: HTTP request http://172.16.55.14:61621/cluster/datacenter?node_ip=172.16.55.2 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.14:61621/cluster/datacenter?node_ip=172.16.55.2 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: HTTP request http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.13 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.13 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: HTTP request http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.15 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.15 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: HTTP request http://172.16.55.15:61621/cluster/datacenter?node_ip=172.16.55.16 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:48:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.15:61621/cluster/datacenter?node_ip=172.16.55.16 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:49:29+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was 3.0.0. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
2013-06-03 23:49:50+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was [u'3.0.0', u'/605807254/conf']. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
2013-06-03 23:50:29+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was 3.0.0. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
2013-06-03 23:50:46+0530 [local] WARN: HTTP request http://172.16.55.14:61621/cluster/datacenter?node_ip=172.16.55.13 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.14:61621/cluster/datacenter?node_ip=172.16.55.13 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: HTTP request http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.14 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.14 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: HTTP request http://172.16.55.16:61621/cluster/datacenter?node_ip=172.16.55.2 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.16:61621/cluster/datacenter?node_ip=172.16.55.2 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: HTTP request http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.15 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.15 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: HTTP request http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.11 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.11 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: HTTP request http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.16 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:50:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.16 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:51:29+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was 3.0.0. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
2013-06-03 23:52:30+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was 3.0.0. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
2013-06-03 23:52:46+0530 [local] WARN: HTTP request http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.11 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.11 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: HTTP request http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.2 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.11:61621/cluster/datacenter?node_ip=172.16.55.2 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: HTTP request http://172.16.55.16:61621/cluster/datacenter?node_ip=172.16.55.13 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.16:61621/cluster/datacenter?node_ip=172.16.55.13 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: HTTP request http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.15 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.15 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: HTTP request http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.16 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.13:61621/cluster/datacenter?node_ip=172.16.55.16 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: HTTP request http://172.16.55.2:61621/cluster/datacenter?node_ip=172.16.55.14 failed: Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:52:46+0530 [local] WARN: Unable to collect datacenter, rack information: Failed query to http://172.16.55.2:61621/cluster/datacenter?node_ip=172.16.55.14 : Connection was refused by other side: 10061: No connection could be made because the target machine actively refused it..
2013-06-03 23:53:31+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was 3.0.0. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
2013-06-03 23:54:32+0530 [] WARN: Unable to find a matching cluster for node with IP 172.16.55.5; the message was 3.0.0. This usually indicates that an OpsCenter agent is still running on an old node that was decommissioned or is part of a cluster that OpsCenter is no longer monitoring.
Here's a dump of one of the egent logs:
INFO [StompConnection receiver] 2013-06-03 23:01:01,890 Reconnecting in 60s.
ERROR [StompConnection receiver] 2013-06-03 23:02:02,934 failed connecting to 172.16.55.5:61620:java.net.ConnectException: Connection refused: connect
INFO [StompConnection receiver] 2013-06-03 23:02:02,934 Reconnecting in 60s.
ERROR [StompConnection receiver] 2013-06-03 23:03:03,962 failed connecting to 172.16.55.5:61620:java.net.ConnectException: Connection refused: connect
INFO [StompConnection receiver] 2013-06-03 23:03:03,962 Reconnecting in 60s.
ERROR [StompConnection receiver] 2013-06-03 23:04:05,006 failed connecting to 172.16.55.5:61620:java.net.ConnectException: Connection refused: connect
INFO [StompConnection receiver] 2013-06-03 23:04:05,006 Reconnecting in 60s.
ERROR [StompConnection receiver] 2013-06-03 23:05:06,035 failed connecting to 172.16.55.5:61620:java.net.ConnectException: Connection refused: connect
INFO [StompConnection receiver] 2013-06-03 23:05:06,035 Reconnecting in 60s.
ERROR [StompConnection receiver] 2013-06-03 23:06:07,063 failed connecting to 172.16.55.5:61620:java.net.ConnectException: Connection refused: connect
INFO [StompConnection receiver] 2013-06-03 23:06:07,063 Reconnecting in 60s.
INFO [conf-requester] 2013-06-03 23:06:42,288 Requesting latest conf from opscenterd
WARN [pool-2-thread-3] 2013-06-03 23:06:42,320 Tried to send message while not connected: /conf-request ["172.16.55.16",["3.0.0","\/509131971\/conf"]]
INFO [StompConnection receiver] 2013-06-03 23:07:07,077 Connected to 172.16.55.5:61620
INFO [main] 2013-06-03 23:13:07,725 Loading conf files: C:\Program Files (x86)\DataStax Community\opscenter\agent\conf\address.yaml
INFO [main] 2013-06-03 23:13:07,834 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_29
INFO [main] 2013-06-03 23:13:07,834 Waiting for the config from OpsCenter
INFO [main] 2013-06-03 23:13:07,834 SSL communication is disabled
INFO [main] 2013-06-03 23:13:07,834 Creating stomp connection to 172.16.55.5:61620
INFO [StompConnection receiver] 2013-06-03 23:13:07,834 Reconnecting in 0s.
INFO [main] 2013-06-03 23:13:07,850 Using 172.16.55.16 as the cassandra broadcast address
INFO [main] 2013-06-03 23:13:07,850 New JMX connection (127.0.0.1:7199)
INFO [StompConnection receiver] 2013-06-03 23:13:07,850 Connected to 172.16.55.5:61620
INFO [main] 2013-06-03 23:13:08,053 Cassandra node just went down
INFO [StompConnection receiver] 2013-06-03 23:13:08,177 Got new config from OpsCenter: {:kerberos_use_keytab true, :rollups300_ttl 2419200, :kerberos_use_ticket_cache true, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :thrift_ssl_truststore nil, :metrics_ignored_column_families "", :cassandra_log_location "/var/log/cassandra/system.log", :thrift_rpc_interface "172.16.55.16", :thrift_ssl_truststore_password nil, :jmx_port 7199, :provisioning 0, :kerberos_debug false, :rollups86400_ttl -1, :api_port "61621", :storage_keyspace "OpsCenter", :kerberos_renew_tgt true, :metrics_ignored_solr_cores "", :thrift_ssl_truststore_type "JKS", :metrics_ignored_keyspaces "system, system_traces, system_auth, dse_auth, OpsCenter", :rollup_subscriptions [], :cassandra_install_location ""}
INFO [conf-requester] 2013-06-03 23:23:07,853 Requesting latest conf from opscenterd
ERROR [StompConnection receiver] 2013-06-03 23:27:43,541 Connection closed unexpectedly:
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(Unknown Source)
at java.net.SocketInputStream.read(Unknown Source)
at java.io.FilterInputStream.read(Unknown Source)
at org.jgroups.util.Util.readLine(Util.java:2777)
at org.jgroups.protocols.STOMP.readFrame(STOMP.java:225)
at org.jgroups.client.StompConnection.run(StompConnection.java:253)
at java.lang.Thread.run(Unknown Source)
INFO [StompConnection receiver] 2013-06-03 23:27:43,541 Reconnecting in 0s.
ERROR [StompConnection receiver] 2013-06-03 23:27:44,571 failed connecting to 172.16.55.5:61620:java.net.ConnectException: Connection refused: connect
INFO [StompConnection receiver] 2013-06-03 23:27:44,571 Reconnecting in 2s.
INFO [StompConnection receiver] 2013-06-03 23:27:46,583 Connected to 172.16.55.5:61620
INFO [main] 2013-06-03 23:30:11,276 Loading conf files: C:\Program Files (x86)\DataStax Community\opscenter\agent\conf\address.yaml
INFO [main] 2013-06-03 23:30:11,385 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_29
INFO [main] 2013-06-03 23:30:11,385 Waiting for the config from OpsCenter
INFO [main] 2013-06-03 23:30:11,385 SSL communication is disabled
INFO [main] 2013-06-03 23:30:11,385 Creating stomp connection to 172.16.55.5:61620
INFO [StompConnection receiver] 2013-06-03 23:30:11,385 Reconnecting in 0s.
INFO [main] 2013-06-03 23:30:11,401 Using 172.16.55.16 as the cassandra broadcast address
INFO [main] 2013-06-03 23:30:11,401 New JMX connection (127.0.0.1:7199)
INFO [StompConnection receiver] 2013-06-03 23:30:11,401 Connected to 172.16.55.5:61620
INFO [main] 2013-06-03 23:30:11,619 Cassandra node just went down
INFO [StompConnection receiver] 2013-06-03 23:30:11,697 Got new config from OpsCenter: {:kerberos_use_keytab true, :rollups300_ttl 2419200, :kerberos_use_ticket_cache true, :rollups60_ttl 604800, :thrift_port 9160, :metrics_enabled 1, :rollups7200_ttl 31536000, :thrift_ssl_truststore nil, :metrics_ignored_column_families "", :cassandra_log_location "/var/log/cassandra/system.log", :thrift_rpc_interface "172.16.55.16", :thrift_ssl_truststore_password nil, :jmx_port 7199, :provisioning 0, :kerberos_debug false, :rollups86400_ttl -1, :api_port "61621", :storage_keyspace "OpsCenter", :kerberos_renew_tgt true, :metrics_ignored_solr_cores "", :thrift_ssl_truststore_type "JKS", :metrics_ignored_keyspaces "system, system_traces, system_auth, dse_auth, OpsCenter", :rollup_subscriptions [], :cassandra_install_location ""}
INFO [conf-requester] 2013-06-03 23:40:11,430 Requesting latest conf from opscenterd
INFO [conf-requester] 2013-06-03 23:50:11,458 Requesting latest conf from opscenterd
I've checked, and there are no firewalls on any of the machines.
Any help would be appreciated. Thanks.