I have 4 Linux nodes with agents that cannot communicate with the OpsCenter Management Console.
Agents were installed via tarball.
All in the same datacenter
SSL is turned off. Have tried both the IP and Broadcast address of the management server.
INFO [main] 2013-08-22 16:00:20,748 Loading conf files: ./conf/address.yaml
INFO [main] 2013-08-22 16:00:20,806 Java vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.7.0_25
DEBUG [main] 2013-08-22 16:00:20,856 Default config values: {:rollups300_ttl 2419200, :my_channel_prefix "/agent", :poll_period 60, :kerberos_hostname nil, :storage_dc nil, :thrift_conn_timeout 10000, :thrift_max_frame_size 15728640, :rollups60_ttl 604800, :stomp_port 61620, :shorttime_interval 10, :longtime_interval 300, :private-conf-props ["initial_token""listen_address""broadcast_address""rpc_address"], :thrift_port 9160, :async_retry_timeout 5, :jmx_host "127.0.0.1", :ec2_metadata_api_host "169.254.169.254", :metrics_enabled 1, :async_queue_size 5000, :autodiscovery_interval 120, :rollups7200_ttl 31536000, :autodiscovery_enabled true, :thrift_ssl_truststore nil, :rollup_snapshot_period 300, :is_package false, :monitor_command "/usr/share/opscent-agent/bin/opscenter_agent_monitor", :thrift_socket_timeout 5000, :cassandra_log_location "/var/log/cassandra/system.log", :local_interface "10.14.148.169", :thrift_ssl_truststore_password nil, :jmx_port 7199, :jmx_metrics_threadpool_size 4, :use_ssl 0, :rollups86400_ttl -1, :nodedetails_threadpool_size 3, :api_port 61621, :kerberos_service nil, :kerberos_client_principal nil, :jmx_thread_pool_size 5, :production 1, :stomp_interface "10.14.148.224", :storage_keyspace "OpsCenter", :rollup_snapshot_threshold 300, :thrift_ssl_truststore_type "JKS", :realtime_interval 5}
INFO [main] 2013-08-22 16:00:20,857 Waiting for the config from OpsCenter
INFO [main] 2013-08-22 16:00:20,857 SSL communication is disabled
INFO [main] 2013-08-22 16:00:20,857 Creating stomp connection to 10.14.148.224:61620
INFO [StompConnection receiver] 2013-08-22 16:00:20,861 Reconnecting in 0s.
INFO [main] 2013-08-22 16:00:20,863 Using 10.14.148.169 as the cassandra broadcast address
DEBUG [main] 2013-08-22 16:00:20,869 Not using separate storage cluster for metrics
INFO [main] 2013-08-22 16:00:20,870 Clearing ssl.truststore
INFO [main] 2013-08-22 16:00:20,870 Clearing ssl.truststore.password
INFO [main] 2013-08-22 16:00:20,870 Setting ssl.store.type to JKS
INFO [main] 2013-08-22 16:00:20,870 Clearing kerberos.service.principal.name
INFO [main] 2013-08-22 16:00:20,870 Clearing kerberos.principal
INFO [main] 2013-08-22 16:00:20,871 Clearing kerberos.useTicketCache
INFO [main] 2013-08-22 16:00:20,870 Clearing kerberos.principal
INFO [main] 2013-08-22 16:00:20,871 Clearing kerberos.useTicketCache
INFO [main] 2013-08-22 16:00:20,871 Clearing kerberos.ticketCache
INFO [main] 2013-08-22 16:00:20,871 Clearing kerberos.useKeyTab
ERROR [StompConnection receiver] 2013-08-22 16:00:20,871 failed connecting to 10.14.148.224:61620:java.net.ConnectException: Connection refused
INFO [main] 2013-08-22 16:00:20,871 Clearing kerberos.keyTab
INFO [main] 2013-08-22 16:00:20,872 Clearing kerberos.renewTGT
INFO [main] 2013-08-22 16:00:20,872 Clearing kerberos.debug
INFO [main] 2013-08-22 16:00:20,874 Starting OS metric collectors (Linux)
DEBUG [thrift-init] 2013-08-22 16:00:20,876 Starting Thrift initialization
INFO [thrift-init] 2013-08-22 16:00:20,876 Connecting to Cassandra cluster: 10.14.148.169 (port 9160)
DEBUG [main] 2013-08-22 16:00:20,881 Starting node detail collection
INFO [main] 2013-08-22 16:00:20,883 Starting Cassandra JMX metric collectors
DEBUG [main] 2013-08-22 16:00:20,884 Adding JMX connector to pool (127.0.0.1:7199)
DEBUG [main] 2013-08-22 16:00:20,885 Adding JMX connector to pool (127.0.0.1:7199)
DEBUG [main] 2013-08-22 16:00:20,886 Ignoring the following keyspaces:
DEBUG [main] 2013-08-22 16:00:20,886 Ignoring the following column families:
DEBUG [main] 2013-08-22 16:00:20,886 Ignoring the following solr cores:
INFO [thrift-init] 2013-08-22 16:00:20,887 Downed Host Retry service started with queue size -1 and retry delay 10s
DEBUG [thrift-init] 2013-08-22 16:00:20,890 Concurrent Host pool started with 0 active clients; max: 1 exhausted wait: 10000
DEBUG [main] 2013-08-22 16:00:20,892 Responding to ping request
INFO [install-location-finder] 2013-08-22 16:00:20,892 New JMX connection (127.0.0.1:7199)
INFO [thrift-init] 2013-08-22 16:00:20,892 Registering JMX me.prettyprint.cassandra.service_Agent Cluster:ServiceType=hector,MonitorType=hectorINFO [install-location-finder] 2013-08-22 16:00:20,892 New JMX connection (127.0.0.1:7199)
INFO [thrift-init] 2013-08-22 16:00:20,892 Registering JMX me.prettyprint.cassandra.service_Agent Cluster:ServiceType=hector,MonitorType=hector
INFO [StompConnection receiver] 2013-08-22 16:00:20,892 Reconnecting in 2s.
DEBUG [main] 2013-08-22 16:00:20,894 Refreshing and sending all node details
INFO [main] 2013-08-22 16:00:20,895 New JMX connection (127.0.0.1:7199)
WARN [clojure-agent-send-off-pool-0] 2013-08-22 16:00:20,897 Tried to send message while not connected: /agents-connection ["10.14.148.169","3.2.1"]
DEBUG [thrift-init] 2013-08-22 16:00:20,900 Created 'cluster' object
DEBUG [thrift-init] 2013-08-22 16:00:20,902 Released Thrift initialization lock
DEBUG [pdp-loader] 2013-08-22 16:00:20,910 blocking on queue - current block count 1
DEBUG [thrift-init] 2013-08-22 16:00:20,914 Creating a new thrift connection to 10.14.148.169(10.14.148.169):9160
INFO [thrift-init] 2013-08-22 16:00:21,056 Connected to Cassandra cluster: Sabre NoSQL Cassandra
INFO [pdp-loader] 2013-08-22 16:00:21,056 in execute with client org.apache.cassandra.thrift.Cassandra$Client@3ba0aaa8
DEBUG [thrift-init] 2013-08-22 16:00:21,056 blocking on queue - current block count 1
INFO [thrift-init] 2013-08-22 16:00:21,061 in execute with client org.apache.cassandra.thrift.Cassandra$Client@3ba0aaa8
INFO [pdp-loader] 2013-08-22 16:00:21,062 Attempting to load stored metric values.
INFO [thrift-init] 2013-08-22 16:00:21,062 Using partitioner: org.apache.cassandra.dht.Murmur3Partitioner
DEBUG [thrift-init] 2013-08-22 16:00:21,063 Done initializing Thrift
DEBUG [main] 2013-08-22 16:00:21,065 Collecting real-time data
WARN [clojure-agent-send-off-pool-0] 2013-08-22 16:00:21,075 Tried to send message while not connected: /node-details ["10.14.148.169",{"last-seen":0}]
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:21,077 Tried to send message while not connected: /node-details ["10.14.148.169",{"last-seen":0}]
DEBUG [main] 2013-08-22 16:00:21,082 Collecting short-time data
DEBUG [main] 2013-08-22 16:00:21,094 Fetching all versions
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:21,097 Tried to send message while not connected: /node-details ["10.14.148.169",{"os-load":0.1,"down-nodes":["10.14.148.170"]}]
DEBUG [pdp-loader] 2013-08-22 16:00:21,100 keyspace reseting from null to OpsCenter
INFO [install-location-finder] 2013-08-22 16:00:21,114 New JMX connection (127.0.0.1:7199)
DEBUG [main] 2013-08-22 16:00:21,115 Fetching datacenter for ip 10.14.148.169
DEBUG [main] 2013-08-22 16:00:21,123 Fetching rack for ip 10.14.148.169
DEBUG [main] 2013-08-22 16:00:21,137 Fetching all tasks
DEBUG [main] 2013-08-22 16:00:21,137 Fetching current drain progress
DEBUG [main] 2013-08-22 16:00:21,138 Attempting to fetch old-style compaction info
DEBUG [main] 2013-08-22 16:00:21,139 Getting new-style compaction status.
DEBUG [main] 2013-08-22 16:00:21,140 Got new-style compaction info
DEBUG [main] 2013-08-22 16:00:21,142 Fetching all outgoing stream data
INFO [install-location-finder] 2013-08-22 16:00:21,153 New JMX connection (127.0.0.1:7199)
DEBUG [main] 2013-08-22 16:00:21,164 Collecting long-time values
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:21,186 Tried to send message while not connected: /node-details ["10.14.148.169",{"bootstrapping-nodes":{},"streams":{},"tasks":{},"mode":"normal","data-held":72802.0,"rack":"rack1","datacenter":"datacenter1","version":{"cassandra":"1.2.6","dse":null,"jobtracker":null,"tasktracker":null,"search":null}}]
ERROR [StompConnection receiver] 2013-08-22 16:00:22,894 failed connecting to 10.14.148.224:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2013-08-22 16:00:22,894 Reconnecting in 6s.
DEBUG [Thread-2] 2013-08-22 16:00:23,106 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
DEBUG [main] 2013-08-22 16:00:23,257 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
INFO [jmx-metrics-2] 2013-08-22 16:00:25,889 New JMX connection (127.0.0.1:7199)
DEBUG [main] 2013-08-22 16:00:23,257 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
INFO [jmx-metrics-2] 2013-08-22 16:00:25,889 New JMX connection (127.0.0.1:7199)
DEBUG [jmx-metrics-2] 2013-08-22 16:00:25,908 New CF list: ["system""NodeIdInfo"], ["system_auth""users"], ["system""batchlog"], ["system""hints"], ["OpsCenter""settings"], ["system""HintsColumnFamily"], ["OpsCenter""rollups60"], ["OpsCenter""rollups7200"], ["system_traces""sessions"], ["system""schema_columns"], ["system""peer_events"], ["system""Migrations"], ["system""LocationInfo"], ["OpsCenter""rollups86400"], ["system""schema_keyspaces"], ["OpsCenter""pdps"], ["PNRDATA_KEYSPACE""pnrInfo"], ["OpsCenter""events_timeline"], ["OpsCenter""rollups300"], ["system""peers"], ["system""Schema"], ["system""range_xfers"], ["OpsCenter""events"], ["system""local"], ["system_traces""events"], ["system""schema_columnfamilies"], ["system""IndexInfo"], ["PNRDATA_KEYSPACE""users"]
DEBUG [jmx-metrics-2] 2013-08-22 16:00:25,911 New Solr Cores:
DEBUG [Thread-2] 2013-08-22 16:00:26,108 Unable to retrieve an EC2 instance type: Connect to 169.254.169.254:80 timed out
DEBUG [main] 2013-08-22 16:00:26,108 Unable to retrieve an EC2 instance type: Connect to 169.254.169.254:80 timed out
ERROR [StompConnection receiver] 2013-08-22 16:00:28,895 failed connecting to 10.14.148.224:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2013-08-22 16:00:28,896 Reconnecting in 14s.
DEBUG [Thread-2] 2013-08-22 16:00:29,110 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
DEBUG [main] 2013-08-22 16:00:29,110 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
DEBUG [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2013-08-22 16:00:30,887 Retry service fired... nothing to do.
DEBUG [jmx-metrics-3] 2013-08-22 16:00:30,905 Getting metrics for system.NodeIdInfo
DEBUG [jmx-metrics-3] 2013-08-22 16:00:32,049 Getting metrics for system_auth.users
DEBUG [main] 2013-08-22 16:00:32,110 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
DEBUG [Thread-2] 2013-08-22 16:00:32,111 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:32,136 Tried to send message while not connected: /node-details ["10.14.148.169",{"os_version":"6.3"}]DEBUG [Thread-2] 2013-08-22 16:00:32,111 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:32,136 Tried to send message while not connected: /node-details ["10.14.148.169",{"os_version":"6.3"}]
DEBUG [Thread-2] 2013-08-22 16:00:32,136 Adding JMX connector to pool (127.0.0.1:7199)
DEBUG [Thread-2] 2013-08-22 16:00:32,137 Adding JMX connector to pool (127.0.0.1:7199)
DEBUG [node-details-1] 2013-08-22 16:00:32,139 Starting real-time collection
DEBUG [node-details-2] 2013-08-22 16:00:32,140 Starting short-time collection
DEBUG [node-details-3] 2013-08-22 16:00:32,159 Starting long-time collection
DEBUG [node-details-1] 2013-08-22 16:00:32,162 Collecting real-time data
DEBUG [node-details-3] 2013-08-22 16:00:32,165 Collecting long-time values
DEBUG [node-details-2] 2013-08-22 16:00:32,168 Collecting short-time data
DEBUG [node-details-2] 2013-08-22 16:00:32,169 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:32,174 Fetching all versions
INFO [main] 2013-08-22 16:00:32,178 Starting Jetty server: {:port 61621, :host nil, :ssl? false, :join? false}
DEBUG [main] 2013-08-22 16:00:32,179 Jetty starting in production mode: true
WARN [clojure-agent-send-off-pool-0] 2013-08-22 16:00:32,192 Tried to send message while not connected: /node-details ["10.14.148.169",{"saved-caches-device":"dm_0","sub_os":"Red Hat Enterprise Linux Server","procs":8,"agent_rpc_broadcast_address":"10.14.148.169","network-interfaces":["lo","eth1"],"os":"linux","hostname":"ltxl0334","arch":"amd64","memory":7869,"has-jna":false,"ec2":{"instance-id":null,"instance-type":null,"ami-id":null,"placement":null},"saved-caches-partition":"\/dev\/mapper\/BaseSystem_ROOT","data-devices":["dm_0"],"commitlog-device":"dm_0","data-partitions":["\/dev\/mapper\/BaseSystem_ROOT"],"other-devices":["sdb","sda"],"commitlog-partition":"\/dev\/mapper\/BaseSystem_ROOT","other-partitions":["\/dev\/sda1"],"os_version":"6.3"}]
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:32,193 Tried to send message while not connected: /node-details ["10.14.148.169",{"os-load":0.09}]
DEBUG [node-details-2] 2013-08-22 16:00:32,194 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:32,201 Fetching rack for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:32,194 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:32,201 Fetching rack for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:32,208 Fetching all tasks
DEBUG [node-details-2] 2013-08-22 16:00:32,208 Fetching current drain progress
DEBUG [node-details-2] 2013-08-22 16:00:32,211 Attempting to fetch old-style compaction info
DEBUG [node-details-2] 2013-08-22 16:00:32,214 Getting new-style compaction status.
DEBUG [node-details-2] 2013-08-22 16:00:32,215 Got new-style compaction info
DEBUG [node-details-2] 2013-08-22 16:00:32,215 Fetching all outgoing stream data
DEBUG [node-details-2] 2013-08-22 16:00:32,221 Send requested, but nothing new to send
INFO [Jetty] 2013-08-22 16:00:32,270 Jetty server started
DEBUG [jmx-metrics-3] 2013-08-22 16:00:33,141 Getting metrics for system.batchlog
DEBUG [jmx-metrics-3] 2013-08-22 16:00:34,236 Getting metrics for system.hints
DEBUG [node-details-3] 2013-08-22 16:00:35,111 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
DEBUG [jmx-metrics-3] 2013-08-22 16:00:35,327 Getting metrics for OpsCenter.settings
DEBUG [jmx-metrics-3] 2013-08-22 16:00:36,417 Getting metrics for system.HintsColumnFamily
DEBUG [node-details-1] 2013-08-22 16:00:37,165 Starting real-time collection
DEBUG [node-details-1] 2013-08-22 16:00:37,166 Collecting real-time data
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:37,169 Tried to send message while not connected: /node-details ["10.14.148.169",{"os-load":0.08}]
DEBUG [jmx-metrics-3] 2013-08-22 16:00:37,507 Getting metrics for OpsCenter.rollups60
DEBUG [node-details-3] 2013-08-22 16:00:38,112 Unable to retrieve an EC2 instance type: Connect to 169.254.169.254:80 timed out
DEBUG [jmx-metrics-3] 2013-08-22 16:00:38,598 Getting metrics for OpsCenter.rollups7200
DEBUG [jmx-metrics-3] 2013-08-22 16:00:39,701 Getting metrics for system_traces.sessions
DEBUG [jmx-metrics-3] 2013-08-22 16:00:40,791 Getting metrics for system.schema_columnsDEBUG [jmx-metrics-3] 2013-08-22 16:00:39,701 Getting metrics for system_traces.sessions
DEBUG [jmx-metrics-3] 2013-08-22 16:00:40,791 Getting metrics for system.schema_columns
DEBUG [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2013-08-22 16:00:40,887 Retry service fired... nothing to do.
DEBUG [node-details-3] 2013-08-22 16:00:41,112 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
DEBUG [jmx-metrics-3] 2013-08-22 16:00:41,880 Getting metrics for system.peer_events
DEBUG [node-details-1] 2013-08-22 16:00:42,169 Starting real-time collection
DEBUG [node-details-1] 2013-08-22 16:00:42,170 Collecting real-time data
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:42,174 Tried to send message while not connected: /node-details ["10.14.148.169",{"os-load":0.07}]
DEBUG [node-details-2] 2013-08-22 16:00:42,221 Starting short-time collection
DEBUG [node-details-2] 2013-08-22 16:00:42,222 Collecting short-time data
DEBUG [node-details-2] 2013-08-22 16:00:42,222 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:42,226 Fetching all versions
DEBUG [node-details-2] 2013-08-22 16:00:42,238 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:42,242 Fetching rack for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:42,248 Fetching all tasks
DEBUG [node-details-2] 2013-08-22 16:00:42,248 Fetching current drain progress
DEBUG [node-details-2] 2013-08-22 16:00:42,249 Attempting to fetch old-style compaction info
DEBUG [node-details-2] 2013-08-22 16:00:42,250 Getting new-style compaction status.
DEBUG [node-details-2] 2013-08-22 16:00:42,252 Got new-style compaction info
DEBUG [node-details-2] 2013-08-22 16:00:42,252 Fetching all outgoing stream data
DEBUG [node-details-2] 2013-08-22 16:00:42,256 Send requested, but nothing new to send
ERROR [StompConnection receiver] 2013-08-22 16:00:42,897 failed connecting to 10.14.148.224:61620:java.net.ConnectException: Connection refused
DEBUG [node-details-2] 2013-08-22 16:00:42,256 Send requested, but nothing new to send
ERROR [StompConnection receiver] 2013-08-22 16:00:42,897 failed connecting to 10.14.148.224:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2013-08-22 16:00:42,897 Reconnecting in 30s.
DEBUG [jmx-metrics-3] 2013-08-22 16:00:42,969 Getting metrics for system.Migrations
DEBUG [jmx-metrics-3] 2013-08-22 16:00:44,058 Getting metrics for system.LocationInfo
DEBUG [node-details-3] 2013-08-22 16:00:44,114 Unable to retrieve an EC2 instance id: Connect to 169.254.169.254:80 timed out
DEBUG [node-details-3] 2013-08-22 16:00:44,131 Send requested, but nothing new to send
DEBUG [jmx-metrics-3] 2013-08-22 16:00:45,147 Getting metrics for OpsCenter.rollups86400
DEBUG [jmx-metrics-3] 2013-08-22 16:00:46,237 Getting metrics for system.schema_keyspaces
DEBUG [node-details-1] 2013-08-22 16:00:47,174 Starting real-time collection
DEBUG [node-details-1] 2013-08-22 16:00:47,174 Collecting real-time data
DEBUG [node-details-1] 2013-08-22 16:00:47,178 Send requested, but nothing new to send
DEBUG [jmx-metrics-3] 2013-08-22 16:00:47,326 Getting metrics for OpsCenter.pdps
DEBUG [jmx-metrics-3] 2013-08-22 16:00:48,415 Getting metrics for PNRDATA_KEYSPACE.pnrInfo
DEBUG [jmx-metrics-3] 2013-08-22 16:00:49,504 Getting metrics for OpsCenter.events_timeline
DEBUG [jmx-metrics-3] 2013-08-22 16:00:50,593 Getting metrics for OpsCenter.rollups300
DEBUG [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2013-08-22 16:00:50,888 Retry service fired... nothing to do.
DEBUG [jmx-metrics-3] 2013-08-22 16:00:51,683 Getting metrics for system.peers
DEBUG [node-details-1] 2013-08-22 16:00:52,178 Starting real-time collection
DEBUG [node-details-1] 2013-08-22 16:00:52,179 Collecting real-time data
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:52,183 Tried to send message while not connected: /node-details ["10.14.148.169",{"os-load":0.06}]
DEBUG [node-details-2] 2013-08-22 16:00:52,256 Starting short-time collection
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:00:52,183 Tried to send message while not connected: /node-details ["10.14.148.169",{"os-load":0.06}]
DEBUG [node-details-2] 2013-08-22 16:00:52,256 Starting short-time collection
DEBUG [node-details-2] 2013-08-22 16:00:52,257 Collecting short-time data
DEBUG [node-details-2] 2013-08-22 16:00:52,257 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:52,261 Fetching all versions
DEBUG [node-details-2] 2013-08-22 16:00:52,273 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:52,277 Fetching rack for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:00:52,283 Fetching all tasks
DEBUG [node-details-2] 2013-08-22 16:00:52,283 Fetching current drain progress
DEBUG [node-details-2] 2013-08-22 16:00:52,284 Attempting to fetch old-style compaction info
DEBUG [node-details-2] 2013-08-22 16:00:52,289 Getting new-style compaction status.
DEBUG [node-details-2] 2013-08-22 16:00:52,291 Got new-style compaction info
DEBUG [node-details-2] 2013-08-22 16:00:52,291 Fetching all outgoing stream data
DEBUG [node-details-2] 2013-08-22 16:00:52,296 Send requested, but nothing new to send
DEBUG [jmx-metrics-3] 2013-08-22 16:00:52,771 Getting metrics for system.Schema
DEBUG [jmx-metrics-3] 2013-08-22 16:00:53,858 Getting metrics for system.range_xfers
DEBUG [jmx-metrics-3] 2013-08-22 16:00:54,946 Getting metrics for OpsCenter.events
DEBUG [jmx-metrics-3] 2013-08-22 16:00:56,033 Getting metrics for system.local
DEBUG [jmx-metrics-3] 2013-08-22 16:00:57,119 Getting metrics for system_traces.events
DEBUG [node-details-1] 2013-08-22 16:00:57,183 Starting real-time collection
DEBUG [node-details-1] 2013-08-22 16:00:57,184 Collecting real-time data
DEBUG [node-details-1] 2013-08-22 16:00:57,186 Send requested, but nothing new to send
DEBUG [jmx-metrics-3] 2013-08-22 16:00:58,206 Getting metrics for system.schema_columnfamilies
DEBUG [jmx-metrics-3] 2013-08-22 16:00:59,294 Getting metrics for system.IndexInfo
DEBUG [jmx-metrics-3] 2013-08-22 16:00:58,206 Getting metrics for system.schema_columnfamilies
DEBUG [jmx-metrics-3] 2013-08-22 16:00:59,294 Getting metrics for system.IndexInfo
DEBUG [jmx-metrics-3] 2013-08-22 16:01:00,381 Getting metrics for PNRDATA_KEYSPACE.users
DEBUG [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2013-08-22 16:01:00,888 Retry service fired... nothing to do.
DEBUG [node-details-1] 2013-08-22 16:01:02,187 Starting real-time collection
DEBUG [node-details-1] 2013-08-22 16:01:02,187 Collecting real-time data
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:01:02,190 Tried to send message while not connected: /node-details ["10.14.148.169",{"os-load":0.05}]
DEBUG [node-details-2] 2013-08-22 16:01:02,296 Starting short-time collection
DEBUG [node-details-2] 2013-08-22 16:01:02,297 Collecting short-time data
DEBUG [node-details-2] 2013-08-22 16:01:02,297 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:01:02,301 Fetching all versions
DEBUG [node-details-2] 2013-08-22 16:01:02,311 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:01:02,314 Fetching rack for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:01:02,320 Fetching all tasks
DEBUG [node-details-2] 2013-08-22 16:01:02,320 Fetching current drain progress
DEBUG [node-details-2] 2013-08-22 16:01:02,321 Attempting to fetch old-style compaction info
DEBUG [node-details-2] 2013-08-22 16:01:02,322 Getting new-style compaction status.
DEBUG [node-details-2] 2013-08-22 16:01:02,323 Got new-style compaction info
DEBUG [node-details-2] 2013-08-22 16:01:02,323 Fetching all outgoing stream data
DEBUG [node-details-2] 2013-08-22 16:01:02,328 Send requested, but nothing new to send
DEBUG [node-details-1] 2013-08-22 16:01:07,190 Starting real-time collection
DEBUG [node-details-1] 2013-08-22 16:01:07,191 Collecting real-time data
DEBUG [node-details-1] 2013-08-22 16:01:07,193 Send requested, but nothing new to send
DEBUG [node-details-1] 2013-08-22 16:01:07,191 Collecting real-time data
DEBUG [node-details-1] 2013-08-22 16:01:07,193 Send requested, but nothing new to send
DEBUG [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2013-08-22 16:01:10,888 Retry service fired... nothing to do.
DEBUG [node-details-1] 2013-08-22 16:01:12,194 Starting real-time collection
DEBUG [node-details-1] 2013-08-22 16:01:12,194 Collecting real-time data
WARN [clojure-agent-send-off-pool-1] 2013-08-22 16:01:12,198 Tried to send message while not connected: /node-details ["10.14.148.169",{"os-load":0.04}]
DEBUG [node-details-2] 2013-08-22 16:01:12,328 Starting short-time collection
DEBUG [node-details-2] 2013-08-22 16:01:12,329 Collecting short-time data
DEBUG [node-details-2] 2013-08-22 16:01:12,329 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:01:12,335 Fetching all versions
DEBUG [node-details-2] 2013-08-22 16:01:12,344 Fetching datacenter for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:01:12,348 Fetching rack for ip 10.14.148.169
DEBUG [node-details-2] 2013-08-22 16:01:12,353 Fetching all tasks
DEBUG [node-details-2] 2013-08-22 16:01:12,353 Fetching current drain progress
DEBUG [node-details-2] 2013-08-22 16:01:12,354 Attempting to fetch old-style compaction info
DEBUG [node-details-2] 2013-08-22 16:01:12,355 Getting new-style compaction status.
DEBUG [node-details-2] 2013-08-22 16:01:12,356 Got new-style compaction info
DEBUG [node-details-2] 2013-08-22 16:01:12,356 Fetching all outgoing stream data
DEBUG [node-details-2] 2013-08-22 16:01:12,359 Send requested, but nothing new to send
ERROR [StompConnection receiver] 2013-08-22 16:01:12,898 failed connecting to 10.14.148.224:61620:java.net.ConnectException: Connection refused
INFO [StompConnection receiver] 2013-08-22 16:01:12,898 Reconnecting in 62s.
DEBUG [node-details-1] 2013-08-22 16:01:17,198 Starting real-time collection
F