Hi, I was able to get one of my Cassandra clusters monitored with OpsCenter Community Edition on Amazon, my second cluster has not gone as easily. There are two big differences in the clusters
1) they are different cassandra versions (the working cluster is 1.2.4, the one not working is 1.2.3 if that matters)
2) the working cluster has OpsCenter installed on one of the seed nodes, the one not working is a stand-alone machine
There have been other odd things so I'm sure there must be some other differences, here is an oddity
3) on the cluster that is not being monitored well, I had to add the local_interface to the address.yaml file (saw this in another post)
4) i had to manually install my agents (not sure if I'd have to do this again now that I've come so far)
I've gotten to the point where the opscenter web console is showing SOME of the statistics (Cluster Reads / Writes, Cluster Latency, Schema and Data Explorer), but some of the information is not showing (0 of 3 agents connected, Storage Capacity and Disk IOPS).
The /var/log/opscenter/opscenterd.log seems to confirm this. I can connect to the various URLs w/ wget but the agents seem to return no data. Is there something inconsistent on this cluster?
[root@ip-10-10-10-218 opscenter]# cat opscenterd.log
2013-07-26 03:54:37+0000 [] INFO: Log opened.
2013-07-26 03:54:37+0000 [] INFO: twistd 10.2.0 (/usr/bin/python2.6 2.6.8) starting up.
2013-07-26 03:54:37+0000 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor.
2013-07-26 03:54:37+0000 [] INFO: set uid/gid 220/498
2013-07-26 03:54:37+0000 [] INFO: Logging level set to 'debug'
2013-07-26 03:54:37+0000 [] INFO: OpsCenter version: 3.2.0
2013-07-26 03:54:37+0000 [] INFO: Compatible agent version: 3.2.0
2013-07-26 03:54:37+0000 [] DEBUG: Main config options:
2013-07-26 03:54:37+0000 [] DEBUG: agents : [('use_ssl', 'false')], webserver : [('interface', '0.0.0.0'), ('port', '8888')], logging : [('level', 'DEBUG')], authentication : [('passwd_file', '/etc/opscenter/.passwd')],
2013-07-26 03:54:37+0000 [] DEBUG: Loading all per-cluster config files
2013-07-26 03:54:37+0000 [] INFO: Loading per-cluster config file /etc/opscenter/clusters/Provider_Services_Development_Cluster.conf
2013-07-26 03:54:37+0000 [] DEBUG: Done loading all per-cluster config files
2013-07-26 03:54:37+0000 [] DEBUG: Per-cluster config for Provider_Services_Development_Cluster:
2013-07-26 03:54:37+0000 [] DEBUG: jmx : [('username', ''), ('password', ''), ('port', '7199')], kerberos_client_principals : [], kerberos : [], agents : [], kerberos_hostnames : [], kerberos_services : [], cassandra : [('username', ''), ('api_port', '9160'), ('password', ''), ('seed_hosts', '10.10.10.87,10.10.10.88')],
2013-07-26 03:54:38+0000 [] INFO: HTTP BASIC authentication enabled
2013-07-26 03:54:38+0000 [] INFO: Starting webserver with ssl disabled.
2013-07-26 03:54:38+0000 [] INFO: SSL disabled
2013-07-26 03:54:38+0000 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888
2013-07-26 03:54:38+0000 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x28f3320>
2013-07-26 03:54:38+0000 [] INFO: morbid.morbid.StompFactory starting on 61619
2013-07-26 03:54:38+0000 [] INFO: Starting factory <morbid.morbid.StompFactory instance at 0x29fd998>
2013-07-26 03:54:38+0000 [] INFO: Configuring agent communication with ssl support disabled.
2013-07-26 03:54:38+0000 [] INFO: morbid.morbid.StompFactory starting on 61620
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: Starting services for cluster Provider_Services_Development_Cluster
2013-07-26 03:54:38+0000 [] DEBUG: Not using SSL for Thrift communication
2013-07-26 03:54:38+0000 [] DEBUG: Not using Kerberos authentication for Thrift
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Not using separate storage cluster
2013-07-26 03:54:38+0000 [] INFO: Metric caching enabled with 50 points and 1000 metrics cached
2013-07-26 03:54:38+0000 [] INFO: Starting PushService
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Adding connection to <CassandraNode 10.10.10.88:9160 @0x27e4950>
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Adding connection to <CassandraNode 10.10.10.87:9160 @0x27e49e0>
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Adding connection to <CassandraNode 10.10.10.88:9160 @0x27e4950>
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Adding connection to <CassandraNode 10.10.10.87:9160 @0x27e49e0>
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Adding connection to <CassandraNode 10.10.10.88:9160 @0x27e4950>
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Waiting for describe_version() results
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: Starting CassandraCluster service
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: agent_config items: {'cassandra_log_location': '/var/log/cassandra/system.log', 'thrift_port': 9160, 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'metrics_ignored_solr_cores': '', 'api_port': '61621', 'metrics_enabled': 1, 'thrift_ssl_truststore_type': 'JKS', 'kerberos_use_ticket_cache': True, 'kerberos_renew_tgt': True, 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'ec2_metadata_api_host': '169.254.169.254', 'provisioning': 0, 'kerberos_use_keytab': True, 'metrics_ignored_column_families': '', 'thrift_ssl_truststore_password': None, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter'}
2013-07-26 03:54:38+0000 [] INFO: OS Version: Linux version 3.4.43-43.43.amzn1.x86_64 (mockbuild@gobi-build-31003) (gcc version 4.6.3 20120306 (Red Hat 4.6.3-2) (GCC) ) #1 SMP Mon May 6 18:04:41 UTC 2013
2013-07-26 03:54:38+0000 [] INFO: CPU Info: ['1799.999']
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Successfully added connection to <CassandraNode 10.10.10.88:9160 @0x27e4950> to the pool
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Successfully added connection to <CassandraNode 10.10.10.87:9160 @0x27e49e0> to the pool
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Successfully added connection to <CassandraNode 10.10.10.87:9160 @0x27e49e0> to the pool
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Successfully added connection to <CassandraNode 10.10.10.88:9160 @0x27e4950> to the pool
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Successfully added connection to <CassandraNode 10.10.10.88:9160 @0x27e4950> to the pool
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:54:38+0000 [] INFO: Mem Info: 1656MB
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: Snitch: org.apache.cassandra.locator.SimpleSnitch
2013-07-26 03:54:38+0000 [] DEBUG: EC2 Instance type: m1.small
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: Cluster Name: Provider Services Development Cluster
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: Partitioner: org.apache.cassandra.dht.ByteOrderedPartitioner
2013-07-26 03:54:38+0000 [] DEBUG: EC2 Instance ami: (unknown)
2013-07-26 03:54:38+0000 [] INFO: Package Manager: Unknown
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: Recognizing new node 10.10.10.88 ('e046ebe8b649f756076e780c84b0cad0')
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: Recognizing new node 10.10.10.87 ('56d34da899abd1626137411f6b4153d7')
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] INFO: Recognizing new node 10.10.10.86 ('73797374656d')
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Sending agent config to 10.10.10.88 synchronously
2013-07-26 03:54:38+0000 [] DEBUG: Not returning SASL credentials for 10.10.10.88
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (PUT): http://10.10.10.88:61621/agent-conf?, body: {'metrics_ignored_column_families': '', 'cassandra_log_location': '/var/log/cassandra/system.log', 'jmx_user': '', 'thrift_port': 9160, 'jmx_pass': '', 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'ec2_metadata_api_host': '169.254.169.254', 'metrics_ignored_solr_cores': '', 'rollup_subscriptions': [], 'api_port': '61621', 'metrics_enabled': 1, 'kerberos_use_ticket_cache': True, 'thrift_ssl_truststore_type': 'JKS', 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'thrift_rpc_interface': '10.10.10.88', 'thrift_user': '', 'provisioning': 0, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter', 'kerberos_use_keytab': True, 'kerberos_renew_tgt': True, 'thrift_ssl_truststore_password': '****', 'thrift_pass': ''}
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Sending agent config to 10.10.10.86 synchronously
2013-07-26 03:54:38+0000 [] DEBUG: Not returning SASL credentials for 10.10.10.86
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (PUT): http://10.10.10.86:61621/agent-conf?, body: {'metrics_ignored_column_families': '', 'cassandra_log_location': '/var/log/cassandra/system.log', 'jmx_user': '', 'thrift_port': 9160, 'jmx_pass': '', 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'ec2_metadata_api_host': '169.254.169.254', 'metrics_ignored_solr_cores': '', 'rollup_subscriptions': [], 'api_port': '61621', 'metrics_enabled': 1, 'kerberos_use_ticket_cache': True, 'thrift_ssl_truststore_type': 'JKS', 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'thrift_rpc_interface': '10.10.10.86', 'thrift_user': '', 'provisioning': 0, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter', 'kerberos_use_keytab': True, 'kerberos_renew_tgt': True, 'thrift_ssl_truststore_password': '****', 'thrift_pass': ''}
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Sending agent config to 10.10.10.87 synchronously
2013-07-26 03:54:38+0000 [] DEBUG: Not returning SASL credentials for 10.10.10.87
2013-07-26 03:54:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (PUT): http://10.10.10.87:61621/agent-conf?, body: {'metrics_ignored_column_families': '', 'cassandra_log_location': '/var/log/cassandra/system.log', 'jmx_user': '', 'thrift_port': 9160, 'jmx_pass': '', 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'ec2_metadata_api_host': '169.254.169.254', 'metrics_ignored_solr_cores': '', 'rollup_subscriptions': [], 'api_port': '61621', 'metrics_enabled': 1, 'kerberos_use_ticket_cache': True, 'thrift_ssl_truststore_type': 'JKS', 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'thrift_rpc_interface': '10.10.10.87', 'thrift_user': '', 'provisioning': 0, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter', 'kerberos_use_keytab': True, 'kerberos_renew_tgt': True, 'thrift_ssl_truststore_password': '****', 'thrift_pass': ''}
2013-07-26 03:54:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.88:61621/agent-conf? failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:54:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.86:61621/agent-conf? failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:54:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.87:61621/agent-conf? failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:54:40+0000 [Provider_Services_Development_Cluster] DEBUG: Current number of column families: 51
2013-07-26 03:54:40+0000 [Provider_Services_Development_Cluster] INFO: Keyspaces: {...)}
2013-07-26 03:54:41+0000 [Provider_Services_Development_Cluster] INFO: Initializing event storage.
2013-07-26 03:54:41+0000 [Provider_Services_Development_Cluster] INFO: Attempting to load all persisted alert rules
2013-07-26 03:54:41+0000 [Provider_Services_Development_Cluster] DEBUG: Attempting to load all persisted scheduled job descriptions
2013-07-26 03:54:41+0000 [Provider_Services_Development_Cluster] INFO: Done initializing event storage.
2013-07-26 03:54:41+0000 [Provider_Services_Development_Cluster] INFO: Done loading persisted scheduled job descriptions
2013-07-26 03:54:41+0000 [Provider_Services_Development_Cluster] INFO: Done loading persisted alert rules
2013-07-26 03:54:41+0000 [Provider_Services_Development_Cluster] INFO: OpsCenter starting up.
2013-07-26 03:55:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:55:38+0000 [] DEBUG: Average opscenterd CPU usage: 6.33%, memory usage: 40 MB
2013-07-26 03:55:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:55:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.88 has an unknown topology, collecting topology info from another node now
2013-07-26 03:55:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88, body: {}
2013-07-26 03:55:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.87 has an unknown topology, collecting topology info from another node now
2013-07-26 03:55:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87, body: {}
2013-07-26 03:55:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.86 has an unknown topology, collecting topology info from another node now
2013-07-26 03:55:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.86, body: {}
2013-07-26 03:55:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:55:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:55:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:55:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:55:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.86 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:55:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.86 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:56:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:56:38+0000 [] DEBUG: Average opscenterd CPU usage: 3.05%, memory usage: 41 MB
2013-07-26 03:56:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:57:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:57:38+0000 [] DEBUG: Average opscenterd CPU usage: 2.49%, memory usage: 41 MB
2013-07-26 03:57:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:57:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.88 has an unknown topology, collecting topology info from another node now
2013-07-26 03:57:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.88, body: {}
2013-07-26 03:57:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.87 has an unknown topology, collecting topology info from another node now
2013-07-26 03:57:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87, body: {}
2013-07-26 03:57:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.86 has an unknown topology, collecting topology info from another node now
2013-07-26 03:57:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.86, body: {}
2013-07-26 03:57:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:57:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:57:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.88 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:57:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.88 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:57:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.86 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:57:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.86 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:58:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:58:38+0000 [] DEBUG: Average opscenterd CPU usage: 2.77%, memory usage: 42 MB
2013-07-26 03:58:39+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:59:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:59:38+0000 [] DEBUG: Average opscenterd CPU usage: 2.83%, memory usage: 42 MB
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.88 has an unknown topology, collecting topology info from another node now
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.88, body: {}
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.87 has an unknown topology, collecting topology info from another node now
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.87, body: {}
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.86 has an unknown topology, collecting topology info from another node now
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.86, body: {}
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.88 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.88 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.86 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.86 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.87 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 03:59:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.87 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:00:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:00:38+0000 [] DEBUG: Average opscenterd CPU usage: 3.53%, memory usage: 44 MB
2013-07-26 04:00:39+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:01:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:01:38+0000 [] DEBUG: Average opscenterd CPU usage: 2.65%, memory usage: 44 MB
2013-07-26 04:01:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:01:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.88 has an unknown topology, collecting topology info from another node now
2013-07-26 04:01:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88, body: {}
2013-07-26 04:01:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.87 has an unknown topology, collecting topology info from another node now
2013-07-26 04:01:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87, body: {}
2013-07-26 04:01:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.86 has an unknown topology, collecting topology info from another node now
2013-07-26 04:01:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.86, body: {}
2013-07-26 04:01:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:01:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.87 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:01:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:01:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:01:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.86 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:01:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.86 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:02:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:02:38+0000 [] DEBUG: Average opscenterd CPU usage: 2.56%, memory usage: 44 MB
2013-07-26 04:02:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:03:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:03:38+0000 [] DEBUG: Average opscenterd CPU usage: 2.65%, memory usage: 44 MB
2013-07-26 04:03:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:03:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.88 has an unknown topology, collecting topology info from another node now
2013-07-26 04:03:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.88, body: {}
2013-07-26 04:03:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.87 has an unknown topology, collecting topology info from another node now
2013-07-26 04:03:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.87, body: {}
2013-07-26 04:03:39+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.86 has an unknown topology, collecting topology info from another node now
2013-07-26 04:03:39+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.86, body: {}
2013-07-26 04:03:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.87 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:03:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.87 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:03:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.86 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:03:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.88:61621/cluster/topology?node_ip=10.10.10.86 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:03:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.88 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:03:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.88 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:04:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:04:38+0000 [] DEBUG: Average opscenterd CPU usage: 2.69%, memory usage: 44 MB
2013-07-26 04:04:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:05:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:05:38+0000 [] DEBUG: Average opscenterd CPU usage: 3.08%, memory usage: 44 MB
2013-07-26 04:05:38+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
2013-07-26 04:05:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.88 has an unknown topology, collecting topology info from another node now
2013-07-26 04:05:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88, body: {}
2013-07-26 04:05:38+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.87 has an unknown topology, collecting topology info from another node now
2013-07-26 04:05:38+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.87, body: {}
2013-07-26 04:05:39+0000 [Provider_Services_Development_Cluster] DEBUG: Node 10.10.10.86 has an unknown topology, collecting topology info from another node now
2013-07-26 04:05:39+0000 [Provider_Services_Development_Cluster] DEBUG: Performing HTTP request (GET): http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.86, body: {}
2013-07-26 04:05:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:05:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.86:61621/cluster/topology?node_ip=10.10.10.88 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:05:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.86 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:05:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.86 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:05:39+0000 [Provider_Services_Development_Cluster] WARN: HTTP request http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.87 failed: [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:05:39+0000 [Provider_Services_Development_Cluster] WARN: Unable to collect datacenter, rack information: Failed query to http://10.10.10.87:61621/cluster/topology?node_ip=10.10.10.87 : [<twisted.python.failure.Failure <class 'twisted.internet.error.ConnectionDone'>>]
2013-07-26 04:06:08+0000 [Provider_Services_Development_Cluster] DEBUG: Collecting node/token list over Thrift
[root@ip-10-10-10-218 opscenter]#