I am running a 3 node development cluster on AWS EC2 using the Datastax AMI. DSE 3.0.1, Cassandra 1.1.9.3.
I upgraded to OpsCenter 3.1. I restarted OpsCenter, clicked "fix" agents, entered credentials and refeshed. All agents connected. OpsCenter loads, but no keyspaces are visible and the error "Error loading alerts: Alerts have not been loaded yet. There may be a connectivity problem with Cassandra" is displayed.
My column families were created with CQL3 and use compound primary keys. I cannot see any schema info after the upgrade.
I would appreciate any ideas on how to resolve this issue!
Log entries:
2013-04-29 18:57:06+0000 [] INFO: Testing SSH connectivity to 10.115.11.199, 10.218.41.204, 10.78.222.11
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 2048 09:47:46:51:d5:f9:30:f2:56:7f:dc:56:d6:18:de:a0 _ (RSA) for 10.218.41.204
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 256 49:ad:ca:6b:1b:17:8f:f6:8e:3b:8a:c6:7e:3c:aa:e5 _ (ECDSA) for 10.218.41.204
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 1024 c8:97:cf:53:3c:aa:e3:c4:74:92:2e:92:7d:0b:31:1f _ (DSA) for 10.218.41.204
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 2048 d7:fa:27:c5:8a:da:3b:69:00:8b:31:87:93:1a:a7:3b _ (RSA) for 10.78.222.11
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 2048 ae:6f:39:a2:37:9d:ea:75:94:f7:09:9e:b5:d7:05:1a _ (RSA) for 10.115.11.199
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 256 e2:9d:2d:f1:fe:2e:86:7b:b3:f5:4a:81:bf:59:fe:d1 _ (ECDSA) for 10.78.222.11
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 1024 aa:d8:90:28:2b:9f:8d:df:84:86:9d:c7:32:4c:a7:fb _ (DSA) for 10.78.222.11
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 256 6c:b4:e1:2b:36:d5:ac:6c:a7:25:b4:77:61:fc:d1:ae _ (ECDSA) for 10.115.11.199
2013-04-29 18:57:07+0000 [] INFO: Obtained fingerprint 1024 c8:93:7b:67:d0:04:14:ea:cd:26:56:6f:ac:27:6e:58 _ (DSA) for 10.115.11.199
2013-04-29 18:57:08+0000 [] INFO: Testing SSH connectivity to 10.115.11.199, 10.218.41.204, 10.78.222.11
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 2048 09:47:46:51:d5:f9:30:f2:56:7f:dc:56:d6:18:de:a0 _ (RSA) for 10.218.41.204
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 2048 d7:fa:27:c5:8a:da:3b:69:00:8b:31:87:93:1a:a7:3b _ (RSA) for 10.78.222.11
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 256 49:ad:ca:6b:1b:17:8f:f6:8e:3b:8a:c6:7e:3c:aa:e5 _ (ECDSA) for 10.218.41.204
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 2048 ae:6f:39:a2:37:9d:ea:75:94:f7:09:9e:b5:d7:05:1a _ (RSA) for 10.115.11.199
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 256 e2:9d:2d:f1:fe:2e:86:7b:b3:f5:4a:81:bf:59:fe:d1 _ (ECDSA) for 10.78.222.11
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 1024 c8:97:cf:53:3c:aa:e3:c4:74:92:2e:92:7d:0b:31:1f _ (DSA) for 10.218.41.204
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 256 6c:b4:e1:2b:36:d5:ac:6c:a7:25:b4:77:61:fc:d1:ae _ (ECDSA) for 10.115.11.199
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 1024 aa:d8:90:28:2b:9f:8d:df:84:86:9d:c7:32:4c:a7:fb _ (DSA) for 10.78.222.11
2013-04-29 18:57:09+0000 [] INFO: Obtained fingerprint 1024 c8:93:7b:67:d0:04:14:ea:cd:26:56:6f:ac:27:6e:58 _ (DSA) for 10.115.11.199
2013-04-29 18:57:09+0000 [DEV_CLUSTER] INFO: Beginning install of OpsCenter agent to 10.115.11.199
2013-04-29 18:57:09+0000 [DEV_CLUSTER] INFO: Installing debian package on 10.115.11.199
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Beginning install of OpsCenter agent to 10.218.41.204
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Beginning install of OpsCenter agent to 10.78.222.11
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Version: {'search': None, 'dse': '3.0.1', 'tasktracker': None, 'jobtracker': None, 'cassandra': '1.1.9.3'}
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Node 10.115.11.199 changed its mode to normal
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Using 10.115.11.199 as the RPC address for node 10.115.11.199
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Using 10.115.11.199 as the RPC address for node 10.115.11.199
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Installing debian package on 10.218.41.204
2013-04-29 18:57:24+0000 [DEV_CLUSTER] INFO: Installing debian package on 10.78.222.11
2013-04-29 18:57:38+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-04-29 18:57:38+0000 [DEV_CLUSTER] INFO: Node 10.78.222.11 changed its mode to normal
2013-04-29 18:57:38+0000 [DEV_CLUSTER] INFO: Using 10.78.222.11 as the RPC address for node 10.78.222.11
2013-04-29 18:57:38+0000 [DEV_CLUSTER] INFO: Using 10.78.222.11 as the RPC address for node 10.78.222.11
2013-04-29 18:57:40+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-04-29 18:57:40+0000 [DEV_CLUSTER] INFO: Using 10.218.41.204 as the RPC address for node 10.218.41.204
2013-04-29 18:57:40+0000 [DEV_CLUSTER] INFO: Node 10.218.41.204 changed its mode to normal
2013-04-29 18:57:40+0000 [DEV_CLUSTER] INFO: Using 10.218.41.204 as the RPC address for node 10.218.41.204
2013-04-29 20:52:01+0000 [] INFO: No handlers could be found for logger "xhtml2pdf"
2013-04-30 13:06:15+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-04-30 13:06:15+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-04-30 13:06:15+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-01 14:59:03+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-01 14:59:03+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-01 14:59:03+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-01 21:42:33+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-01 21:42:33+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-01 21:44:59+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-01 21:44:59+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-01 21:54:52+0000 [] ERROR: Problem while calling CFDataController: unpack requires a string argument of length 8
2013-05-01 21:54:52+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-01 21:54:52+0000 [] ERROR: Problem while calling CFDataController: unpack requires a string argument of length 8
2013-05-01 21:54:52+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-01 21:56:42+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-01 21:56:42+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-01 22:03:42+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-01 22:03:42+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-01 22:18:07+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-01 22:18:07+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-01 22:18:07+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-01 22:18:07+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-02 14:07:12+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-02 14:07:12+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-02 14:07:12+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-02 14:09:30+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-02 14:09:30+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-02 15:34:25+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-02 15:34:25+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-02 15:36:44+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-02 15:36:44+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-02 16:08:10+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-02 16:08:10+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-02 16:08:10+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-02 16:08:10+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-02 17:50:35+0000 [] ERROR: Problem while calling CFDataController: year is out of range
2013-05-02 17:50:35+0000 [] ERROR:
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 845, in getCFPage
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 753, in _unpack_rows
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 771, in _unpack_columns
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 785, in _unpack_col
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftVals.py", line 334, in unpack_date
2013-05-03 02:07:44+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-03 02:07:44+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-03 02:07:44+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-03 14:07:45+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-03 14:07:45+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-03 14:07:45+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-04 02:08:20+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-04 02:08:20+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-04 02:08:20+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-04 14:08:27+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-04 14:08:27+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-04 14:08:27+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-05 02:08:37+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-05 02:08:37+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-05 02:08:37+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-05 14:08:48+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-05 14:08:48+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-05 14:08:48+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-06 02:08:53+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-06 02:08:53+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-06 02:08:53+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-06 14:08:54+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-06 14:08:54+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-06 14:08:54+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-07 02:09:47+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-07 02:09:47+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-07 02:09:47+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-07 15:42:19+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-07 15:42:19+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-07 15:42:19+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-08 15:06:24+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-08 15:06:24+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-08 15:06:24+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-10 18:12:27+0000 [DEV_CLUSTER] INFO: Agent for ip 10.78.222.11 is version u'3.0.2'
2013-05-10 18:12:27+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-10 18:12:28+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-10 19:32:19+0000 [] INFO: Received SIGTERM, shutting down.
2013-05-10 19:32:19+0000 [DEV_CLUSTER] INFO: OpsCenter shutting down.
2013-05-10 19:32:19+0000 [] INFO: (TCP Port 61620 Closed)
2013-05-10 19:32:19+0000 [] INFO: (TCP Port 61619 Closed)
2013-05-10 19:32:19+0000 [] INFO: Stopping factory <morbid.morbid.StompFactory instance at 0x2e2e128>
2013-05-10 19:32:19+0000 [] INFO: (TCP Port 8888 Closed)
2013-05-10 19:32:19+0000 [] INFO: Stopping factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x2e28440>
2013-05-10 19:32:19+0000 [DEV_CLUSTER] INFO: Stopping CassandraCluster service
2013-05-10 19:32:19+0000 [DEV_CLUSTER] INFO: OpsCenter shutting down.
2013-05-10 19:32:19+0000 [] INFO: Main loop terminated.
2013-05-10 19:32:19+0000 [] INFO: Server Shut Down.
2013-05-10 19:32:27+0000 [] INFO: Log opened.
2013-05-10 19:32:27+0000 [] INFO: twistd 10.2.0 (/usr/bin/python2.7 2.7.3) starting up.
2013-05-10 19:32:27+0000 [] INFO: reactor class: twisted.internet.selectreactor.SelectReactor.
2013-05-10 19:32:27+0000 [] INFO: set uid/gid 0/0
2013-05-10 19:32:27+0000 [] INFO: Logging level set to 'info'
2013-05-10 19:32:27+0000 [] INFO: OpsCenter version: 3.1.0
2013-05-10 19:32:27+0000 [] INFO: Compatible agent version: 3.1.0
2013-05-10 19:32:27+0000 [] INFO: Loading per-cluster config file /etc/opscenter/clusters/DEV_CLUSTER.conf
2013-05-10 19:32:27+0000 [] INFO: HTTP BASIC authentication disabled
2013-05-10 19:32:27+0000 [] INFO: Starting webserver with ssl disabled.
2013-05-10 19:32:27+0000 [] INFO: SSL agent communication enabled
2013-05-10 19:32:27+0000 [] INFO: opscenterd.WebServer.OpsCenterdWebServer starting on 8888
2013-05-10 19:32:27+0000 [] INFO: Starting factory <opscenterd.WebServer.OpsCenterdWebServer instance at 0x3fd33f8>
2013-05-10 19:32:27+0000 [] INFO: morbid.morbid.StompFactory starting on 61619
2013-05-10 19:32:27+0000 [] INFO: Starting factory <morbid.morbid.StompFactory instance at 0x3ff2fc8>
2013-05-10 19:32:27+0000 [] INFO: Configuring agent communication with ssl support enabled.
2013-05-10 19:32:27+0000 [] INFO: morbid.morbid.StompFactory starting on 61620
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Starting services for cluster DEV_CLUSTER
2013-05-10 19:32:27+0000 [] INFO: Starting PushService
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Starting CassandraCluster service
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: agent_config items: {'cassandra_log_location': '/var/log/cassandra/system.log', 'thrift_port': 9160, 'thrift_ssl_truststore': None, 'rollups300_ttl': 2419200, 'rollups86400_ttl': -1, 'jmx_port': 7199, 'metrics_ignored_solr_cores': '', 'api_port': '61621', 'metrics_enabled': 1, 'thrift_ssl_truststore_type': 'JKS', 'kerberos_use_ticket_cache': True, 'kerberos_renew_tgt': True, 'rollups60_ttl': 604800, 'cassandra_install_location': '', 'rollups7200_ttl': 31536000, 'kerberos_debug': False, 'storage_keyspace': 'OpsCenter', 'ec2_metadata_api_host': '169.254.169.254', 'provisioning': 0, 'kerberos_use_keytab': True, 'metrics_ignored_column_families': '', 'thrift_ssl_truststore_password': None, 'metrics_ignored_keyspaces': 'system, system_traces, system_auth, dse_auth, OpsCenter'}
2013-05-10 19:32:27+0000 [] INFO: OS Version: Linux version 3.2.0-35-virtual (buildd@allspice) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #55-Ubuntu SMP Wed Dec 5 18:02:05 UTC 2012
2013-05-10 19:32:27+0000 [] INFO: CPU Info: ['2666.760', '2666.760']
2013-05-10 19:32:27+0000 [] INFO: Mem Info: 17079MB
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Cluster Name: DEV_CLUSTER
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Snitch: com.datastax.bdp.snitch.DseDelegateSnitch
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Partitioner: org.apache.cassandra.dht.RandomPartitioner
2013-05-10 19:32:27+0000 [] INFO: Package Manager: aptitude
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Recognizing new node 10.115.11.199 ('113427455640312821154458202477256070485')
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Recognizing new node 10.218.41.204 ('0')
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Recognizing new node 10.78.222.11 ('56713727820156410577229101238628035242')
2013-05-10 19:32:27+0000 [DEV_CLUSTER] ERROR: Error getting keyspaces: [Failure instance: Traceback: <type 'exceptions.KeyError'>: 'cf_test'
/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py:361:callback
/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py:455:_startRunCallbacks
/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py:542:_runCallbacks
/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py:1076:gotResult
--- <exception caught here> ---
/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py:1020:_inlineCallbacks
/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py:408:getNonCompactCQL3ColumnFamilies
]
2013-05-10 19:32:27+0000 [DEV_CLUSTER] ERROR: Error when attempting to create OpsCenter schema: (KeyError) 'cf_test'
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Sleeping for 10s before retrying schema creation
2013-05-10 19:32:27+0000 [] INFO: Unhandled error in Deferred:
2013-05-10 19:32:27+0000 [] Unhandled Error
Traceback (most recent call last):
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 361, in callback
self._startRunCallbacks(result)
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 455, in _startRunCallbacks
self._runCallbacks()
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 542, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1076, in gotResult
_inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/defer.py", line 1020, in _inlineCallbacks
result = g.send(result)
File "/usr/lib/python2.7/dist-packages/opscenterd/ThriftService.py", line 408, in getNonCompactCQL3ColumnFamilies
exceptions.KeyError: 'cf_test'
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Version: {'search': None, 'dse': '3.0.1', 'tasktracker': None, 'jobtracker': None, 'cassandra': '1.1.9.3'}
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Node 10.218.41.204 changed its mode to normal
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Node 10.115.11.199 changed its mode to normal
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Node 10.78.222.11 changed its mode to normal
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Using 10.78.222.11 as the RPC address for node 10.78.222.11
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Using 10.218.41.204 as the RPC address for node 10.218.41.204
2013-05-10 19:32:27+0000 [DEV_CLUSTER] INFO: Using 10.115.11.199 as the RPC address for node 10.115.11.199
2013-05-10 19:32:29+0000 [DEV_CLUSTER] INFO: Agent for ip 10.218.41.204 is version u'3.0.2'
2013-05-10 19:32:29+0000 [DEV_CLUSTER] INFO: Agent for ip 10.115.11.199 is version u'3.0.2'
2013-05-10 19:32:29+0000 [DEV_CLUSTER] IN