I have a couple of cassandra nodes installed with a front end and back end nic on separate networks. The opscenter talks to the front end fine except when I try to look at Hadoop Jobs, I get the error "Error loading jobs: Call to /Dev_Master_Cluster1/hadoop/jobs timed out."
If I click on "View Full Details", it redirects to the opscenter on port 50031 (and it's listening) but it errors out with a "Could not connect" error.
In the logs it appears that opscenter is trying to talk to the backend for it's hadoop queries, is this expected and opscenter needs backend access or did I miss something in the config?
front - 10.1.61.x
back - 10.1.35.x
Logs:
2013-04-12 16:53:27+0000 [Dev_Build_Cluster1] INFO: Using 10.1.61.101 as the RPC address for node 10.1.35.101
2013-04-12 16:53:27+0000 [Dev_Build_Cluster1] INFO: Agent for ip 10.1.35.101 is version u'3.0.2'
2013-04-12 16:53:27+0000 [Dev_Build_Cluster1] INFO: Using 10.1.61.100 as the RPC address for node 10.1.35.100
2013-04-12 16:53:27+0000 [Dev_Build_Cluster1] INFO: Using 10.1.61.102 as the RPC address for node 10.1.35.102
2013-04-12 16:53:27+0000 [Dev_Build_Cluster1] INFO: Agent for ip 10.1.35.100 is version u'3.0.2'
2013-04-12 16:53:27+0000 [Dev_Build_Cluster1] INFO: Agent for ip 10.1.35.102 is version u'3.0.2'
2013-04-12 16:53:48+0000 [Dev_Build_Cluster1] WARN: HTTP request https://10.1.35.102:61621/agent-conf? failed: User timeout caused connection failure.
2013-04-12 16:53:48+0000 [Dev_Build_Cluster1] WARN: HTTP request https://10.1.35.100:61621/agent-conf? failed: User timeout caused connection failure.
2013-04-12 16:53:48+0000 [Dev_Build_Cluster1] WARN: HTTP request https://10.1.35.101:61621/agent-conf? failed: User timeout caused connection failure.
2013-04-12 16:54:22+0000 [Dev_Build_Cluster1] ERROR: Client Connection to the JobTracker failed. Here is the connector: <twisted.internet.tcp.Connector instance at 0x29b3908>.
2013-04-12 16:54:22+0000 [Dev_Build_Cluster1] ERROR: Connection Destination: (protocol, host, port): IPv4Address(TCP, '10.1.35.100', 9290)
2013-04-12 16:54:22+0000 [Dev_Build_Cluster1] ERROR: Current connection binding address: None
2013-04-12 16:54:22+0000 [] Unhandled Error
Traceback (most recent call last):
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/log.py", line 84, in callWithLogger
return callWithContext({"system": lp}, func, *args, **kw)
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/log.py", line 69, in callWithContext
return context.call({ILogContext: newCtx}, func, *args, **kw)
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/context.py", line 59, in callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/python/context.py", line 37, in callWithContext
return func(*args,**kw)
--- <exception caught here> ---
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
why = getattr(selectable, method)()
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/tcp.py", line 628, in doConnect
self.failIfNotConnected(error.getConnectError((err, strerror(err))))
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/tcp.py", line 582, in failIfNotConnected
self.connector.connectionFailed(failure.Failure(err))
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/base.py", line 1044, in connectionFailed
self.factory.clientConnectionFailed(self, reason)
File "/usr/lib/python2.7/dist-packages/opscenterd/hadoop/thrift_util.py", line 173, in clientConnectionFailed
File "/usr/share/opscenter/lib/py-debian/2.7/amd64/twisted/internet/protocol.py", line 334, in clientConnectionFailed
self.retry()
File "/usr/lib/python2.7/dist-packages/opscenterd/hadoop/thrift_util.py", line 253, in retry
File "/usr/lib/python2.7/dist-packages/opscenterd/hadoop/thrift_util.py", line 253, in retry
<snip>
File "/usr/lib/python2.7/dist-packages/opscenterd/hadoop/thrift_util.py", line 253, in retry
File "/usr/lib/python2.7/dist-packages/opscenterd/hadoop/thrift_util.py", line 252, in retry
exceptions.RuntimeError: maximum recursion depth exceeded while calling a Python object