Hi.
Our cluster consists of 32 nodes and each node has 32G mem with CentOS 5.8
Recently, we upgraded our cassandra and opscenter to 1.2.3 and 3.0.1(DCE).
After that, some nodes of our cluster got too many CPU load and finally down.
I saw cassandra system log file, and found OpsCenter-using column family data was the major consumer of heap memory.
this problem did not happen when we used 2.1.x opscenter(with cassandra 1.1.8)
Here is log file in our cluster node.
(This log pattern occurs frequently)
WARN [ScheduledTasks:1] 2013-04-12 22:59:30,414 GCInspector.java (line 142) Heap is 0.9732005111098229 full. You may need to reduce memtable and/or cache sizes. Cassandra will now flush up to the two largest memtables to free up memory. Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically
WARN [ScheduledTasks:1] 2013-04-12 22:59:30,415 StorageService.java (line 3541) Flushing CFS(Keyspace='OpsCenter', ColumnFamily='rollups60') to relieve memory pressure
INFO [ScheduledTasks:1] 2013-04-12 22:59:30,415 ColumnFamilyStore.java (line 637) Enqueuing flush of Memtable-rollups60@1713273404(1088/13308 serialized/live bytes, 68 ops)
INFO [FlushWriter:39] 2013-04-12 22:59:30,417 Memtable.java (line 453) Writing Memtable-rollups60@1713273404(1088/13308 serialized/live bytes, 68 ops)
INFO [FlushWriter:39] 2013-04-12 22:59:30,422 Memtable.java (line 487) Completed flushing /database/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-ib-12329-Data.db (1970 bytes) for commitlog position ReplayPosition(segmentId=1365742099627, position=2948892)
INFO [CompactionExecutor:797] 2013-04-12 22:59:30,424 CompactionTask.java (line 106) Compacting [SSTableReader(path='/database/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-ib-12326-Data.db'), SSTableReader(path='/database/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-ib-12327-Data.db'), SSTableReader(path='/database/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-ib-12328-Data.db'), SSTableReader(path='/database/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-ib-12329-Data.db')]
INFO [CompactionExecutor:797] 2013-04-12 22:59:30,444 CompactionTask.java (line 263) Compacted 4 sstables to [/database/cassandra/data/OpsCenter/rollups60/OpsCenter-rollups60-ib-12330,]. 20,423 bytes to 18,904 (~92% of original) in 19ms = 0.948856MB/s. 645 total rows, 608 unique. Row merge counts were {1:572, 2:35, 3:1, 4:0, }
...
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,065 StatusLogger.java (line 109) ColumnFamily Memtable ops,data
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,065 StatusLogger.java (line 112) system.local 2,52
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,065 StatusLogger.java (line 112) system.peers 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,065 StatusLogger.java (line 112) system.batchlog 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,066 StatusLogger.java (line 112) system.NodeIdInfo 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,066 StatusLogger.java (line 112) system.LocationInfo 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,066 StatusLogger.java (line 112) system.Schema 0,0
...
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,073 StatusLogger.java (line 112) OpsCenter.events 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,073 StatusLogger.java (line 112) OpsCenter.rollups60 68,13308
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,074 StatusLogger.java (line 112) OpsCenter.settings 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,074 StatusLogger.java (line 112) OpsCenter.pdps 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,074 StatusLogger.java (line 112) OpsCenter.rollups86400 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,074 StatusLogger.java (line 112) OpsCenter.events_timeline 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,074 StatusLogger.java (line 112) OpsCenter.rollups300 53,20521
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,074 StatusLogger.java (line 112) OpsCenter.rollups7200 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,074 StatusLogger.java (line 112) system_traces.sessions 0,0
INFO [ScheduledTasks:1] 2013-04-12 22:59:38,075 StatusLogger.java (line 112) system_traces.events 0,0
So we stopped opscenter agent for each node, and now our cluster get well.
Is there any solution for this problem?