I'm using Solr secondary indexes on a ~2k row with about 50 columns. With commitlog_total_space_in_mb set to 1024 on a config change and rolling restart, each commit log is taking about 4 minutes to replay. At 32MB/log that's roughly 2 hours to restart node EACH node. Is this really expected? CPU is 75% idle, disk I/O is non-existent. I'm not sure what's throttling it unnecessarily?
INFO [WrapperSimpleAppMain] 2013-08-14 11:17:39,586 CommitLogReplayer.java (line 119) Replaying /var/clog/cassandra/commitlog/CommitLog-2-1376342658133.log
INFO [WrapperSimpleAppMain] 2013-08-14 11:20:58,850 CommitLogReplayer.java (line 274) Finished reading /var/clog/cassandra/commitlog/CommitLog-2-1376342658133.log
INFO [WrapperSimpleAppMain] 2013-08-14 11:20:58,851 CommitLogReplayer.java (line 119) Replaying /var/clog/cassandra/commitlog/CommitLog-2-1376342658134.log
INFO [WrapperSimpleAppMain] 2013-08-14 11:24:22,366 CommitLogReplayer.java (line 274) Finished reading /var/clog/cassandra/commitlog/CommitLog-2-1376342658134.log