I am posting this here for others to benefit if you happen to experience this. Perhaps someone will have some idea on why this happened.
I did a stock install of the dse 3.1.1 .deb install on ubuntu 10.04 and followed the config directions for a single node. Everything came up nicely, and I spent a day or so creating & dropping various kinds of tables from examples I have grokd on the web. Since the default cassandra.yaml only binds the service ports to localhost, I could not hit the node from the net. So, I edited cassandra.yaml and commented out the 2 lines that bind the ports to localhost. When I tried to restart DSE, I got this is system.log:
INFO 12:12:11,433 Scheduling row cache save to each 0 seconds (going to save all keys).
INFO 12:12:11,537 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-2 (236 bytes)
INFO 12:12:11,537 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-3 (280 bytes)
INFO 12:12:11,573 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ic-21 (5099 bytes)
INFO 12:12:11,577 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columnfamilies-KeyCache-b.db
INFO 12:12:11,589 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ic-21 (3361 bytes)
INFO 12:12:11,593 reading saved cache /var/lib/cassandra/saved_caches/system-schema_columns-KeyCache-b.db
INFO 12:12:11,600 Opening /var/lib/cassandra/data/system/peers/system-peers-ic-1 (108 bytes)
INFO 12:12:11,606 Opening /var/lib/cassandra/data/system/local/system-local-ic-14 (541 bytes)
INFO 12:12:11,607 Opening /var/lib/cassandra/data/system/local/system-local-ic-15 (139 bytes)
INFO 12:12:11,610 reading saved cache /var/lib/cassandra/saved_caches/system-local-KeyCache-b.db
INFO 12:12:12,104 Opening /var/lib/cassandra/data/system_auth/users/system_auth-users-ic-1 (64 bytes)
INFO 12:12:12,115 completed pre-loading (7 keys) key cache.
INFO 12:12:12,123 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1375881240468.log, /var/lib/cassandra/commitlog/CommitLog-2-1375881240469.log, /var/lib/cassandra/commitlog/CommitLog-2-1375881240470.log, /var/lib/cassandra/commitlog/CommitLog-2-1376064619532.log
INFO 12:12:12,137 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1375881240468.log
INFO 12:12:12,237 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1375881240468.log
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
Caused by: java.lang.IndexOutOfBoundsException: index (2) must be less than size (2)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
at com.google.common.collect.RegularImmutableList.get(RegularImmutableList.java:81)
at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at java.util.TreeMap.put(TreeMap.java:547)
at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:397)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:202)
at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:97)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:298)
at com.datastax.bdp.server.DseDaemon.setup(DseDaemon.java:164)
at org.apache.cassandra.service.CassandraDaemon.init(CassandraDaemon.java:375)
... 5 more
Cannot load daemon
Service exit with a return value of 3
Here is the details:
INFO 12:12:08,120 DSE version: 3.1.1
INFO 12:12:08,123 Hadoop version: 1.0.4.8
INFO 12:12:08,124 Hive version: 0.9.0.1
INFO 12:12:08,124 Pig version: 0.9.2
INFO 12:12:08,124 Solr version: 4.3.0.1
INFO 12:12:08,125 Sqoop version: 1.4.2.3
INFO 12:12:08,125 Mahout version: 0.6
INFO 12:12:08,126 Appender version: 1.1
INFO 12:12:08,157 JVM vendor/version: OpenJDK 64-Bit Server VM/1.6.0_20
WARN 12:12:08,158 OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
INFO 12:12:08,158 Heap size: 8375238656/8375238656
The system.log had this in it:
INFO [main] 2013-08-09 12:12:11,610 AutoSavingCache.java (line 140) reading saved cache /var/lib/cassandra/saved_caches/system-local-KeyCache-b.db
INFO [SSTableBatchOpen:1] 2013-08-09 12:12:12,104 SSTableReader.java (line 169) Opening /var/lib/cassandra/data/system_auth/users/system_auth-users-ic-1 (64 bytes)
INFO [main] 2013-08-09 12:12:12,115 CassandraDaemon.java (line 281) completed pre-loading (7 keys) key cache.
INFO [main] 2013-08-09 12:12:12,123 CommitLog.java (line 125) Replaying /var/lib/cassandra/commitlog/CommitLog-2-1375881240468.log, /var/lib/cassandra/commitlog/CommitLog-2-1375881240469.log, /var/lib/cassandra/commitlog/CommitLog-2-1375881240470.log, /var/lib/cassandra/commitlog/CommitLog-2-1376064619532.log
INFO [main] 2013-08-09 12:12:12,137 CommitLogReplayer.java (line 119) Replaying /var/lib/cassandra/commitlog/CommitLog-2-1375881240468.log
INFO [main] 2013-08-09 12:12:12,237 CommitLogReplayer.java (line 274) Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1375881240468.log
I googled'd around for awhile and really could not find anything that pointed me in the right direction, so I just deleted the /var/lib/cassandra folder and started over :-( Not like I lost anything special, but I really wish I knew what caused that because this will happen for sure later when I have something in there I really want.
If anyone has seen this and found a cause, please share!!
-Regards!!