How to create cmdXXX files and mysqld startup scripts following a SERVER_CRASHED trial

This wiki is a guide to taking a trialXXX.log and associated vardir1_XXX.tar.gz from a SERVER_CRASHED status and running startup.sh to successfully generate a cmdXXX file and other useful files.

The best way to demonstrate the use of this script is to show its execution against a crash. In this case the trial number is 222:

1. joel@tokuts1:/data.ssd/qa/761180$ ls | grep 222

trial222.log
vardir1_222.tar.gz

2. joel@tokuts1:/data.ssd/qa/761180$ ~/percona-qa/startup.sh 222 man

Manual RQG mode active, using RQG and sever directories as found in the RQG run.
BASE directory: /data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64
Extracting vardir (./vardir1_222.tar.gz) to ./vardir1_222
Adding scripts: ./start_mtr222 | start_wipe_mtr222 | ./stop_mtr222 | ./cl_mtr222 | ./cl_binmode_mtr222 | ./test_mtr222 | ./dump_mtr222
 -> These bring up the server (using MTR) with the vardir from the original run, and all mysqld options are preseved
 -> These MTR scripts also preseve the --valgrind option (normal scripts need a ./valgrind222 addition still)
Adding scripts: ./start222 | ./stop222 | ./cl222 | ./cl_binmode222 | ./test222 | ./wipe222 (w/o exe attrib) | ./init222 (w/o exe attrib) | ./dump222
 -> These quickly bring up the server with the same vardir (but no original run mysqld options are passed)
 -> Use for testing if an issues reproduces without specific mysqld options passed at startup
 -> Not using MTR also results in a faster startup, and finally, this uses socket connections which gives faster SQL replay, thereby reducing the probability of non-reproducibility
Adding RQG trial run scripts: ./cmd222 ./cmdtrace222
 -> Use the cmd script to start the trial again in the same way as originally executed, to see if the issue is reproducible.
 -> Use the cmdtrace script to generate an sql trace. Do check that the issue reproduces, otherwise the sql trace is useless.
 -> Note the cmdtrace script runs longer then the original trial. This is to compensate for disk writes. Note --duration= may need further increass.

3. joel@tokuts1:/data.ssd/qa/761180$ ls | grep 222

cl222
cl_binmode222
cl_binmode_mtr222
cl_mtr222
cmd222
cmdtrace222
dump222
dump_mtr222
init222
start222
start_mtr222
start_wipe_mtr222
stop222
stop_mtr222
test222
test_mtr222
trial222.log
vardir1_222
vardir1_222.tar.gz
wipe222

4. joel@tokuts1:/data.ssd/qa/761180$ cat start222 (THIS STARTS THE DB WITH BASIC OPTIONS)

/data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64/bin/mysqld --basedir=/data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64 --datadir=/data.ssd/qa/761180/vardir1_222/master-data --core --socket=/data.ssd/qa/761180/vardir1_222/socket.sock --log-error=/data.ssd/qa/761180/vardir1_222/log/master.err 2>&1 &
echo 'Server socket: /data.ssd/qa/761180/vardir1_222/socket.sock with vardir: /data.ssd/qa/761180/vardir1_222'
echo 'Base Directory: /data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64'

5. joel@tokuts1:/data.ssd/qa/761180$cat cl222 (THIS IS THE CLIENT CONNECTION TO THE DB)

/data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64/bin/mysql -A -uroot -S/data.ssd/qa/761180/vardir1_222/socket.sock test

5. joel@tokuts1:/data.ssd/qa/761180$ cat cmd222 (THIS IS THE TRIAL THAT WAS RUN)

ps -ef | grep 'rundir1_222' | grep -v grep | awk '{print $2}' | xargs sudo kill -9 > /dev/null 2>&1
rm -Rf /data.ssd/qa/761180/rundir1_222
rm -Rf /data.ssd/qa/761180/rundir1_222_epoch
mkdir /data.ssd/qa/761180/rundir1_222
mkdir /data.ssd/qa/761180/rundir1_222_epoch
cd /data.ssd/randgen
bash -c "set -o pipefail; perl /data.ssd/randgen/runall.pl --queries=100000000 --seed=7998 --duration=600 --querytimeout=60 --short_column_names --reporter=Shutdown,Backtrace,QueryTimeout,ErrorLog,ErrorLogAlarm --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --mysqld=--loose-tokudb-check-jemalloc=0 --mysqld=--plugin-load=tokudb=ha_tokudb.so --mysqld=--utility-user-password=test --grammar=conf/percona_qa/5.6/5.6.yy --gendata=conf/percona_qa/5.6/5.6.zz1 --threads=15 --no-mask --basedir=/data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64 --notnull --mysqld=--innodb_change_buffering=none --mysqld=--innodb_changed_pages=FORCE --mysqld=--innodb_file_format=barracuda --mysqld=--innodb_file_per_table=1 --mysqld=--innodb_flush_log_at_trx_commit=1 --mysqld=--innodb_flush_method=O_DIRECT --mysqld=--innodb_log_arch_dir=/data.ssd/qa/761180/rundir1_222_epoch --mysqld=--innodb_log_arch_expire_sec=120 --mysqld=--innodb_log_archive=1 --mysqld=--innodb_log_buffer_size=10485761 --mysqld=--innodb_log_file_size=10485761 --mysqld=--innodb_max_bitmap_file_size=20480 --mysqld=--innodb_max_changed_pages=2 --mysqld=--innodb_track_changed_pages=1 --mysqld=--innodb_use_global_flush_log_at_trx_commit=0 --mysqld=--innodb-buffer-pool-populate --mysqld=--query_cache_size=1048576 --mysqld=--query_cache_type=1 --mysqld=--readonly-key-cache-block-size=0 --mysqld=--readonly-loose-max-connect-errors=1 --mysqld=--slow_query_log --mysqld=--transaction-isolation=READ-UNCOMMITTED --mysqld=--userstat --mysqld=--utility-user=roel@% --mysqld=--utility-user-password=test --mysqld=--utility-user-schema-access=mysqlinformation_schema --mysqld=--innodb_purge_threads=32 --mysqld=--innodb_buffer_pool_instances=8 --mtr-build-thread=738 --mask=9760 --vardir1=/data.ssd/qa/761180/rundir1_222 > /data.ssd/qa/761180/cmd222.log 2>&1"tail -n1 /data.ssd/qa/761180/cmd222.log

6. joel@tokuts1:/data.ssd/qa/761180$ cat stop222 (THIS STOPS THE DB)

/data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64/bin/mysqladmin -uroot -S/data.ssd/qa/761180/vardir1_222/socket.sock shutdown
echo 'Server on socket /data.ssd/qa/761180/vardir1_222/socket.sock with vardir /data.ssd/qa/761180/vardir1_222 halted'

7. joel@tokuts1:/data.ssd/qa/761180$ cat start_mtr222 (THIS STARTS THE DB WITH ALL OPTIONS USED)

cd /data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64/mysql-test/
perl lib/v1/mysql-test-run.pl --start-and-exit --start-dirty --skip-ndb --mysqld=--core-file --mysqld=--loose-new --mysqld=--sql-mode=no_engine_substitution --mysqld=--relay-log=slave-relay-bin --mysqld=--loose-innodb --mysqld=--secure-file-priv= --mysqld=--max-allowed-packet=16Mb --mysqld=--loose-innodb-status-file=1 --mysqld=--master-retry-count=65535 --mysqld=--loose-debug-assert-if-crashed-table --mysqld=--loose-debug-assert-on-error --mysqld=--skip-name-resolve --vardir=/data.ssd/qa/761180/vardir1_222 --master_port=16346 --mysqld="--log-output=none" --mysqld="--sql_mode=ONLY_FULL_GROUP_BY" --mysqld="--loose-tokudb-check-jemalloc=0" --mysqld="--plugin-load=tokudb=ha_tokudb.so" --mysqld="--utility-user-password=test" --mysqld="--innodb_change_buffering=none" --mysqld="--innodb_changed_pages=FORCE" --mysqld="--innodb_file_format=barracuda" --mysqld="--innodb_file_per_table=1" --mysqld="--innodb_flush_log_at_trx_commit=1" --mysqld="--innodb_flush_method=O_DIRECT" --mysqld="--innodb_log_arch_dir=/data.ssd/qa/761180/_epoch/1400824232907890610" --mysqld="--innodb_log_arch_expire_sec=120" --mysqld="--innodb_log_archive=1" --mysqld="--innodb_log_buffer_size=10485761" --mysqld="--innodb_log_file_size=10485761" --mysqld="--innodb_max_bitmap_file_size=20480" --mysqld="--innodb_max_changed_pages=2" --mysqld="--innodb_track_changed_pages=1" --mysqld="--innodb_use_global_flush_log_at_trx_commit=0" --mysqld="--innodb-buffer-pool-populate" --mysqld="--query_cache_size=1048576" --mysqld="--query_cache_type=1" --mysqld="--readonly-key-cache-block-size=0" --mysqld="--readonly-loose-max-connect-errors=1" --mysqld="--slow_query_log" --mysqld="--transaction-isolation=READ-UNCOMMITTED" --mysqld="--userstat" --mysqld="--utility-user=roel@%" --mysqld="--utility-user-password=test" --mysqld="--utility-user-schema-access=mysqlinformation_schema" --mysqld="--innodb_purge_threads=32" --mysqld="--innodb_buffer_pool_instances=8" 1st 2>&1 
echo 'If you would like to create a new vardir @ /data.ssd/qa/761180/vardir1_222, then please remove --start-dirty from ./start_mtr222'
echo 'Server socket: /data.ssd/qa/761180/vardir1_222/tmp/master.sock with vardir: /data.ssd/qa/761180/vardir1_222'
echo 'Base Directory: /data.ssd/Percona-Server-5.6.17-rel65.0-603-38-debug.Linux.x86_64'