heartbeat[22503]: 2007/09/20_14:30:26 info: killing /usr/lib/heartbeat/mgmtd -v process group 22523 with signal 15 mgmtd[22523]: 2007/09/20_14:30:26 info: mgmtd is shutting down mgmtd[22523]: 2007/09/20_14:30:26 debug: [mgmtd] stopped heartbeat[22503]: 2007/09/20_14:30:26 info: killing /usr/lib/heartbeat/crmd process group 22522 with signal 15 crmd[22522]: 2007/09/20_14:30:26 info: crm_shutdown: Requesting shutdown crmd[22522]: 2007/09/20_14:30:26 info: do_shutdown_req: Sending shutdown request to DC: host-1.awale.qc.ca cibmon[22516]: 2007/09/20_14:30:27 info: cibmon_diff: [cib_diff_notify] cib_apply_diff confirmed cibmon[22516]: 2007/09/20_14:30:27 info: log_cib_diff: cib_apply_diff: Diff: --- 0.1.25 cibmon[22516]: 2007/09/20_14:30:27 info: log_cib_diff: cib_apply_diff: Diff: +++ 0.1.26 cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + crmd[22522]: 2007/09/20_14:30:27 info: do_lrm_rsc_op: Performing op=share-drbd_0:0_notify_0 key=57:41:49d75c42-75a0-4bf8-ad1c-7b1910776561) cibmon[22516]: 2007/09/20_14:30:27 info: cibmon_diff: [cib_diff_notify] cib_apply_diff confirmed cibmon[22516]: 2007/09/20_14:30:27 info: log_cib_diff: cib_apply_diff: Diff: --- 0.1.26 cibmon[22516]: 2007/09/20_14:30:27 info: log_cib_diff: cib_apply_diff: Diff: +++ 0.1.27 cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:27 info: log_data_element: cib_apply_diff: + lrmd[22519]: 2007/09/20_14:30:27 info: RA output: (share-drbd_0:0:notify:stdout) r0: ACTION=notify drbd[25252][25263]: 2007/09/20_14:30:28 DEBUG: r0 notify: pre for stop - counts: active 0 - starting 0 - stopping 1 crmd[22522]: 2007/09/20_14:30:28 info: process_lrm_event: LRM operation share-drbd_0:0_notify_0 (call=64, rc=0) complete crmd[22522]: 2007/09/20_14:30:28 info: do_lrm_rsc_op: Performing op=share-drbd_0:0_stop_0 key=6:41:49d75c42-75a0-4bf8-ad1c-7b1910776561) crmd[22522]: 2007/09/20_14:30:28 info: process_lrm_event: LRM operation share-drbd_0:0_monitor_120000 (call=61, rc=-2) Cancelled cibmon[22516]: 2007/09/20_14:30:28 info: cibmon_diff: [cib_diff_notify] cib_apply_diff confirmed cibmon[22516]: 2007/09/20_14:30:28 info: log_cib_diff: cib_apply_diff: Diff: --- 0.1.27 cibmon[22516]: 2007/09/20_14:30:28 info: log_cib_diff: cib_apply_diff: Diff: +++ 0.1.28 cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:28 info: log_data_element: cib_apply_diff: + lrmd[22519]: 2007/09/20_14:30:28 info: RA output: (share-drbd_0:0:stop:stdout) r0: ACTION=stop drbd[25264][25276]: 2007/09/20_14:30:29 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[25264][25283]: 2007/09/20_14:30:29 DEBUG: r0: Exit code 0 drbd[25264][25284]: 2007/09/20_14:30:29 DEBUG: r0: Command output: Secondary/Secondary drbd[25264][25292]: 2007/09/20_14:30:29 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 drbd[25264][25305]: 2007/09/20_14:30:29 DEBUG: r0: Exit code 0 drbd[25264][25306]: 2007/09/20_14:30:29 DEBUG: r0: Command output: Connected drbd[25264][25307]: 2007/09/20_14:30:29 DEBUG: r0 status: Secondary/Secondary Secondary Secondary Connected drbd[25264][25308]: 2007/09/20_14:30:29 DEBUG: r0: Calling /usr/sbin/crm_master -D crm_master[25310]: 2007/09/20_14:30:29 info: Invoked: /usr/sbin/crm_master -D cibmon[22516]: 2007/09/20_14:30:29 info: cibmon_diff: [cib_diff_notify] cib_apply_diff confirmed cibmon[22516]: 2007/09/20_14:30:29 info: log_cib_diff: cib_apply_diff: Diff: --- 0.1.28 cibmon[22516]: 2007/09/20_14:30:29 info: log_cib_diff: cib_apply_diff: Diff: +++ 0.1.29 cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:29 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:30 info: cibmon_diff: [cib_diff_notify] cib_apply_diff confirmed cibmon[22516]: 2007/09/20_14:30:30 info: log_cib_diff: cib_apply_diff: Diff: --- 0.1.29 cibmon[22516]: 2007/09/20_14:30:30 info: log_cib_diff: cib_apply_diff: Diff: +++ 0.1.30 cibmon[22516]: 2007/09/20_14:30:30 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:30 info: log_data_element: cib_apply_diff: + drbd[25264][25311]: 2007/09/20_14:30:30 DEBUG: r0: Exit code 0 drbd[25264][25312]: 2007/09/20_14:30:30 DEBUG: r0: Command output: crm_master[25310]: 2007/09/20_14:30:29 info: Invoked: /usr/sbin/crm_master -D lrmd[22519]: 2007/09/20_14:30:30 info: RA output: (share-drbd_0:0:stop:stdout) crm_master[25310]: 2007/09/20_14:30:29 info: Invoked: /usr/sbin/crm_master -D drbd[25264][25313]: 2007/09/20_14:30:30 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf down r0 drbd[25264][25321]: 2007/09/20_14:30:30 DEBUG: r0: Exit code 0 drbd[25264][25322]: 2007/09/20_14:30:30 DEBUG: r0: Command output: lrmd[22519]: 2007/09/20_14:30:30 info: RA output: (share-drbd_0:0:stop:stdout) drbd[25264][25323]: 2007/09/20_14:30:30 DEBUG: r0 stop: drbdadm down succeeded. drbd[25264][25325]: 2007/09/20_14:30:30 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[25264][25332]: 2007/09/20_14:30:30 DEBUG: r0: Exit code 0 drbd[25264][25333]: 2007/09/20_14:30:30 DEBUG: r0: Command output: Not configured drbd[25264][25341]: 2007/09/20_14:30:30 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 drbd[25264][25354]: 2007/09/20_14:30:30 DEBUG: r0: Exit code 0 drbd[25264][25355]: 2007/09/20_14:30:30 DEBUG: r0: Command output: Not configured drbd[25264][25356]: 2007/09/20_14:30:30 DEBUG: r0 status: Not configured Not configured Not configured Unconfigured crmd[22522]: 2007/09/20_14:30:30 info: process_lrm_event: LRM operation share-drbd_0:0_stop_0 (call=65, rc=0) complete crmd[22522]: 2007/09/20_14:30:31 info: do_state_transition: State transition S_NOT_DC -> S_STOPPING [ input=I_STOP cause=C_HA_MESSAGE origin=route_message ] crmd[22522]: 2007/09/20_14:30:31 info: do_state_transition: State transition S_STOPPING -> S_TERMINATE [ input=I_TERMINATE cause=C_FSA_INTERNAL origin=do_shutdown ] crmd[22522]: 2007/09/20_14:30:31 info: verify_stopped: Checking for active resources before exit cibmon[22516]: 2007/09/20_14:30:31 info: cibmon_diff: [cib_diff_notify] cib_apply_diff confirmed cibmon[22516]: 2007/09/20_14:30:31 info: log_cib_diff: cib_apply_diff: Diff: --- 0.1.30 cibmon[22516]: 2007/09/20_14:30:31 info: log_cib_diff: cib_apply_diff: Diff: +++ 0.1.31 cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - ccm[22517]: 2007/09/20_14:30:31 info: client (pid=22522) removed from ccm crmd[22522]: 2007/09/20_14:30:31 info: verify_stopped: Checking for active resources before exit cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: - cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cibmon[22516]: 2007/09/20_14:30:31 info: log_data_element: cib_apply_diff: + cib[22518]: 2007/09/20_14:30:31 WARN: send_via_callback_channel: Cannot find client for token c3fc49c9-1d60-46f1-8a1e-ed64cbbf9fab crmd[22522]: 2007/09/20_14:30:31 info: do_lrm_control: Disconnected from the LRM cib[22518]: 2007/09/20_14:30:31 WARN: do_local_notify: A-Sync reply to 22522 failed: client left before we could send reply crmd[22522]: 2007/09/20_14:30:31 info: do_ha_control: Disconnected from Heartbeat crmd[22522]: 2007/09/20_14:30:31 info: do_cib_control: Disconnecting CIB heartbeat[22503]: 2007/09/20_14:30:31 info: killing /usr/lib/heartbeat/attrd process group 22521 with signal 15 crmd[22522]: 2007/09/20_14:30:31 info: crmd_cib_connection_destroy: Connection to the CIB terminated... crmd[22522]: 2007/09/20_14:30:31 info: do_exit: Performing A_EXIT_0 - gracefully exiting the CRMd attrd[22521]: 2007/09/20_14:30:31 info: attrd_shutdown: Exiting crmd[22522]: 2007/09/20_14:30:31 info: free_mem: Dropping I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=verify_stopped ] crmd[22522]: 2007/09/20_14:30:31 info: free_mem: Dropping I_TERMINATE: [ state=S_TERMINATE cause=C_FSA_INTERNAL origin=verify_stopped ] attrd[22521]: 2007/09/20_14:30:31 info: main: Exiting... crmd[22522]: 2007/09/20_14:30:31 info: do_exit: [crmd] stopped (0) heartbeat[22503]: 2007/09/20_14:30:31 info: killing /usr/lib/heartbeat/stonithd process group 22520 with signal 15 stonithd[22520]: 2007/09/20_14:30:31 notice: /usr/lib/heartbeat/stonithd normally quit. heartbeat[22503]: 2007/09/20_14:30:31 info: killing /usr/lib/heartbeat/lrmd -r process group 22519 with signal 15 lrmd[22519]: 2007/09/20_14:30:31 info: lrmd is shutting down heartbeat[22503]: 2007/09/20_14:30:31 info: killing /usr/lib/heartbeat/cib process group 22518 with signal 15 cib[22518]: 2007/09/20_14:30:31 WARN: disconnect_cib_client: Disconnecting /cib_ro... cib[22518]: 2007/09/20_14:30:31 WARN: disconnect_cib_client: Disconnecting cibmon/cib_callback... cib[22518]: 2007/09/20_14:30:31 info: cib_shutdown: Disconnected 2 clients cibmon[22516]: 2007/09/20_14:30:31 ERROR: cib_native_msgready: Message pending on command channel [22518] cib[22518]: 2007/09/20_14:30:31 info: cib_process_disconnect: All clients disconnected... cibmon[22516]: 2007/09/20_14:30:31 ERROR: crm_log_message_adv: #========= cib:cmd message start ==========# cib[22518]: 2007/09/20_14:30:31 info: initiate_exit: Sending disconnect notification to 2 peers... cibmon[22516]: 2007/09/20_14:30:31 ERROR: MSG: No message to dump cibmon[22516]: 2007/09/20_14:30:31 info: cib_native_msgready: Lost connection to the CIB service [22518]. cibmon[22516]: 2007/09/20_14:30:31 CRIT: cib_native_dispatch: Lost connection to the CIB service [22518/callback]. cibmon[22516]: 2007/09/20_14:30:31 ERROR: cib_connection_destroy: Connection to the CIB terminated... exiting cib[22518]: 2007/09/20_14:30:31 info: cib_process_shutdown_req: Shutdown ACK from host-1.awale.qc.ca cib[22518]: 2007/09/20_14:30:31 info: terminate_ha_connection: cib_process_shutdown_req: Disconnecting heartbeat cib[22518]: 2007/09/20_14:30:32 info: cib_ha_connection_destroy: Heartbeat disconnection complete... exiting cib[22518]: 2007/09/20_14:30:32 info: main: Done ccm[22517]: 2007/09/20_14:30:32 info: client (pid=22518) removed from ccm heartbeat[22503]: 2007/09/20_14:30:32 info: killing /usr/lib/heartbeat/ccm process group 22517 with signal 15 ccm[22517]: 2007/09/20_14:30:32 info: received SIGTERM, going to shut down heartbeat[22503]: 2007/09/20_14:30:32 info: client [/usr/lib/heartbeat/cibmon -d] is not running. heartbeat[22503]: 2007/09/20_14:30:33 info: killing HBFIFO process 22507 with signal 15 heartbeat[22503]: 2007/09/20_14:30:33 info: killing HBWRITE process 22508 with signal 15 heartbeat[22503]: 2007/09/20_14:30:33 info: killing HBREAD process 22509 with signal 15 heartbeat[22503]: 2007/09/20_14:30:33 info: killing HBWRITE process 22510 with signal 15 heartbeat[22503]: 2007/09/20_14:30:33 info: killing HBREAD process 22511 with signal 15 heartbeat[22503]: 2007/09/20_14:30:33 info: Core process 22507 exited. 5 remaining heartbeat[22503]: 2007/09/20_14:30:33 info: Core process 22511 exited. 4 remaining heartbeat[22503]: 2007/09/20_14:30:33 info: Core process 22508 exited. 3 remaining heartbeat[22503]: 2007/09/20_14:30:33 info: Core process 22509 exited. 2 remaining heartbeat[22503]: 2007/09/20_14:30:33 info: Core process 22510 exited. 1 remaining heartbeat[22503]: 2007/09/20_14:30:33 info: host-2.awale.qc.ca Heartbeat shutdown complete. logd[22481]: 2007/09/20_14:30:48 debug: logd_term_action: received SIGTERM logd[22481]: 2007/09/20_14:30:48 debug: logd_term_action: waiting for 0 messages to be read by write process logd[22481]: 2007/09/20_14:30:48 debug: logd_term_action: sending SIGTERM to write process logd[22484]: 2007/09/20_14:30:48 info: logd_term_write_action: received SIGTERM logd[22484]: 2007/09/20_14:30:48 debug: Writing out 0 messages then quitting logd[22484]: 2007/09/20_14:30:48 info: Exiting write process logd[25409]: 2007/09/20_14:54:35 info: logd started with /etc/logd.cf. logd[25411]: 2007/09/20_14:54:35 info: G_main_add_SignalHandler: Added signal handler for signal 15 logd[25409]: 2007/09/20_14:54:35 info: G_main_add_SignalHandler: Added signal handler for signal 15 heartbeat[25430]: 2007/09/20_14:54:35 info: No log entry found in ha.cf -- use logd heartbeat[25430]: 2007/09/20_14:54:35 info: Enabling logging daemon heartbeat[25430]: 2007/09/20_14:54:35 info: logfile and debug file are those specified in logd config file (default /etc/logd.cf) heartbeat[25430]: 2007/09/20_14:54:35 info: ************************** heartbeat[25430]: 2007/09/20_14:54:35 info: Configuration validated. Starting heartbeat 2.1.2 heartbeat[25431]: 2007/09/20_14:54:35 info: heartbeat: version 2.1.2 heartbeat[25431]: 2007/09/20_14:54:35 info: Heartbeat generation: 1188360864 heartbeat[25431]: 2007/09/20_14:54:35 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[25431]: 2007/09/20_14:54:35 info: G_main_add_TriggerHandler: Added signal manual handler heartbeat[25431]: 2007/09/20_14:54:35 info: Removing /var/run/heartbeat/rsctmp failed, recreating. heartbeat[25431]: 2007/09/20_14:54:35 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0 heartbeat[25431]: 2007/09/20_14:54:35 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0 - Status: 1 heartbeat[25431]: 2007/09/20_14:54:35 info: glib: ping group heartbeat started. heartbeat[25431]: 2007/09/20_14:54:35 info: G_main_add_SignalHandler: Added signal handler for signal 17 heartbeat[25431]: 2007/09/20_14:54:35 info: Local status now set to: 'up' heartbeat[25431]: 2007/09/20_14:54:37 info: Link host-1.awale.qc.ca:eth0 up. heartbeat[25431]: 2007/09/20_14:54:37 info: Status update for node host-1.awale.qc.ca: status up heartbeat[25431]: 2007/09/20_14:54:37 info: Link local:local up. heartbeat[25431]: 2007/09/20_14:54:37 info: Status update for node local: status ping heartbeat[25431]: 2007/09/20_14:54:37 debug: get_delnodelist: delnodelist= heartbeat[25431]: 2007/09/20_14:54:37 info: Link host-2.awale.qc.ca:eth0 up. heartbeat[25431]: 2007/09/20_14:54:37 info: Comm_now_up(): updating status to active heartbeat[25431]: 2007/09/20_14:54:37 info: Local status now set to: 'active' heartbeat[25431]: 2007/09/20_14:54:37 info: Starting child client "/usr/lib/heartbeat/cibmon -d" (90,90) heartbeat[25431]: 2007/09/20_14:54:37 info: Starting child client "/usr/lib/heartbeat/ccm" (90,90) heartbeat[25431]: 2007/09/20_14:54:37 info: Starting child client "/usr/lib/heartbeat/cib" (90,90) heartbeat[25442]: 2007/09/20_14:54:37 info: Starting "/usr/lib/heartbeat/cibmon -d" as uid 90 gid 90 (pid 25442) heartbeat[25431]: 2007/09/20_14:54:37 info: Starting child client "/usr/lib/heartbeat/lrmd -r" (0,0) heartbeat[25431]: 2007/09/20_14:54:37 info: Starting child client "/usr/lib/heartbeat/stonithd" (0,0) heartbeat[25431]: 2007/09/20_14:54:37 info: Starting child client "/usr/lib/heartbeat/attrd" (90,90) heartbeat[25431]: 2007/09/20_14:54:37 info: Starting child client "/usr/lib/heartbeat/crmd" (90,90) heartbeat[25431]: 2007/09/20_14:54:37 info: Starting child client "/usr/lib/heartbeat/mgmtd -v" (0,0) heartbeat[25431]: 2007/09/20_14:54:37 WARN: G_CH_dispatch_int: Dispatch function for read child took too long to execute: 70 ms (> 50 ms) (GSource: 0x910b110) heartbeat[25431]: 2007/09/20_14:54:37 info: Status update for node host-1.awale.qc.ca: status active heartbeat[25444]: 2007/09/20_14:54:37 info: Starting "/usr/lib/heartbeat/cib" as uid 90 gid 90 (pid 25444) cibmon[25442]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 15 heartbeat[25449]: 2007/09/20_14:54:37 info: Starting "/usr/lib/heartbeat/mgmtd -v" as uid 0 gid 0 (pid 25449) heartbeat[25448]: 2007/09/20_14:54:37 info: Starting "/usr/lib/heartbeat/crmd" as uid 90 gid 90 (pid 25448) heartbeat[25445]: 2007/09/20_14:54:37 info: Starting "/usr/lib/heartbeat/lrmd -r" as uid 0 gid 0 (pid 25445) heartbeat[25447]: 2007/09/20_14:54:37 info: Starting "/usr/lib/heartbeat/attrd" as uid 90 gid 90 (pid 25447) heartbeat[25446]: 2007/09/20_14:54:37 info: Starting "/usr/lib/heartbeat/stonithd" as uid 0 gid 0 (pid 25446) heartbeat[25443]: 2007/09/20_14:54:37 info: Starting "/usr/lib/heartbeat/ccm" as uid 90 gid 90 (pid 25443) attrd[25447]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 15 lrmd[25445]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 15 ccm[25443]: 2007/09/20_14:54:37 info: Hostname: host-2.awale.qc.ca lrmd[25445]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 17 lrmd[25445]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 10 lrmd[25445]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 12 lrmd[25445]: 2007/09/20_14:54:37 info: Started. attrd[25447]: 2007/09/20_14:54:37 info: register_with_ha: Hostname: host-2.awale.qc.ca cib[25444]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 15 cib[25444]: 2007/09/20_14:54:37 info: G_main_add_TriggerHandler: Added signal manual handler cib[25444]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 17 cib[25444]: 2007/09/20_14:54:37 info: main: Retrieval of a per-action CIB: disabled cib[25444]: 2007/09/20_14:54:37 info: readCibXmlFile: Reading cluster configuration from: /var/lib/heartbeat/crm/cib.xml crmd[25448]: 2007/09/20_14:54:37 info: main: CRM Hg Version: 9a995a8a7052 tip cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] attrd[25447]: 2007/09/20_14:54:37 info: register_with_ha: UUID: 77c6bcf6-f1e9-4e62-a33e-6ddd3530b92e cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] stonithd[25446]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 10 cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] stonithd[25446]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 12 cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: log_data_element: readCibXmlFile: [on-disk] cib[25444]: 2007/09/20_14:54:37 info: startCib: CIB Initialization completed successfully cib[25444]: 2007/09/20_14:54:37 info: cib_register_ha: Signing in with Heartbeat cib[25444]: 2007/09/20_14:54:37 info: cib_register_ha: FSA Hostname: host-2.awale.qc.ca cib[25444]: 2007/09/20_14:54:37 WARN: cib_init: CCM Activation failed cib[25444]: 2007/09/20_14:54:37 WARN: cib_init: CCM Connection failed 1 times (30 max) crmd[25448]: 2007/09/20_14:54:37 info: crmd_init: Starting crmd crmd[25448]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 15 crmd[25448]: 2007/09/20_14:54:37 info: G_main_add_TriggerHandler: Added signal manual handler crmd[25448]: 2007/09/20_14:54:37 info: G_main_add_SignalHandler: Added signal handler for signal 17 stonithd[25446]: 2007/09/20_14:54:38 info: Signing in with heartbeat. mgmtd[25449]: 2007/09/20_14:54:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 mgmtd[25449]: 2007/09/20_14:54:38 debug: Enabling coredumps mgmtd[25449]: 2007/09/20_14:54:38 info: G_main_add_SignalHandler: Added signal handler for signal 10 mgmtd[25449]: 2007/09/20_14:54:38 info: G_main_add_SignalHandler: Added signal handler for signal 12 stonithd[25446]: 2007/09/20_14:54:38 notice: /usr/lib/heartbeat/stonithd start up successfully. stonithd[25446]: 2007/09/20_14:54:38 info: G_main_add_SignalHandler: Added signal handler for signal 17 ccm[25443]: 2007/09/20_14:54:38 info: G_main_add_SignalHandler: Added signal handler for signal 15 mgmtd[25449]: 2007/09/20_14:54:38 info: init_crm cib[25444]: 2007/09/20_14:54:39 info: cib_init: Starting cib mainloop cibmon[25442]: 2007/09/20_14:54:39 info: main: Starting mainloop cib[25444]: 2007/09/20_14:54:39 info: cib_null_callback: Setting cib_diff_notify callbacks for cibmon: on crmd[25448]: 2007/09/20_14:54:39 info: do_cib_control: CIB connection established crmd[25448]: 2007/09/20_14:54:39 info: register_with_ha: Hostname: host-2.awale.qc.ca cib[25450]: 2007/09/20_14:54:39 info: write_cib_contents: Wrote version 0.1.1 of the CIB to disk (digest: 7c02397728dc64d1f796707d398af8a1) cib[25444]: 2007/09/20_14:54:39 info: cib_client_status_callback: Status update: Client host-2.awale.qc.ca/cib now has status [join] cib[25444]: 2007/09/20_14:54:39 info: cib_client_status_callback: Status update: Client host-2.awale.qc.ca/cib now has status [online] cib[25444]: 2007/09/20_14:54:39 info: cib_null_callback: Setting cib_refresh_notify callbacks for crmd: on cib[25444]: 2007/09/20_14:54:39 info: cib_null_callback: Setting cib_diff_notify callbacks for mgmtd: on heartbeat[25431]: 2007/09/20_14:54:39 WARN: 1 lost packet(s) for [host-1.awale.qc.ca] [38:40] cib[25444]: 2007/09/20_14:54:39 info: cib_client_status_callback: Status update: Client host-1.awale.qc.ca/cib now has status [online] heartbeat[25431]: 2007/09/20_14:54:39 info: No pkts missing from host-1.awale.qc.ca! crmd[25448]: 2007/09/20_14:54:39 info: register_with_ha: UUID: 77c6bcf6-f1e9-4e62-a33e-6ddd3530b92e crmd[25448]: 2007/09/20_14:54:40 info: populate_cib_nodes: Requesting the list of configured nodes mgmtd[25449]: 2007/09/20_14:54:40 debug: main: run the loop... mgmtd[25449]: 2007/09/20_14:54:40 info: Started. heartbeat[25431]: 2007/09/20_14:54:40 WARN: 1 lost packet(s) for [host-1.awale.qc.ca] [42:44] heartbeat[25431]: 2007/09/20_14:54:40 info: No pkts missing from host-1.awale.qc.ca! ccm[25443]: 2007/09/20_14:54:40 debug: quorum plugin: majority ccm[25443]: 2007/09/20_14:54:40 debug: cluster:linux-ha, member_count=1, member_quorum_votes=100 cib[25444]: 2007/09/20_14:54:40 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm ccm[25443]: 2007/09/20_14:54:40 debug: total_node_count=2, total_quorum_votes=200 cib[25444]: 2007/09/20_14:54:40 info: mem_handle_event: instance=1, nodes=1, new=1, lost=0, n_idx=0, new_idx=0, old_idx=3 ccm[25443]: 2007/09/20_14:54:40 debug: quorum plugin: twonodes cib[25444]: 2007/09/20_14:54:40 info: cib_ccm_msg_callback: PEER: host-2.awale.qc.ca ccm[25443]: 2007/09/20_14:54:40 debug: cluster:linux-ha, member_count=1, member_quorum_votes=100 ccm[25443]: 2007/09/20_14:54:40 debug: total_node_count=2, total_quorum_votes=200 ccm[25443]: 2007/09/20_14:54:40 info: Break tie for 2 nodes cluster crmd[25448]: 2007/09/20_14:54:40 notice: populate_cib_nodes: Node: host-2.awale.qc.ca (uuid: 77c6bcf6-f1e9-4e62-a33e-6ddd3530b92e) crmd[25448]: 2007/09/20_14:54:41 notice: populate_cib_nodes: Node: host-1.awale.qc.ca (uuid: 1da8804b-8a93-4145-9d40-1133699fe377) ccm[25443]: 2007/09/20_14:54:41 debug: quorum plugin: majority ccm[25443]: 2007/09/20_14:54:41 debug: cluster:linux-ha, member_count=2, member_quorum_votes=200 cib[25444]: 2007/09/20_14:54:41 info: mem_handle_event: Got an event OC_EV_MS_INVALID from ccm ccm[25443]: 2007/09/20_14:54:41 debug: total_node_count=2, total_quorum_votes=200 cib[25444]: 2007/09/20_14:54:41 info: mem_handle_event: no mbr_track info cib[25444]: 2007/09/20_14:54:41 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm cib[25444]: 2007/09/20_14:54:41 info: mem_handle_event: instance=2, nodes=2, new=1, lost=0, n_idx=0, new_idx=2, old_idx=4 cib[25444]: 2007/09/20_14:54:41 info: cib_ccm_msg_callback: PEER: host-2.awale.qc.ca cib[25444]: 2007/09/20_14:54:41 info: cib_ccm_msg_callback: PEER: host-1.awale.qc.ca crmd[25448]: 2007/09/20_14:54:41 info: do_ha_control: Connected to Heartbeat crmd[25448]: 2007/09/20_14:54:41 info: do_ccm_control: CCM connection established... waiting for first callback crmd[25448]: 2007/09/20_14:54:41 info: do_started: Delaying start, CCM (0000000000100000) not connected crmd[25448]: 2007/09/20_14:54:41 info: crmd_init: Starting crmd's mainloop crmd[25448]: 2007/09/20_14:54:41 notice: crmd_client_status_callback: Status update: Client host-2.awale.qc.ca/crmd now has status [online] crmd[25448]: 2007/09/20_14:54:41 notice: crmd_client_status_callback: Status update: Client host-1.awale.qc.ca/crmd now has status [online] cibmon[25442]: 2007/09/20_14:54:41 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:41 info: log_cib_diff: cib_update: Local-only Change: 0.1.1 cibmon[25442]: 2007/09/20_14:54:41 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:41 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:41 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:41 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:41 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:42 notice: crmd_client_status_callback: Status update: Client host-2.awale.qc.ca/crmd now has status [online] cibmon[25442]: 2007/09/20_14:54:42 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:42 info: log_cib_diff: cib_update: Local-only Change: 0.1.1 cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:42 notice: crmd_client_status_callback: Status update: Client host-1.awale.qc.ca/crmd now has status [online] crmd[25448]: 2007/09/20_14:54:42 info: mem_handle_event: Got an event OC_EV_MS_NEW_MEMBERSHIP from ccm crmd[25448]: 2007/09/20_14:54:42 info: mem_handle_event: instance=2, nodes=2, new=2, lost=0, n_idx=0, new_idx=0, old_idx=4 crmd[25448]: 2007/09/20_14:54:42 info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=2) crmd[25448]: 2007/09/20_14:54:42 info: ccm_event_detail: NEW MEMBERSHIP: trans=2, nodes=2, new=2, lost=0 n_idx=0, new_idx=0, old_idx=4 crmd[25448]: 2007/09/20_14:54:42 info: ccm_event_detail: CURRENT: host-2.awale.qc.ca [nodeid=1, born=1] crmd[25448]: 2007/09/20_14:54:42 info: ccm_event_detail: CURRENT: host-1.awale.qc.ca [nodeid=0, born=2] crmd[25448]: 2007/09/20_14:54:42 info: ccm_event_detail: NEW: host-2.awale.qc.ca [nodeid=1, born=1] crmd[25448]: 2007/09/20_14:54:42 info: ccm_event_detail: NEW: host-1.awale.qc.ca [nodeid=0, born=2] crmd[25448]: 2007/09/20_14:54:42 info: do_started: Delaying start, Config not read (0000000000000040) crmd[25448]: 2007/09/20_14:54:42 notice: cluster_option: Using default value '10s' for cluster option 'dc_deadtime' crmd[25448]: 2007/09/20_14:54:42 notice: cluster_option: Using default value '0' for cluster option 'cluster_recheck_interval' crmd[25448]: 2007/09/20_14:54:42 notice: cluster_option: Using default value '2min' for cluster option 'election_timeout' crmd[25448]: 2007/09/20_14:54:42 notice: cluster_option: Using default value '20min' for cluster option 'shutdown_escalation' crmd[25448]: 2007/09/20_14:54:42 notice: cluster_option: Using default value '3min' for cluster option 'crmd-integration-timeout' cibmon[25442]: 2007/09/20_14:54:42 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:42 info: log_cib_diff: cib_update: Local-only Change: 0.1.1 cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:42 notice: cluster_option: Using default value '10min' for cluster option 'crmd-finalization-timeout' cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:42 info: do_started: The local CRM is operational cibmon[25442]: 2007/09/20_14:54:42 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:42 info: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ] crmd[25448]: 2007/09/20_14:54:42 info: update_dc: Set DC to () attrd[25447]: 2007/09/20_14:54:47 info: main: Starting mainloop... crmd[25448]: 2007/09/20_14:54:53 info: do_election_count_vote: Election check: vote from host-1.awale.qc.ca crmd[25448]: 2007/09/20_14:54:54 info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped! crmd[25448]: 2007/09/20_14:54:54 WARN: do_log: [[FSA]] Input I_DC_TIMEOUT from crm_timer_popped() received in state (S_PENDING) crmd[25448]: 2007/09/20_14:54:54 info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ] crmd[25448]: 2007/09/20_14:54:54 info: update_dc: Set DC to () crmd[25448]: 2007/09/20_14:54:54 info: do_election_count_vote: Updated voted hash for host-2.awale.qc.ca to vote crmd[25448]: 2007/09/20_14:54:54 info: do_election_count_vote: Election ignore: our vote (host-2.awale.qc.ca) crmd[25448]: 2007/09/20_14:54:54 info: do_election_check: Still waiting on 1 non-votes (2 total) crmd[25448]: 2007/09/20_14:54:54 info: do_election_count_vote: Updated voted hash for host-1.awale.qc.ca to no-vote crmd[25448]: 2007/09/20_14:54:54 info: do_election_count_vote: Election ignore: no-vote from host-1.awale.qc.ca crmd[25448]: 2007/09/20_14:54:54 info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ] crmd[25448]: 2007/09/20_14:54:54 info: start_subsystem: Starting sub-system "tengine" crmd[25448]: 2007/09/20_14:54:54 info: start_subsystem: Starting sub-system "pengine" crmd[25448]: 2007/09/20_14:54:54 info: do_dc_takeover: Taking over DC status for this partition cib[25444]: 2007/09/20_14:54:54 info: cib_process_readwrite: We are now in R/W mode tengine[25451]: 2007/09/20_14:54:54 info: G_main_add_SignalHandler: Added signal handler for signal 15 tengine[25451]: 2007/09/20_14:54:54 info: G_main_add_TriggerHandler: Added signal manual handler cibmon[25442]: 2007/09/20_14:54:54 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:54 info: log_cib_diff: cib_update: Diff: --- 0.1.1 cibmon[25442]: 2007/09/20_14:54:54 info: log_cib_diff: cib_update: Diff: +++ 0.1.2 cibmon[25442]: 2007/09/20_14:54:54 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:54 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:54 info: update_dc: Set DC to () crmd[25448]: 2007/09/20_14:54:54 info: join_make_offer: Making join offers based on membership 2 crmd[25448]: 2007/09/20_14:54:54 info: do_dc_join_offer_all: join-1: Waiting on 2 outstanding join acks cib[25444]: 2007/09/20_14:54:54 info: cib_null_callback: Setting cib_diff_notify callbacks for tengine: on tengine[25451]: 2007/09/20_14:54:55 info: te_init: Registering TE UUID: 686542a3-7c00-48a2-bdd3-0f9aad92cc0b tengine[25451]: 2007/09/20_14:54:55 info: set_graph_functions: Setting custom graph functions tengine[25451]: 2007/09/20_14:54:55 info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses tengine[25451]: 2007/09/20_14:54:55 info: te_init: Starting tengine crmd[25448]: 2007/09/20_14:54:55 info: update_dc: Set DC to host-2.awale.qc.ca (1.0.9) pengine[25452]: 2007/09/20_14:54:55 info: G_main_add_SignalHandler: Added signal handler for signal 15 pengine[25452]: 2007/09/20_14:54:55 info: pe_init: Starting pengine attrd[25447]: 2007/09/20_14:54:55 info: attrd_local_callback: Sending full refresh crmd[25448]: 2007/09/20_14:54:55 info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ] crmd[25448]: 2007/09/20_14:54:55 info: do_state_transition: All 2 cluster nodes responded to the join offer. crmd[25448]: 2007/09/20_14:54:55 info: update_attrd: Connecting to attrd... cib[25444]: 2007/09/20_14:54:55 info: sync_our_cib: Syncing CIB to all peers cibmon[25442]: 2007/09/20_14:54:55 info: cibmon_diff: [cib_diff_notify] cib_update confirmed crmd[25448]: 2007/09/20_14:54:55 info: update_dc: Set DC to host-2.awale.qc.ca (1.0.9) cibmon[25442]: 2007/09/20_14:54:55 info: log_cib_diff: cib_update: Diff: --- 0.1.2 cibmon[25442]: 2007/09/20_14:54:55 info: log_cib_diff: cib_update: Diff: +++ 0.1.3 cibmon[25442]: 2007/09/20_14:54:55 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:55 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:55 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:55 info: log_cib_diff: cib_update: Diff: --- 0.1.3 cibmon[25442]: 2007/09/20_14:54:55 info: log_cib_diff: cib_update: Diff: +++ 0.1.4 cibmon[25442]: 2007/09/20_14:54:55 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:55 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:56 info: log_cib_diff: cib_update: Diff: --- 0.1.4 cibmon[25442]: 2007/09/20_14:54:56 info: log_cib_diff: cib_update: Diff: +++ 0.1.5 cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:56 info: do_dc_join_ack: join-1: Updating node state to member for host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:56 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:56 info: log_cib_diff: cib_update: Diff: --- 0.1.5 cibmon[25442]: 2007/09/20_14:54:56 info: log_cib_diff: cib_update: Diff: +++ 0.1.6 cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:56 info: do_dc_join_ack: join-1: Updating node state to member for host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:56 info: cibmon_diff: [cib_diff_notify] cib_update confirmed crmd[25448]: 2007/09/20_14:54:56 info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ] tengine[25451]: 2007/09/20_14:54:56 info: update_abort_priority: Abort priority upgraded to 1000000 cibmon[25442]: 2007/09/20_14:54:56 info: log_cib_diff: cib_update: Diff: --- 0.1.6 crmd[25448]: 2007/09/20_14:54:56 info: do_state_transition: All 2 cluster nodes are eligible to run resources. tengine[25451]: 2007/09/20_14:54:56 info: update_abort_priority: 'DC Takeover' abort superceeded cibmon[25442]: 2007/09/20_14:54:56 info: log_cib_diff: cib_update: Diff: +++ 0.1.7 cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:56 info: log_data_element: cib_update: + pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'false' for cluster option 'stonith-enabled' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max' pengine[25452]: 2007/09/20_14:54:56 notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing' pengine[25452]: 2007/09/20_14:54:56 info: determine_online_status: Node host-2.awale.qc.ca is online pengine[25452]: 2007/09/20_14:54:56 info: determine_online_status: Node host-1.awale.qc.ca is online pengine[25452]: 2007/09/20_14:54:56 info: clone_print: Master/Slave Set: ms-drbd_0 pengine[25452]: 2007/09/20_14:54:56 info: native_print: share-drbd_0:0 (heartbeat::ocf:drbd): Stopped pengine[25452]: 2007/09/20_14:54:56 info: native_print: share-drbd_0:1 (heartbeat::ocf:drbd): Stopped pengine[25452]: 2007/09/20_14:54:56 info: group_print: Resource Group: DW pengine[25452]: 2007/09/20_14:54:56 info: native_print: msg_first (heartbeat:msg): Stopped pengine[25452]: 2007/09/20_14:54:56 info: native_print: share-FS (heartbeat::ocf:Filesystem): Stopped pengine[25452]: 2007/09/20_14:54:56 info: native_print: share-drbdlinks (heartbeat:drbdlinks): Stopped pengine[25452]: 2007/09/20_14:54:56 info: native_print: IPaddr_10_0_0_59 (heartbeat::ocf:IPaddr): Stopped pengine[25452]: 2007/09/20_14:54:56 info: native_print: mysql (heartbeat::ocf:mysql): Stopped pengine[25452]: 2007/09/20_14:54:56 info: native_print: dw-httpd (heartbeat:daemontools): Stopped pengine[25452]: 2007/09/20_14:54:56 info: native_print: msg_last (heartbeat:msg): Stopped pengine[25452]: 2007/09/20_14:54:56 info: master_color: Promoted 0 instances of a possible 1 to master pengine[25452]: 2007/09/20_14:54:56 info: master_color: Promoted 0 instances of a possible 1 to master pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-1.awale.qc.ca Start share-drbd_0:0 pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-1.awale.qc.ca share-drbd_0:0_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start share-drbd_0:1 pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca share-drbd_0:1_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-1.awale.qc.ca Start share-drbd_0:0 pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-1.awale.qc.ca share-drbd_0:0_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start share-drbd_0:1 pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca share-drbd_0:1_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start msg_first pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca msg_first_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start share-FS pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca share-FS_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start share-drbdlinks pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca share-drbdlinks_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start IPaddr_10_0_0_59 pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca IPaddr_10_0_0_59_monitor_5000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start mysql pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca mysql_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start dw-httpd pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca dw-httpd_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 notice: StartRsc: host-2.awale.qc.ca Start msg_last pengine[25452]: 2007/09/20_14:54:56 notice: RecurringOp: host-2.awale.qc.ca msg_last_monitor_120000 pengine[25452]: 2007/09/20_14:54:56 info: process_pe_message: Transition 0: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-55.bz2 crmd[25448]: 2007/09/20_14:54:56 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] tengine[25451]: 2007/09/20_14:54:56 info: unpack_graph: Unpacked transition 0: 47 actions in 47 synapses tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 3: share-drbd_0:0_monitor_0 on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 12: share-drbd_0:0_monitor_0 on host-1.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: te_pseudo_action: Pseudo action 26 fired and confirmed tengine[25451]: 2007/09/20_14:54:56 info: te_pseudo_action: Pseudo action 27 fired and confirmed tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 4: msg_first_monitor_0 on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 13: msg_first_monitor_0 on host-1.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 5: share-FS_monitor_0 on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 14: share-FS_monitor_0 on host-1.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 6: share-drbdlinks_monitor_0 on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 15: share-drbdlinks_monitor_0 on host-1.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 7: IPaddr_10_0_0_59_monitor_0 on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 16: IPaddr_10_0_0_59_monitor_0 on host-1.awale.qc.ca crmd[25448]: 2007/09/20_14:54:56 info: do_lrm_rsc_op: Performing op=share-drbd_0:0_monitor_0 key=3:0:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 8: mysql_monitor_0 on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 17: mysql_monitor_0 on host-1.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 9: dw-httpd_monitor_0 on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 18: dw-httpd_monitor_0 on host-1.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 10: msg_last_monitor_0 on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:56 info: send_rsc_command: Initiating action 19: msg_last_monitor_0 on host-1.awale.qc.ca crmd[25448]: 2007/09/20_14:54:56 info: do_lrm_rsc_op: Performing op=msg_first_monitor_0 key=4:0:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) crmd[25448]: 2007/09/20_14:54:56 info: do_lrm_rsc_op: Performing op=share-FS_monitor_0 key=5:0:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) crmd[25448]: 2007/09/20_14:54:56 info: do_lrm_rsc_op: Performing op=share-drbdlinks_monitor_0 key=6:0:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) crmd[25448]: 2007/09/20_14:54:56 info: do_lrm_rsc_op: Performing op=IPaddr_10_0_0_59_monitor_0 key=7:0:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) crmd[25448]: 2007/09/20_14:54:56 info: do_lrm_rsc_op: Performing op=mysql_monitor_0 key=8:0:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) crmd[25448]: 2007/09/20_14:54:56 info: do_lrm_rsc_op: Performing op=dw-httpd_monitor_0 key=9:0:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) crmd[25448]: 2007/09/20_14:54:56 info: do_lrm_rsc_op: Performing op=msg_last_monitor_0 key=10:0:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) crmd[25448]: 2007/09/20_14:54:57 info: process_lrm_event: LRM operation share-drbdlinks_monitor_0 (call=5, rc=7) complete drbd[25453][25486]: 2007/09/20_14:54:57 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 cibmon[25442]: 2007/09/20_14:54:57 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:54:57 info: match_graph_event: Action share-drbdlinks_monitor_0 (6) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: --- 0.1.7 cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: +++ 0.1.8 cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + drbd[25453][25508]: 2007/09/20_14:54:57 DEBUG: r0: Exit code 0 crmd[25448]: 2007/09/20_14:54:57 info: process_lrm_event: LRM operation msg_first_monitor_0 (call=3, rc=7) complete cibmon[25442]: 2007/09/20_14:54:57 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:54:57 info: match_graph_event: Action msg_first_monitor_0 (4) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: --- 0.1.8 cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: +++ 0.1.9 cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + drbd[25453][25511]: 2007/09/20_14:54:57 DEBUG: r0: Command output: Not configured drbd[25453][25530]: 2007/09/20_14:54:57 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 crmd[25448]: 2007/09/20_14:54:57 info: process_lrm_event: LRM operation share-FS_monitor_0 (call=4, rc=7) complete cibmon[25442]: 2007/09/20_14:54:57 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:54:57 info: match_graph_event: Action share-FS_monitor_0 (5) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: --- 0.1.9 cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: +++ 0.1.10 cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + drbd[25453][25544]: 2007/09/20_14:54:57 DEBUG: r0: Exit code 0 drbd[25453][25545]: 2007/09/20_14:54:57 DEBUG: r0: Command output: Not configured drbd[25453][25546]: 2007/09/20_14:54:57 DEBUG: r0 status: Not configured Not configured Not configured Unconfigured drbd[25453][25547]: 2007/09/20_14:54:57 DEBUG: r0 monitor: resource not configured crmd[25448]: 2007/09/20_14:54:57 info: process_lrm_event: LRM operation share-drbd_0:0_monitor_0 (call=2, rc=7) complete cibmon[25442]: 2007/09/20_14:54:57 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:54:57 info: match_graph_event: Action share-drbd_0:0_monitor_0 (3) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: --- 0.1.10 cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: +++ 0.1.11 cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: --- 0.1.11 cibmon[25442]: 2007/09/20_14:54:57 info: log_cib_diff: cib_update: Diff: +++ 0.1.12 cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:58 info: match_graph_event: Action share-drbdlinks_monitor_0 (15) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:57 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: --- 0.1.12 cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: +++ 0.1.13 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:58 info: match_graph_event: Action msg_first_monitor_0 (13) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: --- 0.1.13 cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: +++ 0.1.14 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:58 info: process_lrm_event: LRM operation IPaddr_10_0_0_59_monitor_0 (call=6, rc=7) complete cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:58 info: match_graph_event: Action share-FS_monitor_0 (14) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:58 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: --- 0.1.14 cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: +++ 0.1.15 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:58 info: match_graph_event: Action IPaddr_10_0_0_59_monitor_0 (7) confirmed on host-2.awale.qc.ca mysql[25549][25580]: 2007/09/20_14:54:58 DEBUG: MySQL is not running crmd[25448]: 2007/09/20_14:54:58 info: process_lrm_event: LRM operation mysql_monitor_0 (call=7, rc=7) complete cibmon[25442]: 2007/09/20_14:54:58 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: --- 0.1.15 cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: +++ 0.1.16 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:58 info: match_graph_event: Action mysql_monitor_0 (8) confirmed on host-2.awale.qc.ca daemontools[25550][25596]: 2007/09/20_14:54:58 ERROR: /usr/lib/ocf/resource.d//heartbeat/daemontools is not an executable file lrmd[25445]: 2007/09/20_14:54:58 WARN: There is something wrong: the first line isn't read in. Maybe the heartbeat does not ouput string correctly for status operation. Or the code (myself) is wrong. crmd[25448]: 2007/09/20_14:54:58 info: process_lrm_event: LRM operation dw-httpd_monitor_0 (call=8, rc=7) complete lrmd[25445]: 2007/09/20_14:54:58 debug: RA output [ERROR: /usr/lib/ocf/resource.d//heartbeat/daemontools is not an executable file] didn't match any pattern crmd[25448]: 2007/09/20_14:54:58 info: process_lrm_event: LRM operation msg_last_monitor_0 (call=9, rc=7) complete cibmon[25442]: 2007/09/20_14:54:58 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: --- 0.1.16 cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: +++ 0.1.17 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: - tengine[25451]: 2007/09/20_14:54:58 info: match_graph_event: Action dw-httpd_monitor_0 (9) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: --- 0.1.17 cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: +++ 0.1.18 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: - tengine[25451]: 2007/09/20_14:54:58 info: match_graph_event: Action msg_last_monitor_0 (10) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:58 info: send_rsc_command: Initiating action 2: probe_complete on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:58 info: cibmon_diff: [cib_diff_notify] cib_modify confirmed cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_modify: Diff: --- 0.1.18 cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_modify: Diff: +++ 0.1.19 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: - tengine[25451]: 2007/09/20_14:54:58 info: extract_event: Aborting on transient_attributes changes for 77c6bcf6-f1e9-4e62-a33e-6ddd3530b92e cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + tengine[25451]: 2007/09/20_14:54:58 info: update_abort_priority: Abort priority upgraded to 1000000 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + tengine[25451]: 2007/09/20_14:54:58 info: update_abort_priority: Abort action 0 superceeded by 2 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:54:58 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:54:58 info: match_graph_event: Action share-drbd_0:0_monitor_0 (12) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: --- 0.1.19 cibmon[25442]: 2007/09/20_14:54:58 info: log_cib_diff: cib_update: Diff: +++ 0.1.20 cibmon[25442]: 2007/09/20_14:54:58 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:59 info: match_graph_event: Action mysql_monitor_0 (17) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: --- 0.1.20 cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: +++ 0.1.21 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:59 info: match_graph_event: Action IPaddr_10_0_0_59_monitor_0 (16) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: --- 0.1.21 cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: +++ 0.1.22 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: --- 0.1.22 cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: +++ 0.1.23 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: - tengine[25451]: 2007/09/20_14:54:59 info: match_graph_event: Action msg_last_monitor_0 (19) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: --- 0.1.23 cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: +++ 0.1.24 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: - tengine[25451]: 2007/09/20_14:54:59 info: match_graph_event: Action dw-httpd_monitor_0 (18) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:59 info: send_rsc_command: Initiating action 11: probe_complete on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:59 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ] tengine[25451]: 2007/09/20_14:54:59 info: run_graph: ==================================================== cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:59 info: do_state_transition: All 2 cluster nodes are eligible to run resources. tengine[25451]: 2007/09/20_14:54:59 notice: run_graph: Transition 0: (Complete=20, Pending=0, Fired=0, Skipped=22, Incomplete=5) cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'false' for cluster option 'stonith-enabled' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max' pengine[25452]: 2007/09/20_14:54:59 notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing' pengine[25452]: 2007/09/20_14:54:59 info: determine_online_status: Node host-2.awale.qc.ca is online pengine[25452]: 2007/09/20_14:54:59 info: determine_online_status: Node host-1.awale.qc.ca is online pengine[25452]: 2007/09/20_14:54:59 info: clone_print: Master/Slave Set: ms-drbd_0 pengine[25452]: 2007/09/20_14:54:59 info: native_print: share-drbd_0:0 (heartbeat::ocf:drbd): Stopped pengine[25452]: 2007/09/20_14:54:59 info: native_print: share-drbd_0:1 (heartbeat::ocf:drbd): Stopped pengine[25452]: 2007/09/20_14:54:59 info: group_print: Resource Group: DW pengine[25452]: 2007/09/20_14:54:59 info: native_print: msg_first (heartbeat:msg): Stopped pengine[25452]: 2007/09/20_14:54:59 info: native_print: share-FS (heartbeat::ocf:Filesystem): Stopped pengine[25452]: 2007/09/20_14:54:59 info: native_print: share-drbdlinks (heartbeat:drbdlinks): Stopped pengine[25452]: 2007/09/20_14:54:59 info: native_print: IPaddr_10_0_0_59 (heartbeat::ocf:IPaddr): Stopped pengine[25452]: 2007/09/20_14:54:59 info: native_print: mysql (heartbeat::ocf:mysql): Stopped pengine[25452]: 2007/09/20_14:54:59 info: native_print: dw-httpd (heartbeat:daemontools): Stopped pengine[25452]: 2007/09/20_14:54:59 info: native_print: msg_last (heartbeat:msg): Stopped pengine[25452]: 2007/09/20_14:54:59 info: master_color: Promoted 0 instances of a possible 1 to master crmd[25448]: 2007/09/20_14:54:59 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] pengine[25452]: 2007/09/20_14:54:59 info: master_color: Promoted 0 instances of a possible 1 to master pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-1.awale.qc.ca Start share-drbd_0:0 pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-1.awale.qc.ca share-drbd_0:0_monitor_120000 pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start share-drbd_0:1 pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca share-drbd_0:1_monitor_120000 pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-1.awale.qc.ca Start share-drbd_0:0 pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-1.awale.qc.ca share-drbd_0:0_monitor_120000 pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start share-drbd_0:1 pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca share-drbd_0:1_monitor_120000 pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start msg_first pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca msg_first_monitor_120000 pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start share-FS pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca share-FS_monitor_120000 pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start share-drbdlinks pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca share-drbdlinks_monitor_120000 pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start IPaddr_10_0_0_59 pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca IPaddr_10_0_0_59_monitor_5000 tengine[25451]: 2007/09/20_14:54:59 info: unpack_graph: Unpacked transition 1: 33 actions in 33 synapses pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start mysql tengine[25451]: 2007/09/20_14:54:59 info: send_rsc_command: Initiating action 3: share-drbd_0:1_monitor_0 on host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca mysql_monitor_120000 tengine[25451]: 2007/09/20_14:54:59 info: send_rsc_command: Initiating action 5: share-drbd_0:1_monitor_0 on host-1.awale.qc.ca pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start dw-httpd tengine[25451]: 2007/09/20_14:54:59 info: te_pseudo_action: Pseudo action 12 fired and confirmed pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca dw-httpd_monitor_120000 tengine[25451]: 2007/09/20_14:54:59 info: te_pseudo_action: Pseudo action 13 fired and confirmed pengine[25452]: 2007/09/20_14:54:59 notice: StartRsc: host-2.awale.qc.ca Start msg_last tengine[25451]: 2007/09/20_14:54:59 info: te_pseudo_action: Pseudo action 48 fired and confirmed pengine[25452]: 2007/09/20_14:54:59 notice: RecurringOp: host-2.awale.qc.ca msg_last_monitor_120000 tengine[25451]: 2007/09/20_14:54:59 info: send_rsc_command: Initiating action 34: msg_first_start_0 on host-2.awale.qc.ca crmd[25448]: 2007/09/20_14:54:59 info: do_lrm_rsc_op: Performing op=share-drbd_0:1_monitor_0 key=3:1:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) pengine[25452]: 2007/09/20_14:54:59 info: process_pe_message: Transition 1: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-56.bz2 crmd[25448]: 2007/09/20_14:54:59 info: do_lrm_rsc_op: Performing op=msg_first_start_0 key=34:1:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) drbd[25598][25624]: 2007/09/20_14:54:59 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[25598][25637]: 2007/09/20_14:54:59 DEBUG: r0: Exit code 0 drbd[25598][25639]: 2007/09/20_14:54:59 DEBUG: r0: Command output: Not configured msg[25599][25640]: 2007/09/20_14:54:59 INFO Starting to switch to this node. msg[25599][25643]: 2007/09/20_14:54:59 INFO Node: host-2.awale.qc.ca drbd[25598][25653]: 2007/09/20_14:54:59 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 crmd[25448]: 2007/09/20_14:54:59 info: process_lrm_event: LRM operation msg_first_start_0 (call=11, rc=0) complete crmd[25448]: 2007/09/20_14:54:59 info: build_operation_update: Digest for 0:0;34:1:686542a3-7c00-48a2-bdd3-0f9aad92cc0b (msg_first_start_0) was fd931b345f577de4ab73115496330fb5 crmd[25448]: 2007/09/20_14:54:59 info: log_data_element: build_operation_update: digest:source cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: --- 0.1.24 tengine[25451]: 2007/09/20_14:54:59 info: match_graph_event: Action msg_first_start_0 (34) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: +++ 0.1.25 tengine[25451]: 2007/09/20_14:54:59 info: send_rsc_command: Initiating action 35: msg_first_monitor_120000 on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: - tengine[25451]: 2007/09/20_14:54:59 info: send_rsc_command: Initiating action 36: share-FS_start_0 on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:59 info: do_lrm_rsc_op: Performing op=msg_first_monitor_120000 key=35:1:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:59 info: do_lrm_rsc_op: Performing op=share-FS_start_0 key=36:1:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + drbd[25598][25683]: 2007/09/20_14:54:59 DEBUG: r0: Exit code 0 drbd[25598][25685]: 2007/09/20_14:54:59 DEBUG: r0: Command output: Not configured drbd[25598][25692]: 2007/09/20_14:54:59 DEBUG: r0 status: Not configured Not configured Not configured Unconfigured drbd[25598][25696]: 2007/09/20_14:54:59 DEBUG: r0 monitor: resource not configured crmd[25448]: 2007/09/20_14:54:59 info: process_lrm_event: LRM operation msg_first_monitor_120000 (call=12, rc=0) complete cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:54:59 info: match_graph_event: Action msg_first_monitor_120000 (35) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: --- 0.1.25 cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: +++ 0.1.26 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:54:59 info: process_lrm_event: LRM operation share-drbd_0:1_monitor_0 (call=10, rc=7) complete cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: --- 0.1.26 cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: +++ 0.1.27 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:59 info: match_graph_event: Action share-drbd_0:1_monitor_0 (3) confirmed on host-2.awale.qc.ca tengine[25451]: 2007/09/20_14:54:59 info: send_rsc_command: Initiating action 2: probe_complete on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_modify confirmed Filesystem[25663][25717]: 2007/09/20_14:54:59 INFO: Running start for /dev/drbd0 on /share cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_modify: Diff: --- 0.1.27 cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_modify: Diff: +++ 0.1.28 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_modify: - cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_modify: + lrmd[25445]: 2007/09/20_14:54:59 info: RA output: (share-FS:start:stderr) mount: block device /dev/drbd0 is write-protected, mounting read-only lrmd[25445]: 2007/09/20_14:54:59 info: RA output: (share-FS:start:stderr) mount: /dev/drbd0 already mounted or /share busy Filesystem[25663][25730]: 2007/09/20_14:54:59 ERROR: Couldn't mount filesystem /dev/drbd0 on /share crmd[25448]: 2007/09/20_14:54:59 ERROR: process_lrm_event: LRM operation share-FS_start_0 (call=13, rc=1) Error unknown error crmd[25448]: 2007/09/20_14:54:59 info: build_operation_update: Digest for 4:1;36:1:686542a3-7c00-48a2-bdd3-0f9aad92cc0b (share-FS_start_0) was 0da34e076c2ca241d96b4d38d98c1ef3 crmd[25448]: 2007/09/20_14:54:59 info: log_data_element: build_operation_update: digest:source cibmon[25442]: 2007/09/20_14:54:59 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: --- 0.1.28 cibmon[25442]: 2007/09/20_14:54:59 info: log_cib_diff: cib_update: Diff: +++ 0.1.29 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:59 WARN: status_from_rc: Action start on host-2.awale.qc.ca failed (target: vs. rc: 1): Error cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:59 info: update_abort_priority: Abort priority upgraded to 1 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:54:59 info: update_abort_priority: Abort action 0 superceeded by 2 cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:54:59 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:00 info: extract_event: Aborting on transient_attributes changes for 1da8804b-8a93-4145-9d40-1133699fe377 tengine[25451]: 2007/09/20_14:55:00 info: update_abort_priority: Abort priority upgraded to 1000000 tengine[25451]: 2007/09/20_14:55:00 info: update_abort_priority: 'Event failed' abort superceeded cibmon[25442]: 2007/09/20_14:55:00 info: cibmon_diff: [cib_diff_notify] cib_modify confirmed cibmon[25442]: 2007/09/20_14:55:00 info: log_cib_diff: cib_modify: Diff: --- 0.1.29 cibmon[25442]: 2007/09/20_14:55:00 info: log_cib_diff: cib_modify: Diff: +++ 0.1.30 cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: - cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:00 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:01 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:01 info: log_cib_diff: cib_update: Diff: --- 0.1.30 cibmon[25442]: 2007/09/20_14:55:01 info: log_cib_diff: cib_update: Diff: +++ 0.1.31 tengine[25451]: 2007/09/20_14:55:01 info: match_graph_event: Action share-drbd_0:1_monitor_0 (5) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: - tengine[25451]: 2007/09/20_14:55:01 info: send_rsc_command: Initiating action 4: probe_complete on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:55:01 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ] tengine[25451]: 2007/09/20_14:55:01 info: run_graph: ==================================================== cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:55:01 info: do_state_transition: All 2 cluster nodes are eligible to run resources. tengine[25451]: 2007/09/20_14:55:01 notice: run_graph: Transition 1: (Complete=10, Pending=0, Fired=0, Skipped=18, Incomplete=5) cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'false' for cluster option 'stonith-enabled' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max' pengine[25452]: 2007/09/20_14:55:01 notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing' pengine[25452]: 2007/09/20_14:55:01 info: determine_online_status: Node host-2.awale.qc.ca is online pengine[25452]: 2007/09/20_14:55:01 WARN: unpack_rsc_op: Processing failed op (share-FS_start_0) on host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:01 WARN: unpack_rsc_op: Handling failed start for share-FS on host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:01 info: determine_online_status: Node host-1.awale.qc.ca is online pengine[25452]: 2007/09/20_14:55:01 info: clone_print: Master/Slave Set: ms-drbd_0 pengine[25452]: 2007/09/20_14:55:01 info: native_print: share-drbd_0:0 (heartbeat::ocf:drbd): Stopped pengine[25452]: 2007/09/20_14:55:01 info: native_print: share-drbd_0:1 (heartbeat::ocf:drbd): Stopped pengine[25452]: 2007/09/20_14:55:01 info: group_print: Resource Group: DW pengine[25452]: 2007/09/20_14:55:01 info: native_print: msg_first (heartbeat:msg): Started host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:01 info: native_print: share-FS (heartbeat::ocf:Filesystem): Started host-2.awale.qc.ca FAILED pengine[25452]: 2007/09/20_14:55:01 info: native_print: share-drbdlinks (heartbeat:drbdlinks): Stopped pengine[25452]: 2007/09/20_14:55:01 info: native_print: IPaddr_10_0_0_59 (heartbeat::ocf:IPaddr): Stopped pengine[25452]: 2007/09/20_14:55:01 info: native_print: mysql (heartbeat::ocf:mysql): Stopped pengine[25452]: 2007/09/20_14:55:01 info: native_print: dw-httpd (heartbeat:daemontools): Stopped pengine[25452]: 2007/09/20_14:55:01 info: native_print: msg_last (heartbeat:msg): Stopped pengine[25452]: 2007/09/20_14:55:01 info: master_color: Promoted 0 instances of a possible 1 to master pengine[25452]: 2007/09/20_14:55:01 info: master_color: Promoted 0 instances of a possible 1 to master pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-2.awale.qc.ca Start share-drbd_0:0 pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-2.awale.qc.ca share-drbd_0:0_monitor_120000 pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start share-drbd_0:1 pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca share-drbd_0:1_monitor_120000 pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-2.awale.qc.ca Start share-drbd_0:0 crmd[25448]: 2007/09/20_14:55:01 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-2.awale.qc.ca share-drbd_0:0_monitor_120000 pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start share-drbd_0:1 pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca share-drbd_0:1_monitor_120000 pengine[25452]: 2007/09/20_14:55:01 notice: NoRoleChange: Move resource msg_first (host-2.awale.qc.ca -> host-1.awale.qc.ca) pengine[25452]: 2007/09/20_14:55:01 notice: StopRsc: host-2.awale.qc.ca Stop msg_first pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start msg_first pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca msg_first_monitor_120000 pengine[25452]: 2007/09/20_14:55:01 notice: NoRoleChange: Recover resource share-FS (host-1.awale.qc.ca) pengine[25452]: 2007/09/20_14:55:01 notice: StopRsc: host-2.awale.qc.ca Stop share-FS pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start share-FS pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca share-FS_monitor_120000 pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start share-drbdlinks pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca share-drbdlinks_monitor_120000 pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start IPaddr_10_0_0_59 tengine[25451]: 2007/09/20_14:55:01 info: unpack_graph: Unpacked transition 2: 33 actions in 33 synapses pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca IPaddr_10_0_0_59_monitor_5000 tengine[25451]: 2007/09/20_14:55:01 info: te_pseudo_action: Pseudo action 12 fired and confirmed pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start mysql tengine[25451]: 2007/09/20_14:55:01 info: te_pseudo_action: Pseudo action 13 fired and confirmed pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca mysql_monitor_120000 tengine[25451]: 2007/09/20_14:55:01 info: te_pseudo_action: Pseudo action 51 fired and confirmed pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start dw-httpd crmd[25448]: 2007/09/20_14:55:01 info: do_lrm_rsc_op: Performing op=share-FS_stop_0 key=2:2:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) tengine[25451]: 2007/09/20_14:55:01 info: send_rsc_command: Initiating action 2: share-FS_stop_0 on host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca dw-httpd_monitor_120000 tengine[25451]: 2007/09/20_14:55:01 info: te_pseudo_action: Pseudo action 63 fired and confirmed pengine[25452]: 2007/09/20_14:55:01 notice: StartRsc: host-1.awale.qc.ca Start msg_last tengine[25451]: 2007/09/20_14:55:01 info: te_pseudo_action: Pseudo action 10 fired and confirmed pengine[25452]: 2007/09/20_14:55:01 notice: RecurringOp: host-1.awale.qc.ca msg_last_monitor_120000 tengine[25451]: 2007/09/20_14:55:01 info: send_rsc_command: Initiating action 6: share-drbd_0:0_start_0 on host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:01 info: process_pe_message: Transition 2: PEngine Input stored in: /var/lib/heartbeat/pengine/pe-input-57.bz2 tengine[25451]: 2007/09/20_14:55:01 info: send_rsc_command: Initiating action 8: share-drbd_0:1_start_0 on host-1.awale.qc.ca crmd[25448]: 2007/09/20_14:55:01 info: do_lrm_rsc_op: Performing op=share-drbd_0:0_start_0 key=6:2:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) lrmd[25445]: 2007/09/20_14:55:01 info: RA output: (share-drbd_0:0:start:stdout) r0: ACTION=start drbd[25739][25763]: 2007/09/20_14:55:01 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[25739][25778]: 2007/09/20_14:55:01 DEBUG: r0: Exit code 0 drbd[25739][25782]: 2007/09/20_14:55:01 DEBUG: r0: Command output: Not configured Filesystem[25738][25794]: 2007/09/20_14:55:01 INFO: Running stop for /dev/drbd0 on /share drbd[25739][25796]: 2007/09/20_14:55:01 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 crmd[25448]: 2007/09/20_14:55:01 info: process_lrm_event: LRM operation share-FS_stop_0 (call=14, rc=0) complete cibmon[25442]: 2007/09/20_14:55:01 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:55:01 info: match_graph_event: Action share-FS_stop_0 (2) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:01 info: log_cib_diff: cib_update: Diff: --- 0.1.31 tengine[25451]: 2007/09/20_14:55:01 info: send_rsc_command: Initiating action 34: msg_first_stop_0 on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:01 info: log_cib_diff: cib_update: Diff: +++ 0.1.32 cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:55:01 info: do_lrm_rsc_op: Performing op=msg_first_stop_0 key=34:2:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) drbd[25739][25817]: 2007/09/20_14:55:01 DEBUG: r0: Exit code 0 crmd[25448]: 2007/09/20_14:55:01 info: process_lrm_event: LRM operation msg_first_monitor_120000 (call=12, rc=-2) Cancelled cibmon[25442]: 2007/09/20_14:55:01 info: cibmon_diff: [cib_diff_notify] cib_delete confirmed cibmon[25442]: 2007/09/20_14:55:01 info: log_cib_diff: cib_delete: Diff: --- 0.1.32 cibmon[25442]: 2007/09/20_14:55:01 info: log_cib_diff: cib_delete: Diff: +++ 0.1.33 cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:01 info: log_data_element: cib_delete: + drbd[25739][25819]: 2007/09/20_14:55:01 DEBUG: r0: Command output: Not configured drbd[25739][25826]: 2007/09/20_14:55:01 DEBUG: r0 status: Not configured Not configured Not configured Unconfigured drbd[25739][25830]: 2007/09/20_14:55:01 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf up r0 cibmon[25442]: 2007/09/20_14:55:02 info: cibmon_diff: [cib_diff_notify] cib_modify confirmed cibmon[25442]: 2007/09/20_14:55:02 info: log_cib_diff: cib_modify: Diff: --- 0.1.33 cibmon[25442]: 2007/09/20_14:55:02 info: log_cib_diff: cib_modify: Diff: +++ 0.1.34 cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_modify: - cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_modify: + msg[25818][25851]: 2007/09/20_14:55:02 INFO Completed switching away from this node. msg[25818][25852]: 2007/09/20_14:55:02 INFO Node: host-2.awale.qc.ca crmd[25448]: 2007/09/20_14:55:02 info: process_lrm_event: LRM operation msg_first_stop_0 (call=16, rc=0) complete cibmon[25442]: 2007/09/20_14:55:02 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:02 info: log_cib_diff: cib_update: Diff: --- 0.1.34 cibmon[25442]: 2007/09/20_14:55:02 info: log_cib_diff: cib_update: Diff: +++ 0.1.35 cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:02 info: match_graph_event: Action msg_first_stop_0 (34) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:02 info: te_pseudo_action: Pseudo action 52 fired and confirmed cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:02 info: te_pseudo_action: Pseudo action 49 fired and confirmed cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:02 info: send_rsc_command: Initiating action 35: msg_first_start_0 on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:02 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:04 info: log_cib_diff: cib_update: Diff: --- 0.1.35 cibmon[25442]: 2007/09/20_14:55:04 info: log_cib_diff: cib_update: Diff: +++ 0.1.36 cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:04 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:04 info: match_graph_event: Action msg_first_start_0 (35) confirmed on host-1.awale.qc.ca tengine[25451]: 2007/09/20_14:55:04 info: send_rsc_command: Initiating action 36: msg_first_monitor_120000 on host-1.awale.qc.ca tengine[25451]: 2007/09/20_14:55:04 info: send_rsc_command: Initiating action 37: share-FS_start_0 on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:05 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:05 info: log_cib_diff: cib_update: Diff: --- 0.1.36 cibmon[25442]: 2007/09/20_14:55:05 info: log_cib_diff: cib_update: Diff: +++ 0.1.37 cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:05 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:05 info: match_graph_event: Action msg_first_monitor_120000 (36) confirmed on host-1.awale.qc.ca drbd[25739][25866]: 2007/09/20_14:55:06 DEBUG: r0: Exit code 0 drbd[25739][25867]: 2007/09/20_14:55:06 DEBUG: r0: Command output: lrmd[25445]: 2007/09/20_14:55:06 info: RA output: (share-drbd_0:0:start:stdout) drbd[25739][25869]: 2007/09/20_14:55:06 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[25739][25876]: 2007/09/20_14:55:06 DEBUG: r0: Exit code 0 drbd[25739][25877]: 2007/09/20_14:55:06 DEBUG: r0: Command output: Secondary/Unknown drbd[25739][25885]: 2007/09/20_14:55:06 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 drbd[25739][25898]: 2007/09/20_14:55:06 DEBUG: r0: Exit code 0 drbd[25739][25899]: 2007/09/20_14:55:06 DEBUG: r0: Command output: WFConnection drbd[25739][25900]: 2007/09/20_14:55:06 DEBUG: r0 status: Secondary/Unknown Secondary Unknown WFConnection drbd[25739][25901]: 2007/09/20_14:55:06 DEBUG: r0 start: succeeded. crmd[25448]: 2007/09/20_14:55:06 info: process_lrm_event: LRM operation share-drbd_0:0_start_0 (call=15, rc=0) complete crmd[25448]: 2007/09/20_14:55:06 info: build_operation_update: Digest for 0:0;6:2:686542a3-7c00-48a2-bdd3-0f9aad92cc0b (share-drbd_0:0_start_0) was c0e018b73fdf522b6cdd355e125af15e crmd[25448]: 2007/09/20_14:55:06 info: log_data_element: build_operation_update: digest:source cibmon[25442]: 2007/09/20_14:55:06 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:06 info: log_cib_diff: cib_update: Diff: --- 0.1.37 cibmon[25442]: 2007/09/20_14:55:06 info: log_cib_diff: cib_update: Diff: +++ 0.1.38 cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:06 info: match_graph_event: Action share-drbd_0:0_start_0 (6) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:06 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:07 WARN: status_from_rc: Action start on host-1.awale.qc.ca failed (target: vs. rc: 1): Error tengine[25451]: 2007/09/20_14:55:07 info: update_abort_priority: Abort priority upgraded to 1 tengine[25451]: 2007/09/20_14:55:07 info: update_abort_priority: Abort action 0 superceeded by 2 tengine[25451]: 2007/09/20_14:55:07 info: match_graph_event: Action share-FS_start_0 (37) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:07 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:07 info: log_cib_diff: cib_update: Diff: --- 0.1.38 cibmon[25442]: 2007/09/20_14:55:07 info: log_cib_diff: cib_update: Diff: +++ 0.1.39 cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:07 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:09 info: log_cib_diff: cib_update: Diff: --- 0.1.39 tengine[25451]: 2007/09/20_14:55:09 info: match_graph_event: Action share-drbd_0:1_start_0 (8) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:09 info: log_cib_diff: cib_update: Diff: +++ 0.1.40 tengine[25451]: 2007/09/20_14:55:09 info: te_pseudo_action: Pseudo action 11 fired and confirmed cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: - tengine[25451]: 2007/09/20_14:55:09 info: te_pseudo_action: Pseudo action 14 fired and confirmed cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + crmd[25448]: 2007/09/20_14:55:09 info: do_lrm_rsc_op: Performing op=share-drbd_0:0_notify_0 key=64:2:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) tengine[25451]: 2007/09/20_14:55:09 info: send_rsc_command: Initiating action 64: share-drbd_0:0_post_notify_start_0 on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:09 info: send_rsc_command: Initiating action 67: share-drbd_0:1_post_notify_start_0 on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + lrmd[25445]: 2007/09/20_14:55:09 info: RA output: (share-drbd_0:0:notify:stdout) r0: ACTION=notify drbd[25909][25920]: 2007/09/20_14:55:09 DEBUG: r0 notify: post for start - counts: active 0 - starting 2 - stopping 0 drbd[25909][25922]: 2007/09/20_14:55:09 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[25909][25929]: 2007/09/20_14:55:09 DEBUG: r0: Exit code 0 drbd[25909][25930]: 2007/09/20_14:55:09 DEBUG: r0: Command output: Secondary/Unknown drbd[25909][25939]: 2007/09/20_14:55:09 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 drbd[25909][25952]: 2007/09/20_14:55:09 DEBUG: r0: Exit code 0 drbd[25909][25953]: 2007/09/20_14:55:09 DEBUG: r0: Command output: Connected drbd[25909][25954]: 2007/09/20_14:55:09 DEBUG: r0 status: Secondary/Unknown Secondary Unknown Connected drbd[25909][25955]: 2007/09/20_14:55:09 DEBUG: r0: Calling /usr/sbin/crm_master -v 75 -l reboot crm_master[25957]: 2007/09/20_14:55:09 info: Invoked: /usr/sbin/crm_master -v 75 -l reboot cibmon[25442]: 2007/09/20_14:55:09 info: cibmon_diff: [cib_diff_notify] cib_modify confirmed cibmon[25442]: 2007/09/20_14:55:09 info: log_cib_diff: cib_modify: Diff: --- 0.1.40 cibmon[25442]: 2007/09/20_14:55:09 info: log_cib_diff: cib_modify: Diff: +++ 0.1.41 tengine[25451]: 2007/09/20_14:55:09 info: extract_event: Aborting on transient_attributes changes for 77c6bcf6-f1e9-4e62-a33e-6ddd3530b92e cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: - tengine[25451]: 2007/09/20_14:55:09 info: update_abort_priority: Abort priority upgraded to 1000000 cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + tengine[25451]: 2007/09/20_14:55:09 info: update_abort_priority: 'Event failed' abort superceeded cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_modify: + drbd[25909][25958]: 2007/09/20_14:55:09 DEBUG: r0: Exit code 0 drbd[25909][25959]: 2007/09/20_14:55:09 DEBUG: r0: Command output: crm_master[25957]: 2007/09/20_14:55:09 info: Invoked: /usr/sbin/crm_master -v 75 -l reboot lrmd[25445]: 2007/09/20_14:55:09 info: RA output: (share-drbd_0:0:notify:stdout) crm_master[25957]: 2007/09/20_14:55:09 info: Invoked: /usr/sbin/crm_master -v 75 -l reboot crmd[25448]: 2007/09/20_14:55:09 info: process_lrm_event: LRM operation share-drbd_0:0_notify_0 (call=17, rc=0) complete cibmon[25442]: 2007/09/20_14:55:09 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:09 info: log_cib_diff: cib_update: Diff: --- 0.1.41 tengine[25451]: 2007/09/20_14:55:09 info: match_graph_event: Action share-drbd_0:0_post_notify_start_0 (64) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:09 info: log_cib_diff: cib_update: Diff: +++ 0.1.42 cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:09 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:11 info: cibmon_diff: [cib_diff_notify] cib_modify confirmed tengine[25451]: 2007/09/20_14:55:11 info: extract_event: Aborting on transient_attributes changes for 1da8804b-8a93-4145-9d40-1133699fe377 cibmon[25442]: 2007/09/20_14:55:11 info: log_cib_diff: cib_modify: Diff: --- 0.1.42 cibmon[25442]: 2007/09/20_14:55:11 info: log_cib_diff: cib_modify: Diff: +++ 0.1.43 cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: - cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:11 info: log_data_element: cib_modify: + cibmon[25442]: 2007/09/20_14:55:13 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:55:13 info: match_graph_event: Action share-drbd_0:1_post_notify_start_0 (67) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:13 info: log_cib_diff: cib_update: Diff: --- 0.1.43 tengine[25451]: 2007/09/20_14:55:13 info: te_pseudo_action: Pseudo action 15 fired and confirmed cibmon[25442]: 2007/09/20_14:55:13 info: log_cib_diff: cib_update: Diff: +++ 0.1.44 crmd[25448]: 2007/09/20_14:55:13 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_IPC_MESSAGE origin=route_message ] tengine[25451]: 2007/09/20_14:55:13 info: run_graph: ==================================================== cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: - crmd[25448]: 2007/09/20_14:55:13 info: do_state_transition: All 2 cluster nodes are eligible to run resources. tengine[25451]: 2007/09/20_14:55:13 notice: run_graph: Transition 2: (Complete=19, Pending=0, Fired=0, Skipped=14, Incomplete=0) cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'stop' for cluster option 'no-quorum-policy' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'true' for cluster option 'symmetric-cluster' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'false' for cluster option 'stonith-enabled' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'reboot' for cluster option 'stonith-action' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value '0' for cluster option 'default-resource-stickiness' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value '0' for cluster option 'default-resource-failure-stickiness' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'true' for cluster option 'is-managed-default' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value '60s' for cluster option 'cluster-delay' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value '20s' for cluster option 'default-action-timeout' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-resources' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'true' for cluster option 'stop-orphan-actions' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'false' for cluster option 'remove-after-stop' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value '-1' for cluster option 'pe-error-series-max' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value '-1' for cluster option 'pe-warn-series-max' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value '-1' for cluster option 'pe-input-series-max' pengine[25452]: 2007/09/20_14:55:13 notice: cluster_option: Using default value 'true' for cluster option 'startup-fencing' pengine[25452]: 2007/09/20_14:55:13 info: determine_online_status: Node host-2.awale.qc.ca is online pengine[25452]: 2007/09/20_14:55:13 WARN: unpack_rsc_op: Processing failed op (share-FS_start_0) on host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 WARN: unpack_rsc_op: Handling failed start for share-FS on host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 info: determine_online_status: Node host-1.awale.qc.ca is online pengine[25452]: 2007/09/20_14:55:13 WARN: unpack_rsc_op: Processing failed op (share-FS_start_0) on host-1.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 WARN: unpack_rsc_op: Handling failed start for share-FS on host-1.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 info: unpack_find_resource: Internally renamed share-drbd_0:0 on host-1.awale.qc.ca to share-drbd_0:1 pengine[25452]: 2007/09/20_14:55:13 info: clone_print: Master/Slave Set: ms-drbd_0 pengine[25452]: 2007/09/20_14:55:13 info: native_print: share-drbd_0:0 (heartbeat::ocf:drbd): Started host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 info: native_print: share-drbd_0:1 (heartbeat::ocf:drbd): Started host-1.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 info: group_print: Resource Group: DW pengine[25452]: 2007/09/20_14:55:13 info: native_print: msg_first (heartbeat:msg): Started host-1.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 info: native_print: share-FS (heartbeat::ocf:Filesystem): Started host-1.awale.qc.ca FAILED pengine[25452]: 2007/09/20_14:55:13 info: native_print: share-drbdlinks (heartbeat:drbdlinks): Stopped pengine[25452]: 2007/09/20_14:55:13 info: native_print: IPaddr_10_0_0_59 (heartbeat::ocf:IPaddr): Stopped pengine[25452]: 2007/09/20_14:55:13 info: native_print: mysql (heartbeat::ocf:mysql): Stopped pengine[25452]: 2007/09/20_14:55:13 info: native_print: dw-httpd (heartbeat:daemontools): Stopped crmd[25448]: 2007/09/20_14:55:13 info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] pengine[25452]: 2007/09/20_14:55:13 info: native_print: msg_last (heartbeat:msg): Stopped pengine[25452]: 2007/09/20_14:55:13 WARN: native_color: Resource msg_first cannot run anywhere pengine[25452]: 2007/09/20_14:55:13 info: master_color: Promoted 0 instances of a possible 1 to master pengine[25452]: 2007/09/20_14:55:13 WARN: native_color: Resource share-FS cannot run anywhere pengine[25452]: 2007/09/20_14:55:13 WARN: native_color: Resource share-drbdlinks cannot run anywhere tengine[25451]: 2007/09/20_14:55:13 info: unpack_graph: Unpacked transition 3: 7 actions in 7 synapses pengine[25452]: 2007/09/20_14:55:13 WARN: native_color: Resource IPaddr_10_0_0_59 cannot run anywhere tengine[25451]: 2007/09/20_14:55:13 info: send_rsc_command: Initiating action 8: share-drbd_0:0_monitor_120000 on host-2.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 WARN: native_color: Resource mysql cannot run anywhere tengine[25451]: 2007/09/20_14:55:13 info: send_rsc_command: Initiating action 11: share-drbd_0:1_monitor_120000 on host-1.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 WARN: native_color: Resource dw-httpd cannot run anywhere crmd[25448]: 2007/09/20_14:55:13 info: do_lrm_rsc_op: Performing op=share-drbd_0:0_monitor_120000 key=8:3:686542a3-7c00-48a2-bdd3-0f9aad92cc0b) tengine[25451]: 2007/09/20_14:55:13 info: te_pseudo_action: Pseudo action 39 fired and confirmed pengine[25452]: 2007/09/20_14:55:13 WARN: native_color: Resource msg_last cannot run anywhere tengine[25451]: 2007/09/20_14:55:13 info: send_rsc_command: Initiating action 2: share-FS_stop_0 on host-1.awale.qc.ca pengine[25452]: 2007/09/20_14:55:13 info: master_color: Promoted 0 instances of a possible 1 to master tengine[25451]: 2007/09/20_14:55:13 info: te_pseudo_action: Pseudo action 55 fired and confirmed pengine[25452]: 2007/09/20_14:55:13 notice: NoRoleChange: Leave resource share-drbd_0:0 (host-2.awale.qc.ca) pengine[25452]: 2007/09/20_14:55:13 notice: RecurringOp: host-2.awale.qc.ca share-drbd_0:0_monitor_120000 pengine[25452]: 2007/09/20_14:55:13 notice: NoRoleChange: Leave resource share-drbd_0:1 (host-1.awale.qc.ca) pengine[25452]: 2007/09/20_14:55:13 notice: RecurringOp: host-1.awale.qc.ca share-drbd_0:1_monitor_120000 pengine[25452]: 2007/09/20_14:55:13 notice: NoRoleChange: Leave resource share-drbd_0:0 (host-2.awale.qc.ca) pengine[25452]: 2007/09/20_14:55:13 notice: RecurringOp: host-2.awale.qc.ca share-drbd_0:0_monitor_120000 pengine[25452]: 2007/09/20_14:55:13 notice: NoRoleChange: Leave resource share-drbd_0:1 (host-1.awale.qc.ca) pengine[25452]: 2007/09/20_14:55:13 notice: RecurringOp: host-1.awale.qc.ca share-drbd_0:1_monitor_120000 pengine[25452]: 2007/09/20_14:55:13 notice: StopRsc: host-1.awale.qc.ca Stop msg_first pengine[25452]: 2007/09/20_14:55:13 notice: StopRsc: host-1.awale.qc.ca Stop share-FS pengine[25452]: 2007/09/20_14:55:13 WARN: process_pe_message: Transition 3: WARNINGs found during PE processing. PEngine Input stored in: /var/lib/heartbeat/pengine/pe-warn-81.bz2 pengine[25452]: 2007/09/20_14:55:13 info: process_pe_message: Configuration WARNINGs found during PE processing. Please run "crm_verify -L" to identify issues. drbd[25960][25972]: 2007/09/20_14:55:13 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[25960][25979]: 2007/09/20_14:55:13 DEBUG: r0: Exit code 0 drbd[25960][25980]: 2007/09/20_14:55:13 DEBUG: r0: Command output: Secondary/Secondary drbd[25960][25988]: 2007/09/20_14:55:13 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 drbd[25960][26001]: 2007/09/20_14:55:13 DEBUG: r0: Exit code 0 drbd[25960][26002]: 2007/09/20_14:55:13 DEBUG: r0: Command output: Connected drbd[25960][26003]: 2007/09/20_14:55:13 DEBUG: r0 status: Secondary/Secondary Secondary Secondary Connected crmd[25448]: 2007/09/20_14:55:13 info: process_lrm_event: LRM operation share-drbd_0:0_monitor_120000 (call=18, rc=0) complete cibmon[25442]: 2007/09/20_14:55:13 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:55:13 info: match_graph_event: Action share-drbd_0:0_monitor_120000 (8) confirmed on host-2.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:13 info: log_cib_diff: cib_update: Diff: --- 0.1.44 cibmon[25442]: 2007/09/20_14:55:13 info: log_cib_diff: cib_update: Diff: +++ 0.1.45 cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:13 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: cibmon_diff: [cib_diff_notify] cib_update confirmed tengine[25451]: 2007/09/20_14:55:14 info: match_graph_event: Action share-FS_stop_0 (2) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:14 info: log_cib_diff: cib_update: Diff: --- 0.1.45 tengine[25451]: 2007/09/20_14:55:14 info: send_rsc_command: Initiating action 36: msg_first_stop_0 on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:14 info: log_cib_diff: cib_update: Diff: +++ 0.1.46 cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:14 info: log_cib_diff: cib_update: Diff: --- 0.1.46 cibmon[25442]: 2007/09/20_14:55:14 info: log_cib_diff: cib_update: Diff: +++ 0.1.47 cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: - cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:14 info: match_graph_event: Action share-drbd_0:1_monitor_120000 (11) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:14 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:15 info: cibmon_diff: [cib_diff_notify] cib_delete confirmed cibmon[25442]: 2007/09/20_14:55:15 info: log_cib_diff: cib_delete: Diff: --- 0.1.47 cibmon[25442]: 2007/09/20_14:55:15 info: log_cib_diff: cib_delete: Diff: +++ 0.1.48 cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: - cibmon[25442]: 2007/09/20_14:55:15 info: log_data_element: cib_delete: + cibmon[25442]: 2007/09/20_14:55:16 info: cibmon_diff: [cib_diff_notify] cib_update confirmed cibmon[25442]: 2007/09/20_14:55:16 info: log_cib_diff: cib_update: Diff: --- 0.1.48 cibmon[25442]: 2007/09/20_14:55:16 info: log_cib_diff: cib_update: Diff: +++ 0.1.49 tengine[25451]: 2007/09/20_14:55:16 info: match_graph_event: Action msg_first_stop_0 (36) confirmed on host-1.awale.qc.ca cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: - crmd[25448]: 2007/09/20_14:55:16 info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_IPC_MESSAGE origin=route_message ] tengine[25451]: 2007/09/20_14:55:16 info: te_pseudo_action: Pseudo action 40 fired and confirmed cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:16 info: run_graph: Transition 3: (Complete=7, Pending=0, Fired=0, Skipped=0, Incomplete=0) cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + tengine[25451]: 2007/09/20_14:55:16 info: notify_crmd: Transition 3 status: te_complete - cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + cibmon[25442]: 2007/09/20_14:55:16 info: log_data_element: cib_update: + drbd[26006][26018]: 2007/09/20_14:57:13 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[26006][26025]: 2007/09/20_14:57:13 DEBUG: r0: Exit code 0 drbd[26006][26026]: 2007/09/20_14:57:13 DEBUG: r0: Command output: Secondary/Secondary drbd[26006][26034]: 2007/09/20_14:57:13 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 drbd[26006][26047]: 2007/09/20_14:57:13 DEBUG: r0: Exit code 0 drbd[26006][26048]: 2007/09/20_14:57:13 DEBUG: r0: Command output: Connected drbd[26006][26049]: 2007/09/20_14:57:13 DEBUG: r0 status: Secondary/Secondary Secondary Secondary Connected drbd[26050][26062]: 2007/09/20_14:59:13 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[26050][26069]: 2007/09/20_14:59:13 DEBUG: r0: Exit code 0 drbd[26050][26070]: 2007/09/20_14:59:13 DEBUG: r0: Command output: Secondary/Secondary drbd[26050][26078]: 2007/09/20_14:59:13 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 drbd[26050][26091]: 2007/09/20_14:59:13 DEBUG: r0: Exit code 0 drbd[26050][26092]: 2007/09/20_14:59:13 DEBUG: r0: Command output: Connected drbd[26050][26093]: 2007/09/20_14:59:14 DEBUG: r0 status: Secondary/Secondary Secondary Secondary Connected drbd[26096][26108]: 2007/09/20_15:01:14 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf state r0 drbd[26096][26115]: 2007/09/20_15:01:14 DEBUG: r0: Exit code 0 drbd[26096][26116]: 2007/09/20_15:01:14 DEBUG: r0: Command output: Secondary/Secondary drbd[26096][26124]: 2007/09/20_15:01:14 DEBUG: r0: Calling drbdadm -c /etc/drbd.conf cstate r0 drbd[26096][26137]: 2007/09/20_15:01:14 DEBUG: r0: Exit code 0 drbd[26096][26138]: 2007/09/20_15:01:14 DEBUG: r0: Command output: Connected drbd[26096][26139]: 2007/09/20_15:01:14 DEBUG: r0 status: Secondary/Secondary Secondary Secondary Connected