Monday, July 30, 2012

VM Does Not Respond to Restart or Shutdown Commands


In XenServer, not too often you may have a VM that is not responsive to shutdown or restart commands through XenCenter and most likely has the yellow icon on it indicating that it is in process of doing something.  You may have even tried to restart the host, but the host is waiting for the VMs to shutdown but they won't so you cannot restart the host except by pulling the plug in which you do not want to do.  Below are some steps to try and resolve the issue starting from easy to last resorts.

  1. SSH or go to the console of the host that houses the troubled VM and perform the xe-toolstack-restart command.  If this does not fix your issue go to step 2.
  2. The next possible culprit may be a stuck task.  Perform the following commands to cycle through any tasks that may be keeping the VM stuck:
    1. To list all tasks:  xe task-list
    2. To cancel a task: xe task-cancel uuid=[uuid of task found from the previous step]
  3. After canceling any problematic tasks, check to see if your VM is still unresponsive.  If still unresponsive, proceed to step 4.
  4. If you made it this far, then the problem most likely is that the dom-id for the VM in XenServer control domain is unresponsive and needs to be terminated:
    1. Find the UUID of the VM.  You can do this through XenCenter, the UUID is found on the General tab of the VM.
    2. Connect to the host that houses the VM through the console or SSH.
    3. Run the following command: cd /opt/xensource/debug
    4. Run the following command: list_domains | grep [UUID of the problematic VM]
    5. The output format is [Dom ID] | [VM UUID]  | [VM state]
    6. Run the following command: ./xenops destroy_domain -domid [Dom ID found in previous step]
    7. Run the following command: xe-toolstack-restart
    8. Give it a few seconds and go back to XenCenter, the VM should go to a shutdown state (again give it a few seconds to refresh)
  5. The last and not recommended method is to power-cycle your host.  You shouldn't have to do this step if you follow the above correctly (at least I haven't ever had to do this for a stuck VM).

9 comments:

  1. This has saved my career! Thanks a lot!

    ReplyDelete
  2. Worked like a charm

    ReplyDelete
  3. xe vm-shutdown vm="VM Name" --force....

    This command from the console worked for me ...

    ReplyDelete
  4. Saved my life! Thanks a lot Chris!

    ReplyDelete
  5. I got this when I try the destroy domain
    # ./xenops destroy_domain -domid 172
    Fatal error: exception Domain.Domain_stuck_in_dying_state(172)

    ReplyDelete
    Replies
    1. Try restarting the tool stack and then try again.

      Delete