VMWare vCenter 4.1 Upgrade Notes - Ran into this one the other day as well

URL For furtner information:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026688

VMware VirtualCenter Server service fails after upgrading to vCenter Server 4.1

Symptoms

• The VirtualCenter Server service fails after upgrading to vCenter Server 4.1.
• You see this error in the vpxd logs:
  
  Panic: Win32 exception: Access Violation (0xc0000005)
     Read (0) at address 0000000000000098
     rip: 000000014071edcc rsp: 000000000419c700 rbp: 000000000419e160
     rax: 000000000419c7a8 rbx: 0000000000000018 rcx: 0000000000000020
     rdx: 000000000419c798 rdi: 0000000000000000 rsi: 000000000e8667b0
     r8:  000000000419c7a8 r9:  000000000cab1eea r10: 0000000002d27fd0
     r11: 000000000419c750 r12: 0000000007279700 r13: 0000000007279740
     r14: 00000000032edeb0 r15: 0000000000000004
  
  Note: For more information about the vpxd logs, see vpxd log directory path when vCenter Server 4.0 is installed on Windows Server 2008 (1011938).
  
• In vCenter Server 4.0, operations such as Inventory > Search fails. The vws.log file truncates with a Chunked stream ended unexpected error
• In vCenter Server 4.1, vCenter 4.1 fails immediately after it is started and the vpxd-*.log shows an ASSERT fails at one of the following:
  • ASSERT d:/build/ob/bora-258902/bora/vpx/vpxd/util/vpxdDbLoad.cpp:1059
  • ASSERT d:/build/ob/bora-258902/bora/vpx/vpxd/util/vpxdDbLoad.cpp:1066

Resolution

This issue is caused by corrupt entries for the virtual machine in the vCenter database.

Caution: This article deals with SQL database operations. Before proceeding, ensure that you are familiar with database concepts and that you have adequate backups.

 

To avoid encountering this issue:

1. Download the appropriate script for your environment. The scripts are attached to this article (located in the Attachments section at the bottom of this article).
2. Unzip the scripts.
3. Run the scripts on vCenter Server 4.0 database.
   
   Note: The SQL query must be run with fully qualified database table names. If you have multiple database instances, there may be a need to qualify the tables with the db-instance name and the db-owner name. For example, VPX_VM_CONFIG_INFO in the query should be changed to <DB_NAME>.<DB_OWNER>.VPX_VM_CONFIG_INFO.
   
4. If no result are returned, you can continue with the upgrade
   
   If any virtual machines are reported with inconsistent data, remove and re-register the affected virtual machines.
   1. Power off the virtual machine.
   2. Right-click the virtual machine and choose Remove from inventory.
   3. Click Yes to remove the virtual machine.
   4. Browse to the datastore where the virtual machine is stored.
   5. Right-click the .vmx file for the virtual machine, choose Add to inventory, and follow the on-screen wizard.
      
5. Continue with the upgrade.

If you have upgraded to vCenter Server 4.1 and the server keeps failing:

1. Roll back to vCenter Server 4.0. Connect vCenter Server 4.0 to the backup database.
2. Run the downloaded scripts on the vCenter Server 4.0 database.
   
   Note: The SQL query must be run with fully qualified database table names. If you have multiple database instances, there may be a need to qualify the tables with the db-instance name and the db-owner name. For example, VPX_VM_CONFIG_INFO in the query should be changed to <DB_NAME>.<DB_OWNER>.VPX_VM_CONFIG_INFO.
   
3. Find out if virtual machines are reported with inconsistent data.
4. Install vCenter Server 4.0 against the database backup.
5. Through the vSphere Client, remove and re-register the virtual machine.
   1. Power off the virtual machine.
   2. Right-click the virtual machine and choose Remove from inventory.
   3. Click Yes to remove the virtual machine.
   4. Browse to the datastore where the virtual machine is stored.
   5. Right-click the .vmx file for the virtual machine, choose Add to inventory, and follow the on-screen wizard.
      
6. Re-try the upgrade.

Note: If you experience the issue even though the script returned no virtual machines, this may be a new issue that has not been reported to VMware. Please contact VMware Support, with vCenter support information and a backup of the database for further review by VMware.

Additional Information

The error Win32 exception: Access Violation can occur for many issues. For other issues that generate this error, see:

• VMware VirtualCenter Server service fails after upgrading to vCenter Server 4.1 in an environment configured with Nexus 1000v (1026009)
• Cannot cold migrate a Virtual machine - vCenter service dump - Win32 exception: Access Violation (0xc0000005) (1014778)
• Virtual Center service crashes frequently (1016837)

Attachments

• getinconsistentvminfo_mssql.zip
• getinconsistentvminfo_oracle.zip