A note on #vCenter 5.5 to 6.5 #upgrade – interesting logs and locations to check for troubleshooting #vexpert
If you are migrating a large 5.5 vCenter environment to latest and greatest sexy VCSA using the Migration-Assistant , it can be a tedious wait sometimes. Also things can go horribly wrong.
Here are some logs and their locations I like to monitor during upgrade activities.
(note: This only applies to the Migration Assistant for Windows)
On the server you are running the Migration Tool itself:
The Migration Tool likes to leave its logs under the local application data location in your windows user profile, usually found under ‘C:\Users\<your username>\AppData\Local\Temp\vcsaUiInstaller
Sometimes windows will add a number inbetween in the path so i may looks like \Temp\12\vcsaUiInstaller
The Migration Tool will leave 3 logs here.
The Installer Log is the main running log for the Migration Tool, it will record everything the migration assistant can see, but wont be able to give you a live view of VCSA config steps, as those can only be retrieved after the VCSA appliance has configured itself, and reports back to the Migration Tool. During this time, you will see the Migration Tool run through a loop called ‘pollRpmInstallProgress’ while waiting for the VCSA to become available.
If the VCSA config fails in this step, the Migration Tool wont ever know about it, and you can then only get more logging information from the VCSA logbundle or via the console or SSH directly on the VCSA command line.
The installer log is also great to double-check all the entries you give in the Phase1 and Phase2 UI fields. We have more than once caught simple spelling and syntax mistakes this way.
This log will often give you an immediate cause of a failure during Phase1 or the start of Phase2, such as for instance pre-check failures, in more detail than the Migration Tool UI will. Later in Phase2, it will also copy cloudVM log file from the appliance back into to this log.
The ovftool is the log of the OVA deployment, and is only used for Phase1. Its useful to troubleshoot the OVA deployment, in case the default target ESX or vCenter messages don’t give you enough.
It also contains multiple copies of the entire EULA for the VCSA appliance, if you care to re-read it at your leisure :p
The cloudvm log only appears after the OVA deployment is complete, and the VCSA has configured itself to the point it can report back to the Migration Tool.
It is actually largely a copy of the ‘\var\log\firstboot\cloudvm.log’ file on the VCSA.
This log details RPM install script results that occur on firstboot after the VCSA starts for the first time. It then outputs config details that it got from the OVA deployment. This is where you spot that you made a mistake with an IP address, subnet mask, DNS server etc etc.
Logs on the source (windows) vCenter server, from the Migration Assistant
General note: As the original vCenter servers are decommissioned and turned off at the end of the upgrade, these logs can only be viewed live, or later on the VCSA (or via VCSA log bundle)
The main program that executes the collection and migration of data is called the ‘Upgrade Runner’ and its a set of Python scripts.
On the source vCenter, these scripts are all set up in a temporary location. For example C:\migE607.tmp\PFiles\VMware\CIS\cis_upgrade_runner\libs\sdk\service_manager.py , this is the script used to manipulate windows servives. Such as turning off your vCenter services one by one.
UpgradeRunner will contain large (all?) sections of all other logs, such as the ‘CollectRequirements’ logs. It makes it the single most useful log to check errors that occur during either initial collection of data from the source vCenter and SSO, or errors with the parsing of it.
This log will also show you errors you might get in the Migration Tool UI during Phase2, but usually with a bit more detail. For example, here is the (common) warning message ‘Please ensure extensions are compatible with the new vCenter Server and re-register extensions with the new vCenter Server after upgrade’
UpgradeRunner will also make a note of all the details of your vCenter topology.
Note: UpgradeRunner will not exist, if your VCSA initial config doesn’t make it past the beginning of Phase2.
In other words: if the VCSA never comes up properly, and never contacts the Migration Tool, then Upgraderunner will never start running on the source vCenter. And thus, no upgraderunner.log
There describe in great detail all the steps that take place, per component, on your source vCenter.
This can be extremely useful to find out of something went wrong during the collection part of Phase2 (just prior to the copy of the data to the target VCSA).
For example, if the source VPXD process (vcenter itself) times out while trying to turn off (which is common with ‘hanging’ windows services), you will find that clearly in the ‘CollectRequirements_com.vmware.vpxd_<timestamp> file.
Another example is if the Migration Assistant runs into trouble exporting the database entries from the external MS SQL database. Then you would check the ‘CollectRequirements_com.vmware.vcdb_’ logs.
Logs on the new VCSA appliance that are interesting
It should be noted that the target VCSA collects all the logs from the source vCenter servers that where created there by the Migration Assistant. So you when you produce a VCSA logbundle, you always have a complete collection of all activities that took place.
This log is generated on the new VCSA and describes anything that happes to prepare for the upgrade process. Such as setting up initial values, starting and stopping the SSH demon when needed to copy files from the source vCenter, and any errors validating the source data it received once it arrived. This file is a good place to find errors that occured during the validation (parsing) of the source vCenter data as it tries to import it into the new VCSA datastructures.
Here is an example where it turned out that the default DB partition was just too small to fit the imported DB data: (see this post for more on this: https://thefluffyadmin.net/?p=1155 )
\var\log\firstboot\cloudvm.log – As mentioned above already, very useful file for finding (network) config issues in the VCSA that occur in Phase2 after VCSA first boot. Includes parts of bootstrap.log
\var\log\vmware\upgrade\CollectRequirements_<component> – As described above, these are copies of the logs collected on the source vCenter
\var\log\vmware\upgrade\UpgradeRunner.log – As above, copied from the source vCenter
\var\log\vmware\upgrade\bootstrap.log – This is the record of all the config changes to the VCSA needed to support the migration, such as Partition sizing actions, etc. Is then merged into cloudvm.log
\var\log\vmware\upgrade\import-upgrade-runner.log – The log of the overall import actions.
\var\log\vmware\upgrade\Import_com.vmware.<component> The log of import actions per VCSA component.
\var\log\vmware\upgrade\Postupgrade_com.vmware.<component> The log of post import actions per VCSA component. (ff applicable)
Finally, also have a look at the following resources: