Transmitting on the wire

Adversity doesn't build character, it reveals it.

VMware vVols: How to Enable Trivia Logging

by jhop

I have been working on a rather obscure issue recently that has required an in-depth look into multiple components within VMware ESXi. One of those components was (you guessed it) vVols and how it was handling a very specific task.

Unfortunately I just wasn’t getting the needed information from the “info” level logging that is enabled by default, I needed more verbosity. I decided to start perusing the interweb to see if I could find out how this is done. Throughout my searching though I didn’t find how this could be accomplished. Naturally, being a computer nerd with OCD tendencies, I had to know the answer. 🙂

I was fortunate enough that this journey actually provided fruitful results and was able to find what I was looking for. Even more fortunate, the change is rather easy and not at all scary! It is as simple as editing a config file, very similar to the process for other vCenter / ESXi services that you can find in this VMware KB.

All right, enough boring back story let’s get you what you came here for; how do you change the logging level?

Step 1. Login to your ESXi host via your valid credentials.

Step 2. Using your favorite text editor (I personally like VI) open the following file:

/etc/vmware/vvold/config.xml

Step 3. In the file you will need to locate two different lines, as shown below:

<!-- default log level -->
<level>info</level>    <----- This line....

<!-- default sys log level -->
 <syslogLevel>info</syslogLevel> <------ ...and this line.

Step 4. Once you have found these two lines simply change the “info” to “trivia”.

It should now look as follows for each line:

<!-- default log level -->
<level>trivia</level>

<!-- default sys log level -->
<syslogLevel>trivia</syslogLevel>

NOTE: You do need to change both of these lines, if you only change one or the other it will not enable trivia logging for you.

Step 5. Restart the vvold service on the ESXi host:

/etc/init.d/vvold restart

NOTE: When restarting the vvold process your vVol VMs I/O to storage should not be impacted. Any vVol related management tasks you may have going from that ESXi host, however, would. This would include, but isn’t limited to, VM Power On / Off requests, Storage vMotions, etc.

Step 6. Review your vvold logs (/var/log/vvold.log) and ensure it is reporting trivia logs:

2019-05-31T21:46:00.009Z info vvold[2191898] [Originator@6876 sub=Default] SI:GetVvolContainer successful for sn1-m20-e05-28-vvol-sc, id=3c805425-3410-3526-8b5e-9721f5e7bdaf, maxVVol=0 MB
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=Default.HTTPService] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: false (PresetContentLength -1)
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=Default.HTTPService] HTTP Response: SetKeepAlive(false)
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=Default.HTTPService] HTTP Response: Complete (processed 1184 bytes)
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=Default.HTTPService] HTTP Response: Flush(lastBlock = true)
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=Default.HTTPService] HTTP Response: Setting Content-Length: 1184
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=Default.HTTPService] HTTP Response: Header size is 161
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=Default.HTTPService] HTTP Response: Writing 1345 bytes to stream
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=Default.HTTPService.HttpConnection] ResponseCompleted(true)
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=session pool] [UnregisterSession] UnregisterSession called for session id=52afb9ca-14e0-bc99-d69b-03e1918c4221
2019-05-31T21:46:00.009Z trivia vvold[2191898] [Originator@6876 sub=session pool] [UnregisterSession] UnregisterSession called for session id=52afb9ca-14e0-bc99-d69b-03e1918c4221
2019-05-31T21:46:00.009Z verbose vvold[2191898] [Originator@6876 sub=Default] CloseSession called for session id=52afb9ca-14e0-bc99-d69b-03e1918c4221

In order to return your logging back to “info” level just do the same steps above, simply reversing step 3 and step 4.

DISCLAIMER: Please remember to take your environment into account and ensure that you don’t have critical management tasks happening that could impact production. This should only be performed in dev environments or under the direction of your storage company and/or VMware’s guidance.

Happy troubleshooting and good luck!