We all know about the VMware case numbers. Each SR you open gets a nice number.
Internally, VMware has a problem database. Newly found bugs end up in there. And if you spend a lot of time with VMware support, you will end up hearing a lot about these internal PR (problem reports).
Here is a cool fact you may not know. Hidden in the HTML source of the public release notes that VMware produces, are the actual PR numbers associated with the issue that is described as having been fixed. (or not fixed).
Take the NSX 6.2.0 release notes for example: https://www.vmware.com/support/nsx/doc/releasenotes_nsx_vsphere_620.html
View the source:
And if you scroll down to the fixes, you will find:
Its those DOCNOTE numbers, that are the actual PR numbers. Sometimes they also list the public KB number too. But there are far more interal PR numbers than there are public KB equivalents.
So how can this help you?
Well for one thing, you can start asking intelligent questions of VMware support, like: ‘Has something like my issue been reported before in the PR database?’ (prompting the engineer to go look for it, which they don’t always do on their own accord 😉
Or you can use it as a validation check. If your issue is scheduled to be fixed in an upcoming patch, ask the support engineer for the associated PR numbers! That way, you can verify yourself in the release notes, if the fix was included!
The process of getting a new patch or update through QA is quite involved, and sometimes fixes fall by the wayside. This is not immediately known to everyone inside VMware. So its always worth checking yourself; trust but verify.
This year I need to refresh my VMware VCP cert, so I have started to look around for educational materials to help with this.
I decided several years ago that I would never again buy any IT book in physical form, if that book ran the risk of being outdated quickly. This is especially true for product-specific books. My reasoning is: buying a physical book that has a limited shelf life is wasteful. And it is usually the case that eBook versions are cheaper. Finally, when I study, I tend to do so when the opportunity and motivation arises which could be anywhere at any time. Often when I travel. So I benefit from a flexible digital format that will follow me around my various devices.
VMware refreshes their core vsphere product every few years, so here is a prime example of the kind of subject I would not buy a physical book for.
It’s slightly shocking to me to see how expensive a ‘Mastering vSphere’ book now is. My main issue is that this book has a utility that is severely limited in time. I would have far less trouble dishing this kind of amount out for books that I could proudly put on my shelf the rest of my life. This amount is so off putting, I will forgo purchasing this book this time around, and will seek other means of getting my coverage of the product. But this is a real shame. I like these books, and I appreciate the effort put into then. But these prices are just not worth it.
Warning: This is kind of a rant.
Sometimes I really have to wonder if the engineers who build hardware ever even talk to people who use their products.
Though I love the EMC VPLEX, I get this feeling of a ‘disconnect’ between design and use more strongly with this product than with many others.
This post is a typical example.
I noticed that one of my vplex clusters apparently does not have the correct DNS settings set up.
Now, Disclaimer: I am not a Linux guy. But even if I was, my first thought, when dealing with hardware, is not to treat it as an ordinary Linux distro. Those kind of assumptions can be fatal. When its a complete provided solution, I assume and it is mostly the case,that vendors supply specific configuration commands environments to configure the hardware. It is always best practice to follow vendor guidelines first before you start messing around yourself. Messing around yourself is often not even supported.
So, lets start working the problem:
My first go to for most things is of course google:
Now I really did try to find anything, any post by anyone, that could tell me how to set up DNS settings. I spent a whole 5 minutes at least on Google :p
But alas, no, lots of informative blog posts, nothing about DNS however.
Ok, to the manuals. I keep a folder of VPLEX documentation handy for exactly this kind of thing:
Ok, something more drastic:
3 hits. THREE.. really?
Yes.. I know the management server uses DNS. *sigh*
Oh.. well at least I know that it uses standard Bind now, great!
oh, hi again!
Ok, lets try EMC Support site next:
Uhhmm.. only interesting one here is:
( https://support.emc.com/docu34006_VPLEX-with-GeoSynchrony-5.0-and-Point-Releases-CLI-Guide.pdf?language=en_US )
director dns-settings create, eh??
Getting exited now!
‘Create a new DNS settings configuration’
Uhmm.. you mean like… where I can enter my DNS servers, right? Riiiiight?
Oh.. uh.. what? I guess they removed it in or prior to Geosyncronity 5.3 ? :p
Back to EMC support
So… there is NO DNS knowledge anywhere in the EMC documentation? At all??? Anywhere??
Wait! Luke, there is another!
SolVe (seriously, who comes up with these names) is the replacement to the good ole ‘procedure generator’ that used to be on SupportLink.
Hmm… I dont see DNS listed?
Change IP addresses maybe??
Hmm… not really.. however I see an interesting command: management-server
Oh… I guess you are too good to care for plain old DNS eh?
And this is the point where I have run out of options to try within the EMC support sphere.
And As you can see, I really really did try!
So… the Management server is basically a Suse Linux distro, right?
Uhm… well fuck.
Now, I am logged into the management server with the ‘service’ account. The highest-level account that is mentioned in any of the documentation. of course, it is not the root account.
sudo su – … and voila:
There we go!
Which brings me to another thing I might as well address right now.
The default root account password for vplex management server is easily Googlable. That is why you should change it. There actually is a procedure for this: https://support.emc.com/kb/211258
Which I am sure no one ever anywhere ever has ever followed.. that at least is usually the case with this sort of thing.
Here is the text from that KB article:
The default password should be changed by following the below procedure. EMC recommends following the steps in this KB article and downloading the script mentioned in the article from EMC On-Line Support.
The VPLEX cluster must be upgraded to code version 5.4.1 Patch 3 or to 5.5 Patch 1 prior to running the script.
Note: VS1 customers cannot upgrade to 5.5, since only VS2 hardware is capable of running 5.5. VS1 customers must upgrade to 5.4 SP1 P3, and VS2 customers can go to either 5.4 SP1 P3, or 5.5 Patch 1.
The script, “VPLEX-MS-patch-update-change-root_password-2015-11-21-install” automates the workaround procedure and can be found at EMC’s EMC Online Support.
Instructions to run the script:
Log in to the VPLEX management-server using the service account credentials and perform the following from the management-server shell prompt:
service@ManagementServer:~> chmod +x /tmp/VPlexInstallPackages/VPlex-MS-patch-update-root_password-2015-11-21-install
This script will perform following operation:
service@ManagementServer:~> sudo /tmp/VPlexInstallPackages/VPlex-MS-patch-update-root_password-2015-11-21-install –force
Running the script…
– Updating sudoers
Enter New Password:
Testing password strength…
Changing password for root.
NOTE: In the event that the password is not updated, run the script again with proper password complexity.
service@ManagementServer:~> sudo -k whoami
***Contact EMC Customer Service with the new root password to verify that EMC can continue to support your VPLEX installation. Failure to update EMC Customer Service with the new password may prevent EMC from providing timely support in the event of an outage.
Notice how convoluted this is. Also notice how you need to have at least 5.4.1 Patch 3 in order to even run it.
While EMC KB articles have an attachment section, this script in question is of course not added.
Instead, you have to go look for it yourself, helpfully, they link you to: https://support.emc.com/products/29264_VPLEX-VS2/Tools/
And its right there, for now at least.
What I find interesting here is that it appears both the article, and the script, have been last edited.. .today?
Coincidental. But also a little scary. Does this mean that prior to 5.4.1 Patch 3 there really was no supported way to change the default vplex management server root password? The one that every EMC and VPLEX support engineer knows and is easily Googlable? Really?
I think the most troubling part of all this is that final phrase:
Failure to update EMC Customer Service with the new password may prevent EMC from providing timely support in the event of an outage.
Have you ever tried changing vendor default backdoor passwords, and see if their support teams can deal with it? Newsflash: they can not. We tried this once with EMC Clariion support. Changed the default passwords. We dutifully informed EMC support that we changed them. They assured it this was noted down in their administration for our customer.
You can of course guess what happened. Every single time EMC support would try to get in, and complain that they could not. You had to tell them every single time about the new passwords you had set up. I am sure that somewhere in the EMC administrative system, there is a notes field that could contain our non-default passwords. But no EMC engineer I have ever spoken to would even look there, or even know to look there.
If you build an entire hardware-support infrastructure around the assumption of built-in default password that everyone-and-their-mother knows, you make it fundamentally harder to properly support users who ‘do the right thing’ and change them. And you build in vulnerability by default.
Instead, design you hardware and appliances to generate new and unique strong default passwords on first deployment, or have the user provide them (enforcing complexity). (many VMware appliances now do this). But do NOT bake in backdoor default passwords that users and Google will find out about eventually.
EMC have released an OVF appliance that is meant to allow you to store and browse 30 days worth of VPLEX performance statistics. Version1 is limited to just a few metrics, but it is a very welcome addition to the VPLEX monitoring tools that are available! Requires GeoSynchrony 5.5
Today I was looking up some information on vplex on the EMC support site, my eye was quickly drawn to the following entries:
I have seen no mention of this at all on either Twitter or on the VPLEX community space at EMC: https://community.emc.com/community/products/vplex
This is typical of EMC in my experience, they are terribad at disseminating support information and making new stuff ‘discoverable’.
So what is this thing?
Up till now, you had several ways to monitor, save, and analyze VPLEX statistics.
- The GUI, but that only shows live data, no history, and only shows very few metics and only on high level
- VPLEXCLI: Monitor create, Monitor collect, etc. Powerfull CLI commands, any statistics can be saved. Can create exportable CSV files. But hard to use and understand, and for live monitoring the implementation is truly horrible, scrolling across your screen in a disruptive way, no ‘top’ kind of function here or anything
- EMC VIPR SRM. EMCs statistics and analytics suite. Good for all kinds of EMC products, uses a ‘perpetual’ version of the above mentioned monitor construct. But definitely not a free tool.
- If you have VMware vROPS: EMC Storage Analytics. Adapter for vROPS, but again not free. v3 of this adapter supports vROPS 6.x
- SNMP. Vplex comes with a MIB, but my experience with it so far is that its got some serious compliance (and syntax) issues that are preventing it from working in, for example the vROPS SNMP adapter. (this was my attempt at a ‘cheapo’ EMC Storage Analytics’ 😉
So, nothing we had so far ‘just worked’ as a fast, and -free, gui-based way of seeing some deep statistics. There was something to be said for this not being available in the product itself. It looks like with “EMC VPLEX Performance Monitor” , which is a free OVF download, they are attempting to address this concern.
Lets check the release notes.
VPLEX Performance Monitor is a stand-alone, customer installable tool that allows you to collect virtual volume metrics from a VPLEX Local or VPLEX Metro system. It allows Storage Administrators to see up to 30 days of historical virtual volume performance data to troubleshoot performance issues and analyze performance trends.
The VPLEX Performance Monitor tool is delivered as an OVA (Open Virtualization Format Archive) file that you deploy as a VMware virtual appliance. The virtual appliance connects to one VPLEX system and collects performance metrics for all virtual volumes that are in storage views. Historical virtual volume metrics are stored in a database within the virtual appliance for 30 days. The virtual appliance has a web application which allows you to view the data in charts that show all 30 days of data at once, or allows you to zoom in on data down to the minute.
The VPLEX Performance Monitor charts the following key virtual volume metrics:
Throughput (total read and write IOPS)
Note: The VPLEX Performance Monitor can connect to one VPLEX Local or Metro system at a time. To monitor additional VPLEX systems, deploy a new instance of the tool for each VPLEX Local or Metro system you want to monitor.
Ok, so admittedly, for a version1, not all that much here, no port statistics or backend storage metrics for example. But in most cases, you are gonna be interested in your virtual volumes most of all anyway, so a good start.
Only 1 VPLEX system at a time? We have 2 Metro-Cluster setups in our environment, which translates to 4 engines in total. Does a ‘system’ equate to an engine? I think so, which means I would need 4 of these appliances. Oh well.
30 days is a nice sweet spot for metric saving as far as I am concerned. This appliance is using an embedded database, so don’t expect options to save your data for years. Get VIPR SRM if you want that.
|IMPORTANT Version 1.0 cannot be upgraded. When the next release is available, you must delete the current VPLEX Monitor virtual appliance and deploy the new one. All performance data and user information will be lost.
Take note of the GeoSynchrony 5.5 requirement. This version only came out recently, so I don’t expect many people to be running this yet.
We don’t in any case, so I can’t provide you with an install demo, yet :p
If you have GeoSynchrony 5.5, go give this a try:
(EMC Support account required)
Update 03:03pm: Was googling for “EMC VPLEX Performance Monitor” to see if anyone else had mentioned it yet, came across this video (with 20 views so far, wow!) that showcases the new tool. https://www.youtube.com/watch?v=HiJgmbLkeTU
The next few weeks I will be in study mode.
Next week a 5 day classroom course for VMware vSphere 4.1: Install, Configure & Manage
The week after that its 3 days VMware vSphere 4: Manage for Performance
Then some exam training and on the 25th the VCP4 exam. I am exited and nervous, as I have not studies for an exam or cert for a long time. But I am also confident as I understand a lot of the material already.
In preparation for this I have been gathering some white papers, been reading up on some things and have also been looking for a good book also. I was in the tram today paging through the Kindle app for the Android, and it struck me that the Amazon sphere would be the right place to consume this kind of stuff. I have the Kindle app on my Ipad, my Phone and on 2 PC’s (I don’t yet own an actual Kindle) and the fact that I can pick up wherever I left off, wherever I am and when I have some time spare, appealed to me.
It’s a relief that I can actually find a good use for all these kindle outputs because I was stuck in the dilemma between the physical and the virtual. I will soon be receiving my first leather bound book from Easton Press, but for a while I felt torn between the superior convenience of ebooks, and the wish to have a proper physical book collection. But I realize now I can split the difference; educational books work a lot better as ebooks. So I can find good an honest use for both the old and new at the same time. This is greatly fulfilling to me somehow. I now even have a reason to get a Kindle.. perhaps.
Still in the tram I bought Scott Lowe’s Mastering VSphere 4, Kindle edition, and directed Amazon to send it to all my devices. By the time I got home, it was already downloaded onto my Ipad.
I used to collect a large amount of the MSPress books, but was saddened when I realized that most of them would become obsolete. Getting the ebook versions of these kinds of books might not necessarily be that much cheaper (mastering VSphere 4 set me back $41), but at least its not a waste of paper!
For a long time I justified buying these books by saying that I would rather read from my hand that from the screen, and this is actually true. But The Ipad and Kindle are convenient enough (and even my Android phone is, to be honest), that I cant justify wasting the trees on this kind of book anymore.
I was also thrilled to discover that some of the old Battletech novels are being re-released as Kindle versions. It was impossible to get your hands on, for example, the Blood of Kerensky trilogy in paperback form, it has been out of print for so long. I read most of the novels until Darkage started, but I never got my hands on some of the oldest titles. I hope they bring them all back! What a rich universe to re-discover!