Warning: This is kind of a rant.
Sometimes I really have to wonder if the engineers who build hardware ever even talk to people who use their products.
Though I love the EMC VPLEX, I get this feeling of a ‘disconnect’ between design and use more strongly with this product than with many others.
This post is a typical example.
I noticed that one of my vplex clusters apparently does not have the correct DNS settings set up.
Now, Disclaimer: I am not a Linux guy. But even if I was, my first thought, when dealing with hardware, is not to treat it as an ordinary Linux distro. Those kind of assumptions can be fatal. When its a complete provided solution, I assume and it is mostly the case,that vendors supply specific configuration commands environments to configure the hardware. It is always best practice to follow vendor guidelines first before you start messing around yourself. Messing around yourself is often not even supported.
So, lets start working the problem:
My first go to for most things is of course google:
Now I really did try to find anything, any post by anyone, that could tell me how to set up DNS settings. I spent a whole 5 minutes at least on Google :p
But alas, no, lots of informative blog posts, nothing about DNS however.
Ok, to the manuals. I keep a folder of VPLEX documentation handy for exactly this kind of thing:
Ok, something more drastic:
3 hits. THREE.. really?
Yes.. I know the management server uses DNS. *sigh*
Oh.. well at least I know that it uses standard Bind now, great!
oh, hi again!
Ok, lets try EMC Support site next:
Uhhmm.. only interesting one here is:
( https://support.emc.com/docu34006_VPLEX-with-GeoSynchrony-5.0-and-Point-Releases-CLI-Guide.pdf?language=en_US )
director dns-settings create, eh??
Getting exited now!
‘Create a new DNS settings configuration’
Uhmm.. you mean like… where I can enter my DNS servers, right? Riiiiight?
Oh.. uh.. what? I guess they removed it in or prior to Geosyncronity 5.3 ? :p
Back to EMC support
So… there is NO DNS knowledge anywhere in the EMC documentation? At all??? Anywhere??
Wait! Luke, there is another!
SolVe (seriously, who comes up with these names) is the replacement to the good ole ‘procedure generator’ that used to be on SupportLink.
Hmm… I dont see DNS listed?
Change IP addresses maybe??
Hmm… not really.. however I see an interesting command: management-server
Oh… I guess you are too good to care for plain old DNS eh?
And this is the point where I have run out of options to try within the EMC support sphere.
And As you can see, I really really did try!
So… the Management server is basically a Suse Linux distro, right?
Uhm… well fuck.
Now, I am logged into the management server with the ‘service’ account. The highest-level account that is mentioned in any of the documentation. of course, it is not the root account.
sudo su – … and voila:
There we go!
Which brings me to another thing I might as well address right now.
The default root account password for vplex management server is easily Googlable. That is why you should change it. There actually is a procedure for this: https://support.emc.com/kb/211258
Which I am sure no one ever anywhere ever has ever followed.. that at least is usually the case with this sort of thing.
Here is the text from that KB article:
The default password should be changed by following the below procedure. EMC recommends following the steps in this KB article and downloading the script mentioned in the article from EMC On-Line Support.
The VPLEX cluster must be upgraded to code version 5.4.1 Patch 3 or to 5.5 Patch 1 prior to running the script.
Note: VS1 customers cannot upgrade to 5.5, since only VS2 hardware is capable of running 5.5. VS1 customers must upgrade to 5.4 SP1 P3, and VS2 customers can go to either 5.4 SP1 P3, or 5.5 Patch 1.
The script, “VPLEX-MS-patch-update-change-root_password-2015-11-21-install” automates the workaround procedure and can be found at EMC’s EMC Online Support.
Instructions to run the script:
Log in to the VPLEX management-server using the service account credentials and perform the following from the management-server shell prompt:
service@ManagementServer:~> chmod +x /tmp/VPlexInstallPackages/VPlex-MS-patch-update-root_password-2015-11-21-install
This script will perform following operation:
service@ManagementServer:~> sudo /tmp/VPlexInstallPackages/VPlex-MS-patch-update-root_password-2015-11-21-install –force
Running the script…
– Updating sudoers
Enter New Password:
Testing password strength…
Changing password for root.
NOTE: In the event that the password is not updated, run the script again with proper password complexity.
service@ManagementServer:~> sudo -k whoami
***Contact EMC Customer Service with the new root password to verify that EMC can continue to support your VPLEX installation. Failure to update EMC Customer Service with the new password may prevent EMC from providing timely support in the event of an outage.
Notice how convoluted this is. Also notice how you need to have at least 5.4.1 Patch 3 in order to even run it.
While EMC KB articles have an attachment section, this script in question is of course not added.
Instead, you have to go look for it yourself, helpfully, they link you to: https://support.emc.com/products/29264_VPLEX-VS2/Tools/
And its right there, for now at least.
What I find interesting here is that it appears both the article, and the script, have been last edited.. .today?
Coincidental. But also a little scary. Does this mean that prior to 5.4.1 Patch 3 there really was no supported way to change the default vplex management server root password? The one that every EMC and VPLEX support engineer knows and is easily Googlable? Really?
I think the most troubling part of all this is that final phrase:
Failure to update EMC Customer Service with the new password may prevent EMC from providing timely support in the event of an outage.
Have you ever tried changing vendor default backdoor passwords, and see if their support teams can deal with it? Newsflash: they can not. We tried this once with EMC Clariion support. Changed the default passwords. We dutifully informed EMC support that we changed them. They assured it this was noted down in their administration for our customer.
You can of course guess what happened. Every single time EMC support would try to get in, and complain that they could not. You had to tell them every single time about the new passwords you had set up. I am sure that somewhere in the EMC administrative system, there is a notes field that could contain our non-default passwords. But no EMC engineer I have ever spoken to would even look there, or even know to look there.
If you build an entire hardware-support infrastructure around the assumption of built-in default password that everyone-and-their-mother knows, you make it fundamentally harder to properly support users who ‘do the right thing’ and change them. And you build in vulnerability by default.
Instead, design you hardware and appliances to generate new and unique strong default passwords on first deployment, or have the user provide them (enforcing complexity). (many VMware appliances now do this). But do NOT bake in backdoor default passwords that users and Google will find out about eventually.