Archive for the ‘Rants’ Category

$60 is too much for an IT education book

Sunday, January 31st, 2016

This year I need to refresh my VMware VCP cert, so I have started to look around for educational materials to help with this.

image

I decided several years ago that I would never again buy any IT book in physical form, if that book ran the risk of being outdated quickly. This is especially true for product-specific books. My reasoning is: buying a physical book that has a limited shelf life is wasteful. And it is usually the case that eBook versions are cheaper. Finally, when I study, I tend to do so when the opportunity and motivation arises which could be anywhere at any time. Often when I travel. So I benefit from a flexible digital format that will follow me around my various devices.

VMware refreshes their core vsphere product every few years, so here is a prime example of the kind of subject I would not buy a physical book for.

It’s slightly shocking to me to see how expensive a ‘Mastering vSphere’ book now is.  My main issue is that this book has a utility that is severely limited in time. I would have far less trouble dishing this kind of amount out for books that I could proudly put on my shelf the rest of my life. This amount is so off putting, I will forgo purchasing this book this time around, and will seek other means of getting my coverage of the product. But this is a real shame. I like these books, and I appreciate the effort put into then. But these prices are just not worth it.

My journey to find how to set EMC VPLEX DNS Settings and how to change your default root password.

Tuesday, December 22nd, 2015

Warning: This is kind of a rant.

Sometimes I really have to wonder if the engineers who build hardware ever even talk to people who use their products.

Though I love the EMC VPLEX, I get this feeling of a ‘disconnect’ between design and use more strongly with this product than with many others.

This post is a typical example.

I noticed that one of my vplex clusters apparently does not have the correct DNS settings set up.

Now, Disclaimer: I am not a Linux guy. But even if I was, my first thought, when dealing with hardware, is not to treat it as an ordinary Linux distro. Those kind of assumptions can be fatal.  When its a complete provided solution, I assume and it is mostly the case,that vendors supply specific configuration commands environments to configure the hardware. It is always best practice to follow vendor guidelines first before you start messing around yourself.   Messing around yourself is often not even supported.

 

So, lets start working the problem:

 

My first go to for most things is of course google:

 

Now I really did try to find anything, any post by anyone, that could tell me how to set up DNS settings. I spent a whole 5 minutes at least on Google :p

But alas, no, lots of informative blog posts, nothing about DNS however.

Ok, to the manuals. I keep a folder of VPLEX documentation handy for exactly this kind of thing:

 

 

 

docu52651_VPLEX-Command-Reference-Guide MARCH2014.pdf

 

 

Uhh.. nope.

docu52646_VPLEX-Administration-Guide MARCH2014.pdf

AHA!

 

 

Uhh.. nope.

 

docu34005_VPLEX-Configuration-Guide MARCH2014.pdf

Nope

🙁

 

 

 

Ok, something more drastic:

docu52707_VPLEX-5.3-Documentation-Portfolio.pdf

3 hits. THREE.. really?

 

Yes.. I know the management server uses DNS. *sigh*

 

 

 

Oh.. well at least I know that it uses standard Bind now, great!

 

 

 

 

oh, hi again!

 

 

Ok, lets try EMC Support site next:

Uhhmm..    only interesting one here is:

( https://support.emc.com/docu34006_VPLEX-with-GeoSynchrony-5.0-and-Point-Releases-CLI-Guide.pdf?language=en_US )

director dns-settings create, eh??

Ok then!

Getting exited now!

\

 

‘Create a new DNS settings configuration’

Uhmm.. you mean like… where I can enter my DNS servers, right? Riiiiight?

 

Oh.. uh.. what?  I guess they removed it in or prior to Geosyncronity 5.3 ?    :p

🙁

Back to EMC support

Nope.

 

 

Nope.

So… there is NO DNS knowledge anywhere in the EMC documentation?  At all???  Anywhere??

 

Wait! Luke, there is another!

 

SolVe (seriously, who comes up with these names) is the replacement to the good ole ‘procedure generator’ that used to be on SupportLink.

Hmm… I dont see DNS listed?

Change IP addresses maybe??

Hmm…  not really.. however I see an interesting command: management-server

Oh… I guess you are too good to care for plain old DNS eh?

 

And this is the point where I have run out of options to try within the EMC support sphere.

And As you can see, I really really did try!

 

So…   the Management server is basically a Suse Linux distro, right?

vi /etc/resolv.conf

Uhm… well fuck.

Now, I am logged into the management server with the ‘service’ account. The highest-level account that is mentioned in any of the documentation. of course, it is not the root account.

sudo su – …  and voila:

There we go!

 

Which brings me to another thing I might as well address right now.

The default root account password for vplex management server is easily Googlable. That is why you should change it. There actually is a procedure for this: https://support.emc.com/kb/211258
Which I am sure no one ever anywhere ever has ever followed.. that at least is usually the case with this sort of thing.

Here is the text from that KB article:

The default password should be changed by following the below procedure. EMC recommends following the steps in this KB article and downloading the script mentioned in the article from EMC On-Line Support.

Automated script: 

The VPLEX cluster must be upgraded to code version 5.4.1 Patch 3 or to 5.5 Patch 1 prior to running the script.

Note: VS1 customers cannot upgrade to 5.5, since only VS2 hardware is capable of running 5.5. VS1 customers must upgrade to 5.4 SP1 P3, and VS2 customers can go to either 5.4 SP1 P3, or 5.5 Patch 1.

The script, “VPLEX-MS-patch-update-change-root_password-2015-11-21-install” automates the workaround procedure and can be found at EMC’s EMC Online Support.

Instructions to run the script: 

Log in to the VPLEX management-server using the service account credentials and perform the following from the management-server shell prompt:

  1. Pull down a copy of the “VPLEX-MS-patch-update-change-root_password-2015-11-21-install” script from the specified location above and then, using SecureCopy (scp), copy the script into the “/tmp/VPlexInstallPackages/” directory on the VPLEX management server.
  2. The permissions need to be changed to allow execution of the script using the command chmod +x.

service@ManagementServer:~> chmod +x /tmp/VPlexInstallPackages/VPlex-MS-patch-update-root_password-2015-11-21-install

  1. Run the script as shown below.

Sample Output:

This script will perform following operation:
– Search and insert the IPMI related commands in /etc/sudoers.d/vplex-mgmt.
– Prompt for the mgmt-server root password change.
Run the script with “–force” option to execute it

service@ManagementServer:~> sudo /tmp/VPlexInstallPackages/VPlex-MS-patch-update-root_password-2015-11-21-install –force

Running the script…

– Updating sudoers
– Change root password
Choose password of appropriate complexity.

Enter New Password:
Reenter New Password:

Testing password strength…

Changing password for root.

Patch Applied

NOTE: In the event that the password is not updated, run the script again with proper password complexity.

  1. Following running of the script, from the management server, verify that password change is successful.

Sample output:

service@ManagementServer:~> sudo -k whoami
root’s password:
root

***Contact EMC Customer Service with the new root password to verify that EMC can continue to support your VPLEX installation. Failure to update EMC Customer Service with the new password may prevent EMC from providing timely support in the event of an outage.

Notice how convoluted this is. Also notice how you need to have at least 5.4.1 Patch 3 in order to even run it.

While EMC KB articles have an attachment section, this script in question is of course not added.

Instead, you have to go look for it yourself, helpfully, they link you to: https://support.emc.com/products/29264_VPLEX-VS2/Tools/

And its right there, for now at least.

What I find interesting here is that it appears both the article, and the script, have been last edited.. .today?
Coincidental. But also a little scary. Does this mean that prior to 5.4.1 Patch 3 there really was no supported way to change the default vplex management server root password? The one that every EMC and VPLEX support engineer knows and is easily Googlable? Really? 

I think the most troubling part of all this is that final phrase:

Failure to update EMC Customer Service with the new password may prevent EMC from providing timely support in the event of an outage.

Have you ever tried changing vendor default backdoor passwords, and see if their support teams can deal with it?  Newsflash: they can not. We tried this once with EMC Clariion support. Changed the default passwords. We dutifully informed EMC support that we changed them. They assured it this was noted down in their administration for our customer.

You can of course guess what happened. Every single time EMC support would try to get in, and complain that they could not. You had to tell them every single time about the new passwords you had set up.  I am sure that somewhere in the EMC administrative system, there is a notes field that could contain our non-default passwords. But no EMC engineer I have ever spoken to would even look there, or even know to look there.

If you build an entire hardware-support infrastructure around the assumption of built-in default password that everyone-and-their-mother knows, you make it fundamentally harder to properly support users who ‘do the right thing’ and change them. And you build in vulnerability by default.

Instead, design you hardware and appliances to generate new and unique strong default passwords on first deployment, or have the user provide them (enforcing complexity). (many VMware appliances now do this). But do NOT bake in backdoor default passwords that users and Google will find out about eventually.

The Gillmor Gang Podcast – temp RSS feed and other stuff

Sunday, May 31st, 2009

For those of you following the Gillmore Gang podcast, you may have noticed the feed has not been updated recently.

All queries directed at Steve Gilmore about this, have resulted in him simply pointing us to the live stream of the Gang, as its recorded on Leo Laporte’s site on Friday or Saterday. He gives very little advance warning of when they are going to do a show. And its just down to luck, and your own personal schedule wether you are in the position to listen to this at the time.

After about a month of this, several people, like myself and Dave Winer, have become fed up, and this Friendfeed discussion has resulted in various alternative resources being made available, including a certain “Jack” very helpfully providing us with an alternative RSS feed, coming out of Yahoo pipes.

Here is my own little rant, followed by the complete conversation:

“I cannot understand why they would stop the podcast. I dont care what else they do, as long as I can recieve it as a podcast. The ENTIRE point of podcasting was its time-shifted nature, the fact that we can unshackle ourselves from the -obsolete- broadcasting model. Realtime has its place, but one does not preclude the other. Going live-only is a step back in my opinion. They cut -so- many people out of their listenership, I cant wrap my head around it!”

http://friendfeed.com/davew/4f42347b/torrent-of-this-week-gillmor-gang-just

 

To summorise, here are the various resources currently available:

Internet Archive:
http://www.archive.org/details/GillmorGang/

Yahoo Pipes conversion from the Internet Archive, into an RSS feed:
http://pipes.yahoo.com/pipes/pipe.info?_id=uFOYFoZN3hGItIPx0j6skA

Feedburner version of above RSS feed:
http://feeds2.feedburner.com/TheGillmorGangShow

ISOHunt Search (rss) for any Torrent with the words “Gillmor Gang” in it:
http://isohunt.com/js/rss/%22Gillmor+Gang%22?iht=

Update 20 July 2009:
After the little Spat with Leo Laporte, Steve has not done another Gillmor Gang.
It looks like we shall have to go without for the forseeable future 🙁

Phasor Burn: The Official Unofficial System Administrator Oath

Friday, April 10th, 2009

This is basicly a report of something the Phasor Burn blog linked to.

Very nice blog btw, and he linked to my related rant on there, thank you for that 🙂 The original is from an alt.sysadmin.recovery post from ’99 by Mike Sphar, and it deserves reprinting:

I am hired because I know what I am doing, not because I will do whatever I am told is a good idea.

This might cost me bonuses, raises, promotions, and may even label me as “undesirable” by places I don’t want to work at anyway, but I don’t care.

I will not compromise my own principles and judgement without putting up a fight.

Of course, I won’t always win, and I will sometimes be forced to do things I don’t agree with. My objections will be made known.

If I am shown to be right and problems later develop, I will shout “I told you so!” repeatedly, laugh hysterically, and do a small dance or jig as appropriate to my heritage.

I am not sure its in our own best interest to actually show we feel rather self-satisfied at times when things do go wrong, but we all feel it of course. Sadly, within the complexities of corporate politics, its sometimes quite hard to show clearly how your the disregard of your advise is directly to blame for the issue at hand. But when our moment comes, savor it, but savor it privately 😉

When you touch that server you touch me

Friday, July 25th, 2008

They turned off the HPSIM and general management and alerting server this morning, or at least, unplugged it, cause it was causing this huge network spike at a remote site

I know for a fact that no one besides myself knows what it is exactly that machine does, as its only usefull to me and what I do.

That doesnt mean it isnt explained in the server list in Sharepoint that I made and painstakingly try to keep up to date, that no one bothers to ever look at.

And of course no one bothered to ask during the day what exactly the impact is that they unplugged the server.

I mean, who cares about hardware and remote monitoring of servers anyway. It is, after all, only the most basic part of my job.

That made me feel really appreciated.

HPSIM was reinstalled a few weeks go by one of my collegues. When I explained it took me 2 days to set it up last time I installed it, he was suprised.

I will admit, it doesnt need to take that long. But it was new software to me at the time, and I was carefull, and ran into some awkward service account issues.

Its a very messy collection of software, basicly, so you need to be carefull and precise.

I read the manuals first.

I ended up needing 3 different service accounts. With different levels of rights and access.

He reinstalled HPSIM in about 1 hour. Its his way, he loves to impress with how fast he can do things.

I havent logged on to it in the meantime, because my time was needed elsewhere for the last few weeks. Build activities that go first. Project. Bids. Money.

I warned them in a long email 2 weeks ago, that no one was now doing any active systems administration. No one was keeping an eye on things. No one was cuting the grass.

Fast forward to this morning…

So, I cant dispute that HPSIM or something on that server killed that sites 2mbit WAN line for an hour, daily, between 10 and 11.

I went in over the ILO to have a look, after I asked them to at least plug -that- back in.

HPSIM service wouldnt start, as it couldnt authenticate its domain service account, cause it had no network. This was expected.

What wasn’t expected, was the fact that it was using this collegues domain admin account to start.

And so was the OpenSSH service.

And so was the Sofware update repository service.

I curse myself for not having reinstalled it myself, for one. And I curse myself for not having managed that server myself the past few weeks.

They ask me now, wtf was that server doing? I honestly dont know. I havent managed it for the past few weeks, due to me being allocated to build activities, as they well know.

I hate it. I hate the fact that I dont know.

Even though I have no need to feel responsible, I so very much do. This server was mine, it did this on my watch, at least that is how it feels.

I cant be sure what caused the network spike, and I will never know because they wont let me plug the server back into the network.

This weekend I will reinstall HPSim on a different server. A server that I had racked as spare, for this exact kind of scenario.

It will be reintalled slowly, carefully, with the appropriate documentation at hand, as I did last time.

It will be stable. It will be secure. It will be managed.

It will be beautifull.

And I am not gonna let anyone else on that server. If it ever misbehaves again, they can hold me personally acountable, I want them to, god knows I want them to.

There is only one person in my department with a sense of responsibility for our enviroment.

There is only one person in my department who actually cares things are done correctly.

Every time I place my trust in another technical person, I am dissapointed.

No one else is touching that server from now on.

Happy Sysadmin day.