Last week myself and Erwin had the opportunity to attend the VMware Technical Support Summit in Cork, Ireland.
This is a 2 day event hosted by GSS which is technically oriented. Many interesting sessions by some of the best GSS technical talent, and various breakout sessions to get near one-on-one time with engineers and product leaders.

 

As you can see by the schedule below, it was a wide array of product fields and the technical depth varied between good and amazing.

I was especially blown away by Valentin Bondzio’s talk about CPU accounting in the hypervisor. Technically extremely interesting!
He dove extremely deeply into CPU metrics and what ‘idle’ and ‘use’ really mean from an architecture point of view, and how hyperthreading changes the game.
This was especially gratifying as I have worked with him on a case for over a year that tackles exactly that aspect of hypervisor performance.

The team that support Airwatch gave a very interesting talk that was relevant to a project I am working on.  So afterwards I grabbed all 4 of them into a conference room at the hotel, to discuss our VDI and mobile management design ideas. Extremely valuable opportunity as I bet I will be talking to these guys more in the future!

They also very graciously dropped me and Erwin of at our Hotel afterwards, and then drove us to the city center, where VMware hosted drinks and dinner with live music, which was quite entertaining.

Another talk I was looking forward to was that of Cormac Hogan, and Mark Fitzgerald, senior director of support Cork, presented him with some gifts for just plan being around a long time 😉

 

I had gently badgered various VMware contacts about seeing if we could get a visit of the actual VMware campus, and eventually Mark Fitzgerald himself very graciously drove us over to the VMware office campus and gave us a a personal guided tour of all the GSS offices, which was a great great experience.

We got to meet every Cork GSS team and even visited the test lab including the folks who run that. It was great to put voices and names to faces and to get a real sense of the environment that these engineers work in. Sometimes support engineers are faceless and nameless, but VMware seems to breath a very human and supportive culture that was much in evidence throughout the campus.

 

 

I would much like to thank the entire VMware Cork team and all the GSS engineers for putting on a great summit. And a special thank you to Danka for showing up in the middle of her time-off, just to say hi. Much love to VMware’s best escalation manager!

 

 

Tags:

Thefluffyadmin on May 1st, 2017

The last 18 months I have started listening to podcasts again in a regular fashion. Way way back on this blog, I used to blog about them, and even participated in several,including the long-gone InTheTrenches sysadmin podcast.

The last year or so, my interests have also diversified into the cloud-native space, and my listening choice also reflects this.

Here is the list I currently enjoy, but I am adding more every few weeks.
When the VMware Communities Roundtable podcast is recorded, I am often there live on the chat.  Same goes the vBrownbag EMEA shows.

http://packetpushers.net/podcast/
Where Too Much Technology Would Be Barely Enough
A podcast network consisting of 6 shows mostly around networking.

http://www.thecloudcast.net/
Aaron Delp and Brian Gracely host this award-winning weekly podcast to discuss the technology and organizational changes that are driving the convergence of today’s Data Centers and Cloud Computing. Topics include Cloud Computing | AWS | Azure | GCP | Serverless | DevOps | AppDev | SaaS | IoT. Guests are industry thought-leaders and leaders of companies changing the IT industry.

https://changelog.com/
A weekly podcast that covers the technology and people of open source. It’s about the code, the people, and the community. This show features in-depth interviews with the best and brightest software engineers, hackers, leaders, and innovators. Hosts Adam Stacoviak and Jerod Santo face their imposter syndrome so you don’t have to.

https://twit.tv/shows/this-week-in-enterprise-tech
This Week in Enterprise Tech explores the complex, cutting edge world of enterprise technology. Hosted by Father Robert Ballecer, TWiET features IT professionals explaining the ins and outs of enterprise solutions.

http://www.vmwarevideos.com/vchat
Covering Virtualization and Cloud Computing News, How-To, Events, and Interviews
Our new vChat is a video webcast/podcast and a joint effort between three vExperts – Simon Seagrave of TechHead.co.uk, Eric Siebert of vSphere-Land.com, and David Davis of VMwareVideos.com. The vChat is sponsored by Train Signal – the leader in VMware video training.

The Virtually Speaking Podcast (official VMware podcast)
iTunes and Soundcloud.
The Virtually Speaking Podcast is a weekly technical podcast dedicated to discussing VMware topics related to storage and availability. Each week Pete Flecha (@vPedroArrow) and John Nicholson (@Lost_Signal) bring in various subject matter experts from VMware and within the industry to discuss their respective areas of expertise. If you’re new to the Virtually Speaking Podcast check out all episodes on vSpeakingPodcast.com. Be sure to subscribe to the podcast on iTunes and Soundcloud.

Vmware Communities Roundtable
http://www.talkshoe.com/talkshoe/web/talkCast.jsp?masterId=19367
VMware experts talk about virtualization live every Wednesday at noon CA time. Virtualization news and tech mixed with a “Late Night at VMworld” informality. Join us!

https://vbrownbag.com/
https://vbrownbag.com/vbrownbag-episodes/
Our topics range from certification, emerging tech, and business process. These series provide our listeners with information needed to help them not only pass exams, but learn how to implement new technologies to advance their careers.

https://blog.docker.com/tag/dockercast/

Dockercast is the official Docker Podcast which covers a wide range of topics including products, projects and contributions from active community members and partners with our host, Docker’s very own – John Willis. John Willis (@botchagalupe) is the Director of Ecosystem development for Docker

http://www.spreaker.com/show/explorevm-podcast
Join host Paul Woodward Jr (ExploreVM) as he explores technologies around data centers, cloud computing, automation, networking, and all things nerdy!

http://www.opentechcast.com/
A new technology podcast covering industry news, product reviews, education, certification and homelab advice and discussion

https://intechwetrustpodcast.com/
Five of your favourite community members like calling each other on a weekly base and record their rants on Datacenter News. Although we all are already quite opinionated people, we still love to hear your voice as well through our LinkedIN discussions or enlarge the rant on our Facebook Page.

 

Thefluffyadmin on April 30th, 2017

There are many ways to quickly provision docker hosts to cloud infrastructures. Dockers own native way is to use docker-machine. Being a vSphere admin I was of course interested in the VMware vSphere driver for docker-machine.

The starting resources for this are:

https://docs.docker.com/machine/drivers/vsphere/

https://blogs.vmware.com/cloudnative/how-to-use-docker-machine-in-conjunction-with-vsphere-driver/

While I got this working, one of the issues I struggled with was vCenter rights. The driver wants to do a bit more than just create a new VM. For instance it needs to upload an ISO to the datastore to role the VM, and it needs to be able to manage the VM in different ways. And remove it.

I created a custom vcenter role for this purpose, but as the exact permissions where not listed anywhere, it was process of trail and error.

The debug switch -d is very useful during a ‘docker-machine create’ to identify which steps its going through exactly.

Replicating the steps in the vSphere flex client also helps, as that UI will usually give you a good indication what rights you are missing, either via a warning text, or a grayed option.  (remember log out and back in every time)

The vpxd log on the vcenter might help you see what is being tried, but I find it only partially helpful.

One issue that really killed me for a while, is that the account used by the docker-machine driver, requires not just ‘browse datastore’ and ‘allocate space’ permissions, but also ‘low level file operations’

This is needed in order to upload (and delete) the boot2docker.iso

But it is not enough to apply these permissions only to the datastore, they must be applied to the entire vCenter scope, as per KB027743 , which frankly, is ridiculous. But that is an issue with vCenter rights model, not with the driver perse.

This takes are of vcenter rights overall. At least its better than applying full admin rights to the account you are using.

Down at resource level, in my case a specific cluster of 2 esx hosts, but it might also be a resource pool, I have applied full admin rights for the account.  This may be overkill, but it as far as I have gotten right now. I will update this post if I discover more nuance, or find some way to exactly inventory what rights its actually using the manage the VMs it spawns.

In this post, I discuss a newly found performance bug in Solaris 11, that has since Solaris 11 came out in 2011, severely hampered ESX VM disk i/o performance when using the LSI Logic SAS controller. I show how we identified the issue, what tools were used, and what the bug actually is.

In Short:

A bug in the disk controller driver ‘mpt_sas’ as used in Solaris 11, as used by the VMware virtual machine ‘LSI Logic SAS’ controller emulation, was causing disk I/O to only be handled up to 3 i/o at a time.

This causes severe disk i/o performance degradation on all versions of Solaris 11 up to the patched version. This was observed on Solaris 11 VMs on  vSphere 5.5u2, but has not been tested on any other vSphere version.

The issue was identified by myself and Valentin Bondzio of VMware GSS, together with our customer, and eventually Oracle. Tools used: iostat, esxtop, vscsiStats

The issue was patched in patch# 25485763 for Solaris 11.3.17.5.0, and in Solaris 12

Bug Report ( Bug 24764515 : Tagged command queuing disabled for SCSI-2 and SPC targets  ) : https://pastebin.com/DhAgVp7s

Link to Oracle Internal

KB Article: (Solaris 11 guest on VMware ESXI submit only one disk I/O at a time (Doc ID 2238101.1) ) : https://pastebin.com/hwhwiLRM

Link to Oracle Internal

————————

TLDR below:

Read the rest of this entry »

Tags: ,

Thefluffyadmin on March 17th, 2017

Here is a summery of my experience of speaking at the NLVMUG for the first time.

For someone who always take pride in knowing just that little bit more than the next guy, it is not surprising that a longstanding desire of mine, was to speak at a public event to some kind of unique knowledge. Public conferences, even vendor-specific conferences like VMWare’s VMUG’s and of course VMworld, are very interesting to me precisely because of this. It tends to attract and concentrate some of the most knowledgeable people, and some of the most cutting edge technological knowledge and experiences.

Last year I was invited by @gekort , a great public speaker in his own right, to present a session at the VMware summerschool in Utrecht, at VMwares Dutch main office. Having never previously spoken publicly like that, this was a pretty big deal for me. The sheer fear of being publicly scrutinized on my knowledge of a subject sends me into fits of anxiety 😉
But it was a great experience, and personally for me a great success. It boosted my confidence in my speaking and presentation abilities quite a bit. The feedback that I got was valuable and I took as much of the experience and advice on board as I could. In any case, I knew I wanted to do more of this!  But the main advantage I had was that I was speaking to a set of subjects I was quite comfortable and knowledgeable about, in that case Metro-Cluster and HA.

When it was time to submit a paper to the NLVMUG, the largest VMware user conference in the world, besides VMWorld, it was obvious to myself and Alexander, our co-founder, that we should speak about our NSX experiences over the last 3 years. It is currently our biggest asset as an infrastructure partner, as we are currently in a rather unique position with it, and to be blunt, we really cannot advertise it enough. I am not in essence a ‘network guy’, so I was a bit nervous about the material. I made doubly sure I had fact-checked every single thing I wanted to talk about. I probably spent over 40 hours doing just that.

Simultaneously, my colleague Robin van Altena also submitted a talk about vRealize Network Insight.

We submitted the NSX talk and the vRNI talk as a ‘lightning session’, which is only about 20 minutes. (My talk at the Summerschool was an hour). There where many, many of those slots available at the NLVUG. In retrospect, I think we could have equally well have pitched a full break-out session of 50 minutes, with the material we had.
As it turned out, there was already a full break-out session just before mine by one of the NLVUG leaders, Joep Piscaer , on OGD’s experience with NSX over the last 3 years. the NLVMUG leaders reached out to all new speakers to help coach them a bit, and me and Robin gracefully accepted.
This was quite a valuable Skype session, and the key point that was inparted on us, was the non-commercial nature of the talks. We where there to talk about our own, personal experiences. While we could acknowledge our companies, it would be bad form to explicitly pitch our company or product. This is relatively easy for me, as having to engage in ‘sales talk’ causes a fair bit of cognitive dissonance in my brain, even though I can do it quite well when needed :p

Practicing your talk is essential, as is getting feedback early.  We occasionally have ‘knowledge sessions’ at Redlogic, where people do little presentations of whatever it is they want to share. This was a perfect opportunity to get early feedback on our sessions.

My talk was pretty dense with NSX information. It took me a few personal practice runs, timing myself on the different parts, to get it all under 20 minutes. And you want a minute or two for questions.

The day itself was awesome. I was quite nervous of course. My talk was at 11:00, and that is a great time slot. Anything after lunch, and you risk the change that people are either falling asleep, or have left.  Joep Piscaer’s talk about NSX at OGD was just prior to mine. I knew I would want to refer to his talk in mine, so I made sure to attend it.

His talk was indeed very interesting. There was a lot of overlap with mine, but our talks where also highly complementary for each other, each touching on unique aspects and experiences. He called me and my talk out specifically as a follow up, which was very gracious, and his final slide even referenced me. As I was going to briefly discuss NSX-T, he mentioned that specifically. This made me somewhat nervous as I was only going to spend maybe half a minute on that. I made it a point to give that subject a little more time at the end of my talk, which I did.

If you want to learn more about OGD’s hosted IAAS platform with NSX, check out http://vmwareemeablog.com/nl/ogd-biedt-klanten-maximale-vrijheid-met-eigen-iaas-platform/  and https://ogd.nl/blog/post/2016/08/samen-slimmer-met-ogds-eigen-iaas-platform/ (both in Dutch)

The ‘Dexter’ rooms reserved for the lightning talks are all quite small, only fitting about 40-45 people. As there where a record amount of sessions at the NLVMUG this year, the logistics of the venue had a bit of trouble keeping up. Also, all talks where about 10 minutes behind schedule, so I ended up in line for my own talk 🙂
It is both incredibly encouraging and nerve-racking to see the room filled to capacity, and then another 15 or so people trying to get in. It was standing-room only at the back, and the same was true for Robin’s vRNI talk.
Getting started is always the hardest part, but once I was into the swing of it, I forgot about time and nerves and just went all-in on the knowledge. I didn’t even watch the timer counting down.  My talk was pretty dense and I feel I have a pretty intense style of speaking. I try to scan the room and look people in the eye. I hope that keeps peoples attention. One thing I regret is not having some humorous moments in my slide deck. I need to take a page from Joep and include some memes next time :p
I tend to move around a lot, but the size of the room did not allow for much lateral pacing. Probably a good thing. You don’t want to remain hidden behind the lectern, but you don’t want to obscure the beamer either. I will take this into account with my slides next time; leave some space for my ‘shadow’ if needed.  I was very happy the venue had provided fresh water behind the lectern. But a bottle would have been more practical than the glasses we had. I will take a bottle with me next time. Your mouth will dry out :p

To my surprise, I seemed to stay inside the time perfectly, but I was not entirely sure. I was expecting (and dreading) questions, but I only got 1, which was customer-related and kinda drew a black for me in the moment. (why did our customer choose NSX). It was not the kind of question I had been expecting, and regretfully I had to admin on the spot that I did not know. I actually did remember later, but my mind was focused on product facts, not customer politics.

I asked the room for more questions.. silence. “Ideas?” ..  “did you like it?!” .. and the whole room made enthusiastic and acknowledging noises. That was the best moment of the day 🙂
I heard later, via others, that it had indeed been very well received by people. It also reminded me that is really not enough NSX experience out there right now, and many people are curious.

Also Robin’s talk about vRNI, just after and right next door time mine, was very well attended, with lots of interest. Again a packed and overcrowded room.  He managed to cram in slides and material and exposition, and 4 demo-movies, and stayed right inside 20 minutes. Very impressive!  And demo’s of a product are always very popular, even if they are recorded. It should be noted he recorded these himself, in our own lab. They where not VMware-provided.

The rest of the day was much like any other conference day.. attending sessions, checking hands, live-tweeting, getting plied by vendors, hunting for food and snacks, and networking. I had been invited to a vExpert lunch with Frank Denneman, but I totally forgot about it.  We did have a nice buffet afterwards with the other speakers, and I had some great convos there with folks from ITQ. The day was exhausting but a huge amount of fun, best NLVMUG I have been to, and higher on my list than even VMworld so far.  I will certainly want to speak next year again, and perhaps at other places and events, my mind is already churning with what my next talk will be about!

I will be writing some upcoming blog posts about our NSX experiences, based on my presentation.

Tags: , , , , , , , ,