Posts Tagged ‘vmware’

NLVMUG2018 – Speaking on NSX microsegmentation and a community panel discussion #vexpert

Thursday, March 15th, 2018

Its always exiting to speak publicly and this year I am setting my bar higher by participating in 2 sessions.

First up is a panel discussion that I was very happy to be invited to by Francisco Perez van der Oord, one of the directors of ITQ. We will have a 45 min flow of topics around SDDC, NSX, Cloud, etc, and the general trends of technology as they impact vSphere admins. We titled the session “vSphere – .. and then what next?”  Never participated in a panel discussion on stage before, so that will be an interesting experience. The other participants are, imo, giants in the dutch VMware community: Joep Piscaer of OGD/Jumbo and Viktor van den Berg of PQR, and I feel quite humbled being on stage with them.


My second session is my own talk, 20 minutes, on NSX Microsegmentation in practice. This is a condensed version of the talk I gave at the Infosecurity conference last year.
In it I cover some practical tips about using NSX Microsegmentation, do’s and dont’s, and common Gotcha’s
Its actually quite tough to get all the essentials into 20 mins or so, so it will be dense and fast-paced (as usual for me).

Nervous, but really looking forward to the day. I love the VMUG concept and I love networking and seeing all the community in the flesh again (as apposed to only on Slack/Twitter)

LinkedIn embeds:


VMworld 2017 EU Day 1 (part 2)- #HACKATHON (awesome!!) #vexpert #vmwarecode

Tuesday, September 12th, 2017

So there where two things I knew for a few years.
– Everyone always seems to have an awesome time at Hackathons
– I am not a developer, not even a descent scripter, how could I contribute to something like this?

Well VMware{Code}, who organise the VMworld hackathon, don’t care all that much what you can do going in.
The point is to learn and to have fun

A bit difference for me this year was being in the vExpert community. And as you might imagine, there is quite a bit of overlap between vExperts, VMware{code} community and innovative scripters. Well they where all very encouraging. So I decided to just throw myself in there!

Quite a diverse lineup of teams. I have wanted to get into Ansible for a while, so I immediately saw my oppertunity in Team4


I ended up taking responsibility for the presentation of our teams results, so I made the powerpoint which summarizes what we tried to achieve:



Being both a Linux and Ansible noob, I spent most of my evening trying to get the VIC OVA copied onto my Ubuntu VM to test our deployment parameters.

Also, it took half an hour to deploy VIC each time we tried… this we called.. a constraint :p



However, I have to say that a lot of effort was put into providing us a cool deployment environment. They gave us the option to use the on-prem hardware they had set up in the Hackerspace where we at, OR.. to use VMware on AWS, which was of course very cool.
For practical reasons we ended up going on prem – mostly so it was easier to SSH in.



I partnered with @kev_johnson (of the @OpenTechCast podcast) to create the OVF deployment part. Kev ended up doing 99% of the work. I contributed mostly by googling some variables and pointing out irrelevant things and distracting him. :p 

While we didn’t really manage to test much of our solution, we did put all the Ansible roles and playbook in GIT. This was our main goal, to be able to contribute something to the community. And we succeeded in this. Its not finished, but its a great start!





The main goal was to learn, and that we all certainly did do. I now have a far better understanding of where all the moving parts go for Ansible, and am happy to see its actually not all that complicated. I also learnt how to use GIT which will be extremely useful going forward.



The main thing I could meaningfully contribute was my some powerpoint and the accompanying presentation of the teams results. Within 90 seconds!

Unfortunately, our ultimate gambit of bribing the judges with Belgium and Dutch chocolate products, did not succeed :p


I want to give a MASSIVE thank you to our teamleader and inspirator:  who really helped us get to grips with all this new stuff, and did a significant bit of preperation on his blog.

VMworld Europe Hackathon: Introducing team Automation for All

VMworld Europe Hackathon: Preparation

I want to thank our team, who really pulled together and took their tasks seriously:

Kev Johnson (@kev_johnson – beer connisseur, hoping to learn loads about Ansible as I know *literally* nothing… Not sure what I can bring to the party other than enthusiasm!)
Ozan Orcunus (@vOrcunus) – system architect with a high interest in devops mindset and infra as code concepts, random powercli scripter and virtualization guy.
Chris Lewis (@thecloudxpert) – vExpert, VCIX6, all things SDDC and vRealise Suite – Ansible n00b – merc that may switch teams before the day 😄
Orhan Biyiklioglu (@biyiklioglu) – ex-sysadm new cloud engineer.
Laurent Borgognon (@lbggn / @BruksL) – beer expert – Ansible n00b but want to learn – random Scripter
Nick Goldman (@nickgold) Interested in all Infrastructure automation. Looking to learn about ansible


And finally I want to thank the VMware{code} team and the judges for putting this all together and making this such a fun event!
Jake Robinson, Nikki Roda, Tim Bonneman, William Lam, Alan Renouf, Steve Trefethen and Ricky Trilago and everyone else involved. Great event! Hopefully see you next time!


Speaking at the NLVMUG for the first time #vexpert

Friday, March 17th, 2017

Here is a summery of my experience of speaking at the NLVMUG for the first time.

For someone who always take pride in knowing just that little bit more than the next guy, it is not surprising that a longstanding desire of mine, was to speak at a public event to some kind of unique knowledge. Public conferences, even vendor-specific conferences like VMWare’s VMUG’s and of course VMworld, are very interesting to me precisely because of this. It tends to attract and concentrate some of the most knowledgeable people, and some of the most cutting edge technological knowledge and experiences.

Last year I was invited by @gekort , a great public speaker in his own right, to present a session at the VMware summerschool in Utrecht, at VMwares Dutch main office. Having never previously spoken publicly like that, this was a pretty big deal for me. The sheer fear of being publicly scrutinized on my knowledge of a subject sends me into fits of anxiety 😉
But it was a great experience, and personally for me a great success. It boosted my confidence in my speaking and presentation abilities quite a bit. The feedback that I got was valuable and I took as much of the experience and advice on board as I could. In any case, I knew I wanted to do more of this!  But the main advantage I had was that I was speaking to a set of subjects I was quite comfortable and knowledgeable about, in that case Metro-Cluster and HA.

When it was time to submit a paper to the NLVMUG, the largest VMware user conference in the world, besides VMWorld, it was obvious to myself and Alexander, our co-founder, that we should speak about our NSX experiences over the last 3 years. It is currently our biggest asset as an infrastructure partner, as we are currently in a rather unique position with it, and to be blunt, we really cannot advertise it enough. I am not in essence a ‘network guy’, so I was a bit nervous about the material. I made doubly sure I had fact-checked every single thing I wanted to talk about. I probably spent over 40 hours doing just that.

Simultaneously, my colleague Robin van Altena also submitted a talk about vRealize Network Insight.

We submitted the NSX talk and the vRNI talk as a ‘lightning session’, which is only about 20 minutes. (My talk at the Summerschool was an hour). There where many, many of those slots available at the NLVUG. In retrospect, I think we could have equally well have pitched a full break-out session of 50 minutes, with the material we had.
As it turned out, there was already a full break-out session just before mine by one of the NLVUG leaders, Joep Piscaer , on OGD’s experience with NSX over the last 3 years. the NLVMUG leaders reached out to all new speakers to help coach them a bit, and me and Robin gracefully accepted.
This was quite a valuable Skype session, and the key point that was inparted on us, was the non-commercial nature of the talks. We where there to talk about our own, personal experiences. While we could acknowledge our companies, it would be bad form to explicitly pitch our company or product. This is relatively easy for me, as having to engage in ‘sales talk’ causes a fair bit of cognitive dissonance in my brain, even though I can do it quite well when needed :p

Practicing your talk is essential, as is getting feedback early.  We occasionally have ‘knowledge sessions’ at Redlogic, where people do little presentations of whatever it is they want to share. This was a perfect opportunity to get early feedback on our sessions.

My talk was pretty dense with NSX information. It took me a few personal practice runs, timing myself on the different parts, to get it all under 20 minutes. And you want a minute or two for questions.

The day itself was awesome. I was quite nervous of course. My talk was at 11:00, and that is a great time slot. Anything after lunch, and you risk the change that people are either falling asleep, or have left.  Joep Piscaer’s talk about NSX at OGD was just prior to mine. I knew I would want to refer to his talk in mine, so I made sure to attend it.

His talk was indeed very interesting. There was a lot of overlap with mine, but our talks where also highly complementary for each other, each touching on unique aspects and experiences. He called me and my talk out specifically as a follow up, which was very gracious, and his final slide even referenced me. As I was going to briefly discuss NSX-T, he mentioned that specifically. This made me somewhat nervous as I was only going to spend maybe half a minute on that. I made it a point to give that subject a little more time at the end of my talk, which I did.

If you want to learn more about OGD’s hosted IAAS platform with NSX, check out  and (both in Dutch)

The ‘Dexter’ rooms reserved for the lightning talks are all quite small, only fitting about 40-45 people. As there where a record amount of sessions at the NLVMUG this year, the logistics of the venue had a bit of trouble keeping up. Also, all talks where about 10 minutes behind schedule, so I ended up in line for my own talk 🙂
It is both incredibly encouraging and nerve-racking to see the room filled to capacity, and then another 15 or so people trying to get in. It was standing-room only at the back, and the same was true for Robin’s vRNI talk.
Getting started is always the hardest part, but once I was into the swing of it, I forgot about time and nerves and just went all-in on the knowledge. I didn’t even watch the timer counting down.  My talk was pretty dense and I feel I have a pretty intense style of speaking. I try to scan the room and look people in the eye. I hope that keeps peoples attention. One thing I regret is not having some humorous moments in my slide deck. I need to take a page from Joep and include some memes next time :p
I tend to move around a lot, but the size of the room did not allow for much lateral pacing. Probably a good thing. You don’t want to remain hidden behind the lectern, but you don’t want to obscure the beamer either. I will take this into account with my slides next time; leave some space for my ‘shadow’ if needed.  I was very happy the venue had provided fresh water behind the lectern. But a bottle would have been more practical than the glasses we had. I will take a bottle with me next time. Your mouth will dry out :p

To my surprise, I seemed to stay inside the time perfectly, but I was not entirely sure. I was expecting (and dreading) questions, but I only got 1, which was customer-related and kinda drew a black for me in the moment. (why did our customer choose NSX). It was not the kind of question I had been expecting, and regretfully I had to admin on the spot that I did not know. I actually did remember later, but my mind was focused on product facts, not customer politics.

I asked the room for more questions.. silence. “Ideas?” ..  “did you like it?!” .. and the whole room made enthusiastic and acknowledging noises. That was the best moment of the day 🙂
I heard later, via others, that it had indeed been very well received by people. It also reminded me that is really not enough NSX experience out there right now, and many people are curious.

Also Robin’s talk about vRNI, just after and right next door time mine, was very well attended, with lots of interest. Again a packed and overcrowded room.  He managed to cram in slides and material and exposition, and 4 demo-movies, and stayed right inside 20 minutes. Very impressive!  And demo’s of a product are always very popular, even if they are recorded. It should be noted he recorded these himself, in our own lab. They where not VMware-provided.

The rest of the day was much like any other conference day.. attending sessions, checking hands, live-tweeting, getting plied by vendors, hunting for food and snacks, and networking. I had been invited to a vExpert lunch with Frank Denneman, but I totally forgot about it.  We did have a nice buffet afterwards with the other speakers, and I had some great convos there with folks from ITQ. The day was exhausting but a huge amount of fun, best NLVMUG I have been to, and higher on my list than even VMworld so far.  I will certainly want to speak next year again, and perhaps at other places and events, my mind is already churning with what my next talk will be about!

I will be writing some upcoming blog posts about our NSX experiences, based on my presentation.

VMware Cloud on AWS – More details emerging

Wednesday, March 15th, 2017

—Update 21st March, vBrownbag episode on AWS that these slides are from, is now posted on youtube:

— Update 15th March ~5pm CET, added some extra info and clarified some points—

More details regarding VMware Cloud on AWS are starting to come out of VMware. Tonight I attended an awesome  #vBrownbag webinar on #VMWonAWS,   hosted by Chris Williams‏ (@mistwire) and  Ariel Sanchez‏ ( @arielsanchezmor).

Presenting where Adam Osterholt (@osterholta) Eric Hardcastle (@CloudGuyVMware) and Paul Gifford @cloudcanuck

Here are some of the slides and highlights that stood out for me. Information is not NDA and permission was given to repost slides.


VMware Cross-Cloud Architecture. A nice slide that summorises the VMware strategy going forward. Expect VMware cloud to pop up in more places, like IBM Cloud. More info about VMware cloud strategy here

Important to note here, is that this is a complete service offering, meaning its fully licensed. You do not need to bring your own licenses to the table. So you get the full benefit of technologies like vSAN and NSX as part of the offering.

Skillsets.. this is a huge selling point. Many native cloud deployments require your admins to know AWS or cloud-native specific tools and automation scripting languages. VMware Cloud on AWS (VMWonAWS) removes that barrier-to-entry completely. If you can administer a VMware-based cloud stack today , you can administer VMware Cloud on AWS.

You have access to AWS sites around the world to host VMWonAWS. What is to note however is that, because these are vSphere clusters on bare-metal, where you instantiate your VMware environment is where you are bound in certain ways.

Initial roleout will be Oregon. The followed by an EMEA location. Sometime around mid-2017.  (from announcement to GA in about a year.. not bad!!)

With the recent S3 outage in mind, asked specifically about things like stretched-cluster and other advanced high-availability features inside AWS, and these will not be initially part of the offering. However you can always move your VMs off and on VMWonAWS via x-vmotion. More or that later.

VMWonAWS will use customized HTML interfaces throughout. No flash here! 🙂

But if you are a bit of a masochist and you like the flash/flex client, it will be available to you anyway.

The frontend provisioning component will include its own API interface. What you see below is a mockup and subject to change.

Administering your cluster uses a custom and locked-down version of the already available HTML5 client.

Its important to note here, that VMware will administer, and upgrade their software inside these environments themselves. They will keep an n-1 backward compatibility, but if you have a lot of integration talking against this environment, operationally you will have to keep up with updating your stuff. Think of vRA/vRO workflows and other automation you might have talking to your VMWonAWS instances. This may be a challenge for customers.

Demonstrated below is a typical feature unique to VMWonAWS, the ability to resize your entire cluster on the fly.

Again, above screenshots are mockups/work-in-progress

Your VMware environment is neatly wrapped up in an NSX Edge gateway, which you cannot touch. However, inside your environment, you are able to provision your own NSX networks, manage DFW, edges, etc, and with that all the functionality they offer you. However initially NSX API access will be limited or not available, so it may be hard to automate NSX actions out of the gate.

The Virtual Private Cloud (VPC) you get is divided into 2 pools of resources. Management functions are separated from compute.

Remember that all of this is bare-metal, managed and patched by VMware directly.

VMware manages the VPC with their stuff in it. Your get access to it via your own VPC, and the two are then linked together.

They give you a snazzy web frontend interface with its own API  do the basic connectivity config and provisioning.

So how do you connect up your new VMWonAWS instance with your on-premises infrastructure?

End-to-end, you are bridging via Edges.. but there is obviously a little more involved. Here are the high-level steps that the customer and VMware/Amazon take to hook it all up.


The thing to remember here is that your traffic to the VMware VPC is routed through your customer VPC. Its ‘fronts’ the VMware VPC.

Link the vCenters together, and now you can use x-vmotion to move VMs back and forth. And remember, no NSX license is required on-prem to do this.

If you already have NSX, you can of stretch your NSX networks across. this allows live x-vmotions (cross-vcenter vmotion).

If you do not have NSX on-premise, you will deploy a locked-down NSX edge for bridging, but vmotions would be ‘cold’.

Encryption will be available between the Edge endpoints. No details on this yet.

As standard NSX edges are being used on both ends, you can do things like NAT, so you can do overlapping IP spaces if you so choose. That is not something native AWS VPC’s allow you to do.

Because your always have your own native AWS VPC, you can leverage any other native AWS service.

But you can do some crazy-cool things too, that will be familiar to native AWS users.  You can, for example, leverage regional native AWS services, for example S3, inside VMWonAWS VMs. These resources are connected inside AWS, using their own internal routing. So this kind of traffic does not neet to go back out over the internet.

VMs inside VMWonAWS can make use of the Amazon breakout for their internet connectivity. Or you can backflow it through your own on-premises internet.

Some additional notes on APIs:

There is no backup function built into this, so you are expected to backup your own VMs hosted inside VMWonAWS. Do facilitate this, the VADP API for backups  is available to leverage, as per normal.

Some notes on vSAN:

vSAN is used as underlying storage. All Flash. VMware does not yet know what the default setup of this will be in terms of FTT (failure To Tolerate_ level or dedupe. But you will have control over most of it, to decide for yourself what you want.


Using vmkramdisk to fix rare HA-FDM Agent tardisks vmware_f.v00 uninstall issue

Tuesday, September 6th, 2016

On ESX 5.5U3, I recently ran into an annoying issue with HA. vSphere had recently been updated, but the hosts had not been all yet received the very latest version of the FDM (fault domain manager, aka ‘HA’) agent.
During some routine maintenance work, a particular host was taken in and out of maintenance mode a few times. Eventually it was observed to no longer properly complete HA configuration. Checking the host status in the UI, it would seemingly get stuck in the install phase of the latest FDM agent.

Checking the FDM installer log ( /var/run/log/fdm-installer.log ) , I found the following:

fdm-installer: [40283] 2016-08-25 11:16:13: Logging to /var/run/log/fdm-installer.log
fdm-installer: [40283] 2016-08-25 11:16:13: extracting vpx-upgrade-installer/VMware-fdm-eesx-2-linux-4180647.tar
[40283] 2016-08-25 11:16:13: exec rm -f /tmp/vmware-root/ha-agentmgr/upgrade
[40283] 2016-08-25 11:16:13: status = 0
[40283] 2016-08-25 11:16:13: exec cd /tmp/vmware-root/ha-agentmgr/vpx-upgrade-installer
[40283] 2016-08-25 11:16:13: status = 0
fdm-installer: [40283] 2016-08-25 11:16:13: Installing the VIB
fdm-installer: [40283] 2016-08-25 11:16:18: Result of esxcli software vib install -v=/tmp/vmware-root/ha-agentm
fdm-installer: Error in running rm /tardisks/vmware_f.v00:
fdm-installer: Return code: 1
fdm-installer: Output: rm: can’t remove ‘/tardisks/vmware_f.v00’: No such file or directory
fdm-installer: It is not safe to continue. Please reboot the host immediately to discard the unfinished update.
fdm-installer: Please refer to the log file for more details.
fdm-installer: [40283] 2016-08-25 11:16:18: There is a problem in installing fdm vib. Remove the vib…
[40283] 2016-08-25 11:16:18: exec esxcli software vib remove -n=vmware-fdm.vib
No VIB matching VIB search specification ‘vmware-fdm.vib’.
Please refer to the log file for more details.
[40283] 2016-08-25 11:16:19: status = 1
fdm-installer: [40283] 2016-08-25 11:16:19: Unable to install HA bundle because esxcli install return 1


This was decidedly odd. I checked the /tardisks mount, and could, indeed, not found any vmware_f.v00 file. It was trying to ‘remove’ (unmount, as it turns out) a file that did not exist. And this was breaking the uninstall process.

This page was useful in understanding the sequence of events:

What I can only speculate as to what happened, is that at some point in the sequence of taking the host in and out of maintenance, the FDM uninstall somehow failed to complete properly, and left the host image list in a strange, invalid state.

Querying the host in this state, it listed the old FDM agent as still installed:

# esxcli software vib list | grep -i fdm
vmware-fdm                     5.5.0-3252642                       VMware  VMwareCertified   2016-02-03

Yet a force uninstall of the VIB would fail with the same error.

fdm-uninstaller: [] 2016-08-24 11:42:30: exec /sbin/esxcli software vib remove -n=vmware-fdm
Removal Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed:
VIBs Removed: VMware_bootbank_vmware-fdm_5.5.0-3252642
VIBs Skipped:
fdm-uninstaller: [] 2016-08-24 11:43:58: status = 1
fdm-uninstaller: [] 2016-08-24 11:43:58: exec /sbin/chkconfig –del vmware-fdm

Together with VMware support, we tried various tricks, including copying a fresh imgdb.tgz from a different host to /bootbank , and running the latest installer and uninstaller of the FDM agent manually.
By the way, the source that vCenter uses for the FDM agent installer and uninstaller, is (on Windows) “Program Files\VMware\Infrastructure\VirtualCenter Server\upgrade”

If you wish to try to run these files directly on an ESX host, simply copy them to the host /tmp and chmod them to 777. They are then executable.

But in all cases, the FDM installer first will always try an uninstall of the previous verison, which always includes trying to unmount /tardisks/vmware_f.v00

Now /tardisks is a bit of a strange bird, and deserves some explanation. This VMware research paper turned out to be a very excellent document in understanding what /tardisks actually is and does:

In short, it is a directory that hosts mounted TAR files, that are loading at boot time from /bootbank (or /altbootbank). These TAR files are mounted as live filesystems, using what VMware calls VisorFS. Which makes the mounted TAR files behave as part of the file system. This has various administrative and management advantages as the paper linked above explains.

It is therefore not possible to simply copy a missing file to /tardisks in order to force the FDM uninstaller to properly complete.

You can list which TAR filesystems ESX has mounted, by running the command  esxcli system visorfs tardisk list


This list will be the same as the filelist of /tardisks

Of note: when you re-install FDM, just after the install, the ‘system’ flag will be set to false, until you reboot. After a reboot, it will be set to true like all other modules.

On a normal host, you will find the FDM VIB listed here.

In our case, this entry was missing, even though the Vib list command showed it as installed.

So it seemed to me that if ESX needed to mount these TAR files at boot time, there was probably a command it used to do this.
Or in any case, I found it likely such a command should exist, if only for troubleshooting purposes.
I wondered that if I could mount this TAR manually, the uninstaller might proceed normally.
A few minutes of google-fu later, I stumbled on this:
Creating and Mounting VIB files in ESXi

Now the VMware engineer noted that the vmkramdisk command has been deprecated since 4.1, but to both our surprise (and delight) it was still there in 5.5, and still did its job.

We manually mounted the /bootbank/vmware-f.v00 using the command vmkramdisk /bootbank/vmware-f.v00

Immediately you will find vmware-f.v00 listed under /tardisks, and using esxcli system visorfs tardisk list

And as predicted, the installed passed through the uninstall this time, without a hitch, and then installed the new version of the HA agent. We rebooted the host just to be sure it would properly load the new VIB each time. And it did, and managed to initiate HA in the cluster without any issues thereafter.