Archive for the ‘Rants’ Category

Desicionmaking on the new Proxy solution

Wednesday, May 28th, 2008

For your enjoyment, here is a, slightly edited, email I just sent to the department head and various other decision makers. It goes over some of the options we need to consider to solve the current issues with our internet access.

Names and places have been changed to protect the guilty 😉

And please exuse the spelling. I was in a hurry and I really dont care about spelling as much as I do content.


Hi all,

We are currently faced with some decisions that need to me made in regard to the Internet Proxy solution for the Netherlands and Belgium.

This is the current situation in regard to the proxy servers in Lala City and Chipville.

Server Lala City: LA-Server-S99
Server Chipville: CHIPVILLE-Server-S99

Both servers are HP DL360 G2 servers, and are now approaching 8 years of age. They are very out of warranty, and no hardware support can be expected from HP anymore regarding these.
Both servers run Windows 2000 standard
The Proxy software on both servers is ISA server 2000, running on the SQL MSDE engine. This software is still supported by Microsoft, but has been superceded by 2 newer versions.
In addition, we currently run the Surfcontrol web-filtering software, as a plug-in for ISA.
This software allows us to tightly control web-behaviour, for example to allow certain users access to certain sites, and to block entire catagories of websites, or web-protocols.
We have built up a pretty extensive rule-set over the years on both machines, and both rulesets are largely identical.
The company “Surfcontrol” was aquired by Websense in 2007, and since that time the Surfcontrol software is no longer supported, no patches or service packs are being offered for download, and no licences are being extended or sold, forcing all former Surfcontrol customers, including us, to look for alternatives.
The software combination on these servers has causes us some issues in the past. Some elements of Surfcontrol have always been buggy, and as the hardware has aged, it has become unreliable.
Furthermore, the decision to use SQL MSDE has causes problems, because of its inherrent 2gb limit.

Lala City
The Lala City proxy server is due to be replaced with new hardware, located in SiteB. This action is outstanding as part of the TCR move project.
As part of this, a new server was purchased, together with W ISA server 2006, and SQL 2005
At the moment, no replacement for Surfcontrol has yet been purchased, although Dick Dickerson did get a cost estimate for the Websense software, based on a single server, 500 users, and 3 years of licensing. (included as attachement)
A decision on this has been on the back burner, due to the fact that we where also planning on moving the current ISA server to SiteB anyway, and using the Chipville ISA server as a backup.

The old Proxy server in Chipville is in a similair state to the one in Lala City. Although one of its 2 disks (that run in a mirror) has failed since last week.
This causes a serious risk to internet service continuity. It also represents a risk to the TCR move project, as this server is now no longer a reliable fallback while we move the Lala City server.

We need to decide how to proceed going forward.

The time factor
We have only a limited time to come up with a solution. Currently the situation in Chipville is more pressing, because of the hardware failure of the server there.
The big-bang server move from TCR Lala City is sqeduled less than a month from now, and we need a stable and supportablesolution at the very least in Chipville before that time, and idealy a solution for Lala City aswell.

There are a number of options:

Option 1. Keep the current servers
The Lala City server can be moved to SiteB and continue to operate from there, serving Internet users (non-citrix) in the Netherlands.
However, the hardware and software is no longer supported, the software is in an unstable state due to past problems with Surfcontrol, and the ISA MSDE database.
Due to the advanced age of the hardware, it is only a matter of time before it fails. Moving it might actually break it too.
The Chipville server cannot operate as-is, on a failed hardware mirror. This absolutely needs to be replaced, more or less disqualifying this option.
Due to the above, I cannot recommend this option in any way.

Option 2. Outsource the Proxy service to European Datacenter / UK
This would involve redirecting all internet traffic from Netherlands and Belgium to an outside, centralised Proxy system for internet access.
This would simplify our support model somewhat, and remove the technical burden of supporting the solution ourselves.
The downside though, is that we no longer have direct control over what is allowed/disallowed over the Internet.
By default, as far as I have heard, no rules are in place for both the UK and European Datacenter proxy solutions, meaning that there are no limits on what people can do with the Internet connection, it would be a free-for-all, whereas right now, we have strict limits on usage.
This option should be considdered. But the question has to be asked why the web-filtering function was ever needed in the first place. If web-filtering and control remains a business requirement, that this options cannot be considdered.

Option 3. Hybrid In-country hosting / European Datacenter hosting
I have been made aware of a version of the European Datacenter hosting scenario, that includes re-directing in-country internet traffic to European Datacenter, but in combination with a local Proxy/web-filter server, running the Websense software. This would involve installing a local server with the “Websense” filtering software, and “chaining” it to the Websense Proxy server in European Datacenter. Many countries apparently already follow this model.
This has the advantage of retaining local control of a rulebase, allowing us to continue to restrict internet use where nessesary, but with the advantage of not needing local Internet line for basic Internet use anymore. MEGACORP(TM) also can retain an amount of corperate internet-use control, via the gateway in European Datacenter, as all internet fraffic eventually moves through there to get out. Currently MEGACORP(TM) does not pose any global restrictions on the Internet gateway in European Datacenter, as far as I have heard.
This option should be considdered, however it will take some time to study and set up properly. The support model may be complicated because of the fact you are dealing with possible web-filtering and proxying in 2 different locations, supported by 2 different organisations. It would however, also require that local websense software be purchased and supported. I have also been told by some, that the connection via European Datacenter is very slow and not that usefull for many operational tasks.  This could hurt us, as we run a number of line-of-business web-based applications over the internet. (Hp Shipview, etc)
We would also benefit from the fact that the websense software can be centerally managed from 1 console, making it very easy keep the netherlands and Belgium ruleset identical, and simplifying reporting and failover.
I would recommend this option if we can be sure the performance is adequate for our business needs, and if the support model can be agreed apon quickly. The major downside of this solution currently is that it will take time to set up, and we dont have much time anymore.

Option 4. New installation In-Country
This involves basicly rebuilding the 2 Proxy servers on new hardware, and installing fresh, current and supported Proxy and web-filtering software.
In this scenario we would use our own local Intranet lines in SiteB and Chipville.
We would directly support the solution, and maintain direct control over the web-filter ruleset, this is the most simple support scenario.
Hardware for this is already available: The replacement of the Lala City server was already part of the TCR move project, as is the licence for ISA 2006 and SQL 2005.
Hardware for Chipville is also already available on site, in the form of a 3-year-old IBM server, however, this server may soon fall out of hardware support (needs to be checked).
Apart from the ISA and SQL license that would be needed for the Chipville server, we need new web-filtering software for both servers, again, IF the business still deems this a requirement.
If they dont, then this solution would provide unfiltered internet access to all (non-citrix) internet users in Netherlands and Belgium.
For the web-filtering requirement, I would at this time advise to als go with the Websesne software, as they are currently regarded as the market leader, and their software is well supported ans well known in the industry. (they are incorperating a lot of the Surftcontrol concepts as part of the aquesition )
We need to look at the current available hardware for this. Almost all the hardware we have is 3 years old or older, so it may be advisable to considder purchasing a new piece of hardware for this solution in Chipville.
This option should be considdered. It has the advantage of retaining central control and will be quick to set up, once the software has been purchased. The downside is that the Websense software is expensive, so we may want to considder looking at alternatives, even though it has becomes a defacto standard within MEGACORP(TM). Again, we have a time-constraint problem here.
We would also benefit from the fact that the websense software can be centerally managed from 1 console, making it very easy keep the netherlands and Belgium ruleset identical, and simplifying reporting and failover.
I would recommend this option first and foremost, and it is the prefered solution technically, considdering the circumstances.

Again, i wish to stress the timeconstraints we have, less than a month before big-bang, we want a new solution up and running within the next 3 weeks!

Messing with IBM xSeries 336 1U and IBM in general, my experiences so far…

Tuesday, July 26th, 2005

IBM xSeries 336
Pictures @ eWeek

So in my current job I get to play with some cool toys, like the IBM xSeries 336 model 8837, and the EXP 400 Disk cabinet. I must say its the first time I have had some real hands-on experience with server hardware, and then to get to work with stuff thats pretty new is quite cool.

Add to that, the fact that its my job to get to know this stuff very well, something I am not accustomed to, is very cool. My past jobs where all pretty pretty boring compared to this, so very rarely was I able to play with rack-mounted servers, let alone new ones.

The company I am with is stardardising on IBM, and these servers represent the first IBM’s this department has laid their hands on.  The first 336 will be used as a WSUS server, which I have written de deployment plan for aswell. The two others we unpacked today, will be used in a cluster solution together with EXP 400 cabinet running a few different RAID configurations. These systems will be running a large logistics application that we dont know that much about yet, even though we, or rather, my female collegue, will be installing and setting up.

I am quite impressed with the hardware so far, but finding my way though the forrest of IBM’s BIOS, driver, and firmware updates, has been quite a hassle. To be blunt, IBM’s support site is a complete mess, with different versions of software included in different downloads, all currencurrently called the latest versions, while seperate downloads are almost impossible to find. Add to that a confusing naming convention, a very amateuristic download system, the most stupid website search function ever, confusing and contradictory documentation; its just been very tiring finding all the proper bits together to get these systems up to date once they are out of the box.

Here is a typical example

IBM offers a downloadable ISO on the IBM site that is suppose to autodetect and update ALL of your systems Firmware and Bios in a single CD-Boot cycle. Its called update express. But even in the newest version, it in fact contains firmware and bios versions that are outdated by single releases featured elsewhere on their support pages.

Oke, no biggy, we can always collect each update individually. Or can you? Most updates come as either a Zipped floppy drive image (the x336 doesnt have a floppy drive), or, thank god, an ISO. So getting our new X336’s up to spec, out of the box, requires at least 6 reboot cycles using 6 different bootable D-roms (!). Even installing IBM director was NO help.. nothing included at all that even hinted at any kind of automatic update system built in, let alone any system that could update firmware live, or, god forbid, remotely!

The only ray of light seems to be IBM directors built in software-distribution system, very very basic, but supporting a standard that IBM has to distribute driver updates (at least!), using a special package format. Pitty though that so far, only 2 of the 12 or so driver downloads for the 336 have this package format included in any way at all.

However, included on the UpdateXpress CD, is a little app called UpdateExpress Live.  This will, get this, automaticly download the updates you need for you. Does it scan your system first to determine what you need? No. Does it integrate with IBM director in any way? No. Does it even install anything? No. It just downloads them for you.
Not that it works mind you. The application contains a single hard-coded web-service URL that its looking for at the IBM site. And guess what.. its giving a timeout.. as in.. there is nothing awnsering on the IBM side.

So, just out of curiosity, I called the IBM support desk, got patched thought to Dubai or somewhere, and I actually ending up guiding the support guy though the steps of running the UpdateExpress Live app, just so he finnally understood what I was talking about! He obviously had no idea about how this was suppose to function. Then he proclaimed that they didnt support the UpdateExpress software, and proceded to guide me to … get this, the website feedback form! Its been a week and I have not recieved any kind of reply so far. I am hessitant to call again.

Here is another, really stupid example.

Try looking for the EXP-400 disk cabinet on the IBM site, then try google. Here’s another. IBM’s web-team have no clue.

Neither am I, initially at least, that impressed with IBM’s main system management software, the IBM Director itself. Sure it can read a lot of data sources and supports a lot of hardware alerting standards, but its interface is one of those examples of really really slow java programming. Its so slow to use the console, even on the server itself. But its web-interface is worse!  Clunky, slow, very limited, totally un-intuitive to use, and ugly.. its just not something I am very happy using.  All in all, it is powerfull though, and perhaps in time, as I learn to use more of its feature set, such as theRemote Deployment Manager, which may solve some of my above problems, providing we dont have to pay more.

So my experiences with IBM so far have been bumpy. Things will undoubtably get easier as I get to learn more how their miriad software and management systems work, and how to navigate their confusing web-presence.

Oh by the way, did I mention that lights-off, out-of-band management with IPMI is cool? Ever hearf of SAC?

Tech-Ed Irony

Tuesday, July 5th, 2005

I started my new job on monday. Its looking very cool, but I will relate that in a later post.

Anyway, to get there, I travel by train to Schiphol, our international Airport, and grab the bus to Hoofdorp, which is only 2 minutes away. In fact where I work is more or less at the end of the runway!

So today I was waiting for my buss, and I notice all these coaches with Microsoft sighs in front of them.

I so wish to attend Tech-Ed, but my employer would never pay my way. So typical that I have to see all those cool people (some perhaps from this site even) get on the bus.. and I cant go with them!


Bumping into cool Agile tools

Friday, April 29th, 2005

Currently, I am on one of those really standard tasks in a migration process: Software Intake.

My role in this is writing application installation instructions for the helpdesk (or for packaging), whereafter these instructions are tested and then approved by the key-user.

Now generally, I considder this rather boring an tedious work, however, you get to be exposed to a very large array of different software, which I do find usefull. I often learn a lot about what kind of software is out there during this kind of project. Would not want to do it for longer than a month or so though.

Anyhow.. one of the applications that passed my desk yesterday was Junit. (Homepage, Sourceforge)

Now, not being a developer at all, and not knowing the first thing about Java really, I had no idea what this thing was, or how to write an instruction for it. But I quickly realized that this little tool was representative of some of the better stuff I have been exposed to over the last year..

Junit is of course a Unit Test tool, used to test small bits of code at a time.

From Wikipedia:
computer programming, a unit test is a method of testing the correctness of a particular module of source code.

The idea is to write test cases for every non-trivial function or method in the module so that each test case is separate from the others if possible.

The unit testing concept is part of the Extreme Programming method of software engineering. Various unit testing frameworks, based on a design by Kent Beck, have come to be known as xUnit testing frameworks and are available for many programming languages and development platforms. Unit testing is the building block to test driven development (TDD). Extreme Programming and most other methods use unit tests to perform black box testing.

Note that the Extreme Programming community has renamed unit tests to “programmer tests”.

Now it was only recently that I was introduced to some of these Agile methodologies like Extreme Programming and Scrum, and I got rather enthousiastic about it all.. as enthousiastic as a non-developer can get about such things I suppose.

But you have to understand, that my opinion of how IT in general is practiced here in the Netherlands, is not very high, and a symptom of this is this ‘disconnect’ between what I understand to be best-practices and what I have read about them online, opposite the fact that the average admin or dev in the Netherlands (that I have met) often wont even have heard of any of this kind of stuff, let alone follow any of it.
(Arrogance alert: Remember, this is my own perception, and I may be wrong. But the more Dutch companies I am exposed to, the more I get this feeling; – that there is an overwhealming lack of awareness of people in IT, of the larger IT community, and what is happening in that space. And thus, the more convinced I become that people like me are the exception in this ‘awareness’, and not the rule)

Therefor, you can possibly imagine my delight to see some evidence of developers within a Dutch company actually being connected with the bigger picture, in that they are not only familiar with concepts such as Agile methods, but are actually going out and finding the tools to make it work for them.

Datacenter Move post 2

Monday, April 25th, 2005


So it turns out, we are gonna move all the servers to 1 location. Expect a new rack diagram soonish 😉

I hate politics.

Moving it all to 1 place is both a blessing and a curse. Naturally, its logistically easier for us. But it also means we are facing potential power and airco issues at the single location.

Moving to the 1 location was a political desicion, and not made with the best of technical considerations in mind.

On top of that, the big Citrix migration project, that we are depending on being finished by the time we move the bulk of our servers, is looking less and less likely to complete on time.

This means, in short we will be moving a lot more servers and equipment we are currently scoped for. I would very much like to prepare for the worst case, but politics are getting in the way again.

For example, I originally planned to move at least 1 IBM Bladecenter along with our NAS to the new location. These are needed to support the Citrix farm, in case the Prague project doesnt finish in time.

IMG_0909 The Bladecenters

However the project steering group told us this was not in scope, as the Prague project have indicated they would be finished, even though everyone knows they will never make it.

The reasoning is mostly to do with money. Moving the Blacecenter and the NAS, would place a power burden on the new TCR, that would require a more powerfull generator. These are fucking expensive.

So if the Prague move project -claims– they will finish end of May, then why spend thousands of euro’s on a new generator that wont be needed.

Well, because, dear project stearing group, the Prague migration project -wont– be finished on time, and guess what, your gonna have to buy it anyway, so lets by it now and give ourselves some breathing room.

I dont really care about the politics. All I know, is that when push comes to shove, its gonna be US that do all the hard work.

I dont understand this insistance to stand on principle, with all the risks associated with that. Why not play it safe, we are talking about all the Benelux operations of the company that are at stake if we dont mitigate some of these risks.

I cant understand their thinking at all, it seems insanely risky and dangerous to me.

To make things even more silly, there is now talk of at least one of the WMS’s (warehouse management systems) that the Prague project was supposedly gonna migrate we are going to have to move.

The only reason is that, apparently, they are starting to see they wont make it in time.

IMG_0617 The Alpha server running one of our many WMS’

The really funny thing, is that that particular WMS runs ons an Alpha box, with a similair power requirement to the Blade Center.. taking the serverroom over the powerlimit too! Whahah!

Anyway, back to the technology

The new HP servers and racks + options arrived, and the last week has been spent building it all up.

This is me at one of the 3 sexy HP TFT7210R 1U console options.

Cables are still messy, its al temporary till our network guys can put in the new core. Some of that will be happening tonight!

I labeled all the servers, and set up the ILO cards with the advanced licences, and gave them static IP adreses in a new management VLAN we created.

Currently we are on a temporary switch, and only room enough to hook up 4 servers at at time. Should be better after tonight when they bring the new core online.

I spent the last 2 days documenting stuff, deviding the licences, and trying to install the OS on these machines remotely, using ILO.

This didn’t go according to plan. At all.

I posted about this on the new Ars Technica forum “the Server room”

From my thread “ad-hoc Remote Windows installation stategies using ILO”

We have set up a number of servers in a remote location. They are all HP DL360 G5’s with advanced ILO licenses.

Now this works great, but when I tried to remotely install, I run into problems.

I can mount an ISO of the SmartStart CD, and it boots as it should, its a little slow loading of course, but I can get through all the config steps.
But at the screen where the SmartStart says it is copying files to server, it often hangs, or at least, takes forever.
On the rare occasion it gets past this, it wont, for some reason, recognize the Windows OS installer ISO image.

Now both problems might be related to latency, or to a limitation in the way the ISO files are being mounted through the ILO.

Perhaps what I am doing is not supported (I know it isn’t with IBM), but I have not found anything in the Smartstart or ILO documentation that says I cannot do Windows OS install remotely.

So, if I cant figure out why this isn’t working, I am going to have to build some alternative way of remotely installing Windows. Probably via a distribution share, and network boot media like BartPE or something similar.

I don’t have any commercial product available for OS imaging, unfortunately. And don’t have the timescale to purchase these either.

Nor can I use a PXE boot option at this time, because of network limitations. And even then, i dont have time to set up RIS.

What are the kind of solutions you employ for remote OS provisioning?

Well I have gotten a number of responces so far, none of them very helpfull. By best bet is that latency us causing the issues, but I dont have any hardware at location just yet, that I can set up as a distribution point.

I will elaborate on the way I want to use ILO in a sperate post. I have yet to figure out the best way to deal with this.

In the meantime, tomorrow, I will be going to the location and starting the OS installs manually. That at least is a sure way of getting them installed.

Oh, finally, want to see why we need to move in the first place?

This is through the window, the office next to ours.

See the cable tray that was previously in the ceiling? The one marked with the red/white tape?

Thats all the copper and fiber of the current datacenter going through there. I am scared shitless they are gonna damage them while they strip the building.