Today I am at a customer again, for whome I had set up their first network and domain, based on Microsoft Small Business Server 2003.
They have called me in today to solve some of the 'critical errors' their SBS server is giving them. Because they seem to be pretty generic, and probably common errors, I thought a ITT blog post might be cool, so I will be formating this post as a kind of 'dairy' of the day.
(The SBS installation is in Dutch, so I am forced to traslate errors ad-hoc, I try to name by English language equivelant where I can though!)
Just came in, plugged me laptop (which is not part of domain), starting coffee drip, and started me Outlook with the profile I have prepared for my mailbox here. The admin mailbox is used to send all the standard reporting and alarts of SBS too, and a rumage though it does indeed show a number of issues. Apart from any issues I find here, I have also been told the server is slow in responding most of the time.
The first error I am gonna tackle is the following:
Subject: Process (store.exe) Alert on ServerName
Alert on ServerName at date time
The store.exe process is allocating more memory than usual.
Check to see if you are having problems with e-mail. If so, stop and then restart the Microsoft Exchange Information Store service. You can disable this alert or change its threshold by using the Change Alert Notifications task in the Server Management Monitoring and Reporting taskpad.
One of the first google results (I use google before MSKB, because MS KB is indexed well by google) I run into is Carlie Anthe's weblog. He seems to be on the MS SBS team, so his blog is gonna prove usefull. He also points to a link that I had not run into yet: Downloads for small Business Server 2003 (I had gotten all the updates so far from Windows Update).
Actually, I looks like his blog isnt updated that frequently, and doesnt contain that much information. There does seem to be mention of the problem I have identified already, that the store.exe process is triggering memory alerts, since the installation of Exchange SP1:
Via a trackback I quickly run into the self-proclaimed "Diva" of SBS, Susan Bradley. Very vert helpfully, Susan has provided a google search box, with her site as a pre-defined target, saving me the trouble of doing that kind of search myself this time. I do search and find quite a meriad of mentions of this issue.
It seems to me that first thing is first, lets determine how much memory store.exe is actually consuming. Also, I would like to know if it really is actually causing a problem in performance. If it is, it may explain why the server appears sluggish to some users.
Determining how much memory store.exe is using now, is not hard of course. Personally, I usually grab Sysinternals process explorer for this, cause it gives me more and better info than the default task manager app, I might grab it later, for now I only want a memory usage overview.
This SBS server has 1gb of memory. As you can see, store.exe is using about 300mb of memory. The second largest process appears to be IIS, at 50mb. Is 300mb much? Is this too much? This post, pointing to this fix, makes me wonder, as microsoft simply turns off the store.exe alert. If you are not alerted to a problem, does that make the problem go away?
This associated KB article explains a bit more.
In the original release version of Exchange 2003, the virtual memory that is allocated to the Jet database cache is allocated as "mapped bytes." However, in Exchange 2003 SP1, the virtual memory that is allocated to the Jet database cache is allocated as "private bytes." Therefore, the memory size that is reported as used by the Store.exe process is dramatically larger in terms of "private bytes." For example, an Exchange computer that has about 4 gigabytes (GB) of RAM might report an increase of about 900 megabytes (MB) of private bytes memory after you install Exchange 2003 SP1.
This also gives me an estimate of what to expect in terms of private byte usage totals. 900mb for 4gb? So 300mb for 1gb sounds reasonable then. But I need to be sure. We will return to this later, lets apply the patch first, after all, I need some results before the end of the day.
Patch doesnt require a reboot (I should have checked beforehand, you never know!). I can quess what it has changed in regard to the store.exe alerting, so I check to see if it has simply disabled the alart, or upped the threshold.
Yup... it simply disabled it. Its no biggy I guess, if the server runs into serious memory issues, other alerts will start to fire. As it is, the server seems to have a reserve of 100-150mb left. Its not much, but it seems to fall within the default operating paramaters of SBS.
But I am stilll curious weather all is really fine and dandy in memory land.
Now memory and processes is not something I am super familiar with, so I need to dig some technet references on how to measure and analyze memory usage. One of the best resources I came across for this kind of this, was my paperback version of the Windows 2000 Server Resource Kit, and the digital version of the chapter I am talking about is here.
(please take note of the non-default scales)
As you can see, this system seems to be doing fine. No paging at all, loads of working set pages available, and in this view, about 200mb of unnasigned memory. If this server is slow, its not because of a memory problem, at least, not on the 'surface'.
Coffee break. Hmm.. no more coffee. Set a new can going, will return to it later.
I examen some of the other errors in the SBS report. They are mostly referring to eventlog issues. Some are pretty mundane, such as the missing printer driver the server looks for, the moment I connect with my laptop over RDP. By default, the RDP client tries to forward printer functionality from the server to my locally connected printer on my laptop, but it cant find the driver it needs. This error can be safely igonored of course, I am not actually home, and I dont wanna print anything via the laptop anyway.
This one is definatly more interesting:
Type gebeurtenis: Fout
Bron van gebeurtenis: MSExchangeDSAccess
Categorie van gebeurtenis: Topology
Process MAD.EXE (PID=3784). All Domain Controller Servers in use are not responding:
A quick scan of Google reveals KB article 321318.
I doubt the mentioned scenario at the top of the article is the case here.. people are using mail without any problems, the store seems mounted and all is well. So this seems a one-off event to begin with. The timing I find curious also. 4.01am.
High workload due to backup?
Well we have the SBS NT backup starting at 23:00 every night, the Iomega backup that takes the daily SBS NT backup, and sticks it on tape (or rather: "Rev Drive" every other day.), starts every morning at 5:30am. So I cant really see a connection there.
I spend 15 minutes trying to find the backup log for this Iomega backup software, only to find that its stored as part of the backup-setup, and worse, that each backup setup is stored per user, in the user profile. I also start to doubt that the backup would function at all without the front-end Gui app being active. This means that the only reason the backup has been working at all, was that the RDP session of the director of the company, under whose session we setup the Iomega backup, was still connected.
I already had a hunch that this Iomege Rev Driver backup was not going to be an ideal server solution.
There is another backup-related error in the logs, that the remote storage manager cannot write to the medium in G. This would be the rev drive. But the issue is easily explained. It occurs every other day at about 18:00. Looks like the guy here is ejecting the drive, where he perhaps should be using a different method. I must say, I had not looked at this at all, and had assumed you could simply just eject and remove the drive. Will check the manual later on.
I send the next hour testing some backups, and messuring system performance to get an idea of what the load is. During the Iomega backup, the server runs up some pretty high page-fault scores, even though available memory doesnt seem to decrease much. Something in the way memory is handled perhaps, that I am not aware of. Backup estimates to be about 45 minutes. We break for lunch.
Backup is probably finished, but I cant log back into the directors DRP session cause of password, he is still lunching. Page faults seem to have gone again. Looks like its overload from the backup process.
Looking at the contents of the Rev Drive, it seems to have worked oke.
Time to look at the NT backup.
Well the Server manager reports that backups seems to be working well most days, however, on some days it simply fails, but distrubingly doesnt write anything to the backup log, like the monday backup:
Back-up runner is gestart.
NTBackup wordt gestart: ntbackup.exe backup "@C:Program FilesMicrosoft Windows Small Business ServerBackupSmall Business Backup Script.bks" /d "SBS back-up gemaakt op 28-3-2005 om 23:00" /v:yes /r:no /rs:no /hc:off /m normal /j "Back-uptaak van Small Business Server" /l:s /f "F:Windows BackupBackup FilesSmall Business Server Backup (01).bkf" /UM
LOGBOEKBESTAND VAN NTBACKUP: C:Documents and SettingsSBS Backup UserLocal SettingsApplication DataMicrosoftWindows NTNTBackupdatabackup09.log
=============<BEGIN VAN LOGBOEKBESTAND VAN NTBACKUP>===============
=============<EIND VAN LOGBOEKBESTAND VAN NTBACKUP>===============
Er zijn fouten opgetreden tijden de back-up.
Raadpleeg het artikel over probleemoplossing voor back-ups op de volgende webpagina: http://go.microsoft.com/fwlink/?LinkId=18414 voor meer informatie over mislukte back-ups.
Back-up is beëindigd op maandag 28 maart 2005 om 23:00
Back-up runner is voltooid.
The backup log refers to the standard SBS 2003 trouble shooting document, and the first issue in the document is:
The NTBackup log is blank.
NTBackup.exe is being manually ended from the Task Manager, or NTBackup.exe encountered an error during launch.
Solution: Run NTBackup manually, and load the Small Business Backup Script.
To run NTBackup manually and load the script
- Click Start, click Run, type ntbackup, and then press Enter. The Backup or Restore Wizard launches.
- On the Welcome to the Backup or Restore Wizard page, click Advanced Mode.
- Click the Backup tab.
- From the Job menu, choose Load Selections.
- In the File Name box type %sbsprogramdir%backup.
- Click Small Business Backup Script.bks to select, and click Open.
- On the Backup tab, click Start Backup.
If the backup succeeds, run the Windows Small Business Server Backup Configuration Wizard from the Backup taskpad in Server Management. If the problem persists, click Start, click Server Management, click the Information Center link, and then click either Community Website or Technical Support to get information about the problem.
If the backup fails, consult the error message for further information about the problem.
Well I am able to manually run a backup oke using the SBS script, And last nights backup went well also.
There is no mention at all in the eventlog of why the backup fails, only that it fails, and then it throws eventlog errors:
I trow out the backup configuration, and use the wizzard to create a fresh one.
I have a good rummage around google for other clues. There seem to be lots of references to error 800423f2, but I cant find any mention of this error in the logs, so i dont think its relevant to my case.
http://www.talkroot.com/showthread.php?s=4b887f59b10dfdbb8b9b541cfe62fb36&p=211670#post211670 provides a possible awner.
Apon checking the rights assigned to the accounts, group membership, and group policy modeling, it looks like this might indeed be the issue. The backup is being run in the context of the SBS Backup "backup user" account. This account does not have any special rights, though, according to the above article, someone suggests that this account must have at least "log on as a service" and "log on as a batch file" rights.
Let me find an authoritive source to confirm this before I try it.
Cant find any. Cant find any resource to tell me what rights the SBS backup-user account should have.
The only thing I can do, is try to give the account some more rights (log on as a service, as a batch job), turn on Verbose logging in NTbackup, and wait to see what happens.
Another problem they seem to be having is very slow access to files and folders on the fileshare.
I try to access some shares from a laptop, and can see it is indeed very slow to respond, and to produce a directory listing.
Eventlog on the server doesnt show anything. However, the eventlog on the client shows a very common error indeed: The Redirector Failed to determine the connection type. KB article 315244
Oke.. I had turned on netbios over tcp/ip on the servers network interface. Lets turn it off and see of that helps. As far as I can tell, no services in this company depend on the netbios name resolution.
Well it didn't help, at least not on his laptop. Turns out that no one else was really having a lot of trouble with the network, not as much as him at least. The last 2 hours I made a vbs login script to create driver mappings, thus trowing their network use back a decade, but with the upshot of pre-making sessions before they actually double click on their shortcuts, speeding things up considderably in some cases.
For him, his network slowness turned out to be the fact that on his laptop, the Windows Netware drivers where installed. We removed them and preso!
Well.. all in all not a terribly productive day. We'll just have to see weather my teeaking of the NTbackup will create more successfull backups.
The Iomega rev drive is just a guess. If it suddenly decides to stop reading its tape in the middle of the week, there is very little I can do about it. Its just not a proper service-based backup.
The combination of Turning off Netbios over TCP/IP, making a loginscript with mappings, has speeded share access up, at least I think so. Again, time will tell.
As for the other issues he came to me with.. I have emptied the event logs. Gives SBS something new to complain about. One last tweak is that I am forwarding the SBS reports to my gmail for now, will allow me to keep a tab on things.
As for explaining all of the above to this customer. Well.. its all very much beyond them. I am sure that the next little red cross that appears, will be ample reason to call me over again.