As the CPU usage was not excessive (20% on Exchange vs. typical usage of around 3-5%; 15% usage on the DCs compared to typical 1-3%) it had not impacted on performance, so I only noticed it from a perfmon trace I leave running against all servers. I logged onto a DC, ran task manager (taskmgr.exe) and found the task with highest CPU: lsass.exe. A quick scan through the event logs on the affected servers didn’t turn up anything significant.

I then ran filemon to see what files lsass was accessing and give me a quick insight into where the problem might lie. Almost all the accesses were relating to ntds.dit – the AD database. This microsofty’s blog post had some good advice on tracking down CPU issues, but I couldn’t really use the tip of unplugging the server from the network so Wireshark was a better option. I chose to run wireshark on the DC to check if the AD activity was generated on the DC itself or if a remote server was querying AD. As it turned out, Exchange was generating a lot of queries relating to other forests in the domain. Exchange is renowned for being very stroppy about having good access to a GC and the information from Wireshark made me suspect that Exchange had become upset after a weekend without access to a GC in our AD site.

I moved over to the Exchange 2003 server and checked the high CPU services on there. Top of the pops was mad.exe – the Exchange System Attendant. The excellent Microsoft Exchange Team blog (you had me at ehlo) had a useful article entitled The Cliff Notes on System Attendant (MAD.EXE). That confirmed my suspicions that Exchange was going a little bit haywire with AD queries so as a quick fix I restarted the system attendant service and its dependant services from the Windows services console (services.msc).

The high CPU persisted for a couple of minutes and then subsided. Another quick scan through the Exchange server’s event logs showed a splattering of ExchangeAL errors in the Application event log like this one:

Event Type: Error
Event Source: MSExchangeAL
Event Category: LDAP Operations
Event ID: 8270
User:  N/A
Description:
LDAP returned the error [10] No Such Attribute when importing the transaction
dn: <GUID=**********>
changetype: Modify
msExchPoliciesIncluded:delete:{**********},{26491CFC-9E50-4857-861B-0CB8DF22B5D7}
msExchPoliciesIncluded:add:{**********},{26491CFC-9E50-4857-861B-0CB8DF22B5D7}
msExchALObjectVersion:329
objectGUID:**********
-

… but they subsided after a couple of minutes too and the storm was over. Mad.exe was back down to 0.00% CPU and the DCs were behaving themselves. Job done.

Take for example the last few updates (Feb, Feb re-release and March). Like good little soldiers we upgraded one evening soon after February came out. Unfortunately it completely stomped all over our ability to add new teaching staff thanks to a bug which Capita themselves took quite a few weeks to really get a handle on. It also made three commonly-used menu options crash SIMS.net completely. We were told that a patch was being prepared but it would be unlikely to be finished until the March release, so Capita advised us to restore our SIMS server from backup tapes. Ahem…

I’m sure you can imagine the look of glee on the faces of our administrative staff as we told them we were giving them the opportunity to repeat all the work they’d done in SIMS/SIMS.net since we applied the February update several days earlier! Hmm. So that evening we restored from a backup taken before the update and for reasons unfathomable to me (possibly workstation-end) it somehow managed not to fix the problem. Luckily Capita then chose to release a patch to fix the problem. After waiting for everyone to get out of SIMS that night I found that it wasn’t created for our version of the database (we hadn’t been issued the Feb re-release). A call to Capita next morning lead to the re-release and patch being made available via SOLUS. Patch applied that evening. Didn’t fix anything. Oh well, there’s always the March update…

So a couple of days later the March update arrives. Much fanfare. That goes on quite smoothly (they’ve broken SOLUS in a couple of places but it’s nothing too tragic) and it’s all installed before the caretakers start jangling their keys at me. Unfortunately it’s not made a jot of difference to the problems that are now plaguing three of our most active SIMS users.

So now we are at the stage of requiring a site-specific fix, which means uploading our 1.2GB (zipped) database at ~50KB/s over what appears to be the only cobbled section of the information superhighway. Currently 15 failed upload attempts down and I’m starting to lose the will to live. Tomorrow morning I will probably give in and send it by DVD.

To give Capita credit where it’s due, their support people seem intelligent, responsive and genuinely keen to fix problems. It’s just a shame that they seem to have too many bug creators (programmers) for their team of bug spotters (testers).

I’ve spent nearly a day’s effort trying to get offline files to encrypt as they are supposed to in Windows XP. The advantage of doing this is that any data which has synchronised to a laptop from your network is protected when the laptop is offsite. Even removing the hard drive and connecting it up to another computer won’t yield access to the offline files. The only sure way of getting access to the files is to get the user’s password.

It seems that the only way to get the encryption of offline files working is to manually log on to each laptop as an administrator and turn the option on:

Windows Explorer > Tools > Folder Options > Offline Files tab > Encrypt offline files to secure data (tick)

This works quite nicely thank you and when connecting to \\testcomputer\c$\windows\CSC (the real location of your offline files) the files all show up with green filenames – cryptography applied! Unfortunately, doing that on each of 100 laptops sounds like as much fun as a Daniel O’Donnell concert and undoubtedly longer. I also have my doubts about whether the laptops would Ghost nicely afterwards.

Luckily there is a Group Policy setting at:

Computer Configuration > Admin Templates > Network > Offline Files > Encrypt the Offline Files Cache

(enabled|disabled|not configured)

On the downside, the setting doesn’t work. At all. In fact, all it does is greys-out the Encrypt offline files to secure data checkbox in the Windows XP GUI. I have verified that even with the above GPO setting in place and the testcomputer’s Resultant Set of Policy (rsop.msc) report showing as much, the files on disk are not encrypted at all.

There are several reports on the web from people suffering the identical problem, but seemingly none with a satisfactory resolution. Suggested resolutions include:

Make sure the first user to log on after the policy is put in place is a member of the Administrators group

Tried it, made no difference.

Reset the offline files cache:

Windows Explorer > Tools > Folder Options > Offline Files tab > Ctrl+Shift+Click on Delete Files…
Tried it, the client-side cache was reset, but the encryption problem was unaffected.

Contact Microsoft Product Support Services for proof that the data is encrypted:

Ahem, no. I can prove that it isn’t with only a few minutes work and zero cost.

Make sure the partition is NTFS:

Yup!

And not compressed:

Nope!

Try applying the fix from MS KB810859 (The “Encrypt the Offline Files cache” Group Policy setting does not take effect when a user logs on to a Windows XP-based computer):

I don’t get the event specified in the event logs so it doesn’t seem to be relevant. I’m also using an admin account which should prevent the scenario described. On top of that you need to contact PSS for the fix, apply it to each of your workstations (not via WSUS) and then do some rather nasty looking ADSIedit manoeuvres! Nopety nope.

Quite a crap effort there Microsoft! Maybe if I just use Vista instead….

Requirements: 

Server: Windows Server 2003 Standard / Enterprise (I used standard) DC
Clients: Windows XP Pro SP2, wireless NIC supporting WPA
Access point: Most new wireless APs / routers will do. I used a Linksys WRT54GL with modified firmware from http://www.dd-wrt.com/ 

You will need to be a domain admin for your own domain, but you don’t need to be an enterprise admin. 

Summary: 

To get the secured wireless working we need to have a RADIUS server running which authenticates the wireless computers against Active Directory. In WS2003, RADIUS is provided by the Internet Authentication Service (IAS) which is a built-in windows component, but isn’t installed by default. 

To allow the laptops to verify that the server is what it claims to be, we need to set up certificates. In WS2003, this is done via Certificate Services which again is included with the OS, but not installed by default. 

Next, we need to tell the access point(s) where our RADIUS (IAS) server is and vice versa and then use Group Policy to tell the XP clients how to authenticate.

The finished wireless network will then be using WPA, PEAP (MS-CHAPv2) and AES.

Procedure: 

On the Domain Controller 

Setting up IAS: 

1)     Launch Add/Remove programs > Windows components > Networking Services > Details> Tick Internet Authentication Services > OK, NEXT
2)     Start > Programs > Administrative Tools > Internet Authentication Services
3)     Right click Internet Authentication Services (local) on left hand side > “Register server in Active Directory” > OK
4)     Right click Internet Authentication Services (local) > Properties > Ports – make a note of the ports used for Authentication and Accounting: you may need them for setting up your Access Point later
5)     Right click on Radius Clients > New Radius Client > Pick a friendly name and a static IP address you can use for the Access Point. Be sure not to choose something in your DHCP range. Select RADIUS Standard and enter a shared secret. Microsoft recommend 22 characters or more generated by a random password generator.
6)     Click on Remote Access Logging > Right click on Local file > As a minimum, select Accounting Requests and Authentication Requests and check the Log File tab settings.
 

Setting up Certificate Services:

7)     Launch Add/Remove programs > Windows components > Certificate Services > Details > Tick Certificate Services CA > YES > OK > NEXT
8)     In the Windows Component Wizard window:
a.      Choose “Standalone Root CA”. If you are an enterprise admin and know how to set up autoenrollment, you might want to choose “Enterprise Root CA”, but standalone should work in all cases. NEXT.
b.      Choose a “common name” such as StTriniansRootCA, increase the validity period from 5 years (I used 25), NEXT
c.      Click YES to stop the IIS service.
d.      Choose YES to installing ASP to allow web enrolment – we’ll be using that later
e.      FINISH
 

Creating and installing a server certificate: 

This process creates a certificate to prove the identity of your IAS server. By default the certificate only lasts 1 year before it expires. For instructions on increasing this value, visit http://support.microsoft.com/?id=254632
 

9)     Open Internet Explorer at http://YOURSERVER/certsrv
10)  Click “Request a certificate”
11)  Click  “advanced certificate request”
12)  Click “Create and submit a request to this CA.”
13)  The “Advanced Certificate Request” page must be filled in carefully, in particular the following fields:
Name – The fully qualified name of your server as the wireless clients see it, e.g. yourserver.yourdomain.int
Type of Certificate Needed – Choose “Server Authentication Certificate”
Create new key set – Select this option
CSP – Choose “Microsoft RSA/Schannel Cryptographic Provider”
Key Size – 1024 should be fine. Bigger numbers give better security, but increase the processing power required.
Mark Keys as exportable – Tick this
Store certificate in the local computer certificate store – Tick this
14)  Click Submit >
15)  The next screen tells you to come back later when your certificate has been approved, so go back to the Certification Authority snap-in, expand your Root CA > Pending Requests > Select the request > Right click > All tasks > Issue. It should move into the Issued Certificates container.
16)  Open Internet Explorer at http://YOURSERVER/certsrv (again)
17)  Click “View the status of a pending certificate request”
18)  Follow the link to install the certificate you just created.
 

Creating a Remote Access Policy: 

19)  Create a security group (e.g. WirelessComputers) in the Active Directory Users and Computers snap-in and add all your wireless laptops and PCs as members of the group.
20)  Start > Programs > Administrative Tools > Internet Authentication Services
21)  Right click on Remote Access Policies > New Remote Access Policy
22)  NEXT
23)  Make sure the “Use the wizard…” option is selected and type in a name, such as “Wireless Access to the St. Trinians network”. NEXT.
24)  Select “Wireless”, NEXT
25)  Select Group, click Add… and find your WirelessComputers security group. NEXT
26)  Select “Protected EAP (PEAP)” and click Configure…
27)  Pick the certificate with your IAS server’s fully qualified domain name e.g. yourserver.yourdomain.int
28)  “Enable fast reconnect” is optional, but most sources I have found suggest that you tick it.
29)  “Secured password (EAP-MSCHAP v2)” should be the only item in the list
30)  Click OK
31)  NEXT
32)  Finish
 

Pushing Wireless Networking policies out to workstations: 

Some steps required for this section vary depending on whether the Group Policy Management Console (GPMC.msc) has been installed on your server or not, so those steps are described in a general way. 

33)  Create a new Group Policy Object and link it to the OU containing your wireless computers
34)  Disable the user portion of the GPO (not necessary, but good practice for speeding up application of the policy)
35)  Edit the policy
36)  In the Group Policy Object Editor snap-in, navigate to Computer Configuration > Windows Settings > Security Settings > Wireless Network (IEEE 802.11) Policies
37)  Right-click Wireless Network (IEEE 802.11) Policies > Create Wireless Network Policy
38)  NEXT
39)  Type a name, e.g. “St. Trinians Secure Wireless Network”, NEXT
40)  Tick “Edit properties”, FINISH
41)  In the properties of the policy, select the General tab
42)  Set “Networks to access” to “Access point (infrastructure) networks only”
43)  Tick “Use windows to configure…”
44)  Untick “Automatically connect to non-preferred networks”
45)  Click the “Preferred networks” tab
46)  Click “Add…”
47)  Pick the SSID you will be using to identify the network, e.g. SchWlan1 This will need to match the SSID set up on your access point(s).
48)  Set “Network Authentication” to “WPA”
49)  Set “Data Encryption” to “AES”
50)  Click the IEEE 802.1x tab
51)  Set the EAP Type to “Protected EAP (PEAP)”
52)  Click Settings
53)  Tick “Validate server certificate”
54)  Tick “Connect to these servers” and type in the name of your IAS server
55)  Under “Trusted Root Certification Authorities”, find the Root CA you created earlier and tick it
56)  Tick “Do not prompt user to authorize new servers or trusted authorities”
57)  Select “Secured password (EAP-MSCHAP v2)”
58)  Tick “Enable Fast Reconnect”
59)  OK
60)  Untick “Authenticate as a guest…”
61)  Tick “Authenticate as a computer…”
62)  Computer Authentication: “Computer only” (This setting will prevent the computer being disconnected and reconnected while the user is logging in.)
63)  OK
64)  OK
65)  In the Group Policy Object Editor, move down to Public Key Policies and right click on Trusted Root Certification Authorities
66)  Import…
67)  NEXT
68)  Type \\YourDCName\CertConfig\ and click Browse…
69)  Pick the Root CA certificate, OPEN
70)  NEXT
71)  NEXT
72)  FINISH
73)  Close the Group Policy Object Editor
74)  To apply the policies, connect the wireless computers via a wired connection, log in, run “gpupdate /force” then reboot.
Setting up the wireless access point: 

Obviously this depends very much on the model of the access point, so these instructions are very generic. 

75)  Look for an option for setting up WPA with Radius. On the superb DD-WRT firmware this is under Wireless > Wireless Security.
76)  If there is an option to choose between AES and TKIP, choose AES
77)  For the RADIUS server address, enter the IP address of your domain controller running IAS
78)  For the RADIUS port, enter the port number you made a note of earlier on. By default this will probably by 1812.
79)  Enter the WPA shared key: this should match exactly with the “shared secret” you used when setting up IAS.
80)  Set up the SSID to match what you put in your Wireless Networking policies sent to the clients, e.g. SchWlan1. If you want disable broadcasting of the SSID it will make your network less visible as a target to casual snoopers, but it might make it more difficult to troubleshoot problems with your setup.

You will need:

• A web server which runs ASP pages
• Outlook Web Access for Exchange 2003 (other versions not tested but may work)
• Internet Explorer installed on your clients (doesn’t need to be the default browser)

First, create an empty text document called exchredir.asp and put the exchredir.asp code listing into it. (I’ve had to store the code listing in a Google Notebook, because Wordpress is a complete tart about quoting code) Make sure you edit the text in red to reflect your OWA server’s name; text in green can be customised to suit your setup, but isn’t critical.
Drop exchredir.asp into a new folder on your ASP server where people can get at it, but make sure that you turn off anonymous access to the folder so that the script can pick up their username. To do that go into the IIS management console, find the directory containing exchredir.asp in the treeview on the left, properties > directory security tab > anonymous access and authentication control > edit > untick anonymous access.
Finally, your computer needs to know that mailto: links are opened by our script so we need to register the “URL:MailTo Protocol” filetype in Windows Explorer (Tools > Folder Options > File types).

Use the “Advanced” button to edit the action for “open” so that it reads, including quotes:

“c:\program files\internet explorer\iexplore.exe” http://server/path/exchredir.asp?mailto=%1

which makes the setting for all users of that computer.

For a large number of users you may need to manipulate the registry keys at:

HKEY_CURRENT_USER\SOFTWARE\Classes\mailto\shell\open\command
HKEY_LOCAL_MACHINE\SOFTWARE\Classes\mailto\shell\open\command

more info here: How to implement a per-user default mail client in Windows XP?

Now try testing a mailto link on a web page, such as the one at the end of the first paragraph of: http://www.bbc.co.uk/blogs/ouch/ . It should pop up a new email message in OWA with the To: field filled in for you.

Limitations:

At the moment I don’t know of a way of passing the subject into a new email message in OWA and it may not be possible at all, so I just drop the subject line – you’ll have to type it in yourself.

As you can see from the screenshot above, the program doesn’t just give you a simple bar or pie chart of your top-level directories, it also produces a view known as a Treemap. The Treemap displays a coloured rectangle for each file on your drive – larger files have larger rectangles. The rectangles are also clustered together into folders: if you look carefully at the screenshot above, there is a white rectangle surrounding about half of the treemap’s area. All files within that rectangle are inside the same parent folder (in this case “c:\program files”) You can hover your mouse over any part of the treemap and the status bar will show you which file it represents.

But why the wacky colours? The colours in the treemap represent different filetypes (determined by their filename extension) so that you can see at a glance which type of file is occupying the most disk space. Ingenious!

So how did this help my ailing server? I ran WinDirStat from a share on our network whilst logged on to the server and started a scan of the c:\ drive. A few seconds later the treemap showed up and two large files were standing out:

c:\pagefile.sys (the windows swapfile) – 1.5GB

c:\Program Files\Websense\bin\xid_trace.txt (a mystery file!) – 1GB

A bit of googling told me that xid_trace was just a logfile generated by our Web filtering software (Websense) on the servers which perform authentication (known as DC Agents). Every time a user had requested a page from the Internet, a line had been logged in xid_trace.txt to record the event (as well as our standard database logs). Ouch. I zapped the file from within WinDirStat and added it to my list of logfiles to prune periodically. If only applying service packs was so quick and painless!

That is all.