Exchange 2003 high CPU (mad.exe)
Written on May 18, 2009 – 6:47 pm | by Sahmeepee
I noticed recently that both of my DCs and my Exchange 2003 server were running at higher CPU than normal. Only one of the DCs is a GC and it had recently fallen over due to a hardware failure, but had since been running for a few days without problems.
As the CPU usage was not excessive (20% on Exchange vs. typical usage of around 3-5%; 15% usage on the DCs compared to typical 1-3%) it had not impacted on performance, so I only noticed it from a perfmon trace I leave running against all servers. I logged onto a DC, ran task manager (taskmgr.exe) and found the task with highest CPU: lsass.exe. A quick scan through the event logs on the affected servers didn’t turn up anything significant.
I then ran filemon to see what files lsass was accessing and give me a quick insight into where the problem might lie. Almost all the accesses were relating to ntds.dit – the AD database. This microsofty’s blog post had some good advice on tracking down CPU issues, but I couldn’t really use the tip of unplugging the server from the network so Wireshark was a better option. I chose to run wireshark on the DC to check if the AD activity was generated on the DC itself or if a remote server was querying AD. As it turned out, Exchange was generating a lot of queries relating to other forests in the domain. Exchange is renowned for being very stroppy about having good access to a GC and the information from Wireshark made me suspect that Exchange had become upset after a weekend without access to a GC in our AD site.
I moved over to the Exchange 2003 server and checked the high CPU services on there. Top of the pops was mad.exe – the Exchange System Attendant. The excellent Microsoft Exchange Team blog (you had me at ehlo) had a useful article entitled The Cliff Notes on System Attendant (MAD.EXE). That confirmed my suspicions that Exchange was going a little bit haywire with AD queries so as a quick fix I restarted the system attendant service and its dependant services from the Windows services console (services.msc).
The high CPU persisted for a couple of minutes and then subsided. Another quick scan through the Exchange server’s event logs showed a splattering of ExchangeAL errors in the Application event log like this one:
Event Type: Error
Event Source: MSExchangeAL
Event Category: LDAP Operations
Event ID: 8270
User: N/A
Description:
LDAP returned the error [10] No Such Attribute when importing the transaction
dn: <GUID=**********>
changetype: Modify
msExchPoliciesIncluded:delete:{**********},{26491CFC-9E50-4857-861B-0CB8DF22B5D7}
msExchPoliciesIncluded:add:{**********},{26491CFC-9E50-4857-861B-0CB8DF22B5D7}
msExchALObjectVersion:329
objectGUID:**********
-
… but they subsided after a couple of minutes too and the storm was over. Mad.exe was back down to 0.00% CPU and the DCs were behaving themselves. Job done.
Tags: active directory, Exchange, exchange 2003
