A Collection of Random Thoughts: Exchange Performance and Tuning

Tuesday, June 07, 2005

Exchange Performance and Tuning - Paul Bowden
Paul's the guy that did lots of work on the Best Practices Analyzer, so this fits right in his area of expertise.

Storage Performance.

7 out of 10 times where there are performance issues, the cause is storage. Usually too few spindles to meet the demand (IOPS required exceed disk ability)

Exchange is I/O hungry.
1 database spindle for every 100 users.

Actions to take:

Work out your I/O profile - Optimizing storage white paper. Typical profile of sending 30 messages per day, receiving 100 messages per day, 100mb mailbox size will result in approx. 0.75 IOPS.
Tell your storage vendor how many IOPS you will produce
set caching controllers to 100% write / 4KB cache page size

The old rule of "put your logs on dedicated RAID1, put your databases on dedicated RAID5" doesn't always hold true any more.

Read/write ratio is typically 75% read/ 25% write

Use DiskPar to align tracks (performance boost) (Windows 2000, Windows 2003 Resource Kit) - use this if you are using DAS. Some more advanced storage (e.g. SAN's) you may not have to worry about this.

Typical tools to check performance are Loadsim and Jetstress. Jetstress is much better to validate storage architecture. You give it the variables and let it go and it will hammer the storage system.

A common bottleneck can be the Host Bus Adapter (HBA)

Warning signs of disk bottlenecks would be:
Average Disk Latency is higher than 20ms

Perfmon - look at the Logical Disk counters instead of the Physical disk counters.

Fast Access to Active Directory

If Exchange is slow accessing the AD directory service, store performance suffers and mail delivery is slow.
Duble chec that the Directory Service Access (DSAccess) list only contains local domain controller or global catalog servers.

Remember that round robin is used
Round trip times shiould be less than 10ms.

Rule of thumb is that you should have 1 GC processor for every 4 Exchange processors.

Verify that the /3gb switch is being used on GC's with 1gb or more of memory - this was news to me.
Increase JET cache from 512mb to 1gb

More of the Active Directory is cached in memory

*In case there is any confusion about the /3gb switch, it can be used on any Windows 2000 and 2003 systems with the exception of Windows 2000 Standard.

Correct use of /3gb switch
Mailbox and PF servers
when there is 1gb or more of RAM
Windows 2003 - implement /USERVA=3030

FE and Bridgehead servers
Do NOT implement regardless of memory!
Instead, you can tune the Windows memory settings. By default, XP/2003 will set memory usage for System cache. Installing Exchange will switch that over to Programs. On FE and Bridgehead/Routing servers, it's actually beneficial to switch that back to System Cache. This is because most of what a FE/Bridgehead will be doing is queuing mail, etc.

Using the /USERVA=3030 - reducing user mode memory and re-claiming 42mb. Allows for more System PTE's and makes kernel mode memory much happier.

HeapDecommitFreeBlockThreshold
Should be set toe 0x40000 on ALL Exchange servers.
By default, this is set to 0. This can cause fragmented bits of the process. Changng this setting tells the OS that it should only return virtual memory when it is in a much larger block.

PAE and /3GB.

PAE allows access to memory beyond 4gb
Hardware-enforced Data Execution Prevention

Three ways to enable PAE
/PAE switch in the boot.ini
Automatic enabling on newer machines with Windows 2003 and hot-add memory
Enable DEP in BIOS

Note: Exchange will NOT take advantage of memory beyond 4gb, so you need to be careful when enabling PAE. It puts additional pressure on the kernel, especially Free PTE's (System Page Table Entries). Running out of PTE's can cause serious problems, such as server dropping off network, or one of the HBA's suddenly disappearing.

Enabling Hyper-threading - DO IT!
setting in BIOS
Sometimes defaults to OFF
Can improve Exchange processor efficiency by 25%
Works with Windows 2000 and Windows 2003.

Deploy Exchange 2003 Service pack 1

30% reduction in I/O pattern. SP2 will be reducing I/O even more!
Drastic reduction in log file replay times

From 30 seconds to less than 2 seconds

Support for 8-way hyper-threading servers
Sets MaxDSNSize to 10mb by default - by default, there is no limit. This could cause issues where your server is bogged down with NDR's. Setting this registry value will still allow Exchange to send the DSN, but if the original message is more than 10mb, it will not be attached to the NDR.
Detection of single-bit flip errors (ECC correction within databasse)

Hardware Device Drivers

Disable unused hardware devices
Especially consider network adapters and storage cards that are not being used.
Use identical hardware in the server - results in one driver for multiple devices
Be especially careful with graphics adapters!

can result in serious PTE reduction
keep drivers up to date
switch to vga.sys driver - don't need blazing graphics on servers.

Setting the /3gb switch.
119,000 - goes down to 14,000 PTE's
Warning zone - if you get down to ~5,000 PTE's
Danger zone - if you get down to ~3,000 PTE's
If you run out of System PTE's, it will likely result in a Blue-screen.

Windows 2003 - can use /USERVA=3030. Windows 2000, use SystemPages registry parameter (steals from the Paged Pool)

TEMP Conversions

For POP3, IMAP, and dedicated PF servers, redirect TEMP path to a dedicated fast disk.
Stand-alone = System TEMP path

Cluster = Cluster account TEMP path

Tune Antivirus scanners
Get latest avice from your antivirus vendor on optimum configuration for your server.

Number of scanning threads
memory/disk file size threshold

Remember to keep your antivirus binaries up-to-date

Avoids problems such as

Virtual memory fragmentation
non-paged pool errors
service failures

Online Maintenance Schedule

Skew online maintenance for large mailbox servers so there is at least a 15 minute gap between start times.
Skew online maintenance and online backup schedule so that here is zero overlap

Backup will kill maintenance threads if they collide

Always make sure that online maintenance gets a chance to run
NEVER disable online maintenance.

Log Buffer tuning
Exchange server 2003 defauts to 500 buffers (at least 512bytes)

If disk latencies are reasonable, but log stalls are high, increase log buffer value in Active Directory
This is especially critical for deployments that use replicated storage
Use a multiple of 64 and never more than 9000

Mailbox Store Cache

Store has a two hour cache for re-reading mailbox configuration settings

Mailbox limits and thresholds

Use Mailbox Cache Age Limit registry parameter to reduce wait time. (or you can restart IS) :-) Recommended to set this to perhaps 15 minutes.

Use Dedicated Routing servers

Queueing messages use paged pool memory and increases CPU utilization
Ensure that large mailbox servers can immediately offload messages to a local dedicated routing server
For routing servers, split SMTP queue onto dedicated fast disk
only implement CheckConnectorRestrictions registry parameter on routing servers

Reduce LinkState size

Avoid creating or removing routing groups

Use REMoniotr - inject to nullify old data if necessary (PSS tool)

Avoid 7-bit conversions
Pairs of Exchange Server 2003 bridgeheads on native mode orgs perform 8-bit MIME transfers to each other over RGC's.
All other combinations will force each message to go through 7-bit conversion, resulting in ~30% conversion overhead.

Implement Message Size Limits
Understand Client-add ins - most will increase the load on the server.
VAR solutions such as BES usually require store-level tuning.

Bottom line here is that there were actually a number of suggestions that I hadn't seen before. Paul always presents great sessions, and this was no exception.

- posted by Ben Winzenz @ 11:10 AM

Comments:

Do NOT, I repeat DO NOT ever turn PAE on on an Exchange Server.

MS has KB on this at http://support.microsoft.com/?id=827281

I spent 6 months chasing down random crashes on my Exchange Servers and it turned out to be caused by PAE being enabled.

# posted by

Anonymous : 1:25 PM, June 07, 2005

That KB article refers to not using the /nopae switch and states that the /nopae switch stops working after Windows 2003 SP1 is installed. In Paul's presentation, he actually said that having PAE enabled shouldn't cause problems. Sure, Exchange can't take advantage of the additional memory, but there are scenarios where PAE can be beneficial. For instance, as the presentation referred to, if you enable Data Execution Protection (DEP), that will enable PAE.

# posted by

Ben Winzenz : 4:30 PM, June 07, 2005

Update:

I just spoke with Paul. As of about a week ago, Microsoft now supports using PAE with the /3GB switch. However, this can get you into a precarious situation with not having much in terms of System PTE's free. To overcome this problem, you can tune the /USERVA= switch and lower the number. I believe the number now recommended with PAE enabled (not controlled by boot.ini) is USERVA=2970. This would give 60mb back to System VM instead of just 42mb with 3030 as the value. Hope that helps.

# posted by

Ben Winzenz : 5:03 PM, June 07, 2005

The webcast is coming soon!

# posted by

Anonymous : 9:39 AM, June 22, 2005

My Blog

About Me

Subscribe to this Blog

Contact Me

Other Blogs

Search

Archives